Reviewing the Execution Logs of Katalon Studio
While the output logs of Katalon Studio are extremely helpful, it’s not possible (as far as I can tell) to copy that output directly. The reason being, I want to run a baseline test to capture multiple data points. After a code deployment, I want to run the tests again and verify the results are the same. The logs look great in Katalon, but the raw logs contain a lot more information.
On my Mac, I can use Terminal and the Unix command, Grep, to start filtering. The cleanest file to work with is the JUnit_Report.xml file, located in the Reports directory. Inside, there are lines marked as:
[MESSAGE][PASSED] [MESSAGE][WARNING] [MESSAGE][FAILED] [MESSAGE][ERROR]
The real file will contain lines that look like this:
2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - today = new java.util.Date(): null 2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - yesterday = today.previous(): null 2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - todayDate = today.format("MM/dd/yyyy"): null 2018-03-25 10:02:27 - [MESSAGE][PASSED] - Delayed 2 second(s) 2018-03-25 10:02:27 - [MESSAGE][WARNING] - Filter Results: 1679 2018-03-25 10:02:27 - [MESSAGE][PASSED] - Text of object 'Object Repository/Page_/Dashboard/Footer-Total Number of Users Returned' is: '1-25 of 1679 users' 2018-03-25 10:02:27 - [MESSAGE][WARNING] - Pagination Results: 1679 2018-03-25 10:02:27 - [MESSAGE][PASSED] - SUCCESS: The Filter Results Matches the Pagination Results]]></system-out>
The drawback are the [TEST_STEP][PASSED] lines. Those need to be filtered out.
At the Terminal prompt:
grep "MESSAGE" JUnit_Report.xml > firstpass.txt
This creates a text file that contains the [MESSAGE] lines.
This is good, but there will be hundreds of [MESSAGE][PASSED] lines, which really aren’t that important. That’s not a problem as those lines can be filtered out in LibreOffice using the AutoFilter.
Load the file, select AutoFilter and the option for Standard Filter. Set the filter so the lines that Do not contain the word PASSED are displayed.
Now we have our Warning, Error and Failed messages together. When the second test is completed and the results filtered in the same manner, they can be pasted into the next column to be compared line by line. Or, the EXACT function can be run to compare the two strings.
To illustrate, here is the Log Viewer from Katalon:
And here are the same results in LibreOffice:
If needed, the same filtering method can be used to see the Failed and Error messages. But this gives the ability to run a test, capture data, run the test again and compare the results.
Other articles of interest:
- Marking tests as Passed, Failed or in Error using MarkFailed, MarkPassed, MarkError in Katalon Studio
- Output status messages and test information by writing to the Log File Viewer in Katalon Studio
- Checking for Page Load with a Custom Keyword
- Creating and Calling Methods for Test Cases in Katalon Studio
- Getting the number of pages in a pagination ribbon
- Quasi Performance/Load Testing with Katalon Studio
- Wait For Alert, Verify Alert Present in Katalon Studio
- A Try Catch example in Katalon Studio
- New Libraries to Import into Katalon Studio
- Selenium based Custom Keyword to Count Pagination in Katalon Studio