Katalon

Adding Timing Flags to Test Cases within Katalon

I've made reference to this before, but due to the success I've had using it, I wanted to bring it up again.

Within my tests cases I have been adding markers to time how long it takes to complete an action. This is usually connected with the Save action or something similar. For example, how long does it takes to save the form data? Or how long does it take to move from one page to another.

Using the "feature flag" idea, I place the start of the code right before there is an action to click the button. After the click action, there is a waitForElementVisible action, which waits for an email address input field to become visible. Once we can take action on it, the Save action is complete and we can record the time between those two events.

if (timedTest==true){
    timeStart = new Date()
}
WebUI.click(findTestObject('btn-Save Cart))
WebUI.waitForElementVisible(findTestObject(input-Email Address'), 90)
if (timedTest==true){
    timeStop = new Date()
    TimeDuration duration = TimeCategory.minus(timeStop, timeStart)
    log.logWarning("Execution Time: " + duration)
}

The same process works when moving through pages. Click the link or button that moves to the next page and wait for an item to appear that proves the page has been loaded. Take the time between those two events and output it.

Additionally, I have been outputting these times to a text file so I can gather some benchmark data. As an example, the following is listed within the timedTest flag.

    File file1 = new File('/FileLocation/savetimes.txt')
    file1 << "Execution Time: " + timeStop +"," + duration"

This has been a huge benefit for our performance testing. I can run the same test dozens of times of and record how long it takes. I don't have to guess or write it all down manually. It all becomes part of the test and gets tracked over time.

After we make a change, I run the same tests again and record the new results. This easily creates the Before and After benchmarks.

You can even add a small bit of code to give you a visual cue things are going south.

    String timeCounter=duration
    timeCounter=timeCounter.replaceAll(' seconds','')
    float timeValue=Float.valueOf(timeCounter)
    if (timeValue>15){
        log.logError('ERROR: The save time for the item is higher than expected')
        KeywordUtil.markError('ERROR: The save time for the item is higher than expected')
    }

Katalon Studio may not be a performance or load balancing tool, but you can easily track metrics and show processes that are taking longer than normal. This probably isn't part of a regression test, but by combining the feature flags with the TimeDuration library you can record all sorts of metrics about your test which can be incredibly value.

Other articles of interest:

Adding Feature Flags to Test Cases

Feature flags are by no means a new concept, but when used within QA tests, they provide a lot of flexibility and extend the functionality without having to write and maintain multiple test cases.

I've recently added the TimeDuration library to several tests to record how long it takes to perform a "save" action. It's great for me, but I don't want this to run all the time, so it's being wrapped in a feature flag. I start with a simple list of flags at the top of the code like:

boolean timedTest=true

And then later in the code, there is an IF block if (useDataBase==true){ to see if the time should be recored. By default these are off for regression tests. But when I want to use them for functional testing, I change the flag.

Taking that a step further, I've been using feature flags to section off functionality. Let's say we are creating an order. That order needs a list of items. For a regression test, the list needs to be generic enough to apply to multiple scenarios and users.

However, with a feature flag, the list can be be specific for my functional test without having to write a whole new test case or disrupt the existing one. Right now, I am adjusting them at the test level.

Here is a simple example to either read a spreadsheet for generic items or query the database for a specific user and location.

//Read inventory from the database file or xls spreadshseet?
boolean useDataBase=true

if (useDataBase==true){
    WebUI.callTestCase(findTestCase('Populate Custom Inventory List - DB'), [:], FailureHandling.CONTINUE_ON_FAILURE)
}else{
    WebUI.callTestCase(findTestCase('Populate Inventory Items List'), [:], FailureHandling.CONTINUE_ON_FAILURE)
}

Another example would be whether or not to add a custom item to an order. A custom item has specific fields associated with it and isn't used in regression testing.

//add a custom item to the order?
boolean addCustomItem=false
if (addCustomItem==true){
    //Wait for Add Inventory Item button to be visible
    WebUI.waitForElementVisible(findTestObject('btn-Add Custom Item'), 15)

<additional code here>

    //Click to Add Custom Item to Order
    WebUI.click(findTestObject('btn-Save Custom Item'))
}

While these aren't complicated examples, they are the start of consolidating code for functional and regression testing. In the case above, my original test created a certain type of order. With a few small additions, the test is dynamic enough to fill in extra forms when the testing is for my benefit. But with the flags turned off, it acts as a regression test that works for 80-90% of the users in the system. This gives me extra testing capability without having to write and maintain two sets of tests.

Other articles of interest:

Formatting Execution Times with TextSoap

As noted, I have a couple of tests that use the TimeDuration library to time how long certain actions take. For example, the time it takes to Save a large number of items. After running a whole series of test suites, I had a lot of information in the log files. As an experiment, I searched through all the logs and pulled out the “Execution Time:” marker. This gave me multiple instances of how long it took to save that large chunk of data.

I then decided to try out TextSoap to see what sort of formatting I could apply to this text. Could I make some sort of report that showed the really long save times and the longer than average save times?

Simply put, you absolutely can.

Using a bit of RegEx and then applying some formatting, I can call out the really long test that took 3 minutes. Clearly something went totally wrong there.

.*minutes.*

I can then highlight the times that took over 20 seconds.

.*\b[2-9][0-9]\b.*

I can then highlight the times between 10 and 19 seconds as pretty close to problematic.

.*\b[1][0-9]\b.*

In conjunction with these, I can apply text formatting to make a pretty cool little report. From the screenshot, the Minutes time gets red text and gets converted to uppercase.

The 20+ second condition also gets the same treatment.

For 10-19 seconds, get formatted in orange, but no change of case.

So in short order, I was able to use grep to find the Execution Time text and save that to a file. Then TextSoap applies a set of conditional filters and I have a pretty useful report of my automation testing. Now that the “cleaner” is built, this process takes less than 30 seconds to complete.

This could be done several ways and with several different tools. But it seemed like a fun exercise for TextSoap and a way to try out it’s features. And it totally worked.

By the way, my RegEx skills are nigh non-existent. I used regex101.com and comments from Stackoverflow to put these together.

Other articles of interest:

Programmatic Database query with Katalon

As mentioned previously, making dynamic database queries was a big goal for this year. Katalon makes this quite easy with a UI to set up the connection and a straightforward way of connecting to the data itself.

To start the process, select Data Files, New Test Data and select Database as the source.
Taking the configuration string from before, we connect to the database using this screen and provide the default query. This creates a default table of data that Katalon will work with when executing tests.

Within our code, we use the TesData object to get at our database source.
TestData inventoryDB = findTestData('Data Files/database object name')

We can now get the number of rows in the database, the names of the columns and read data from each column as needed. We need to reference each column with an index number rather than it’s name. Even though Katalon displays "item_number" as the column title, internally that is column 1.

However, we can still use item_number by getting a list of the columns and finding it’s index. I am adding 1 to the index since the List starts at 0 and 0 isn’t a valid column.

From there, we read information from the database using the getValue statement:
inventoryData=inventoryDB.getValue(columnIndex, loop)

From this very simple example we can get the counts of rows and column, then read the correct item from the list.

int loop=1, columnIndex=1, rowCount=1
String inventoryData=''
//Connect to Database Object in Data Files
TestData inventoryDB = findTestData('Data Files/Inventory Items Query')
rowCount=inventoryDB.getRowNumbers() //Row count for the number of results returned
List columnNames=inventoryDB.getColumnNames() //A List of the column names
columnIndex=columnNames.indexOf("item_number")+1 //Get the index of the column name we want

for (loop = 1; loop <=rowCount; loop++) {
    //Read the column data from the database and assign to a List
    inventoryData=inventoryDB.getValue(columnIndex, loop) //Read data directly from the database query
    GlobalVariable.inventorySKU[loop]=inventoryData
}

Other articles of interest:

Connecting Katalon to a Postgres Database

One of our goals for this year was to connect Katalon into our Postgres instance. It turned out to be quite simple, but we ran into a couple of errors before we got all the magic to happen.

When using the standard connection string, we get an hba_conf error along with ssl=true errors. This can be fixed in the connection string by appending the two following parameters:

sslfactory=org.postgresql.ssl.NonValidatingFactory
ssl=true

The connection string to be placed in Katalon looks like:

jdbc:postgresql://hostname:5432/dbname?sslfactory=org.postgresql.ssl.NonValidatingFactory&ssl=true

With this connection string in place, we were able to connect to the database without issue and execute queries.

Other articles of interest:

Recent Comments

  • Output status messages and test information by writing to the Log File Viewer in Katalon Studio (6)
    • Saranya: Good One. Could you plz share link to access all your katalon related blogs. ThankQ
    • Rodrigo Calabretta: I’m using the KeywordUtil.markFailed or KeywordUtil.markError and my test stops is being shown as Error and If I use in the @afterTestCase the testCaseContext.getTestCaseSta tus() to show the status test case...
  • Working with Dates and Date Formatting in Katalon Studio (14)
    • Don Pedro: That could be a little tricky. First, is the text/format always going to be the same? You will need to do some parsing. Separate the first but of text at 2019. After that, split the text again to get rid of Central Standard...
    • Don Pedro: It should be of the same Date type as today. You could then do some calendar trickery with Calendar /*Calendar c = Calendar.getInstance(); //c.set(Calendar.MONTH, 1) //Set the month, 0=January, 11=December println...
    • josh: I’m testing a page with a string like “Updated November 21, 2019 16:25:32 PM Central Standard Time.” How might I capture the date and time and compare it to current time to verify that the update time was within...
    • zakir hussain: today = new Date() lastWeek = (today – 1.week) can u please help me in understanding what is the return type of lastweek? and I wanted to set nextyear date how can i do it
    • ALLAN FORD: Useful !
    • Don Pedro: Not sure I understand what you are trying to do.
    • Don Pedro: formattedDate = mydate.format(“M/dd/yyy& #8221;)
  • Setting up Environment Profiles in Katalon Studio (2)
    • Don Pedro: Not quite sure I understand. The term “Execution Profile” is normally used now, but they are still GlobalVariables and are created in the same way for version 7 as previous. You make a new execution profile, then...
    • Aparna: Hello, Your documentation is excellent and has helped me understand many things. But this article above seems outdated and I need help in creating my own Global variables and all the documentation out there seems to be point to...
  • Here is my drive cluster (2)
    • Don Pedro: While I still have those drives, the drive cluster looks nothing like that now. Almost all of my external drives have had the cases removed and they have been installed into a series of MediaSonic 8 Bay Drive Enclosures. I...
    • Marin Boucher: Hi! Verry funny blog with “go to the point” on many post! I spent good time reading some of your post. Regarding this one about NAS, “many NAS” in fact, I would be curious to see a screenshot of...
  • How To Disable the Quicken Registration Prompt (27)
    • debra: didn’t work for me… my 2007 – which I am keeping on my older mac.
  • Adding entries to an open Excel spreadsheet during runtime (1)
    • IanG: Hi: Seems like nobody else has described (in detail) how Katalon can be configured to test an API by reading the first entry from a multi-row multi column XLSX spreadsheet, executing a test, writing the result to another column (or...