Migrating Platforms – A Katalon Success Story

Several months ago, we began mapping out a migration strategy to move from one service platform to another. It would include new databases, new security authentication, a complete data load and new import and export functionality. Basically, new everything.

Since it’s a new platform, it needed a full regression test. It could be done by hand, but that would be slow and issues could easily fall through the cracks. The goal was to introduce automation, so back in December we started with Katalon as our tool of choice to handle the work.

And so we set off to automate as much as possible. It started as a brute force endeavor. Just to get started, tests were created to read data so we could compare Before and After results to confirm we didn’t lose anything. With those complete, it became possible to fill in forms. After a bit more experience and knowledge, we could count items on the page and validate their content. It was then possible to sum columns and confirm internal calculations to site values. Each test built on the last.

The tests continued to expand in depth and complexity until there was a solid set of tests that validated hundreds of data points. A full run could be completed in less than 10 minutes. Doing that same list of tests manually takes an hour to complete. Plus, the tests could be run in the background giving time back for other tasks.

As we practiced the migration, the scripts were quick to uncover issues. They even revealed a few of bugs due to how frequently they were running. After a week of repeated test runs, the environment was stable and we were quite confident to move the process to Staging.

To confirm it’s readiness, the same set of tests were run repeatedly and the small hiccups were ironed out before turning it over to the customer. What they saw was a nearly flawless upgrade experience.

Over the weekend, we did the real upgrade, with the final validation performed by Katalon and the automation scripts. In a fraction of the time, we were able to check dozens of links, read hundreds of sales figures, fill in multiple user forms, confirm each page loaded as expected and compared dozens of known data elements. We also had written confirmation of what we tested so everyone could agree on the sign-off.

I would say that our implementation of Katalon has been a big success. There is still a lot more to learn and more robust tests to write. But, as of now, we have a solid test suite that definitely validates whether the site is working or not.

Other articles of interest:

Reviewing the Execution Logs of Katalon Studio

While the output logs of Katalon Studio are extremely helpful, it’s not possible (as far as I can tell) to copy that output directly. The reason being, I want to run a baseline test to capture multiple data points. After a code deployment, I want to run the tests again and verify the results are the same. The logs look great in Katalon, but the raw logs contain a lot more information.

On my Mac, I can use Terminal and the Unix command, Grep, to start filtering. The cleanest file to work with is the JUnit_Report.xml file, located in the Reports directory. Inside, there are lines marked as:

[MESSAGE][PASSED]
[MESSAGE][WARNING]
[MESSAGE][FAILED]
[MESSAGE][ERROR]

The real file will contain lines that look like this:

2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - today = new java.util.Date(): null
2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - yesterday = today.previous(): null
2018-03-25 09:42:40 - [TEST_STEP][PASSED] - Statement - todayDate = today.format("MM/dd/yyyy"): null
2018-03-25 10:02:27 - [MESSAGE][PASSED] - Delayed 2 second(s)
2018-03-25 10:02:27 - [MESSAGE][WARNING] - Filter Results: 1679
2018-03-25 10:02:27 - [MESSAGE][PASSED] - Text of object 'Object Repository/Page_/Dashboard/Footer-Total Number of Users Returned' is: '1-25 of 1679 users'
2018-03-25 10:02:27 - [MESSAGE][WARNING] - Pagination Results: 1679
2018-03-25 10:02:27 - [MESSAGE][PASSED] - SUCCESS: The Filter Results Matches the Pagination Results]]></system-out>

The drawback are the [TEST_STEP][PASSED] lines. Those need to be filtered out.

At the Terminal prompt:

grep "MESSAGE" JUnit_Report.xml > firstpass.txt

This creates a text file that contains the [MESSAGE] lines.

This is good, but there will be hundreds of [MESSAGE][PASSED] lines, which really aren’t that important. That’s not a problem as those lines can be filtered out in LibreOffice using the AutoFilter.

Load the file, select AutoFilter and the option for Standard Filter. Set the filter so the lines that Do not contain the word PASSED are displayed.

libreoffice-standard-filter

Now we have our Warning, Error and Failed messages together. When the second test is completed and the results filtered in the same manner, they can be pasted into the next column to be compared line by line. Or, the EXACT function can be run to compare the two strings.

To illustrate, here is the Log Viewer from Katalon:

katalon-log-viewer

And here are the same results in LibreOffice:

libreoffice-log-viewer

If needed, the same filtering method can be used to see the Failed and Error messages. But this gives the ability to run a test, capture data, run the test again and compare the results.

Other articles of interest:

Custom Keywords for Custom Functions

A custom keyword is similar to a function. It takes in information and returns a single result. It would be a repeatable block of code used in several different test cases. In my case, I’ve set up custom keywords to create Contact information.
As outlined previously, the site makes use of Contacts and making them a little more real world would be ideal. I first created a Test Case that set the value of several Global Variables. That worked quite well, and to be honest, I don’t see anything wrong with it. But, as an exercise, I wondered if the same could be accomplished using a Custom Keyword. Turns out it can.
The steps for setting up a Custom Keyword can be found here:
https://docs.katalon.com/display/KD/Define+custom+keywords
And while that shows exactly how to do it, it doesn’t really explain why. But, after a few experiments, things are little clearer to me. To that end, here is some code for my Custom Keyword – getPhoneNumber()
To give context, I created a new package called, createUserDetails.
My Keywords are stored in, contactInformation.
The first step is to mark the code as a Keyword.
The next is to give the Keyword a name.
After that is the code the Keyword will run when called. Keep in mind, it can only return a single value.

@Keyword
def getPhoneNumber() {
//Generate phone number
int areaCode=Math.abs(new Random().nextInt(799)) + 200;
int numPrefix=Math.abs(new Random().nextInt(899)) + 100;
int numSuffix=Math.abs(new Random().nextInt(9000)) + 1000;
String phoneNumber=String.valueOf(areaCode) + String.valueOf(numPrefix) +String.valueOf(numSuffix)
return phoneNumber
}

For my case, I need a name, company, street address, phone, etc. To that end, I created Custom Keywords for the following:

/*This package is used to generate Customer Keywords to populate Contact and Prospect Information
* The keywords used are:
* getStateAbbr() – Pick a 2 letter state abbreviation
* getPhoneNumber() – Generate a 10 digit phone number
* getCityName() – Pick a City Name
* getStreetName() – Generate a street address such as 1313 Mockingbird Ln
* getProperName() – Pick a Full Name for the user
* getCompanyName() – Pick a Company Name for the user
* createEmail(String companyName, String fullName) – Takes 2 arguments, companyName and fullName to create a “valid” email address for the user
* getZipCode() – Generates a random 5 digit zip code
*/
Those are defined in the Custom Keyword package. With that done, within the Test Case, the Keywords are called in the following manner.

userFullName=CustomKeywords.'createUserDetails.contactInformation.getProperName'()
log.logWarning(userFullName)

streetAddress=CustomKeywords.'createUserDetails.contactInformation.getStreetName'()
log.logWarning(streetAddress)

cityName=CustomKeywords.'createUserDetails.contactInformation.getCityName'()
stateAbbr=CustomKeywords.'createUserDetails.contactInformation.getStateAbbr'()
zipCode=CustomKeywords.'createUserDetails.contactInformation.getZipCode'()
log.logWarning(cityName + ' ' + stateAbbr + ', ' + zipCode)

Each variable will be set to the result of the Custom Keyword. In the first example, userFullName will be set to the text that makes up a valid looking first and last name.
The getStreetName() Keyword does several things and put several pieces of information together, even though it only returns a single answer. It generates a number for the address, name of the street, adds a suffix, then concatenates all the pieces together to create a single text string like 1313 MockingBird Ln.
Like a Test Case, a Custom Keyword can have parameters passed to it. In my code, I create a name and company for a user. To make it even more valid, I pass those as variables to my createEmail Keyword.

emailAddress=CustomKeywords.'createUserDetails.contactInformation.createEmail'(companyName, userFullName)

The emailAddress Keyword takes the name and parses it into two strings. The “.” is put between the names. The company and the “@” are appended to the name. A domain suffix is added and the string is converted to lowercase.
Granted, the whole thing is less than 15 lines of code, but it’s 15 lines of code I only have to type and update once. From now on, a single line does all that work for me. And if I want to add suffixes or change my methodology, it’s all in one place. That’s pretty powerful.

Other articles of interest:

Calling Another TestCase in Katalon Studio

One of the many great features of Katalon Studio, is the ability to call other test cases using the CallTestCase command.

This would make the test case act like a procedure, where it can be called to execute a series of steps, then return to the main program after the work is done.
This means the automation code can be more modular, more dynamic and with the right planning, far more code can be reusable.
I’ve started to bring this into my own tests as seen by the example of reading the tabs of the customer profile. Here is another example to toggle the status of a user.
I made two test cases, one to “lock” a user and another to “unlock” a user. I can then read the text of the button and call to the appropriate test case.

//Call the appropriate function to toggle the status of the user
if (buttonText == 'Unlock') {
log.logWarning('Budget is Locked, running Unlock code')
WebUI.callTestCase(findTestCase('Admin Tasks/Unlock User'), [:], FailureHandling.STOP_ON_FAILURE)
} else {
log.logWarning('Budget is Unlocked, running Lock code')
WebUI.callTestCase(findTestCase('Admin Tasks/Lock User'), [:], FailureHandling.STOP_ON_FAILURE)
}

A test case can also take parameters when it’s called. If you had another test case that acted on a Name and ID, those would be passed as part of the command:

WebUI.callTestCase(findTestCase('Tasks/LogNameAndID'), [('userID') : '1091504 ', ('userName') : 'Bob Smith'], FailureHandling.STOP_ON_FAILURE)

By using CallTestCase, test cases can be called when needed, not just in a linear fashion. This means they can react to what’s on the page and skip over tests that aren’t applicable. A test can be written once and used multiple times.
As another example, I have a contact generation test case. We have several Contact forms and instead of putting the same text in over an over again, I created a test case that uses Lists, text parsing and random numbers to generate a somewhat unique user details.
I have a list for first names, last names, company and state. I use random numbers for the zip, phone, and street number. When called, a name, address, phone, etc are generated and assigned to the Global Variables I have defined for the project.
It would have code like the following:

def states=['AL','AK','AZ']
def cities=['Dentsville','Woodcreek','Las Lomitas']
//Generate phone number
areaCode=Math.abs(new Random().nextInt(799)) + 200;
numPrefix=Math.abs(new Random().nextInt(899)) + 100;
numSuffix=Math.abs(new Random().nextInt(9000)) + 1000;
GlobalVariable.phoneNumber=String.valueOf(areaCode) + String.valueOf(numPrefix) + String.valueOf(numSuffix)
//Create an email address from first name, last name and company name
tempEmailAddress=firstName+'.'+lastName+'@'+domainName
GlobalVariable.emailAddress=tempEmailAddress.toLowerCase()
//Pick a city
rndCity=Math.abs(new Random().nextInt(cities.size()));
GlobalVariable.cityName=cities[rndCity]

To create a new user, I call the Contact Generation test case then set the text of the input fields to the values of the variables. It works quite nicely.
As time goes by, I will try to use the CallTestCase more frequently and work to my test cases more modular. It’s already starting to happen in a couple of places and it’s a very powerful tool.

Other articles of interest:

A Try Catch example in Katalon Studio

Since Katalon has the markError and markFailed commands, I don’t know if the Try Catch example is really needed. But, I did make use of it when trying to determine if a page loaded or generated an error.

After I click the link to load the page, I set the Try to the WebUI.verifyTextPresent(“Month To Date Sales Detail”, false). I need to determine if the text, Month To Date Sales Detail appears on the page. If it doesn’t, it means the page has an error and not only will the tests fail, but the data used for the page is invalid and needs manual investigation.

The Try is set to the text on the page. If that generates an error, the Catch statement then reads the Title of the page. This is written to the Warning log and I use the Throw command to break out of the test.

//Look for the text, Month To Date Sales Detail on the Page.
//If it's not there, there is a problem loading the page and the test should exit as the rest of the steps will fail
WebUI.click(findTestObject('Page_/Sales Dashboard/Monthly Details/Monthly Sales Accordion'))
try {
 elementPresent=WebUI.verifyTextPresent("Month To Date Sales Detail", false)
}
catch (Exception e) {
 title = WebUI.getWindowTitle()
 log.logWarning('ERROR: The title of the Sales Detail Page is:=' + title)
 throw new AssertionError('ERROR: The Sales Detail Page did not load correctly', e)
}

While this works and handles the error, I’m not sure if it’s the best method I could have come up with. Would it have been better and more consistent to use the GetText command with the CONTINUE_ON_FAILURE and mark the test as failed if the text didn’t appear? This would be more in line with how I check to see which tabs are available on a customer profile. This test was written quite awhile before my tabs example. This was more a brute force validation method, whereas the other seems a little more elegant and dynamic.

Either way, here is a very simple Try Catch example to determine of a page loaded correctly.

Other articles of interest:

Recent Comments