Test Automation Best Practices

This post is about test automation and best practices. Have you ever failed with test automation? No? I don´t believe that. But still then I would have an advise: Read this post, to avoid any fails in the future.

Software is today often the heart of many companies and software applications must be tested before going into production. The applications are also more and more complex. Trying to test everything manually is a big challenge, as you would need a lot of testers to test every feature with every release cycle. Manual testing can become very extensive and time consuming and costs a company a lot of money. Business users know the software, but they are not testers. And testers tend to test features which are not always relevant for the application under test. Do the manual testers know the happy path?

So what is the best way to ensure, that the application goes into production with as less bugs as possible?
Here comes test automation into play. Automating test cases means, you can run more tests in shorter time and so improve the quality of the software application.
Robots can run the test cases 24/7 and even parallel. But not everything can be automated, so let the manual testers do test only new yet unstable features and let them do some exploratory testing.

But test automation is not easy and has some pitfalls. So where to start, where are the challenges and pitfalls?

Here I want to cover functional testing on the GUI or API level. Non-functional test automation like performance, load or security tests are out of scope here.

Where To Start with Automation

Choose the right testing framework which fits best for your application,
Engage testers and automation engineers with the skills to automate your application,
Start with unit tests your need to run very often and which are monotonous and tedious to execute manually,
Unit tests are cheap and easy to write,
Find the most vulnerable parts of the application and start automating there,
Automate the testing of all methods and classes, this way you can ensure functionality on a low level,
Use design patterns such as page objects,
Communicate well the right expectations to stakeholders, what are the costs, what are the benefits and time savings,
Have a clear testing strategy and determine the goal of the test automation.

Test Automation Challenges

Test Automation can reduce testing time significantly, when properly implemented. Usually functionsl regression tests, which needs to be run many times are automated. This may be unit tests or integration tests but also system and end-2-end tests, which needs to be run after every build to make sure the application is not broken after a change.

But test automation is not something out of the box everybody can create and run. You needs testers for designing the tests and automation engineers with coding skills in Java, JavaScript or any other languages used and you need a well documented test plan in order to create the right scripts.

But test automation engineer with these skills are rare and these people are expensive. In that case you can think about low-code or no-code automation tools, where you don´t have the burden of needing automation engineers with coding skills-

Why can Automation Fail?

Test automation is hard to get right and can very easily fail,
Choosing the wrong automation tools for your application to test,
No proper planning before you start creating automated test cases,
Underestimating the time and costs to setup and put the automation in place,
When you try to overautomate, that will not work,
100% automation should not be the goal
Automation can give a false sense of security,
Don´t rely on passing tests, but more on finding bugs through automated test cases,
When you did not set the right expectations,
When you have many assertions and interactions in one test.

What are the Biggest Challenges?

Human factor,
Set the expectations right, as 100% automation is never possible,
Decide early, which test cases can be automated and how to automate them, otherwise you may spend too much time on test cases which are hard to automate
Don´t start with the most complex test cases, do cherry-picking at the beginning otherwise you may fail early
Understanding, where manual testing is still needed (like UAT or exploratory tests),
Testing itself can´t be automated, only checking or verifying a functionality can be automated,
Automation is not static, you need to maintain your automated test cases,
When to use parallel execution as time saver,
In Webtesting: No IDs, dynamic elements, popups, cookies, re-captcha,
Time-out handling for your test cases,
One application test engineer is not enough, engage the whole team in automation, as collaboration is important here
Collaboration, as test cases need to be reviewed and signed-off,
Test environment has to be static, so you need cleaning procedures,
When tests are failing due to wrong or buggy environment settings,
Test the test: test scripts needs to be tested as well, as the automated test case can be buggy,
Test Automation for older and fixed application versions.

Pitfalls of Test Automation

Test cases are not stable enough and deliver different results on the same application,
The application changes very often, old tests are not repeatable after a change and must be rewritten,
Test automation takes long and the test is run only once
When using the wrong tools,
When created by testers who do not know the application

As a result of all these positive and negative outcomes of test automation, here some best practises to avoid all the pitfalls:

Functional Test Automation Tips

Test automatiom is not about testing, it is about checking the as much of the functionality to determine the quality state of the application.
Gather accurate requirements before writing the first automated test case,
Determine the right scope for test automation for every release cycle,
Create an accurate test plan and the right test strategy,
Make sure that test cases are accurate and precise,
Develop an automation framework, which is easy to understand and to maintain,
Create libraries for test cases for reusability (Login preocedure is always the first and best candidate)
Use data driven tests, this allows you to test a complete dataset with only one test case
Never use personal logins for testing, you should have a test account
Never use production data for testing, when using data driven tests,
Setup a correct test environment
Mock any API Calls, when they´re not available in the test environment
Find synthetic data sets for automation
All test scenarios should be realistic and make sure they can be automated,
The more generic your test scripts are, the more reusable they will be,
Automate only stable functions and not these which are still changing on a daily basis,
Avoid useless tests with no outcome,
Does the test run fast or is there a way to speed it up?
Automate test cases so they resistent to any changes on the GUI,
Be prepared that only 20% of the scripts will have a pass rate >90%,
Keep things simple, follow the KISS priinciple,
Only one assertion per script (one result of the test case),
The more interactions you have, the less stable will the test case be,
One test case should not run longer than 30 seconds and the test suite not more than 10 minutes.
Don´t forget about test reports, where everybody can see all the results for a release cycle
All test cases should be categorized in at least 3 categories, so you can do a proper risk analysis for the release
Before automating the first test you should collaborate with business users to find out, which functionalities of the application are used very often

Goals of Continuous Testing

Continuous testing is the testing after every build, most often it is part of the CI/CD pipeline.
Continuous testing may include unit and integration testing but also End-2-end tests to ensure that the build does not break the whole functionality.
Check all functionalities as early as possible,
Quick delivery of changed or new features with quality approved code,
Test automation starts to make sense, when the scripts are included in a CI/CD pipeline,
A build is successful, when all automated tests for this build do pass,
Is only successful, when the automation scripts are stable,
Allows to increase platform coverage when all tests are run against different platforms (OS, Browsers, Versions)
With continuous testing you get faster feedback about the quality of the application.
Measure the time for the whole test cycle and use parallel test whenever necessary, especially when the time slots are very tough