Test Automation in Production Environment
If you have landed up to this article, I assume that you are an automation test engineer and very well aware of the advantages of automation testing. If not, you can have a look at Manual vs Automation testing.
This article deals with the automation strategy and implementation in the production environment. As an automation engineer, we set up the test environment as a part of the automation life cycle and run scripts on that. We also have a staging environment that is supposed to simulate production.
In most organizations, staging is a miniature or replica of production and they try to make it in sync as much as possible. It is mandated to test and run scripts on staging and sign off before each release.
Then ‘why test in production’?
- Testing in production gives confidence that the application is working fine.
- It gives the tester real-time scenarios where the test cases are not predefined and users keep on changing data.
- Edge cases related to network failure, slow connections, high traffic, etc can be found and tested.
- To monitor API response during peak traffic.
- Catch bugs at an early stage and fix them before they impact user experience at a much bigger level.
In organizations, a different instance of the database is kept, synced with production data on a scheduled basis. This further masks the sensitive information like user personal details. The problem with this approach is that often the staging cluster size is much smaller than the production. Now the configuration options for eg – load balancer, queues, the size of the thread pool, etc. are pretty different than production. The chances of getting issues in scenarios where traffic is high and multiple systems are involved, become very less in staging. I believe this gives a fair idea of why we must test in production.
Though production testing can be as basic as executing sanity checklist manually or as detailed as we may want. Here I will not be discussing the manual approach. So let’s directly jump on to how automation can help test the production environment. In this article, I will try to answer a few questions related to this –
- The automation approach to test in production
- Risks of testing in production systems
- Advantages of automation
- Tools to automate production test cases
Approach of automation in production- Testing in production is another quality guard around the AUT.
While we should always test at each stage of SDLC, a failure might happen in production. If as a tester we regularly run our automation suite in production, the root cause, and fix is quick and early without impacting the business as it could at a later stage.
- Post-Live Sanity– An approach could be to create a sanity checklist of the most critical functions of the application and create/ reuse the test scripts for them. Having a sanity suit ready and schedule it whenever a deployment is made. This can be further integrated into the CI/CD pipeline.
- Monitoring- Monitoring can be done with a set of functional test scripts scheduled for production. The key to choose test scripts is that those should cover user behaviour and should not manipulate data in production. With the DevOps integration and CI/CD evolution, Many organizations have come up with the concept of ‘ QA in Prod’ which means necessary changes made to the application to support testing in production. Another way is to mock real-time data to run test scripts.
- Synthetic Monitoring- Synthetic monitoring is used to health check the application with respect to actual user behaviour. This is critical to provide seamless user experience by finding and fixing issues as quickly as possible. Synthetic monitoring test scripts are based on past user behaviour data. It tends to find problems users might have with the application. It addresses the following issues-
- Is the website up and running?
- Are the website functions working properly?
- Is there any failure and if yes where it failed?
- What is website performance in terms of API responses?
- Are the 3rd party components working?
- Alerting- Alerting systems are used to send alerts if something goes wrong in production. Alerts can be sent in any form- email, SMS, call, etc. There are many tools available to configure alerts. One of the most popular tools is NewRelic.
Risks of automating in production
The process of testing in production depends on several factors. The major factor is the application itself and what exactly needs to be tested. Test Engineers should keep in mind that there should not be any manipulation of production data. Businesses could experience a loss of transactions or commingling of test data with production data. Experts agree that a major risk of testing in production is the business risk. A bad user experience, security issues or system crashes could all lead to a loss in profit or tarnish a brand.
Major risks are-
- Exposing potential vulnerabilities to the public.
- Loss of data/ Manipulated data.
- Poor user experience due to instability of application.
- Security issues or application crashes.
It is critical to address these risks and design a strategy accordingly.
Advantages of Automation in Production– Automation Testing in production has many advantages-
- Scheduled monitoring and alerting of issues before it can impact user behavior.
- To maintain the desired quality of the application.
- To verify the performance of the application in high traffic and response time of web pages.
- To support continuous testing in agile model.
Tools to automate testing in production- Automation in production may involve-
- Creating a sanity checklist first and write test scripts for the same.
- Creating a monitoring suite with integrated test scripts and integrated alerting system.
- Integrate the automation suit in CI/CD pipeline.
CloudQA’s truMonitor is a complete tool to achieve above requirements.
- Simulate business-critical customer journeys, use the recorded test cases to check critical flows
- Ensure that your web application is available and performs well
- Reduce your hours of manual troubleshooting down to seconds by analyzing hundreds of test executions, thousands of objects, millions of dependencies, and billions of events in seconds.
How does it work?
CloudQA can monitor both your test cases flow and URLs
- Navigate to TruMonitor, Add a test case
- Search and select the test case from the list, set the frequency and save
- Click on the test case to view results
- The results have the following information
- Availability analysis, Performance analysis, SLA report and KPI Trends.
- Execution information – You can set up Notifications for any functional failure during monitoring and also get performance alerts based on the Threshold value.
- Clicking on status will direct you to the step results page.
- Navigate to Monitoring report for detailed analysis of your application.
All the Critical errors in the test cases flow, can be reported to you through SMS.
RECENT POSTS
Guides
How Test Automation Accelerates Time-to-Market and Boosts Software Quality
How Test Automation Accelerates Time-to-Market and Boosts Software Quality
How To Select a Regression Testing Automation Tool For Web Applications
Regression testing is an essential component in a web application development cycle. However, it’s often a time-consuming and tedious task in the QA process.
Switching from Manual to Automated QA Testing
Do you or your team currently test manually and trying to break into test automation? In this article, we outline how can small QA teams make transition from manual to codeless testing to full fledged automated testing.
Why you can’t ignore test planning in agile?
An agile development process seems too dynamic to have a test plan. Most organisations with agile, specially startups, don’t take the documented approach for testing. So, are they losing on something?
Challenges of testing Single Page Applications with Selenium
Single-page web applications are popular for their ability to improve the user experience. Except, test automation for Single-page apps can be difficult and time-consuming. We’ll discuss how you can have a steady quality control without burning time and effort.
Why is Codeless Test Automation better than Conventional Test Automation?
Testing is important for quality user experience. Being an integral part of Software Development Life Cycle (SDLC), it is necessary that testing has speed, efficiency and flexibility. But in agile development methodology, testing could be mechanical, routine and time-consuming.