Automation Detailed Analysis

Test Automation in Production Environment

If you have landed up to this article, I assume that you are an automation test engineer and very well aware of the advantages of automation testing. If not, you can have a look –

This article deals with the automation strategy and implementation in the production environment. As an automation engineer, we set up the test environment as a part of the automation life cycle and run scripts on that. We also have a staging environment that is supposed to simulate production.

In most organizations, staging is a miniature or replica of production and they try to make it in sync as much as possible. It is mandated to test and run scripts on staging and sign off before each release.

Then ‘why test in production’?

  1. Testing in production gives confidence that the application is working fine.
  2. It gives the tester real-time scenarios where the test cases are not predefined and users keep on changing data.
  3. Edge cases related to network failure, slow connections, high traffic, etc can be found and tested.
  4. To monitor API response during peak traffic.
  5. Catch bugs at an early stage and fix them before they impact user experience at a much bigger level.

In organizations, a different instance of the database is kept, synced with production data on a scheduled basis. This further masks the sensitive information like user personal details. The problem with this approach is that often the staging cluster size is much smaller than the production. Now the configuration options for eg – load balancer, queues, the size of the thread pool, etc. are pretty different than production. The chances of getting issues in scenarios where traffic is high and multiple systems are involved, become very less in staging. I believe this gives a fair idea of why we must test in production.

Though production testing can be as basic as executing sanity checklist manually or as detailed as we may want. Here I will not be discussing the manual approach. So let’s directly jump on to how automation can help test  the production environment. In this article, I will try to answer a few questions related to this –

  1. The automation approach to test in production
  2. Risks of testing in production systems
  3. Advantages of automation
  4. Tools to automate production test cases

 Approach of automation in production- Testing in production is another quality guard around the AUT. 

While we should always test at each stage of SDLC, a failure might happen in production. If as a tester we regularly run our automation suite in production, the root cause, and fix is quick and early without impacting the business as it could at a later stage.

  • Post-Live Sanity– An approach could be to create a sanity checklist of the most critical functions of the application and create/ reuse the test scripts for them. Having a sanity suit ready and schedule it whenever a deployment is made. This can be further integrated into the CI/CD pipeline.
  • Monitoring- Monitoring can be done with a set of functional test scripts scheduled for production. The key to choose test scripts is that those should cover user behaviour and should not manipulate data in production. With the DevOps integration and CI/CD evolution, Many organizations have come up with the concept of ‘ QA in Prod’ which means necessary changes made to the application to support testing in production.  Another way is to mock real-time data to run test scripts.
  • Synthetic Monitoring- Synthetic monitoring is used to health check the application with respect to actual user behaviour. This is critical to provide seamless user experience by finding and fixing issues as quickly as possible. Synthetic monitoring test scripts are based on past user behaviour data. It tends to find problems users might have with the application. It addresses the following issues-


    1. Is the website up and running?
    2. Are the website functions working properly?
    3. Is there any failure and if yes where it failed?
    4. What is website performance in terms of API responses?
    5. Are the 3rd party components working?
  • Alerting- Alerting systems are used to send alerts if something goes wrong in production. Alerts can be sent in any form- email, SMS, call, etc. There are many tools available to configure alerts. One of the most popular tools is NewRelic.

Risks of automating in production

The process of testing in production depends on several factors. The major factor is the application itself and what exactly needs to be tested. Test Engineers should keep in mind that there should not be any manipulation of production data. Businesses could experience a loss of transactions or commingling of test data with production data. Experts agree that a major risk of testing in production is the business risk. A bad user experience, security issues or system crashes could all lead to a loss in profit or tarnish a brand.

Major risks are-

  1. Exposing potential vulnerabilities to the public.
  2. Loss of data/ Manipulated data.
  3. Poor user experience due to instability of application.
  4. Security issues or application crashes.

It is critical to address these risks and design a strategy accordingly.

Advantages of Automation in Production– Automation Testing in production has many advantages-

  1. Scheduled monitoring and alerting of issues before it can impact user behavior.
  2. To maintain the desired quality of the application.
  3. To verify the performance of the application in high traffic and response time of web pages.
  4. To support continuous testing in agile model.

Tools to automate testing in production- Automation in production may involve-

  1. Creating a sanity checklist first and write test scripts for the same.
  2. Creating a monitoring suite with integrated test scripts and integrated alerting system.
  3. Integrate the automation suit in CI/CD pipeline.

CloudQA’s truMonitor is a complete tool to achieve above requirements.

How CloudQA TruMonitor can help in designing a complete Production automation tests- CloudQA helps to find and fix issues before they impact your end-users. CloudQA will monitor and send alerts during low-traffic periods
  • Simulate business-critical customer journeys, use the recorded test cases to check critical flows
  • Ensure that your web application is available and performs well
  • Reduce your hours of manual troubleshooting down to seconds by analyzing hundreds of test executions, thousands of objects, millions of dependencies, and billions of events in seconds.

How does it work?

CloudQA can monitor both your test cases flow and URLs

  1. Navigate to TruMonitor, Add a test case
  2. Search and select the test case from the list, set the frequency and save
  3. Click on the test case to view results
  4. The results have the following information
  • Availability analysis, Performance analysis, SLA report and KPI Trends.
  • Execution information – You can set up Notifications for any functional failure during monitoring and also get performance alerts based on the Threshold value.
  • Clicking on status will direct you to the step results page.
  1. Navigate to Monitoring report for detailed analysis of your application.

All the Critical errors in the test cases flow, can be reported to you through SMS.


Dynatrace vs CloudQA Synthetic Monitoring Tools Comparison

Most website description of the tools only add to confusion and you can lose a ton of time if you pick a tool that doesn’t match your expectations. So, before you put all your faith the in the most popular synthetic monitoring tools it is better to question whether they are really worth your time.
In this article we’ll compare Dynatrace synthetic monitoring which one of the popular options and CloudQA TruMonitor which brings promising features that pack a punch. Staring with a short  introduction for both synthetic monitoring tools and then a juxtaposition of most relevant features for an in-depth tool comparison.

HubSpot claims, if the site brings in $100,000 / day in revenues, 1 sec page improvement will bring in extra $7,000. That’s how much a second is worth; it can mean a 7% increase or decrease in your revenues.


Dynatrace – Synthetic Monitoring:

Short synopsis: Dynatrace is an Application Performance Management (APM) software that runs for on-premise and cloud applications. Dynatrace – Synthetic Monitoring is one of its performance analysis tools.

How does it work: Record and playback tool for creation of monitoring scenarios.

Standout features: Monitoring for multiple locations in a single execution.

Challenges: Basic record and playback tool to build monitoring scenarios.

Pros: Global coverage, rich dashboard and detailed reporting.

Cons:  Doesn’t work on complex web applications like Single Page Application, no provision to edit a scenario after being created and high costs.

Bottomline: Dynatrace – Synthetic Monitoring tool is useful ONLY for very basic monitoring and can be very costly to operate.

CloudQA TruMonitor:

Short synopsis: CloudQA is a cloud based web testing and performance analysis platform. TruMonitor is a synthetic monitoring tool by CloudQA.

How does it work: Record and playback tool for creation of monitoring scenarios.

Standout features: Powerful recorder and agility in management.

Challenges: There is no out of the box APM but CloudQA works well with 3rd party APM tools from Google (name of the tool) & NewRelic.

Pros: Very easy to setup and use. Works with even the most complex web applications and is cost effective.

Cons: Geographical monitoring from XX locations compared to YY offered by Dynatrace

Bottomline: TruMonitor is the easiest and the most effective synthetic monitoring solution available in the market.  

Side by side feature comparison:

CategoryFeatureDynatrace – Synthetic MonitoringCloudQA – TruMonitor
CreationRecording ScenariosBasic recording of user actionsAdvanced recording of user actions with support for assertions and hovers
MonitoringBrowsersReal browser supportReal browser support
Location Based MonitoringMonitor multiple locations around the world Simultaneous monitoring of same scenarios across multiple regionsMonitor multiple locations around the world
ComplexitiesSupports basic web applicationsMonitor complex web applications like Single Page Application
MaintenanceEditing ScenariosOnly addition of validation and editing CSSPerform complex actions- Adding scripts, store variables, random variables, alerts, waits etc
3rd Party IntegrationsOut-of-the-box integrations with popular toolsOut-of-the-box integrations with popular tools
Scheduling FrequencyChoice of 5, 10, and 15, 30 and 60 minute frequenciesChoice of 5, 10, and 15, 30 and 60 minute frequencies
ReliabilityFalse Positive ResistanceBased on deviation from historical baseline metricRetrial of monitoring, automatic waits and robust CSS selectors
ReportingDashboardCan create and share custom dashboardBasic dashboard
Performance MetricsAvailability, uptime, full loaded time, network time, client time, server time, and resource timings.Availability, uptime, full loaded time, network time, client time, server time, and resource timings.
Performance Analysis– Line graphs, pie charts and histograms based date and time series analysis of Load times for URLs, and User actions – Interactive graphs with user action and URL filters – Run trends for status on application glitches while monitoring a scenario – Drill-down for glitches with screenshots – Other graphs– Line graphs and histograms based date and time series analysis of Load times for URLs, and User actions – Interactive graphs with user action and URL filters – Run trends for status on application glitches while monitoring a scenario – Drill-down for glitches with screenshots
Alerts and NotificationEmail, Dynatrace mobile app and custom integrations like Slack, PagerDuty and JIRASMS, email, and custom integrations like Slack, PagerDuty and JIRA
Competitive BenchmarkingUsing Apdex ratingsOn request
OtherSupport (Basic Subscription)– Product onboarding – Live chat – 8 Hours / 5 Working Days available– Product onboarding – Dedicated Support Assistant – Live chat – 8 Hours / 5 Working Days available
InfrastructureOn premise and CloudOn premise and Cloud
Price$ 0.35 per 25 runs$ 0.14 per 25 runs


With growing performance expectations and web complexities, synthetic monitoring is indispensable for companies who want to stay ahead of the competition. Unlike other monitoring tools, TruMonitor offers a comprehensive solution for analysis of user journey; even on complex web applications. Contact us to learn more on how you can leverage synthetic monitoring to become proactive in delivering quality digital experience.


Share on facebook
Share on google
Share on twitter
Share on linkedin
Share on email