Case Study
Continuous Automation Case Study
At a glance
- Customer: A leading national healthcare company in the US, with four integrated web applications in scope
- Challenge: Weekly and bi-weekly releases with many tickets, all regression testing done by hand, across browsers and devices
- What CloudQA did: Automated regression for all four applications on a codeless platform integrated with Travis CI, added API-level tests for the middleware, tracked defects in JIRA, and monitored critical production flows every 15 minutes with alerts in Slack
- Outcome: Continuous regression in the CI/CD pipeline, defects found earlier, results archived for audit, and a significantly better mean time to repair for production issues
Client Context:
A leading national healthcare client in the US. Four applications were in scope for test automation. All the regression testing was earlier being done manually.
Scope of Work:
- Automated Testing of all the four applications
- Executed the project in Agile SDLC
- Defect tracking has been done using JIRA Tool
- Selenium accelerator custom platform built with .NET core and using Progressive automation testing approach, started with early automation
- Middleware test automation executed to find defects at the API level
Business Challenge/ Driver:
- High number of tickets planned for Weekly/Bi-Weekly release cycle which has to be verified across browsers/devices for four integrated web application
- Regression testing to be automated with all manual regression to be automated
- Performing regression tests across various browsers/ platforms
- All the projects run in agile mode, frequent production releases, hence shorter test cycles
- Identify and resolve production issues in a timely manner
CloudQA Solution/ Approach:
Using a codeless robust automation platform using Selenium 3.4 & .NET core integrated with Travis (CI Tool) with exhaustive test reporting
- Found defects early in the project life cycle reducing the overall cost spent on the project
- Key workflows, as well as time-consuming test scenarios, are automated
- Continuous integration/ continuous development (CI/CD) automated regression suite
- Monitoring critical flows in production every 15 minutes to provide timely notifications about any functional or performance issues in Slack
Service level tests created for testing REST based middleware
Cross browser testing – Used the automated regression suite to test the application across browsers enabling user confidence across all the browsers right
Benefits Delivered:
- The automation platform is being extended/ customized for other applications of the Client and has been integrated with Travis CI Tool for automated runs
- Continuous regression through CI/CD Automation framework reduces the efforts of repetitive functional tests prerequisites and expedites overall functional test execution
- Continuous Integration helps in faster identification and resolution of defects, with a good reporting structure
- Ensures optimized testing while maximizing test coverage
- CI/CD enables testing teams to release deployable software at any point in time with the automatic build and compilation processes.
- Test results archived for auditing and analysis purpose through a test report server
- Meantime to repair for production issues improved significantly
Lessons for your team
- Start automating early in the project rather than after the application is finished.
- Test the API layer as well as the user interface; many defects are cheaper to catch there.
- Keep watching critical flows in production, not only in test environments.
- In regulated industries, archive test results so audits are easy.
Frequently Asked Questions
What was automated in this project?
Regression testing for four integrated web applications, cross-browser runs of the same suite, and service-level tests for the REST middleware.
How were the tests run?
The automated regression suite was integrated with Travis CI so it ran as part of the CI/CD pipeline, with detailed reports kept on a test report server.
How were production issues caught?
Critical flows were monitored in production every 15 minutes, with notifications about functional or performance issues sent to Slack.