AI in Testing: The third wave of automation
The evolution of agile methodology enforced the enterprises to innovate and deliver at lightning speed. While delivery cycle time is decreasing, the technical complexity required to deliver positive user experience and maintain a competitive edge is increasing—as is the rate at which we need to introduce compelling innovations.
To meet the continuous integration and delivery needs, we have turned to continuous testing backed by automation but how do we test when these trends continue and gaps widen? As this is the time of digital transformation, we need Digital Testing to meet the quality needs of future driven by AI, IoT, robotics and quantum computing.
If we look at how testing practices changed over time, till 2018 it was focused on CI/CD, scalability and continuous testing.
Now the expectations from testing are more about real-time risk assessment. To cope up the testing expectations in current scenarios, Artificial intelligence (AI), imitating intelligent human behavior for machine learning and predictive analytics, can help us get there.
Beyond Continuous Testing
If we analyze the journey since agile came into the picture, it has completely changed the way applications are delivered. Before agile, there used to be a release in a month or sometimes more than a month. With agile companies are aligned to have a two- weeks sprint and make a release in two weeks. To meet this, Continuous Testing came into the picture where automation suits were developed for regression and sanity testing. This supported quick deliveries and fast-paced testing cycles.
Now as the world is moving towards Digital transformation, the pressure to anticipate market requirements and build a system which is predictive and scalable enough to cater to future trends, going beyond continuous testing is inevitable. Testing will need additional assistance to accelerate the process. AI, imitating intelligent human behavior for machine learning and predictive analytics, can help us get there.
What is Artificial Intelligence?
Lets first understand what does artificial intelligence mean. Forrester defines AI as-
“A system, built through coding, business rules, and increasingly self-learning capabilities, that is able to supplement human cognition and activities and interacts with humans natural, but also understands the environment, solves human problems and performs human tasks. “
In simple words, AI enables machines to learn through data giving them the capability to make a decision. The algorithms are not written to solve a particular problem rather they are designed in such a way to enable the system to make a decision based on data.
How AI can be used in software testing?
Using AI and machine learning to automate-
a) Unit tests – Unit testing is very important to make sure that the build is stable and testable. With AI-powered unit test tools like RPA, a developer can get reduce the flaky test cases and maintenance of unit tests.
b) API testing- API testing saves time and effort by getting into the root cause of the issue. The problem with UI tests is that they are not reliable anymore as UI keeps changing in agile, while API tests give a deeper insight into the application and directly hit the root cause of an issue eventually making the application more robust.
There are many tools which are using artificial intelligence to help take the complexity out of API testing by converting manual UI tests into automated API tests, lowering the technical skills required to adopt API testing and helping organizations build a comprehensive API testing strategy that scales.
c) UI testing- The first step in automation is to convert manual UI tests into automated tests. There are tools which leverage AI to run the test cases on multiple platforms and browsers and also learn from the functional flow, reducing the maintenance effort and making testing more reliable.
Some of the most popular tools are mentioned below-
AI Powered testing tools- There are various testing tools which are using AI, though not harnessing the best of AI, they are still able to help testers a lot-
1. Applitools- It is an AI-powered visual testing and monitoring tool that can run tests on different browsers and platforms. It uses AI to identify the meaningful changes in UI and also identify them as bugs/ desired changes.
It also leverages ML/AI-based for automated maintenance (being able to group together similar groups of changes from different pages/browsers/devices)
2. Testim- It leverages machine learning into the most critical part of automation which is execution and maintenance of tests.
3. Sealights- Sealights uses AI and machine learning to analyze the code and run tests which cover the impacted area. It can be any kind of test- unit, functional, performance, manual, etc.
It provides a useful insight ‘Quality Risks’ which focuses user efforts on the things that matter by letting him or she knows exactly which files/methods/lines have changed in the last build that wasn’t tested by a specific test type (or any test type).
4. Test.AI- Test.AI is building as a tool that will add an AI brain to Selenium and Appium. It was created by Jason Arbon, co-author of How Google Tests Software and the founder of Appdiff. Tests are defined in a simple format similar to the BDD syntax of Cucumber, so it requires no code and no need to mess with element identifiers.
- AI just like a real person, IDENTIFIES the screens and elements in your app.
- AI EXECUTES user scenarios—test on-demand whenever you’re ready
- AI RECOGNIZES elements so that even if things change, your test doesn’t break.
5. MABL- Like the other AI-based test automation tools, MABL can automatically detect whether elements of your application have changed, and dynamically updates the tests to compensate for those changes. You just need to show the workflow that has to be tested and MABL does the rest.
6. Retest- Retest propagates an innovative testing approach, which is a combination of “intelligent” monkey testing and “difference testing” and works actually more like a GUI version management than conventional testing.
This tool does Monkey testing whereby the monkey( is called Surili) is artificially intelligent and can be trained by users by capturing user actions.
7. ReportPortal- ReportPortal, as the name suggests, is an AI-powered automation tool which focuses more on report analysis and management. As per its website it-
- Manage all your automation results and reports in one place
- Make automation results analysis actionable & collaborative
- Establish fast traceability with defect management
- Accelerate routine results analysis
- Visualize metrics and analytics
- Make smarter decisions together
8. Functionlize- Functionlize provides an overall solution for seamless automation with less/no efforts in maintenance all with the help of AI. Its AEA tool finds and fixes the broken test scripts thus eliminating the manual maintenance.
Functionize uses machine learning for functional testing and is very similar to other tools in the market regarding its capabilities such as being able to create tests quickly (without scripts), execute multiple tests in minutes, and carry out in-depth analyses.
It also gives scalability to test suites by maintaining them in the functionlize test cloud.
Challenges of AI in Testing
The machine learning process is completely dependant on the data thus leading to a large volume of the dataset. AI model test scenarios should be equipped to identify and remove human bias which often becomes part of training and testing datasets.
There is a lack of awareness about AI and Machine Learning process and proper training is required to the testers.
AI Testing in 2020 and beyond
As we have progressed from a linear waterfall model to agile, the future is all about AI and machine learning technologies. As a tester, we need to be upfront and start digging more about the various aspects of AI, take the hands-on in AI-powered tools and utilize them.
There are so many places where AI has already paved its way whether it be chatbots or Amazon’s Alexa, we need to be very keen about how we are going to the device out test cases to test such applications and deliver quickly.
With the increasing demand of AI-powered testing tools, testers might need training at earliest. Having advance knowledge of AI and its applications will be very helpful.
Talk to our Test Engineers
Fast track your ecommerce monitoring
How could Service Virtualization enhance your Agile Testing Strategy?
How could Service Virtualization enhance your Agile Testing Strategy?
How often you push your “Phone Updates” to a later date as an update install means your phone would not be available to work. Even though the updates just take a couple of minutes, but in a fast-paced world where a minute delay could mean a lot, non-availability of services for a fraction of seconds frustrate you.
Often the similar situation is faced by a Tester/QA while performing his daily testing jobs – environments are not up, dependencies are ambiguous, developers/business users need the SIT environment and ‘n’ number of such reasons.
That’s where Service Virtualization comes into the picture!
In fact, Service Virtualization is all about continuity and boosting agility!
What is Service Virtualization?
Service Virtualization in laymen terms is providing services in a virtual manner. Just like on the internet you have virtual friends, virtual classroom or virtual 3D view of an art gallery, in a similar fashion the services exist for a tester/QA to test, but are not the real services.
Service Virtualization by its means simplifies the issues faced by a Service-Oriented Architecture [SOA] testing strategy. By providing a virtual set-up of services, it speeds up the testing and development process.
A quick example that I could think of is “market data” provided by Bloomberg, Reuters, ACTIVE, WOMBAT that is the lifeline of many investment firms. But for a tester to test the market data scenarios they are connected either with a simulator or a different service that stores the recorded data [ a replica of live data].
By connecting to a virtual service, testing is continuous, and the live environment does not get affected.
How Service Virtualization Could Enhance Agile Testing
Technology is making things simpler for the end users, but complexity is hidden beneath under wraps giving a tough time to developers and testers. Be it chatbots, Pokemon Go or a virtual reality venture showcasing you hotel suites from your living room, the building/testing of tools and apps is not simple enough. To streamline the testing/developing process service virtualization when combined with #Agile becomes handy –
- Get Continuous – Bid goodbye to delays and waiting time for services from production, DevOps could now leverage virtual services and boost continuous testing.
- Integration is Seamless – With virtual services present at your disposal, integration testing is just a matter of connection and few clicks.
- Real Time Analysis – With virtual services, the analysis could be done real time helping in finding issues at an early stage.
- Cost Efficient – With increased complexity of components, simulating them for development/testing environment could be a risky affair. Service virtualization helps in reducing that cost but still provides the benefit of a real-life scenario.
- Enables Parallel Testing – With virtual services, testers/QA could test the pre-prod/UAT environment in parallel accelerating time-to-market and providing a quality software.
- Competitive Advantage – With Service Virtualization firms could process parallel development, quick fixes and boost quality that provides an added benefit when compared to companies following the linear development strategy.
- Good fit to ever changing requirements – In agile when requirements are ever-evolving, virtual services with their flexibility and easy-to-use parameters helps in maintaining the continuity and still follow the Agile process.
- Collaborate, Share and Grow – With multiple teams working on the assets, it gets tough to share the resources in Agile, but service virtualization facilitates on-demand access to resources anytime to any group.
- Test anytime – With virtual services you can test anytime, be it functional, integration or performance testing.
As in our last post, we saw “ The Best Practices of Continuous Agile Testing” we know continuity is the key for any software/tool to survive. With service virtualization, we are a step closer to make it continuous agile and quality-assured!
LIKE THIS POST SHARE IT WITH YOUR FRIENDS
Here’s a Quick Way to Get Agile-Checked in 2019!
Solve Business Problems by being Agile-Checked! Would you?kinkar brain was cluttered, being a technical head he was committed to delivering a project within six months, and after working for five months on a project when they were ready for the first UAT, the code refused to float because of incompatibility of browsers.
Had he been Agile-Checked; he could have avoided the last-minute pressure. Seems unbelievable? Let’s run through some facts.
What is Agile Checked?
The two things that drive a business are
- What Do Customers want?
- How Quickly can you present them with Assured Quality?
Rest all are supporting parameters like skills, logistics, monetary fund’s, etc.
Being a business owner, you do not have much control over “What customers want” as they can demand a “Trip to Moon.”But being a businessman you do have a control on how quickly you present “A Trip To Moon” with Assured Quality! That’s where Agile Manifesto AgileManifesto comes into the picture.
Being Agile-Checked is all about
- How frequently you push your software/application into a live site
- Concrete Planning with user stories and set timelines
- Working in smaller chunks that could be rolled back in case of issues
- Your Client Could view “How the Trip to Moon “is being Planned i.e. setting the expectations right!
- Tracking and evaluating manhour that helps in SMART planning
- Continuous Execution and Deployment
- Working as a TEAM – that involves each and keeps them motivated.
- Growing your business as you involve stakeholders
- Continuous testing that helps in improving the quality
How To Be Agile-Checked?
To be educated about SCRUM/Agile is easy as many courses and certifications could help you, but all that learning can only be put into practice when you know the “Best Practices” to be followed –
- Collaborate with Customers as team
- Review your Product Backlog before every iteration
- Keep it time-boxed
- Let business and technology be compatible
- Weigh each user story with three parameters – Complexity, time and Business Value
- Keep SCRUM short and crisp
- Automate testing
- Have strict programming guidelines or go for pair programming
- Adopt to Test Driven Development
- Let the Burndown charts not burn your skills
- Package your code as a gift that is easy to deploy and ready to use
- Let the Sprint demo meeting be as the code is going LIVE
- The best communication is through speech
- Change is welcome, be adaptive
Does Agile-Checked Help Me to Deliver Quality?
The short Answer is Absolutely YES, and the long answer is let me explain with an example.
Koovy, the head of the development team, is working on the new product that needs to be delivered within six months, they bagged this project because they bid less time against their competitors. So now Sales team has done their job it’s time for the Tech-heads to prove themselves. Let’s compare the three models under different parameters –
|Expected Met Actual|
If we pick a Waterfall, model –
It could be risky, as considering we devote 4.5 months to designing, developing and unit testing we would need another months’ time to test and fix the bugs, now if there are any changes to the code or design changes that would not have enough time to set. So, could lead to uncertainty!
If we Pick V-Model –
With flexible requirements, V-model could prove to be a timely affair, as the verification and validation happen in parallel, any change would also need to be reworked. Just for example if there is a change in the login screen, the parallel task of user manual also needs to be redone, and as we have limited time, this model may not prove to be effective.
If We Pick Agile- Methodology-
If we structure the work and divide it into deliverable modules, we could complete the work within the estimated time, would be ready for any surprises and clients could review the work in parallel, helping them to know what’s being cooked and if it matches their expectations.
Well, there are no clear ground rules defined as which methodology should be picked up under which circumstances, however with ever-evolving technology it’s highly recommended to be adaptive to changes and mark your application/software as Agile-Checked. Would you?
LIKE THIS POST SHARE IT WITH YOUR FRIENDS
Fast track your testing process
Microservices - The New Kid On The Block
Micro Services – The new kid on the block
As the Agile form of software development is making waves and ruling the roost in the software game, a new player is slowly emerging – microservices. It is similar to the many small iterations that characterize the Agile development module. So what exactly is microservices?
To understand this, let us first take into account the two basic styles of software development. First, we have a monolithic design, which is a single application entirely contained in a single process.
This sort of application has several groups of classes that are interdependent. Thus, if changes are made to one class, it affects all the other classes.
Next, we have the microservices architecture. Here, the application has micro-services that come together to make up the whole or combine their processes to execute a task.
Every single microservice has its own specialized, customized responsibility within the application’s structure. They are not dependent on each other, and thus, can be deployed and tested individually instead of having to rewrite the entire application.
Applications in Testing
Microservices have various applications in the automated testing field. Let us first examine testing in isolation. Since microservices are isolated from the application, the entire application need not be changed when you change a microservice, and therefore you can test a microservice in isolation. This is an easy place to start your automated testing journey as it will not affect your entire product, only a part of it.
This does not mean that you don’t have to test your entire application after testing the microservice in isolation. You still have to do that to ensure the microservice has integrated as well as its predecessor, but this is very easy and doable to start to the automated testing journey that will save you a lot of time in the development field.
In an application that is based on microservices, end-to-end testing becomes complicated. For your end-to-end testing needs, a good idea is to follow the Pareto Principle that states that 80% of the consequences stem from 20% of the causes. Thus, the approach you should take is to identify your core services that are absolutely essential and automate them, as you need the least margin of error in testing there.
Pro-tip: Do your testing in production
If you want to keep the quality of your output high, you need to test while in production to end up with the best possible version of your product. Microservices development module is fluid, as every piece does its own work, and behaves in isolation. Thus, unless you test during production, you will not have a clear idea of how the services are interacting with each other, nor will you be able to test this interaction.
Another important aspect that you will need to incorporate into your development module is an excellent monitoring and alerts infrastructure. This is because you need to quickly get on top of any issues that might crop up. Since every single service needs to work in order for your product to achieve an end result, you need to be able to identify, isolate and fix any problem that comes up with lightning speed.
This is cut down if you have a good testing in production framework that includes monitoring and alerts as well. Finally, if your monitoring and production testing is spot on, then not only will you be able to respond to the issue quickly, but you might be able to devolve to a more stable build even before your users know that a problem has occurred.
If you want to learn more about being more productive with Test Automation, contact us at CloudQA ([email protected])