Importance of Continuous Testing (CT) in a DevOps World

Continuous Testing

Continuous Testing is the process of repetitive testing of a product through execution of automated tests, starting from build creation to system testing and finally to product release.

Continuous Testing follows a diverse path to testing a software product or an application before its release, ensuring that it is free from defects.  The process involves creation of a quick feedback loop, finding out the critical business risks associated and determine whether the software is ready to progress through the delivery pipeline at any given time.

Ultimately it helps the company cater to the needs of dynamic markets and ever-changing business requirements.

Comparative study with Traditional process…

Most of the traditional IT companies usually face hurdles trying to achieve deadlines and on-time deliveries. A lot is at stake as they work round-the-clock for accurate implementation of new features as per customer specifications. They generally end up with last minute issue fixes only at the time of release. Traditionally in IT organizations, teams with varying responsibilities (developers, testers and operations engineers) typically play their job roles in separate and siloed functional areas as distinct entities. For example, developer completes coding and development tasks as per functional requirement specifications and commits the code to a centralized repository like SVN or Git.

The QA team then starts preparing their test plans. Whether it is a Manual test process or an Automated Regression Test Script, it takes a long time to execute these tests, as all the development work must be completed before QA can start working on them. This causes long delays in identifying the defects and then the identified defects are again resolved by developer teams before the product can be released.

DevOps

But now in this Cloud era, fierce and competitive business environment has seen a major shift in the way IT organizations operate. Often they need to release products in much shorter cycles – daily or weekly or monthly basis. According to a study, more than 30% of the companies surveyed have started to follow continuous delivery or DevOps practices for their product delivery process. The number is quite significant considering DevOps practice to be a recent trend.

DevOps enables IT organizations to create, build, test, deploy and release software faster and reliably to its customers. By embracing DevOps, organizations undergo a paradigm shift in their culture as it removes team and tool silos, facilitates cross-team communication and collaboration among development and operations teams. This helps to identify issues early in the lifecycle and quicker resolution, faster delivery and proper management overall process.

The traditional Waterfall methodology for software development, which was linear and sequential had given rise to spiral sprint-based Agile for shorter and accelerated development cycles.

Further, by embracing DevOps culture, Ops (Operations) teams have eased out the pain-points developers face by provisioning necessary infrastructure just-in-time. But if Agile and DevOps have been both Developer centric, does that imply software testing and QA teams have been neglected?

Figure 1: DevOps Workflow

The DevOps practice of Continuous Testing synchronizes Testing/QA with Dev and Ops processes, optimized to achieve immediate market demand and business needs.

Continuous Testing – Way to reach DevOps

Continuity of each process and continuous feedback are the main aspects of DevOps. With accelerated development and infrastructure in place, the delivery pipeline needs upgradation for the feedback mechanism that would otherwise fail the entire DevOps objectives.

Thus, software testing that acts as a gatekeeper in identifying defects, robustness and mitigate critical business risks, need to be automated and implemented early in the cycle. DevOps in fact, has always encompassed the role of testers in the Dev part and this has led to the rise of Continuous Testing (CT) bridging the gap between Continuous Integration (CI) and Continuous Deployment (CD).

Steps to achieve Continuous Testing

1. Prepare test plan to validate the design of product as per customer specification

While a requirement specification gets approval for development from the Managerial end, the specification document is distributed among entire production team. As soon as the developers start the product design, the QA team starts setting up their test plan like validation test for designing Security risks and so on.

2. Transform automation test suite from a shared project to an integrated built-in deployment tool

Like product application code, automation scripts need to be staged in a version control tool. It enables integration of all automated test suite within a build deployment tool from where centralized execution and report generation could be possible.

3. Follow a good layered structure of automated test suite at each checkpoint for a quick (defect/fix) feedback before production release

Automation is essential for Continuous Testing. But automated testing is not the same as Continuous Testing. An Automated Regression Test on GUI is often inadequate when providing greater insight into the level of risks associated with the application, when critical issues continuously emerge from the production pipeline, failing to go/ no go business decisions.

Sometimes, issues passed during testing as per conditions laid out in a test suite might be unacceptable to a business. Failure in understanding the nature of risks while writing the Test Suite can affect the outcome. So, making a series of pipeline processes for production is not suitable here. Rather, it needs to be an end-to-end quality control process that executes activities in a parallel manner. In fact, for each build, a full regression is very time consuming.

Continuous Testing is layered around several types of testing by QA teams with varying scenarios – Early Test, Quick Test, Test Often and Full system test.

The following section elaborates on these classifications:

• Build Verification Test

A Build deployment tool (Jenkins, Hudson etc.) generates a build after each integration. It confirms whether the build has any update related error, whether the server has been properly up and running after build deployment, whether the application is properly responding and can be logged-in with valid credentials.

• Smoke tests and Sanity tests

After verifying the build, a Smoke test hardly takes 15-20mins to execute the key features of the application to check for any blocking issues. If everything goes correctly, the QA team concentrates on Sanity test that includes testing new implementation and bug fixes. The QA team needs to work hard during the selection of test cases to automate Smoke tests and Sanity test for a build. This is because, most builds get rejected are sent back to the development teams for defect fixation.

• Sprint Level Regression

It includes test cases for frequently used features, and critical parts of a product within a modest span. If the overall execution provides good report, then the build is considered for deployment process.

• Full regression (End to End Regression):

The test incorporates end-to-end testing of all core functionalities of the product. We usually undertake full regression execution during the weekend. Multiple machines are used for running a set of automated scripts to minimize the time delay in getting feedback. So this is the ultimate step of Continuous Testing process. If any failure occurs, teams are notified immediately for taking fast recovery measures.

Figure 2: Continuous Delivery Workflow

Kovair Implementation

Kovair Intelligent DevOps is the implementation of continuous delivery in the ALM process for a product’s release. It integrates tools like Jenkins, Team City for build creation and deployment.

After deployment, early testing is employed as the first phase of continuous testing. Reports regarding each stage of testing generated and notified to concerned teams. A Set of automated scripts are grouped for each stage, thus resulting in an optimized test result. Through implementation of Continuous Testing, QA team can create a quick feedback channel for whole deployment process.