Risk Based Test Management

It's only fair to share...

Risk based testing has become more relevant recently as most of the teams are following agility concepts and the ‘time to market’ span is getting shorter. Iterative methods are being followed for incremental deliveries based on the priorities defined by the stakeholders. In such a situation, risk identification and mitigation is playing a big role in making a suitable Test Plan to get aligned with Delivery and Operations teams. Risk based Test management is a potential solution to accelerate the timely delivery focusing the Business criticality and Customer Requirements.

“Potential risk” is not the same as “risk.” 

A Potential risk is the substantial uncertainty which may occur in future and endanger the project objectives. All projects are not subject to same kind of risks. Nothing remains constant and risks change over time. So organizations need to forecast and assess the Potential risks before the decisions are made.

Benefits of Risk based Testing

Quantitative and Qualitative Approaches of Risk Analysis

Risk can be measured in two ways: Qualitative and Quantitative. When risk is measured quantitatively, it is based on actual values: cost or time or combined effects of both. Quantitative process involves the probability distributions and the impact areas. For example, if an e-Commerce site is offline for a day it loses business. Historical and projected data can quantify the loss for the organization in sales. Collecting the data to perform a risk assessment is time consuming and cumbersome and hence expensive. Qualitative risk assessment is done on an arbitrary and user defined scale e.g. High, Medium and Low scale or 1 to 10 numerical scale. In Qualitative risk analysis process stakeholders’ expert judgement is used rather than mathematical approaches. For most of the organizations, it is used to make accurate expenditure decisions.

Risk Assessment – The Six Sigma Way

Six Sigma methodology provides a structured approach that is capable of identifying risks and finding the inadequacies which may affect the desired outcomes. Six Sigma risk management tools follow a data driven analysis with the magnitude of potential events. Some of the Six Sigma risk management tools are given below which help organizations to identify their risks, impacts of the risks and the likelihood of occurrence by analyzing historical data.

Cause and Effect Matrix

The Cause and Effect matrix is a useful tool for root cause analysis. Participation of all stakeholders is required for brainstorming and forming a Fishbone diagram by identifying possible causes. Numerical scores help to understand which activities create the risk and the critical steps present in the process.

Failure Mode Effect Analysis [FMEA]

FMEA is a systematic and qualitative tool, widely used in early development cycle for analyzing potential reliability or quality problems. FMEA is calculated on three factors:

  1. Severity : It encompasses what will be the impact on the customer
  2. Occurrence: Measured in a scale of 1 to 10, ranks indicate how frequent this incident is likely to occur
  3. Detection: the probability of incident being detected

Participants define a scale of numeric values e.g. 1 to 10 [low to high] for each criterion and calculate the Risk Priority Number [RPN] and start prioritizing them on Pareto principle.

Risk Governance

The major role of risk governance is to define an acceptable level of risk for an organization. In general this is set by the senior management through discussion with the stakeholders. After setting a benchmark for the risk tolerance level, known as Risk Appetite, risk assessment is done to identify the risk values higher than the defined acceptable value and additional actions are taken to mitigate them.

Risk Responses

Murphy’s Law states: “Anything that can go wrong, will go wrong.”

So in order to mitigate risk and eliminate what has gone wrong, there has to be an effective way to respond and take corrective actions.

There are four ways to respond to a risk:

  1. Acceptance: We can assess the risk, do nothing and continue as is.
  2. Avoidance: We can accept the risk and avoid it by not participating in any risky activity.
  3. Transference: We can transfer the risk to another entity.
  4. Mitigation: We can mitigate risks by adding controls or modify the risky activity by changing the likelihood or impact.

The goal of risk mitigation is to reduce the likelihood from a higher value to lower one or to the level of impact or both. The mitigation process consists of following steps:

  1. Brainstorming the possible controls.
  2. Assessing benefits.
  3. Estimating cost.
  4. Evaluating the resultant likelihood.
  5. Impact and residual risk.

Figure 1: Reducing the likelihood and impact of Risks

While it is difficult to find or to select tools that address the Risks mentioned above, the Kovair Integrated Test Management attempts to deliver on the following aspects.

Kovair Integrated Test Management

Kovair iTM (Integrated Test Management) solution features ease of customization along with strong reporting capabilities. Stakeholders can identify the risks associated with the critical business requirements and provide their views. Kovair also provides a good overview on the impact areas by establishing the traceability between the work-items.


Highlighting some of the key features offered by Kovair iTM:

Collaborative Review

A major stakeholder [moderator] can open a forum and select other stakeholders as reviewers. The reviewers can post their comments, vote, and like/dislike and subscribe to the items. They can get notifications on change. This provides stakeholders a platform for contextual discussion and real time tracking of on-going reviews.

Impact Analysis

Work-items can be linked by custom relationships and the impacts associated with the artifacts can be analyzed. Based on priority, high impact areas can be identified and mitigation plan can be derived. Test Plan gets changed with the identified impact areas.

End-to-End Traceability

Kovair Traceability relationship is a powerful control to get a centralized overview of the cross-artifact relations. Bi-directional relation between the business Requirements and the Test artifacts helps to trace the changes affecting each other, thereby increasing the efficiency to measure the residual risks associated with the overall project.

Test Execution

Kovair Test management supports both manual and automated test execution. Kovair Omnibus platform allows users to integrate Test artifacts from best-of-breed Test management tools. Automatic execution of Test cases can be triggered based on events like changed priorities. The following diagram demonstrates the capabilities of the Omnibus Integration platform to allow users to choose from the best-of-breed tools that they wish to choose.

Figure 2: Test Management Lifecycle using Kovair Omnibus Integration Platform

Comprehensive Metrics

Kovair iTM has a powerful report engine.  Real time metrics, summary graphs can highlight the actions to be taken. Custom Dashboards can be configured with the drill down facility for management personnel pertaining to their needs to make critical business decisions.

Figure 3: Kovair Analytics Dashboard

Kovair iTM solution remains instrumental for various challenges of software testing teams. All the activities starting from test planning to test execution and bug reporting can be done through Kovair iTM. As different organizations follow different workflows, Kovair iTM provides the flexibility to configure workflows accordingly and set business rules without writing any single piece of code.

With the cultural shift in testing practices, organizations are looking for shorter ROI cycles, and the value proposition from Kovair iTM is it’s capability to stay flexible, adjust to the company’s specific business needs and provide clear visibility of metrics that utilize time and money in the most optimal way.

It's only fair to share...

Shubhrojit Sarkar is the Manager - Testing and QA at Kovair Software. He has overall 7+ years of experience in Software industry, predominantly in Testing, and QA. Shubhrojit heads the Testing division at Kovair and his key responsible areas include guiding testing teams for both manual and automated tests, managing integrated testing framework for clients, and many more. He loves trekking and photography at his leisure time.