Five Steps to Better QA Results

Five Steps to Better QA Results
Satish Nagarajan, Senior Testing Advisor – October 10, 2017

The Quality Assurance (QA) function is nearly universal in the Managed Care industry. Some large and complex Payers have multiple and separate QA and testing teams.  QA and testing is often mandated by contract or regulation and usually required as part of good software development practices.  Many senior Business and IT leaders view the QA function as valuable to their mission. Getting QA right is important!

While the QA function is widely recognized as valuable and mission enabling for delivering Health Insurance, in practice QA teams are not universally beloved.  Some people in most organizations feel their QA activities …

…take too much time

…cost too much

…don’t prevent (all, many, important) defects

…get in the way of progress

QA functions are expected to be more efficient and effective to support the mission and business strategy.

Five Steps to Improve Performance and QA Team’s Reputation

Step 1: Fit for Purpose Methodology

Every QA function should have a well defined, well engineered and well understood set of testing methodologies.  The methodologies should be engineered to be “fit for purpose” across the following dimensions:

Match to the SDLC (agile, waterfall, combination, etc.)

Nature of the change (custom code, configuration, setup, etc.)

Impact of the change (a few internal customers using a minor function vs. all external customers)

The fit for purpose methodology needs to be supported by engineered processes, appropriate tools and properly led teams. 

Step 2: Establish and Align Expectations

Every QA function has customers and stakeholders.  Do they have a common and aligned set of expectations?  Does business and IT agree to the role and expectations of QA? Do developers and testers understand and agree on the expectations of QA and their respective roles?

While industry standards and best practices exist for appropriate QA methodologies, expectations are specific to each institution. Organizational expectations can be different in many of the following ways:


Risk tolerance

Contractual requirements

Change tolerance

Business strategy 

Competitive situation


Available resources

Defining and aligning expectations is an ongoing process of discovery.  It is easy to conclude that the expectation is to both have your cake and eat it; however that is not real expectation setting and alignment. Proper expectation setting and alignment requires a periodic review of recent projects – especially spectacular failures – and using them to derive mismatched expectations and doing the deep analysis to realign expectations.

Step 3: Best Possible Requirements

Quality is often defined as fit to requirements. The quality of the requirements defines a practical ceiling to the quality achievable by the delivered product.  For instance if the “communicated” requirements only provide a 90% match to the “actual” requirements, then the final quality does not usually even reach 90% fitness to “actual” requirements. Focusing on the best possible requirements that are clearly documented and validated by the customer provide the best possible start to the SDLC.

Requirements should be…






Clearly identified

Well maintained

Thoroughly communicated

Taking appropriate care and using well engineered well controlled processes to capture, manage, decompose and communicate requirements is always worth the time, effort and money.

Step 4: Enforce Entrance Criteria

Most organizations have test strategies and test plans which include a section for Entrance Criteria; however only a minority of Payer organizations appear to routinely enforce defined entrance criteria. If entrance criteria are known and defined why are they ignored so often?  

Here are some reasons:

Entrance criteria are not known to the people who need to fulfill them

Entrance criteria are treated as pro forma

Entrance criteria are routinely waived

No one is empowered to enforce entrance criteria

Entrance criteria are viewed as bureaucratic impediments to progress

Why do Entrance Criteria exist and who do they benefit?  Entrance Criteria suffer from having a number of stakeholders who have very different reasons to support or ignore them.  The following table provides a partial accounting:

In almost all situations the perceived costs are mirages or artifacts of a dysfunctional culture. The direct benefits of well defined, communicated and enforced entrance criteria are large for all stakeholders. There may be rare situations where Entrance Criteria should be completely or partially waived, but properly engineered methodologies should allow for an emergency pathway that have minimal entrance criteria where the cost vs. risk has been carefully evaluated.

Step 5: Clear Results Reporting
The most important attribute of any QA process is transparency. The methodology, process, activity and results should all be well defined, widely communicated and understood by all stakeholders. There are two primary results that the QA process produces that are of interest to the widest group of stakeholders:

1. Test activity metrics
a. What has (is) been (being) tested?
b. Testing progress (Planned, Executed, Passed, Failed, Blocked, Cancelled)

2. Quality metrics
a. Open defects (by status, severity, age, module, source)
b. Defect detection rate – a good measure of whether we are seeing accelerating or diminishing returns
c. Readiness versus exit criteria
d. Process compliance metrics such as: entrance criteria met, test plan approved, test cases reviewed, root cause analysis completed, etc.

The purpose of testing is to allow appropriate decision makers to make a risk-reward trade-off. Clear and timely QA metrics reporting that is well communicated and well understood provide a key data point to this type of decision making.


QA is an important mission sensitive function for most Healthcare Payers. A more effective and efficient QA function is universally desired and sought after. The five steps described above have several interesting features:

• They are directional (not binary). Any levels of improvement in these directions are going to deliver benefits.
• There are many paths towards these goals. These principles can be implemented using a variety of methodologies, processes and tools.
• The steps are democratic. They do not require any a prior capability and at any organization irrespective of size, sophistication or location.

These steps are complicated and non-trivial and require enterprise-level and senior leadership commitment. However, any material movement in these directions are sure to be amply rewarded in increased QA function effectiveness and efficiency.