Five Mistakes to Avoid in Functional Testing

Five Mistakes to Avoid in Functional Testing
Satish Nagarajan, Senior Testing Advisor – September 4, 2017

Everyone is doing it…

Functional testing that is. It is necessary and ubiquitous. It may be called different things but it is the core test activity for all test disciplines. At the end of functional testing these things should be true:

  • High frequency and high impact functions of the system under test should be validated with a high degree of confidence.

  • System under test should be ready for integration, performance and user acceptance testing

…many are doing it wrong

Frequently, at the end of the designated functional test phase, even if all test cases have been executed and exit criteria are met, expectations are missed:

  • Not all functional defects are identified

  • The system under test is not ready to move to the next test phase

  • A lot of testing effort has to be repeated in later phases

All of these problems are symptoms of the following five major mistakes.

Mistake 1: Test Data is Not Realistic

Modern Managed Care Operations (MCO) computer systems are highly configurable and data driven. The behaviors of the underlying system (workflow, rules, etc.) are responsive to the transaction and configuration data that is present. Test data that is not representative of the full complement of real life does not appropriately exercise the system. Here are some common mistakes to avoid:

Using incompletely setup members and providers

  • Members should be completely setup and enrolled in real and valid benefit packages with realistic ages, gender, family configurations and addresses.

  • Providers of all types should be setup with complete demographics, IDs, practice locations and contracts

Overusing the same members and providers

  • Avoid the easy, simple and lazy practice of using the same fake members and providers for all test cases. A number of system behaviors are sensitive to medical history, gender, age and provider.

  • The member and provider sample should be representative of the production experience and should cover all Lines Of Business (LOBs), provider types (PCP, contracted specialist, non-contracted specialist, DME, Lab, Radiology, ASC, LTC, Nursing Home, Hospital, ER etc.)

Using simple and similar claims

  • Office visit claims tend to be overused for claims testing. These claims usually do not require referrals or prior authorizations. They also do not exercise complex claims pricing logic, clinical editing or benefit rules and therefore do not exercise sufficient breadth and depth of the claims system.

  • The sample of claims used to appropriately test a claims system should include professional and facility claims from all common provider types, with a wide range of contract and network status for a wide range of medical services.

Mistake 2: System is Incompletely or Inappropriately Configured
Many projects suffer delays and in the interests of making up for delays teams often start Functional Testing with incomplete or inappropriate configuration. This is a case of the path to bad outcomes being paved with great intentions. Testing with incomplete and inappropriate (test only) configuration is usually worse than not testing. Modern payer computer systems are entirely driven by organization specific configuration, so the “system under test” will not behave like the production system. This is testing under false pretense which is at best inefficient and at worst waste.
It is appropriate to initiate functional testing before all configurations are complete as long as the following are true:

  • All “production candidate” foundational configuration is complete

  • A sufficient breadth of functionality is configured.

  • The SDLC used to generate this configuration is defined and stable

  • The method and tools used to deliver additional configuration is incremental and auditable and is also the method that will be used to deliver configuration updates to the production system.

Mistake 3: Incomplete Coverage of High Frequency and High Impact Scenarios
Modern healthcare IT platforms are complex systems with behavior defined by a combination of transactional data, configuration and multiple automated agents. Small changes to certain parts of this system can lead to substantial differences in behavior. Functional testing is an important early detection point for defects and unexpected system behavior. Three keys to functional testing success are: Coverage, Coverage and Coverage:

  • Coverage of transaction variety: transaction type, submitter, member, provider, medical service, dates, Line of Business (LOB), etc.

  • Coverage of configuration variety: benefit package, provider contract, workflow rules, clinical edits

  • Coverage of system agents: EDI, direct entry, 3rd party system, reconciliation, etc.

  • (Bonus) Coverage of context: Claims history, medical history, eligibility history, provider history etc.

Mistake 4: Limiting Testing to the “Happy Path”
It is appropriate that most functional testing effort is focused on the high frequency normal transactions, e.g. the well formatted accurate timely claim for an eligible member from a contracted provider. However a lot of the important behavior that needs to be evaluated during functional testing involves transactions that have errors or should raise warnings and alerts. In the parlance of QA this is usually referred to as negative and boundary testing. The following are some key considerations for negative and boundary testing:

Consider thresholds

  • Benefit dollar thresholds

  • Annual visit limits

  • Claims payable amount threshold

  • Timeliness threshold

Consider common transaction errors

  • Missing or incomplete demographics

  • Missing ID

  • Duplicate transaction

  • Wrong amount

Consider important alerts and warnings

  • High dollar amount

  • VIP member

  • Clinical necessity

  • Fraud, waste and abuse related alerts

  • COB and Third Party Liability

  • Provider under sanction

  • Ineligible member

Mistake 5: Lack of Requirements Traceability
Requirements traceability is important for all testing activities for a couple of reasons. Well defined traceable requirements first define the scope and approach for defining test cases and second they allow QA and their stakeholders to track the progress of testing across multiple phases.
If there are well defined requirements which are granular and properly traced to test cases and test suites then it is relatively easily to track progress of testing in terms of system functionality. In this case QA and all stakeholders can appropriately focus the remaining test phases to focus on the remaining unknowns. This is especially valuable for projects that are experiencing schedule pressure – which is a majority of all projects these days.
Here are some important features of requirements traceability from a testing perspective:

  • A well defined and enforced SDLC supports requirements engineering and traceability through all phases

  • Requirements traceability is supported by a tool kit that is available to all stakeholders

  • Requirements are under robust change and version control

  • Requirements traceability tools are well integrated with the test management tools

  • Test methodology requires and encourages tracing test cases to the most granular requirements

  • Test execution reporting includes a view showing requirements coverage

Functional testing is an important ubiquitous test phase at many healthcare insurers. Making this test phase efficient and effective is an important goal for many QA leaders. The recommendations here represent a distillation of QA best practices.