Verification, Validation of Microbiology Methods
By Frederic J. Marsik, PhD, ABMMM
The key attribute of a laboratory method is its ability to produce accurate and precise results over extended periods of time with an appropriately rapid turnaround time. Accuracy and precision are readily measurable. The laboratory must have a detailed verification protocol for the evaluation of a method before beginning the verification procedure. In addition, a systematic approach (validation) must be in place for continuous monitoring of established methods which demonstrates that the testing process and consistency of results have not changed and that testing personnel remain competent to perform tests and report results. The processes of verification and validation, therefore, are part of the quality assurance program for the laboratory.
When verification and validation procedures are not carefully planned to ensure that regulatory requirements are met and corporate philosophies are maintained, noncompliance could result. Like other important activities in a complex organization, verification and validation must be managed. This is especially true since both activities are multi-disciplined. Relevant definitions also must be understood (Table).
Having an impact on validation and verification are the Clinical Laboratory Improvement Amendments of 1988 (CLIA '88). CLIA '88 extended federal regulation to cover all laboratories that examine human specimens for the diagnosis, prevention or treatment of any disease or impairment or for the assessment of the health of human beings.1
Sections 493.1201 through 493.1285 of subpart K of the regulation set forth the quality control (QC) requirements. Section 493.1213 describes the requirement for the establishment and verification of method performance specifications. After Sept. 1, 1992, a laboratory that introduces a new procedure for patient testing that uses a method developed in house, a modification of the manufacturer's test procedure or a method that has not been cleared by the FDA must, prior to reporting patient test results, verify or establish the performance specifications for the following performance characteristics, as applicable:
* analytical sensitivity,
* analytical specificity to include interfering substances,
* reportable range of patient test results,
* reference ranges and
* any other characteristic required for test performance and interpretation of results.
In addition, control procedures for diagnostic tests must be established on the basis of the verified performance specifications. Documentation of the verification of manufacturers' specification or establishment of performance specifications for tests developed in house must be maintained by the laboratory.
Selection of a Lab Method
Once an institution has made the decision to offer a new test, the next step is to select the method by which the test will be performed. Following are some steps that one may choose to follow when deciding on the method.
1. Define the purpose for the test. Common purposes for tests are:
Screening: This type of test is used for testing large populations of patients. Generally, screening tests have high clinical sensitivity and negative predictive value (NPV). Positive results with such tests require confirmation by a more specific test.
Confirmation: Confirmation is used after obtaining a positive screening result to ensure the accuracy of the initial result. Specificity and positive predictive value (PPV) are generally the consideration for such tests.
Diagnosis: Diagnostic tests are used for the evaluation of persons suspected of having a given disease state or characteristic. The specificity of such tests needs to be high (>95 percent) if results cannot be easily supported by clinical data.
2. Decide what analyte (e.g. organism, antigen, nucleic acid) is to be detected. It's not unusual that a new test may be more sensitive and/or specific than an existing "gold standard" test.
3. In consultation with the end user of the test (e.g. physician) and information from steps 1 and 2, determine the medical usefulness of the test (e.g. improve patient care, shorten hospital stay).
4. Survey the technical and medical literature for performance claims of various methods. When reviewing the literature, confirm that the method described is actually the test that is to be evaluated.
5. Other considerations are cost; if the method is accepted by the microbiology community; practicality (e.g. can the test be performed on all shifts when needed?); specimen requirements; quantities of reagents and controls needed for the test; shelf life of reagents and controls before and after opening; availability of supplies, service, technical support; possible safety hazards to personnel performing the test; whether the reference range is appropriate for the test; and how it will be determined for the institution if it is not appropriate.
6. Perform in-house verification. Verification of a test serves to establish that the performance parameters of the test are satisfactory. See the Figure for a flow diagram of the verification process.
Verification of Microbiology Methods
Verification is accomplished by performing the new or revised test method in parallel with a reference method that has an established and satisfactory level of accuracy. The test verification method should be designed such that the test results of the verification indicate one of three possibilities:
1. The test is acceptable for routine use.
2. Further verification studies are required.
3. The test is unsuitable for routine use until its performance parameters can be verified.
There are certain commonly used microbiology items not considered instruments, kits or test systems under CLIA '88 and may not require verification prior to use: media or individual reagents used as part of the identification process (e.g. catalase, oxidase). Instead, these items may be monitored through the QC protocols of the laboratory.
The laboratory, however, may choose to perform in-house verification. Such a case might be verification of the performance of a medium (e.g. medium for the isolation of Neisseria gonorrhoeae) that has a history of failures.
Following are suggested methods for the verification of commonly performed microbiology tests. These examples are not all-inclusive. The extent one may need to go to to verify the performance of a test is dependent on how widely the test has been used and accepted in the microbiology community, extent and results of published evaluations and impact of an incorrect test result on the patient.
* Specimen collection and transport methods
The quality of results generated by microbiology methods depends on the quality of the specimen received by the clinical laboratory. It's critical, therefore, that collection and transport devices be evaluated for their performance.
The verification of their performance can be done in a number of ways. One is to inoculate the existing and new devices with known concentrations of organisms commonly recovered by the laboratory, then quantitatively determine the number of surviving organisms after defined periods of time. The times at which recovery is determined should be based on the times associated with transport of the device to the laboratory under real conditions. If specific storage conditions are used for the real specimens, the conditions need to be included in the evaluation. In conjunction with the in vitro evaluation, patient specimens can be collected in parallel and the qualitative recovery of organisms from the devices determined.
* Antimicrobial Susceptibility Test Methods
Antimicrobial susceptibility test results are one of the most important results generated by the microbiology laboratory; they may directly affect the therapy chosen for treatment of an infection. The performance of a susceptibility test system should be carefully verified.
A number of recommendations for verification of susceptibility test systems have been published.2-4 Evaluation of the test method should be done using microorganisms that are commonly encountered in the institution's patient population. The microorganisms should be both susceptible and resistant to the antimicrobials routinely prescribed at the health care facility.
The evaluation needs to be designed to allow detection of the following types of errors:
* Very major. The most serious error, this is where the reference method indicates that the microorganism is resistant to an antimicrobial and the method being evaluated indicates that the microorganism is susceptible to the antimicrobial. This type of error can only be detected by testing resistant microorganisms. Very major error rates should be calculated using the number of resistant microorganisms as the denominator rather than the total number of microorganisms tested. Very major errors determined for a large sample (n = >35) of known resistant microorganisms should be <3 percent.3
* Major. The method being evaluated indicates the microorganism is resistant to the antimicrobial and the reference method indicates that it's susceptible. Major errors can only be detected by testing organisms susceptible to the antimicrobial agents.
* Minor. The method being evaluated indicates an "intermediate" result and the reference method indicates either susceptible or resistant or vice versa.
The combination of major and minor errors attributable to the new method should be <7 percent when determined for a large known-susceptible population or a large unselected sample of clinical isolates (a minimum of 100 isolates).3 It should be realized that more stringent criteria are suggested by the National Committee for Clinical Laboratory Standards (NCCLS)5 and required by the FDA for manufacturers' susceptibility test systems (<1.5 percent for very major errors and <3 percent for major errors). The laboratory may consider meeting these higher standards but it will involve testing more isolates.
For those cases where an intermediate interpretive category exists (a one-dilution error in the MIC might spuriously appear as a very major or major error), there should be >90 percent agreement within one dilution between the methods being evaluated.3 Growth failures in the test method and reference method should not exceed 10 percent.2
In the event that the chosen limits are exceeded for any drug-isolate combination, the method should be considered unverified and a corrective investigation should be undertaken. When appropriate, the manufacturer should be consulted. Following corrective action, the new revised method should be run again in parallel with the reference method using a minimum of 20 isolates.6 The isolates used should be those that can clearly demonstrate whether the system works or not.
* Diagnostic Tests. Verifying a diagnostic test (nonculture tests for detection of microorganisms, microbial antigens, nucleic acids or antibodies from clinical specimens) may be relatively straightforward if the analyte is common in the population. If it is uncommon in the population, however, the manufacturer of the method, other laboratories or commercial sources may be able to provide specimens to be used in the evaluation. It may also be appropriate to reconsider whether the test should be offered.
* Verification of commercially obtained test methods. Commercially obtained test methods must be verified by performing the method exactly as the manufacturer indicates in order for them not to be considered "home brew." The new test method should be performed in parallel with the existing or reference method on a minimum of 20 specimens containing the analyte and 50 specimens that lack the target analyte.6 Weakly positive specimens should be included for the evaluation.
Diluting a strongly positive specimen can make a weakly positive specimen. Comparing the new method with the existing or reference method can be done by calculating the number of true-positive, true-negative, false-positive and false-negative results obtained. By using these results, the sensitivity, specificity, predictive values, accuracy and precision of the new or revised test may be calculated. The test may be considered verified if it meets the criteria established for the method prior to its evaluation and the sensitivity and specificity are no lower that 5 percent below those of the reference method, those appearing in peer-reviewed literature or those claimed by the manufacturer in the package insert. When possible, data published in peer-reviewed literature should be used in the evaluation of the method. However, results in the peer-reviewed literature should be carefully analyzed.7
If the results of the evaluation do not meet the appropriate criteria, the method is considered unverified. Following corrective action, the method should be run again in parallel against the existing or reference method and the results interpreted as before. Consideration should always be given to increasing the sample size tested when repeating the verification.
* Verification of home-brew test methods. Verification of a test may become necessary even for what may be considered a minimal modification (e.g. a minimal change in incubation temperature or length of incubation or both). Whether verification is required is an individual call. Verification of a home-brew test can be done as stated under "Verification of commercially obtained test methods" except that a larger number of samples need to be tested. It's recommended that a minimum of 50 specimens containing the target analyte and 100 specimens without the target analyte be tested.6 Verification is achieved by meeting the same criteria as stated under "Verification of commercially obtained test methods."
Microbial Identification Tests
The "Commercial Methods" category includes antisera, antigens, chemicals, stains, instruments, reagents or kits used to directly identify microorganisms or indirectly identify them from such things as the metabolic products they produce while growing.
Evaluation of these methods may be approached in a number of ways. Three suggestions are:7
1. One week of consecutive parallel testing (minimum of 50 strains) performed in parallel with the existing (reference) method.
2. Testing the method in parallel with the reference method by using known representative strains (stock cultures). A minimum of 15 species commonly isolated from specimens processed at the test site for a total of 50 or more tests should be done.
3. Testing split samples, run concurrently at another laboratory. Confirm that 20-50 organism identifications (12-15 different species) agree.
The appropriate QC organisms need to be run with each method during the verification. These control organisms should not be considered part of the total number of test organisms. All discrepancies need to be arbitrated. The arbitration of discrepancies can be done a variety of ways; one way is to involve a third laboratory.
For those methods that identify the species level, both the level of agreement between the new and reference method and the types of errors or disagreements should be evaluated. It should be expected that there be at least a 90 percent agreement between systems. A higher level of agreement (>95 percent) should be expected when evaluating systems that identify members of the family Enterobacteriacea.8
When verifying a method that detects a particular analyte, the sensitivity and specificity of the method need to be calculated. In most cases, testing 20 microbial isolates that contain the target analyte and 20 isolates that do not contain the isolate should suffice.6 In these cases, the method may be considered verified if the sensitivity and specificity are no lower than 5 percent below those values of the reference method.
Verification of home-brew methods for the identification of microorganisms is done as described above. The major difference is that a larger number of organisms--at least 200--should be tested.6 The same standards of performance as noted above should be achieved with home-brew tests.
Perhaps one of the most difficult tasks is verifying the performance of a blood culture method. Since this is one of the most critical analyses for patient care performed by the microbiology laboratory, it needs to be done very diligently.
One way to make verification of a new blood culture method easier is to keep good records on the performance of the existing method. The records will identify what organisms are commonly isolated and, more importantly, which are difficult to isolate. This information will allow for a more rigorous verification procedure. The following suggestions for verifying a blood culture method are directed toward automated systems:
* Will the method support the growth of organisms commonly seen in the patient population served by the laboratory? All aspects of the method, media, atmosphere and incubation temperature need to be evaluated to confirm the performance of the method.
* Will the method detect the microorganisms of interest in at least as timely a manner as the existing method?
A parallel verification procedure is the most rigorous and demanding of personnel throughout the institution. Collecting duplicate patient's specimens or splitting specimens generally does this. This type of verification procedure allows the method to be evaluated against the existing system under actual patient and laboratory conditions.
The length of time parallel running of the methods is conducted depends on the institution. It's best to run the comparison until it's felt that there's been sufficient time to collect data on a large number of microorganisms. During or after the initial portion of the verification, it's best to ensure that the instrument is detecting a minimum of 98 percent of organisms in the patient population.6 This can be done by blind subculture of all negatives. A positivity rate of 10 percent necessitates subculturing 500 bottles of each medium.
Also during the verification, duplicate sets of blood cultures set up identically with 20 species of microorganisms clinically relevant to the test site can be run to check the performance of the method.6 It's important to include organisms that the existing system has difficulty recovering. The new system can be considered verified if the sensitivity and time to detection are no lower than 5 percent below that of the existing method.6
Validation of Microbiology Methods
Validation of a microbiology method is an ongoing process after verification of the method has shown it to perform appropriately. Validation results provide the information to assure that a test method continues to perform correctly.
Unlike verification, validation is not a specific process addressed in CLIA '88. However, all the components of the process (QC, proficiency testing, verification of employee competency and instrument calibration) are present.
Validation can have one of three outcomes:
1. the method continues to perform in an acceptable manner,
2. further investigation is warranted or
3. immediate corrective action needs to be taken.
If validation indicates that further evaluation is warranted or immediate corrective action is necessary, performance of the method for patient diagnosis should immediately be discontinued. Any corrective action necessary should be initiated by the laboratory and may involve participation by the manufacturer. Lot numbers and expiration dates should be recorded for all reagents and materials used in validation procedures.
Validation Process Components
There are essentially seven components that comprise the validation process. They include:
1. QC organisms. All QC organisms indicated by the manufacturer to be used on a routine basis must be used. These QC organisms may come from a variety of sources. If they are obtained from a commercial source, the laboratory history (when obtained, how preserved, number of passages) should be maintained. If, however, the organism is from a non-reference source, the aforementioned information plus the source of the organism and characterization should be documented.
2. QC analyte. An analyte can be a metabolic product, nucleic acid, enzyme, antigen, etc. Commercial kits have the QC analytes provided. In this case, the analyte should be identified with a lot number, its concentration, titer (where appropriate), date of preparation, how it is to be used and storage information. For a QC analyte developed by the laboratory, the above information needs to be available as well as a clear history of its development as a QC reagent. Frequency of QC testing should follow the recommendation set forth by the manufacturer. For home-brew tests, positive, negative and other appropriate controls should be determined from the appropriate reference sources such as the NCCLS, Wayne, PA, Manual of Clinical Microbiology9 and/or Clinical Microbiology Procedures Handbook.10
3. Proficiency test (PT) samples. To maintain certification, every laboratory needs to participate in an approved PT program. An internal blinded PT program may supplement the external PT program when deemed necessary.
4. Instrument calibration. Every instrument in the microbiology laboratory needs to be calibrated on a regular basis. The manufacturer's instruction and schedule for calibration must be followed.
5. Blood culture systems. It's imperative that all blood culture methods be quality controlled. Generally, it's sufficient to follow the manufacturer's recommendations. However, if there's suspicion that the method is not sufficiently recovering certain organisms in a timely manner, in-house QC testing needs to be instituted.
6. Personnel competency. It's not enough to know that a particular method or instrument system is performing correctly. It must also be determined that the individuals performing the methods or operating the systems are correctly doing their job. Personnel performance, therefore, must be consistently evaluated.
7. QC reagents not available. It's best to avoid this type of situation; possible approaches are:
* Split samples sent for testing and send to a reference laboratory, then compare results. It's advised to do this at least twice a year.
* Split samples and have the samples tested blinded by two or more people in the laboratory.
* Save known positive and negative samples and use them for QC.
These approaches will depend on how comfortable personnel are with performing the method and how serious the consequences would be if the results were inaccurate.
The frequency of testing and actions to be taken after control failures should follow the manufacturer's recommendations or those of an appropriate regulatory or advisory agency.
Whether the laboratory is performing verification, validation or training personnel, complete and detailed records of these activities must be kept. The documentation should be clear, concise prose with tables and drawings rather than words when appropriate. When deficiencies are documented, it's critical that the corrective action be taken and results be recorded. Recording information without evidence that it has been reviewed and acted upon when necessary is a common way for laboratories to receive deficiencies during inspections.
Verification and validation records should be kept for a minimum of two years. It's beneficial, however, to keep certain records (such as blood culture statistics) for as long as that method is being used. *
Dr. Marsik is a review microbiologist with the Center for Drug Evaluation and Research, Food and Drug Administration, Rockville, MD. Dr. Marsik prepared this paper in his private capacity; no official support or endorsement by the FDA is intended or inferred.
1. Health Care Financing Administration. Medicare, Medicaid and CLIA programs. Regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA). Fed Regist 1992; 57:7002-7186.
2. Ferraro MJ, Jorgensen JH. Instrument-based antibacterial susceptibility testing, P. 1379-1384. In: PR Murray, EJ Baron, MA Pfaller, et al. (ed.). Manual of Clinical Microbiology, 6th ed. Amer Soc for Microbiol, Washington, DC. 1995.
3. Jorgensen J. Selection criteria for an antimicrobial susceptibility testing system. J Clin Microbiol 1993;31:2841-2844.
4. Murray PR, Niles AC, Heeren RL. Comparison of a highly automated 5-h susceptibility testing system, the Cobas-Bact, with two reference methods: Kirby-Bauer disk diffusion and broth microdilution. J Clin Microbiol 1987;25: 2372-2377.
5. National Committee for Clinical Laboratory Standards. Development of In-vitro Susceptibility Testing Criteria and Quality Control Parameters. Approved guideline M23-A. National Committee for Clinical Laboratory Standards, Wayne, PA, 1994.
6. Elder BL, Hansen SA, Kellogg JA, et al. Cumitech 31, Verification and Validation of Procedures in the Clinical Microbiology Laboratory Coordinating editor BW McCurdy. Amer Soc Microbiol, Washington, DC, 1997.
7. Miller JM. Evaluating biochemical identification systems. J Clin Microbiol 1991;29: 1559-1561.
8. Miller JM, O'Hara CM. Substrate utilization systems for the identification of bacteria and yeasts, p.103-106. In: PR Murray, EJ Baron, MA Pfaller, et al (ed.). Manual of Clin Microbiol 6th ed. Amer Soc Microbiol, Washington, DC, 1995.
9. Murray PR, Baron EJ, Pfaller MA, et al. Manual of Clin Microbiol, 6th ed. Amer Soc Microbiol, Washington, DC, 1995.
10. Sewell DL (ed.). Quality assurance, quality control, laboratory records, and water quality, p. 13.0.1-13.4.11. In: HD. Isenberg (ed.) Clin Microbiol Procedures Handbook, Vol. 2. Amer Soc Microbiol, Washington, DC, 1992.
GOOD VERIFICATION, VALIDATION PROCEDURES
* Reduction in rejection, reworks,
resample and retest
* Reduction in costs
* Increased throughput
* Fewer complaints
* Improved employee awareness
Technical: The degree of conformity of a measurement to a standard or true value; a measure of analytical capability.
Clinical: The ability of a method to rule in or out a specific disease or analyte. Accuracy and test efficiency are synonymous and can be expressed mathematically as a percent:
number of correct results \ total number of results x 100
* Gold standard:
The best available approximation of the truth. The accuracy of the test is accepted as reasonable but not 100 percent accurate. In situations where the true disease status of a patient is not known and there's a discrepancy between the results of the test being evaluated and the gold standard test (reference test), it may be appropriate to display the agreement and disagreement between the two methods in graphic or tabular form. The disagreement between the methods may be further investigated by performing another test or following the patient's condition over an appropriate amount of time. When it cannot be clearly determined whether the new test is better than the gold standard, it may be appropriate to use a cost-benefit analysis to pick the appropriate test.1-4
* Home-brew method:
A method developed in house or any method that incorporates modifications of the manufacturer's package insert instructions.
* New method:
Any method not previously offered by a laboratory.
* Old method:
A method that has been in use prior to Sept. 1, 1992, the effective date of the Clinical Laboratory Improvement Amendments of 1988 (CLIA '88).
The degree of agreement among individual test results when the same procedural steps and reagents are used to test the same sample. Mathematically, precision can be expressed as a percent: number of repeated results in agreement \ total number of results x 100
* Predictive value:
The positive predictive value (PPV) of a test is the probability that a patient with positive test results has disease or the presence of an analyte in the specimen. The negative predictive value (NPV) of a test is the probability that a patient with a negative test result does not have the disease or the presence of an analyte in the specimen. Predictive values can vary significantly with the prevalence of the disease or analyte unless the test is 100 percent sensitive (for NPV) or specific (for PPV).
The frequency of a disease in the population of interest at a given point in time.
* Quality control:
Routine performance checks of methods and personnel performance, using known organisms or analytes to ensure that a method and the personnel doing the method are performing as expected. Quality control is an integral part of the test validation process.
* Reference method:
A method that has been accepted by the microbiology community in which exact and clear descriptions of the necessary conditions and procedures are given for the accurate determination of one or more values. A currently used method is unacceptable as a reference method unless there's onsite or peer-review journal documentation of an acceptable level of accuracy and precision of the method.
Analytical sensitivity: Defined at the 0.95 confidence level ("2 standard deviations) and may be referred to as the "detection limit." In microbiology, the detection limit may be correlated to the number of colonies in culture or the lowest quantity of antigen or antibody a test can detect.
Clinical sensitivity: The percent test positivity in a population of affected patients. Mathematically, sensitivity is expressed as a percent:
# of positive results \ # of positive results + false negative results x 100
Analytical specificity: The ability of an analytical method to detect or quantitate only that analyte that it was designed to measure.
Clinical (diagnostic) specificity: Refers to the proportion of negative results obtained when a test is applied to patients known to be free of the disease. Mathematically, specificity is expressed as a percent:
# of true-negative results \ # of true-negative + false-positive results x 100
The ongoing process that provides the information that a test is performing correctly. The components of validation are quality control, proficiency testing, verification of employee competency, instrument calibration and correlation with clinical findings.
A one-time process completed before the test or system is used for patient testing. Verification requires determination or confirmation of the test performance characteristics, including sensitivity, specificity and, where appropriate, the predictive values, precision and accuracy of the test.
* Verification protocol:
A written plan stating how verification will be conducted, including test parameters, product characteristics, test equipment and reagents and decision points on what constitutes acceptable test results.
1. Ilstrup D. Statistical methods in microbiology. Clin Microbiol Rev 1990;3:219-226.
2. Hadgu A. The discrepancy in discrepant analysis. Lancet 1996;348: 592-593.
3. National Committee for Clinical Laboratory Standards. Specifications for Immunological Testing for Infectious Diseases. Approved guideline I/LA 18-A. National committee for Clinical Laboratory Standards, Wayne, PA, 1994.
4. LeBar WD. Keeping up with new technology: New approaches to diagnosis of Chlamydia infection. Clin Chem 1996;42: 809-812.