EVALUATION OF AUTOMATED TEST METHODS

Effective Date: 
Tue, 05/01/2012
Reviewed: 
Mon, 11/06/2017
Policy: 

PURPOSE

Before introducing a new instrument into the laboratory, evaluation studies must be performed in order to:

• Examine its performance under laboratory conditions

• Substantiate the manufacturer’s performance claims

• Compare the new method with the method presently in use in the laboratory

• Determine acceptable precision and accuracy

• Determine the reportable range for each analyte to be tested

Procedure: 

Precision

 Precision is the same as reproducibility and is defined in terms of standard deviation (SD) or coefficient of variation (%CV). Within-Run” precision is the result of running the same sample several times in the same run. "Day-to-Day” precision is evaluated as the facility utilizes an analyzer over multiple days, as in the case of the Piccolo, whereby precision is measured over a five day period. Precision may also be used to establish acceptable SD and %CV of its quality control materials.

Within Run Precision

1. Analyze multiple aliquots of control material or a single patient sample. This analysis should be performed in a single run. A "run” is a sequence of samples analyzed consecutively without interruption (unless recommended operation required such interruption).

2. Examine the data for possible outliners. Exclude from the data set only results which are:

a. Related to documented errors such as results obtained when an instrument is indicating “ERROR” status.

b. Obvious transcription or decimal point error.

3. Perform calculations for the mean, observed standard deviation and coefficient of variations (%CV).

Evaluation of Within-Run Precision

Establish a "precision goal" by setting an allowable standard deviation (or %CV) for the test method. The manufacturer’s performance claims, the required precision at medical decision levels, and recommendations found in the scientific literature may all be considered when setting the allowable standard deviation (%CV). If the observed SD or %CV is less than or equal to the allowable SD or %CV, the within-run precision is considered acceptable. If the observed precision is not acceptable, you should attempt to locate the source(s) of imprecision. Repeat testing may be warranted.

Day-to-Day Precision

The day-to-day precision is a more realistic assessment of test performance under routine operating conditions than the within-run precision. The standard deviation is usually somewhat higher than that observed for the within-run testing.

1. Analyze at least two levels of control or standard material (or serum pools) each working day over a five day period. One of the controls should be near the lower analytical range of the analyzer, the other at the upper end of the analytical range. (The Abaxis Piccolo utilizes three standards supplied by NOVA with target values at the low, midpoint and high end of the reportable range.)

2. Examine the data for possible outliners as described above for within-run precision.

3. Perform calculations for the mean, observed standard deviation and %CV for each set of data.

Evaluation of Day—to-Day Precision:

As described above for within-run precision, an allowable standard deviation (or %CV) should be established and used to evaluate the day-to-day precision. The results will be considered acceptable if the observed SD or %CV is less than or equal to the allowable limits.

Linearity Testing

Linearity testing will be performed on the new test method to validate the analytical range of the test method. The analytical range of the test method should be wide enough to include most of clinical results through the origin whenever possible.

Refer to the "Linearity Testing and Calibration Verification" policy and procedure for a detailed discussion of the linearity testing procedure and its evaluation.

Parallel Studies

Parallel studies are performed to determine the relative bias (accuracy) between the method under evaluation and the method currently in use, if another method or analyzer is being utilized within the laboratory. Analytical specificity, the ability of a test method to determine solely the component(s) it claims to measure, is related to accuracy.

1. Analyze specimens from approximately 10-20 patients.

2. Specimens should be fresh, if possible, or stored under proper conditions. Ideally, the concentration range of the samples should span the clinically meaningful range from below the expected normal range to substantially above it.

3. Analyze all samples by both methods.

4. Determine bias by statistical methods.

Statistical Data Analysis

Linear regression analysis of the data should be performed with the aid of a statistical software program. This analysis will be used to determine an estimate of the bias between the two methods based upon the best straight line through the data using the formula

y = a + bx

where:  x= the reference value (plotted on the X-axis of a graph)

y= the new method value (plotted on the Y—axis of a graph)

b = the slope of the plotted line

a=I the Y-intercept.

If the relationship between the two methods is perfect (result x is equal to result y, the slope would be a line drawn at 45 degrees and the Y-intercept would pass through O. Variation about the regression line is caused by different kinds of systematic error: proportional, constant, and random error.

1. The linearity of the relationship between the two methods determines the slope (b) of the regression line. The slope should be 1.0 +/ -0.1. Proportional error causes changes in the slope.

2. Ideally the Y-intercept (a) = 0.0. Generally a Y-intercept of less than 1.00 will be considered acceptable but for determinations with high value results the Y-intercept may be considerably higher and still be considered acceptable. Constant error causes changes in the Y·intercept.

3. The correlation coefficient (r), is an index of the relationship between x and y and is used to determine whether the data spans a wide enough analytical range for linear regression analysis to be considered valid. Ideally, the value of (r) is 1.00 but the range of values may be considered adequate if (r) is greater than or equal to 0.90. If (r) is less than 0.90 the range of data may not be wide enough for linear regression analysis to be valid. The range may be extended by testing additional samples or the data may be analyzed by an alternate statistical method (See below).

4. S y/x is "Standard Error” between x and y. S y/x ideally equals 0.0. The smaller the standard error, the more reliable the results. Random error causes S y/x to increase.

5. To help evaluate the acceptability of the linear regression results, it is best to define the amount of allowable error at various medical decision levels. The systematic error at these levels can then be calculated. For example if slope (b) = 0.982 and Y-intercept (a) = 1.2 mg/ dl the amount of error at a medical decision level (example: 100 Q mg/dl) can be determined:

y = 2 + 0.982 (100) =Z 99.4

The difference between X and y provides an estimate of the systematic error at the critical concentration (y - x Z 0.6). If the allowable error had been set at 5.0 mg/ dl, the accuracy would be judged acceptable because the observed systematic error (0.6 mg/ dl) is less than the defined allowable error of 5.0 mg/ dl.

Alternate Statistical Method

lf the linear regression analysis is not valid due to inadequate analytical range or if a linear regression program is not available, a modified t-test procedure (Griffiths, et al) may be used for determination of bias between the test methods. The z-value is calculated by the following equation:

y – u

z =                           _______________

s / square root of n

where:  y = mean of the raw test method results

u = mean of the raw reference method results ·

s = SD of the raw test method results

n = number of pairs results

if z > 1.96 the test method exhibits significant positive bias. If x< -1.96 the test method exhibits significant negative bias. When -1.96 < z there is no significant bias between the two methods.

Normal Ranges (Reference Intervals)

l. Collect approximately 20 serum specimens representative of the total population of “healthy" individuals.

2. The results to these 20 samples will be used to validate the established reference intervals.

3. The study may be expanded to include more specimens if:

• The results of the validation study are inconclusive

 • The parallel studies have demonstrated a significant positive or negative bias.

• There are no established reference intervals for the test method being evaluated.

• Reference intervals cannot always be established within a facility, due to the critical nature of its own patient population. In this case, manufacturer’s stated reference ranges or ranges obtained from a reference lab or literature may be utilized, with the approval of the laboratory director.

Summary

The test method evaluation should include a summary of precision and accuracy, linearity for validation of reportable range and patient correlation when applicable. This evaluation should be documented, and all worksheets, instrument printouts, and related documents shall be retained for the life of the method plus two years for regulatory inspection purposes.