Feature

What's in this Blood?

How good are medical measurements and how good do they need to be?

Alan Hart

Some chemical tests of medical significance can be done at home. The most common of these are estimates of blood glucose by diabetics; there are a number of test-kits for blood glucose, suitable for home use, on the market.

Pharmac, the government body charged with monitoring pharmaceuticals and their usage, recently commissioned a study of the effectiveness of test-kits used in New Zealand. The results were duly reported in the media. In some of these reports there seemed to be some surprise that results of blood glucose tests varied, and an expectation that tests using these kits, or any other method for that matter, would give the "true" value for glucose in a blood sample.

One report quoted a mother as expressing concern, on testing her son's blood with two different test-kits three times in a row, that the readings were all different. It would have been very surprising if all the tests had given the same answer! What can we expect from chemical measurements and what are some of the concepts that are used to judge the quality of such measurements?

Absolute Truth Unknowable

The first thing to grasp is that no matter how good an analytical procedure is, no matter how hard we try, the absolute "truth" about the amount of glucose in a blood sample, the amount of protein in urine, of dye in lipstick, of ozone in the atmosphere or whatever, is unknowable. No matter how competent the analyst is, some errors beyond the analyst's control will always creep into an analysis. What we can expect is that a measurement will be done with a stated accuracy -- that there will be some statement of how close the measurement lies to the "true" answer.

If we can't determine the "true" value how can we give an estimate of the accuracy? There are a number of approaches to this problem. We can compare the answer we get with our measurement technique to answers obtained using a "standard" technique which everyone agrees gives the best answer possible. We can use this technique ourselves to analyse our sample. We can analyse a sample which has been carefully prepared so that it contains, as far as is possible, a known concentration of the substance at issue in order to test the validity of our technique.

Through the use of such approaches, a consensus develops about the best way to make a particular measurement. Experience in making a measurement and relating the answers, for example, to the condition of patients, also helps in reaching a consensus about what the answer should be. So we might say that our method for determining the concentration of compound X in blood has an accuracy of 10%: in 95% of cases, our answer will be within 10% of the "true" answer.

The next thing to grasp is that if we do repeated measures of the same sample, we won't get the same answer each time, even if we always use the same method. If we did a large number of repeated measures we might find that the answers were normally distributed about the average of our answers -- a plot of the frequency of their occurrence against their magnitude would conform to a bell-shaped curve.

The concept of precision is used to describe the extent to which repeated measures of the sample vary -- this will be reflected in the width or narrowness of the bell-shaped curve. Estimates which vary a lot are regarded as imprecise, those which only vary a little bit are regarded as precise.

What's a little and what's a lot? Precision is often expressed as Coefficient of Variation (sometimes called Relative Standard Deviation by clinical and analytical chemists) which is the standard deviation of the estimates divided by the mean, or average, of the estimates, the answer being expressed as a percentage. The smaller the CV -- the smaller the scatter about the mean -- the higher the precision.

A particular method might give us estimates which are inaccurate but precise; we can also have high accuracy and poor precision or poor accuracy and poor precision. Obviously what we want are estimates which are both accurate and precise.

A consensus has developed about what's an appropriate precision for various situations. Analyses carried out in a well-equipped clinical biochemistry laboratory should have a high precision, say a Coefficient of Variation (CV) of 1% or less. A CV of 5% or even 10% might be acceptable for estimates carried out in the home.

Why Variation?

Why do the results of measurements vary? The causes are not always obvious. Taking blood glucose measurements as an example, one way to think about the problem is to regard any value for the glucose level displayed by the measuring device as being primarily determined by the actual level of glucose but also by lots of other small (hopefully!) effects arising, for example, from defects in the device, variation in blood composition, temperature at which the test is done, and others we won't even know about or suspect. These "errors" are viewed as being random in nature, making it impossible to predict whether any one of a series of measurements will give a result that lies above or below the average of the set of measurements.

Sometimes we might find that the averages of our answers from a set of analyses were consistently higher or lower than our idea of what the answer should be. If this was the case, we would say that our estimates were biased. This situation would be worth looking into, as bias is often due to a persistent mistake on the part of the analyst, or a mechanical or electrical fault in equipment, or some other non-random effect which may be possible to overcome.

A couple of other properties that chemical measurements which seek to determine the concentration of something should have are specificity and sensitivity. An analysis which is highly specific for a substance will only react to the presence of that substance and not to other similar substances (sometimes called "interferents"). Poor specificity is often the cause of inaccuracy. An analytical technique should also be sensitive -- small changes in the amount of something should result in easily detectable changes in the information provided by the technique, such as the intensity of colour or the current displayed on a meter.

To get back to accuracy and precision, the highest accuracy and precision are obtained with carefully performed tests in professional laboratories. These can be expensive, sometimes time-consuming (modern developments in automation and miniaturization have made this less of an issue) and require professional training. It would be impractical to do these analyses at home but, in some cases, it may be helpful -- and diabetes is the prime example -- for patients to be able to monitor their own metabolism.

Home test-kits have to be made in such way that they can be used by lay people. It will almost inevitably be more difficult to achieve the same levels of accuracy and precision with a home test kit than with an analysis in a laboratory. There can easily be a high risk of "operator error": the sample might be placed on the device incorrectly, the display might be read incorrectly, the temperature might be too high or too low and so on. Nonetheless, manufacturers are rightly required to make their devices meet certain standards of accuracy and precision when correctly used.

The real issue in considering the performance of a home test-kit is the clinical significance of the information it provides. Does it help the user and their medical advisor improve their management of the user's condition? If a condition can be treated or diagnosed by knowing that the concentration, on a scale of 0 to 100, of compound X in the blood is somewhere between 0 and 10 or between 90 and 100, then a test to monitor this situation need be less accurate and precise than a situation where it is necessary to distinguish between a concentration of 3.50 and 3.75.

Economics also comes into the picture. Home-test kits have to be affordable to whoever is paying -- the patient, the medical insurance company, the government-funded health system. The economic benefits of making the measurements or diagnosis will also be an issue. If test-kits can be manufactured at an affordable price, capable of reaching sufficient accuracy and precision to allow clinically useful decisions to be made, then there is little point in a company making a substantial investment to produce devices that reach higher levels of accuracy and precision. They may be too expensive for the market to bear.

Any measurement has some uncertainty attached to it. The degree of accuracy and precision are measures of that uncertainty. The uncertainty is unavoidable, and, in itself, it is not something to be frightened of. What is important, is that the uncertainty is understood and controlled as far as possible, and taken into account when making, for example, medical decisions.

Alan Hart is a scientist at AgResearch, Grasslands Research Centre, Palmerston North.