I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.

(Lord Kelvin)

First, we must insist on risk calculation being expressed as distributions of estimates, and not as magic numbers that can be manipulaed without regard to what they really mean. We must try to display more realistic estimates of risk to show a range of probabilities. To help do this we need tools for quantifying and ordering sources of uncertainty and for putting them in perspective.

(W.D. Ruckelshaus)

In this age of powerful information technology, decision-makers in all fields have instant access to masses of quantitative data to help them formulate their policies. These will usually be tested for their quality, by their success (or otherwise) in the achievement of their stated goals. But how much testing is there of the quality of the information on which the decisions are based? All over the industrial and administrative worlds, quality has become a keyword, and significant resources are devoted to quality-control of processes and procedures. Yet the quantitative information that is crucial to all such exercises has almost always escaped the same critical scrutiny.

Issues of uncertainty, and closely related, those of quality of information, are involved whenever research related to policy is utilized in the policy process. As these issues are new, we do not yet possess practical skills for dealing with them effectively. There are a variety of possible responses among users, contradictory among themselves, but all of them to be seen in ordinary practice. The simplest, and still most common, is when decision-makers and the public demand at least the appearance of certainty. The scientific advisers are under severe pressure to provide a single number, regardless of their private reservations. The decision-makers’ attitude may be typified by the statement of a US Food and Drug Administration Commissioner, "I’m looking for a clean bill of health, not a wishy-washy, iffy answer on cyclamattes" (Whittemore, 1983). Of course, when an issue is already polarized, such simplicity does not achieve its desired ends. With such contested policy numbers in play, debates may combine the hyper-sophistication of scholastic disputations with the ferocity of sectarian politics. The scientific inputs then have the paradoxical property of promising objectivity and certainty by their form, but producing only greater contention by their substance.

It is easy to imagine, in a general way, what we mean by "quality" of information, on the analogy of the quality in any other input. Will it perform reliably at a reasonable cost? and, equally important, do we have good and proper evidence that it will do so? The absence of good quality is betrayed by high uncertainty. If we simply do not know, or we are not sure about the testing and certification of the input, then its quality is low, whether it be a plastic moulding or a set of numbers. Up to now, tests for the quality of quantitative information have been very undeveloped, in comparison with those in all other spheres. There are the standard statistical tests on sets of numbers in relation to an hypothesis; and there are highly elaborated formal theories of decision-making in which "uncertainty" is manipulated as one of the variables. But none of these approaches help with the simple question that a decision-maker wants to ask, when confronted with a screenful of information: is this reliable, can I use it safely?

There are two related reasons for the lack of methods in this important field. One is in our common assumptions about the way we know about the world. Science is based on numbers, therefore numbers are necessary for the effective study of the world; and we then assume that numbers, any numbers, are sufficient as well. In such an environment, it is not easy for anyone to imagine analyzing the uncertainties in numbers, and still less easy for him or her to get an audience. Although there is a rich literature of sceptical jokes about statistics, we still use them, usually quite uncritically, because there is nothing better to hand. One can even hear an argument for pseudo-precision in numerical expression (say, using three or four digits on very rough data), on the grounds that one has no idea of the uncertainties, therefore any "error bar" would be only a guess, and so therefore let all the digits stand! If this policy were adopted in manufacturing, the consequences would be severe; are we sure that they are not equally so in finance and administration?

We have devised a notational system for quantitative information by which these difficulties can, to some extent at least, be overcome. It is based in large part on the experience of research work in the matured natural sciences. Contrary to the impression conveyed by popularisers and philosophers, the success of these sciences is not so much due to their being "exact" or certain, but rather to their effective control of the inherent uncertainties (of which inexactness is one sort) in their data and theories. Academics who teach and research in the sciences involved in decision-making, including economics and systems analysis, have found it difficult to resist "physics-envy", and are then hampered by an anachronistic and fantasized vision of the methods and results of physical science.

Of course, the myth of unlimited accuracy in quantitative sciences has many uses, principally for maintaining the plausibility of projects that have lost most or all contact with reality. For example, only with the debate over "Star Wars" did the public learn that computer programming is a highly uncertain and inexact art. With the proliferation of "models" of all sorts in natural and social science, we must recognize a new sort of pseudo-science, which we may call GIGO (Garbage In, Garbage Out). This can be defined as one where the uncertainties in the inputs must be suppressed, lest the outputs become indeterminate. Anyone whose work has suffered through reliance on low-quality quantitative information will have some idea of how much harm it can cause, and some inkling of how widespread is this particular form of incompetence in our society.

Remedying this situation is a big job; it will involve practitioners and teachers, and perhaps philosophers as well. In some ways the problem is built into our system of numbers, which was designed for counting and for simple calculation, and not for performing and expressing estimates. A simple joke tells the story. A museum attendant was overheard telling a group of schoolchildren that a fossil bone in the collection is sixty-five million and twelve years old. Someone asked him how he could be so precise, and he explained that when he came on the job, twelve years previously, he was told that the bone is sixty-five million years old. So he then did the sum:

The paradox here is that although in context the sum is ridiculous, our system of arithmetical notation constrains us to write it that way. We need a simple notation in which our calculations can reflect, and not violate, our common sense; and where we can express our estimates with clarity.

The notational system "NUSAP" enables the different sorts of uncertainty in quantitative information are displayed in a standardized and self-explanatory way. It enables providers and users of such information to be clear about its uncertainties. Since the management of uncertainty is at the core of the quality-control of quantititative information, the system "NUSAP" also fosters an enhanced appreciation of the issue of quality in information. It thereby enables a more effective criticism of quantitative information by clients and users of all sorts, expert and lay.

The NUSAP system is based on five categories, which generally reflect the standard practice of the matured experimental sciences. By providing a separate box, or "field", for each aspect of the information, it enables a great flexibility in their expression. By means of NUSAP, nuances of meaning about quantities can be conveyed concisely and clearly, to a degree that is quite impossible otherwise. The name "NUSAP" is an acronym for the categories. The first is Numeral; this will usually be an ordinary number; but when appropriate it can be a more general quantity, such as the expression "a million" (which is not the same as the number lying between 999,999 and 1,000,001). Second comes Unit, which may be of the conventional sort, but which may also contain extra information, as the date at which the unit is evaluated (most commonly with money). The middle category is Spread, which generalizes from the "random error" of experiments or the "variance" of statistics. Although Spread is usually conveyed by a number (either + , % or "factor of")it is not an ordinary quantity, for its own inexactness is not of the same sort as that of measurements.

This brings us to the more qualitative side of the NUSAP expression. The next category is Assessment; this provides a place for a concise expression of the salient qualitative judgements about the information. In the case of statistical tests, this might be the significance-level; in the case of numerical estimates for policy purposes, it might be the qualifier "optimistic" or "pessimistic". In some experimental fields, information is given with two + terms, of which the first is the spread, or random error, and the second is the "systematic error" which must estimated on the basis of the history of the measurement, and which corresponds to our Assessment. It might be thought that the "systematic error" must always be less than the "experimental error", or else the stated "error bar" would be meaningless or misleading. But the "systematic error" can be well estimated only in retrospect, and then it can give surprises. Fig. 1 shows how the successive recommended values of a well-known fundamental physical constant bounced up and down, by rather more than the "experimental error", several times before settling down.

Fig. 1. Succesive "recommended values" of the fine-structure constant.

Finally there is P for Pedigree. It might be surprising to imagine numbers as having pedigrees, as if they were showdogs or racehorses. But where quality is crucial, a pedigree is essential. In our case, the pedigree does not show ancestry, but is an evaluative description of the mode of production (and where relevant, of anticipated use) of the information. Each special sort of information has its own pedigree; and we have found that research workers can quickly learn to formulate the distinctions around which a special pedigree is constructed. In the process they also gain clarity about the characteristic uncertainties of their own field. Although we have not yet had the opportunity to test this with non-expert groups, we are equally sure with some preliminary training, any client or user could learn how to elicit the pedigree of information being provided by an expert.

The pedigree is expressed by means of a matrix; the columns represent the various phases of production or use of the information, and within each column there are modes, normatively ranked descriptions. These can be numerically graded, so that with a coarse arithmetic, a "quality index" can be calculated for use in Assessment if desired. For general statistical information, the pedigree is laid out as in the Table, where the top row has grade 4 and the bottom two, 0. For a numerical evaluation, average scores of 4 downwards are rated as High, Good, Medium, Low and Poor.

The Pedigree Matrix for Statistical Information

The first column describes how the job is defined, for any competent statistician knows that "just collecting numbers" leads to nonsense. The whole Pedigree matrix is conditioned by the principle that statistical work is (unlike some traditional lab research) a highly articulated social activity. So in "Definition and Standards" we put "negotiation" as superior to "science", since those on the job will know of special features and problems of which an expert with only a general training might miss. It is important to be able to descibe low-quality work; and so "symbolism" in statistics is something which any comprehensive scheme must allow for. Similarly, a "Task Force" gets a higher rating than a "Direct Survey", for the latter (like a census) may produce information that is not tailored to the problem at hand. The other two columns relate to the more directly social aspects of the work. "Institutional Culture" describes the relations between the various levels in hierarchy of command and control; and we allow for the phenomena variously described by "Clochemerle" or "Schweik". Since quality-assurance is an essential part of any productive process, including that for information, we have a column for "Review". This needs no explanation.

Thus the pedigree matrix, with its multiplicity of categories, enables a considerable variety of evaluative descriptions to be simply scored and coded. In our book Uncertainty and Quality in Science for Policy (Kluwer Academic, 1990), we illustrated the NUSAP system with a somewhat imaginary example of the history of the statistics on hand-pumps for drinking water in a Third-World country. The earlier efforts had a distinctly low Pedigree profile; inertia, symbolism and fiat were prominent, along with the absence of effective review; but by the end, with the lessons of exprience, improvements could be recorded.

The use of NUSAP could also highlight crucial features of the process. For example, the problems of definition can be explored; is a "pump" one that is listed in an old census, one that is ordered from abroad, one that is registered as delivered, one that is installed, or one that is actually in full and satisfactory use? At the other end of the process, NUSAP alerts us to the meaningfulness or otherwise of numerical expressions. If we see a number like 11,287 coming out of an unsophisticated statistical exercise, we should be able to recognize hyper-precision. On the other hand, a correctly framed estimate like "11,300:pumps: +5%" has certain uses, but the less precise statement, "<111/2:K-pumps" may be more appropriate for policy purposes. In prose, this is "somewhat less than 111/2 thousand pumps, where the aggregated unit of counting is a thousand-pumps, with a Spread of a quarter-thousand. Thus with NUSAP we are able to provide numerical statements with considerable nuance of expression.

We know enough about the use of numbers to be aware that no single system will prevent incompentence and abuse of statistics. But with the improvement of competence all around, and especially with the arming users and clients with an instrument of analysis, NUSAP will at least make it possible for the debate to be conducted at a higher level. We cannot claim that an improvement in this one area of practice will transform industry and administration for the better; but we do believe that everyone will be better off when they know what they are doing, in the management of the uncertainty and quality of their quantitative information.

(Note on references: all citations will be found in the book Uncertainty and Quality in Science for Policy, by S.O. Funtowicz and J.R. Ravetz)