Title: Method of small p-values

Used in: Discrete distributions, binomial, poisson, fisher exact, five by two, hypergeometric, negative binomial, tables program, distributions program, multinomial

Used for: exact double sided testing, exact kappa, exact ordinal

This footnote is about the method of small p-values to determine the cumulative double-sided probability in discrete, counted, data. It concerns the tests in discrete distributions, such as the binomial, poisson, and hypergeometric tests, the multinomial test for a single dimensional array; and the Fisher test and other exact tests for crosstables. For most of these procedures the exact point probably, the probability of a specific observed situation, exactly the distribution of cases you observe, given the nil hypothesis, is relatively easy to calculate. The nil hypothesis in the case of most discrete distributions and the multinomial test will be a hypothetical case given by the user against which the data is tested, in the case of the Fisher it will be the model of independence between rows and columns. For the method of small p-values first the exact point probability for the nil hypothesis producing the observed table is calculated. Second, a computer program will calculate all possible alternative outcomes given the set conditions. In the case of a table the number of possible outcomes is determined by the distribution of cases in the table margin, and in particular by the number of cases in the smallest margin. In the case of a distribution the number of outcomes is determined by the sample size in the binomial and hypergeometic, the maximum possible observed count in the poisson, maximum possible population size in the negative binomial. Note that in the case of a table or the binomial test the number of possible outcomes is finite, but in the case of the poisson test the number of alternative outcomes is infinite. If the p-value of an individual alternative is smaller than the original p-value, this alternative is more different from the nil hypothesis than the observed situation. To obtain the cumulative double-sided probability according to the method of small p-values, all p-values of alternative outcomes more different compared with the observed situation are accumulated and the sum gives you the result, the sum (cumulative probability!) of p-value's of all alternative outcomes the same or more different as the situation observed.

Discussed above is the very general situation of testing for the likelihood of this or a larger difference between the expected and the observed data. This very general hypothesis test is the most used in research. However, the method of producing large numbers of alternatives and shifting them into more or less extreme compared with an observed situation, can also be used to develop exact tests for more specific situations. For example, in SISA tables the method is used to provide an exact alternative to the kappa measure of agreement, by accumulating the p-values of tables which show more agreement, and in SISA tables and the online 2*5 Ordinal procedure an exact alternative is given for the Gamma, by accumulating p-values of tables which show more ordinality.

Programs (like SPSS) which give exact probabilities using the method of small p-values sometimes take a sample from all possible alternatives and an exact probability value can be calculated within confidence bounds. The reason is that the number of calculations using exact methods can easily run in the many, many, millions, particularly for larger tables. The not totally exact exact methods are very reliable but might on rare occasions produce different results. SISA always calculates the real exact p-value, using the full number of alternatives, but is therefore limited to small tables. In general the method of small p-values will be superior compared with alternative methods to calculate exact double-sided p-values.

Agresti A. A Survey of Exact Inference for Contingency Tables. Statistical Science 1992;7(1):131-177.

Agresti A. Categorical data analysis. New York & Chichester: John Wiley and Sons 1990.

Berry KJ, Mielke PW. Jr. Exact Cumulative Probabilities for the Multinomial Distribution. Educational and Psychological Measurement 1995;55(5):769-772.

Reynolds HT. The analysis of cross-classifications. New York, London: Free Press, Collier Macmillan 1977.

TOP of page

SISA Research Paper