Reading a Journal Article; How Do We Know That the Results Represent a Statistically Significant Finding?

Basic Statistical Analysis for Anesthesiologists

When we read the biomedical literature we depend, to some extent, on the journal editors to guide us about the veracity of the observations that are being presented. Beyond that, it is largely the responsibility of the reader to determine whether the data that is being presented has been appropriately derived and analyzed. We have spoken before about bias that may have been introduced into the investigation. Some elements of bias may be apparent from the close reading of the article. Some may not be.

The statistical analysis of observations in any given study allows the reader to determine whether the findings are significant based on accepted values routinely used. This does not determine whether the finding is clinically significant in the population that is evaluated or whether there is statistical or clinical significance in any given population that a clinician may be considering. As I have said before, very few single articles provide evidence that should change current practice. The best evidence is obtained from multiple randomized controlled trials that have been analyzed as a group.

Clinicians require a basic knowledge of the basis of a statistical analysis in order to understand the language of medical research and that will be the focus of this short piece.

The Language of Data Evaluation

Medical studies, especially randomized controlled trials, present data in many different ways but some of the information in a statistical analysis is invariable:

1.     Data from a study is presented using two distinct processes – a description of the sample that is being used for the analysis, and a determination of the conclusions that can be drawn from the data. Both of these aspects may be presented in tabular form

2.     As a general rule the characteristics of groups considered in a study should be as close to each other as possible. A significant difference between groups is a bias that may render the findings of the study meaningless.

3.     Variables are, of course, anything that can be measured or manipulated in a study for the purpose of demonstrating an effect. These characteristics may be presented as nominal, ordinal, interval, and/or ratio. Interval data may be presented as continuous or discrete.

4.     Randomization represents the attempt by the investigator to assure that bias is not introduced into the study by choosing patients that are intrinsically more or less likely to respond in a given way.

5.     Blinding attempts to remove any bias that is produced by the investigator, the observer, or other clinicians that may change the characteristics of care associated with the trial.

6.     Data that is presented in a continuous fashion numerically has characteristics, which describe the central tendency of a group of data points. This represents the tendency of an individual observation to be clustered around a central point – the mean, median, and/or the mode. Only the mode may be used with all types of data. The mean requires interval/ratio measurements while the median ordinal may be used with interval/ratio or ordinal data.

7.     In concert with a description of the central tendency or data is a description of the variability of the entire data set around a central value. This variability can be defined in several different ways, the most common being the standard deviation. Other methods may be noted based on the characteristics of the data set – the range, the variance, or the standard error of the mean.

8.     The appropriate use of these characteristics of the data, that is descriptions of the central tendency and the variability of the data are important in the interpretation of the data that is presented.  For a clinician, interpreting a data set with an indication of high variability may suggest difficulty in understanding the true clinical significance of the study.

(P<.05)

There is some probability that in examining the data derived from clinical trials we arrive at a value that is only indicative of chance. That is, the value in reality does not represent a true scientific finding but a random observation. How do we determine whether the finding is statistically significant or not?

First, investigators usually presume that there is no difference between an experimental group and a control group. This is termed the null hypothesis. In analyzing data we must prove that the individual values were drawn at random from the same population. When this requirement is met, we use statistical tests to determine the likelihood that the value that we have derived represents a statistically significant finding, that is that it is not representative of chance alone.

By using statistical analysis, if the probability is determined to be less than one in twenty, giving a P (probability) value of less than .05 (P<.05) then the null hypothesis is rejected and one can conclude that there is indeed a difference in the group. That said, even with careful analysis of data that is randomized, blinded, and analyzed, there always exists a certain probability of accepting or rejecting the null hypothesis in error, either a type 1 or a type 2 error.

Type One Error represents the probability of incorrectly rejecting the null hypothesis when, in fact the null hypothesis is true. (Remember that the null hypothesis assumes that there is no difference between two groups.) This value, the probability of making this mistake is termed the alpha and is usually not more than 5% (Alpha =0.05).

Type Two Error is the probability that one fails to reject a null hypothesis that is in fact false. This implies that there is a real difference in the experimental and the control group but analysis of the data did not demonstrate this difference. Obviously, if the original analysis rejects the null hypothesis, then no Type Two Error can be made.

Randomization, blinding, measures of central tendency, and of variation,  variables, demonstration of the probability that the results are not solely due to chance, and errors in this determination; these are basic concepts that are required in determining whether the journal article that you are reading meets the basic standard of scientific investigation. We will talk about these elements in detail and other, more complete, offerings.

Rae Brown, M.D.

January 22, 2015