Statistical inference, sampling variation and confidence intervals Part 2

This additional article is optional, for those who want a greater depth of understanding of confidence intervals - how they are calculated, what they mean and what factors affect them.

Calculation and interpretation of a confidence interval

The aim of a randomised controlled trial is to estimate the effectiveness of a new treatment in comparison to a control treatment, in a specific population. This population is our target population and includes all current and future patients. When conducting a randomised controlled trial, we know that we could have taken lots of different samples from our population of interest and that different samples would have resulted in different estimates of the treatment effect.

Calculating and reporting a confidence interval around our estimate of treatment effect observed in a single trial, provides us with a range of values that the true population treatment effect could take, which are compatible with the data from our single trial.

Standardly, 95% confidence intervals are reported. A 95% confidence interval is calculated such that if the trial were replicated numerous times, 95% of the resulting confidence intervals would include the true population treatment effect. So for our single trial and single 95% confidence interval, there is a 95% probability that this interval is one of the ones that contains the true value.

In practice, such intervals are generally interpreted as the range of values within which we can be 95% confident that the true population treatment effect lies. This is not quite correct and it is better to think of it as a range of values that the true treatment effect is most likely to take, given the data in our single trial. This more appropriate interpretation still enables us to draw useful inferences from our sample, since values outside of the confidence interval are unlikely (but not impossible), given the data observed in our trial.

The confidence interval also enables us to determine the clinical importance of this full range of possible values, better informing clinical practice.

Let’s go through the hypothetical example described in the previous video in a bit more detail. We want to know how effective a new treatment is at reducing pain compared to the current standard treatment in patients with osteoarthritis after 3 months of treatment. We recruit 100 patients and randomise 50 to receive the new treatment and 50 to receive the current standard treatment. Pain at 3 months is scored by participants on a 0 to 100 scale (100 is the worst possible pain, 0 is no pain at all).

Figure 1: Diagrammatic representation of hypothetical trial

Figure 1: Diagrammatic representation of hypothetical trial

The mean pain score at 3 months follow-up in the new treatment group is observed to be 50 points, and the mean pain score in the standard treatment group is 63 points (Figure 1). So, the difference in means is -13 points. Those receiving the new treatment have, on average, 13 points less pain at follow-up than those receiving the standard treatment. This is our best estimate of the unknown population treatment effect, the effect we would see if we were to treat all current and future osteoarthritis patients with the new treatment rather than the current standard treatment. And if the trial is free from bias, this will be an unbiased estimate of that treatment effect.

But we know that if we had taken a different sample of 100 patients, we would have got a slightly different result. So, we calculate a 95% confidence interval - a range of values that the true population treatment effect could take, which are compatible with the data from our single trial.

The range of values, or the upper and lower limits of the confidence interval depend on the magnitude of the sampling variation (the greater the sampling variation, the wider the confidence interval). Sampling variation is determined by several factors. Firstly, it is dependent on how much variability there is in the outcome measure across patients. So in our example, this refers to the amount of variability in pain scores at 3 months after the start of treatment, between the osteoarthritis patients.

We generally describe the variability in our outcome measure by the standard deviation (SD) and estimate it from our trial participants. The SD measures how much participants’ pain scores differ from the mean value for the group, the larger the SD the more spread out or variable pain scores are (see Figure 2).

Figure 2: Illustration of standard deviation (SD) of pain scores in population of patients with osteoarthritis.

Figure 2: Illustration of standard deviation (SD) of pain scores in population of patients with osteoarthritis.

The less that 3 month pain scores vary between osteoarthritis patients, the less sampling variation there would be between different possible samples drawn from the population, so the closer the estimates of treatment effect from different possible samples would be. Consequently, the smaller this sampling variation, the narrower (or more precise) the confidence interval is, as we can be more confident that the true population treatment effect lies within a narrower range of values.

For our example, let’s assume an SD of 20 points on the pain scale at 3 months amongst those in the new treatment group and an SD of 20 points amongst those in the standard treatment group.

Sampling variation and the confidence interval is also determined by the size of the sample, or the number of participants in the trial. The more participants selected from a population and recruited into a trial, the more representative the trial participants will be of the population. If we were to take lots of very large samples and conduct multiple trials (say with 10,000 patients in each trial), the observed estimates of treatment effect are likely to be very similar between the trials. If instead, we were to conduct multiple trials with only 10 participants, the variability in estimates of treatment effect across the different trials (the sampling variation) would be much greater. So a trial with a small number of participants will result in a much wider confidence interval than a much larger trial.

In our current example we have a total sample size of 100, with 50 patients in each treatment group. The formulae used to calculate the 95% confidence interval in this instance is provided in Box 1 (similar formulae are available for other measure of treatment effect).[1,2]

Box 1: Calculation of a 95% confidence interval (95% CI)

Box 1: Calculation of a 95% confidence interval (95% CI).

So, in our hypothetical trial with an estimated treatment effect of -13 points on the pain scale, our 95% CI is -21 to -5 points. Assuming no bias, the population treatment effect is likely to lie between -21 and -5 points, so on average those receiving the new treatment are likely to have pain scores at 3 months at least 5 points less than those on standard treatment and it could be as much as 21 points less. The confidence interval demonstrates that any detrimental effect of the new treatment compared to the standard treatment is unlikely, as such an effect would not be compatible with the observed data (a detrimental effect of the new treatment would be indicated by a positive value for the upper limit of the confidence interval). We can also say that it is unlikely that there is no difference at all between the treatments (which would be signified by the interval including the value of zero).

Clinical relevance

But we can’t fully interpret this confidence interval without considering the clinical relevance of this range of likely values? What is the smallest difference between treatments that would be important to health professionals and patients? 1 point? 2 points? Probably not. Is a difference of 5 points important? Would that effect the choice of treatment?

Suppose that on our pain scale it has been agreed by patients and health professionals that the smallest difference between the treatments that would be clinically important is ±10 points. Then whilst our observed estimate of -13 would be clinically important, our confidence interval from -21 to -5 tells us that we can’t rule out a smaller benefit (from -9 to -5) that wouldn’t be clinically important. If it is a more expensive treatment or has more side effects than the standard treatment there may be no benefit to its introduction into practice. At the same time the upper limit of the confidence interval tells us that actually the beneficial effect of the new treatment over and above the standard could be as big as 21 points, which would be of substantial clinical importance. So we can’t draw such clear conclusions as we might originally have thought.

Impact of sample size

Let’s have a closer look now at what happens when we increase the sample size of our trial. Let’s say we now recruit 800 patients, so 400 in each group. We observe the same mean pain score in both treatment groups as before (50 points within the new treatment group and 63 within the standard treatment group), the variability in pain scores is the same as before (an SD of 20 in both treatment groups), and the difference between the treatment means is still -13 points in favour of the new treatment. The only thing that has changed is an increase in sample size. Our 95% confidence interval is now much narrower, ranging from approximately -16 to -10 points (see Box 2).

With our larger sample size, we have a more precise estimate of our treatment effect. A difference of -13 points is still our best estimate, but now we can say that, assuming no bias, the new treatment is likely to lead to an average pain score of at least 10 points less than for the standard treatment and it could be as much as 16 points less. Unlike in the previous example with 100 patients (Box 1), the lower and upper limit of the 95% confidence interval (and all values in between), demonstrate a clinically important beneficial effect of the new treatment compared to the standard treatment.

As a final example let’s now reduce the sample size right down to 30, so 15 per treatment group. Again, we have the same mean pain scores and the same observed difference in means but the confidence interval now becomes much wider and far less informative (see Box 2). We can now say that, assuming no bias, the true difference in means is likely to be between -27 and +1 points. This positive upper limit means that the new treatment could on average result in a score marginally worse than the standard treatment, but that it could also result in a massive 27 point benefit. We are now unable to rule out a small (albeit clinically irrelevant) detrimental effect of the new treatment compared to control, or no difference, or a small clinically irrelevant beneficial effect of the new treatment, but neither can we rule out a large beneficial effect of new treatment compared to control. This is not the sort of result we want to get at the end of a long and expensive trial.

Box 2: Impact of Calculation of a 95% confidence interval (95% CI) for different sample sizes.

Box 2: Impact of Calculation of a 95% confidence interval (95% CI) for different sample sizes.[2]

Impact of variability

As stated above, the variability of the data also impacts on the width or precision of the confidence interval. Using the formulae provided in Box 1, have a go at changing the standard deviation on each treatment group from 20 points to 15 and 25 points. What impact does it have on the width of the confidence interval? You should see that as the variability in pain scores reduces, the width of the confidence interval also reduces and when the variability increases, so does the width of the confidence interval.

Confidence Intervals for different measures of treatment effect

Our examples here have considered a continuous outcome and a difference in means as our measure of treatment effect. Standard formulae also exist for other measures of treatment effect such as difference in risks, odds ratios and risk ratios frequently used for binary outcome data (for example when the outcome is death/survival).[3]

Additional Resources

For Canvas related technical support please see the 24/7. The 'Help' icon on the left-hand navigation