Shouldn't we know better now how effective vaccination is? Our blogging statisticians Sascha Feth and Jochen Fiedler discuss statistical basics on vaccine effectiveness. The matching blogpost to read on the German podcast episode.
© iStockphoto
In our podcast, we have already talked about how vaccine effectiveness is calculated. We mainly referred to the registration studies. Now millions of people have been vaccinated and we have a lot of new data. Shouldn't we know better now how effective vaccination is?
Now that millions of people in Germany alone have been vaccinated since December 2020, there have been many reports of observed effectiveness of vaccinations as a measure of efficacy. In some cases, these efficacies were dramatically lower compared to the values from the approval studies. For example, there is a report of only 39 percent effectiveness against SARS-CoV-2 infections and 40.5 percent effectiveness against symptomatic Covid-19 disease. Only in the protection against hospitalizations, or severe courses was still 90 percent effectiveness found.
Do these values prove a decreased protection due to the delta variant, or the fact that the first vaccinated persons are already no longer immune? Israel, for example, was one of the countries that fully vaccinated large portions of the population early on. But be careful, because these quick, plausible-sounding conclusions leave one aspect unconsidered: Vaccinated people probably behave differently than unvaccinated people, partly because they have different rights or experience hurdles in everyday life.
Regulatory studies of vaccines go to great lengths to ensure that there are no bias effects. In the case of the study: those who think they have been vaccinated and feel safe usually behave more carelessly afterwards than unvaccinated persons or vaccinated persons from the control group who received the placebo and do not know what they were injected with. The difference in behavior would then be superimposed on vaccination effects, and it would be unclear what contributed what to the effectiveness. Therefore, these studies are randomized and double-blind. Thus, neither subjects nor physicians know whether the real vaccine or the placebo is administered. Furthermore, the control group – i.e. those who do not receive the vaccine being studied – is not only administered a saline solution, but also, for example, a vaccine against another disease. In AstraZeneca's pivotal study, for example, a combination vaccine against meningococcus was used in part. As a result, there are no systematically different reactions to the injection. Otherwise, conclusions could be drawn from the (non-)occurrence of reactions (such as fever, headache or chills) as to whether one is in the placebo group or not. In addition, there are other measures, such as taping the syringes, so that it is not possible to draw conclusions about the injected substance on the basis of the color.
Based on the effort expended, one can assume that the control and experimental groups do not behave significantly differently and compare the infection rates of the two groups. In very simplified terms, an estimator for vaccine effectiveness can then be found using the following equation:
EImpf = 1 - AExperimantal / AKontroll
where AExperimantal is the proportion of infected people among the vaccinated and Akontroll is the proportion of infected people among the unvaccinated. A vaccine effectiveness of 90 percent then means that the proportion of infected people among the vaccinated is only one tenth of the proportion of infected people in the control group.
This effort put into registration studies shows that one must interpret observations »in the field«, i.e., after the active ingredient has been approved and inoculated, with more caution. First of all, one must consider which parameter exactly one is looking at. In the media, the proportion of vaccinated people among the sick is often mentioned, which in itself is a very misleading figure. As an example, if 100 percent of the people in a population are fully vaccinated, the proportion among those infected is also 100 percent - regardless of whether the vaccine effectiveness is 0 or 90 percent. Thus, this proportion alone in no way indicates the vaccine effectiveness or the individual risk of a vaccinated person becoming infected with SARS-CoV-2.
For these two variables, which are likely to be more interesting for most recipients, a little more knowledge is needed. For didactic reasons, let us briefly assume that vaccinated and unvaccinated persons behave similarly and that the groups are composed similarly – with respect to age, previous diseases and other possibly relevant factors. Then, vaccine effectiveness could again be determined relatively easily: One compares the proportion of infected persons among the vaccinated with the proportion of infected persons among the unvaccinated, analogous to the registration studies. The individual risk of infection of a vaccinated person can now also be roughly and simplified estimated with additional knowledge by applying Bayes' theorem:
R = (Avaccinated and infected * Ainfected) / Avaccinated
Where Avaccinated and infected is the proportion vaccinated among those infected; Ainfected is the proportion infected in the total population, i.e., prevalence; and Avaccinated is the proportion vaccinated in the total population.
| We have used Bayes' theorem before – after taking a deep breath – here on the blog [Only availalbe in German]: Warum Sie nicht Aussagesicherheit sagen sollten |
The assumptions made above are, of course, out of touch with reality. On the one hand, especially older people and people with pre-existing conditions have been increasingly vaccinated, while there is still no approved vaccine at all for children under twelve. On the other hand, vaccinated and unvaccinated people now know about their vaccination status – unlike in the approval studies. That alone will often trigger different behaviors. In addition, vaccinated people sometimes have more rights, or lower barriers, to attend public places or events, or to travel. For example, unvaccinated people in Germany must (have) a test wherever the 3G rule applies, unless they count as recovered.
This point seems very plausible and simple at first glance, but in reality it can become quite complex. This is due to the fact that both individual vaccination effectiveness and contact numbers can depend on common factors, such as age. For our following thought experiment, let us assume that these effects do not play a role. Furthermore, we assume a mean individual vaccine effectiveness of 90 percent and further assume that vaccinated people have twice as many contacts as unvaccinated people.
So how many contacts on average do vaccinated individuals need to have the same individual risk of infection as unvaccinated individuals? The answer is quite simple: ten times as many, because due to the 90 percent vaccine effectiveness, the individual risk of infection is only one tenth per contact. This would result in the same proportion of infected among the vaccinated and unvaccinated, despite a very high vaccine effectiveness! Of course, we do not assume that vaccinated people now have ten times as many contacts as unvaccinated people. But what about smaller values? That, too, can be calculated quite easily: If vaccinated people have twice as many contacts, they are 80 percent less likely to become infected than the unvaccinated group; if they have three times as many contacts, they are 70 percent less likely, and so on. It is also irrelevant how exactly the vaccination works: That is, whether 90 percent of the vaccinated are completely immune or whether all vaccinated individuals have a 90 percent reduced risk of infection.
What does this mean for field observations? We have to take into account how much differently vaccinated and unvaccinated people behave, e.g., in terms of their contact numbers. Estimating these contact numbers is not very easy and requires some effort. The quantitative impact is thus difficult to estimate reliably. At the same time, we do not assume that the significantly lower vaccine efficacies, compared to the registration studies, can be explained by different social behaviors alone.
Nevertheless, please keep in mind that media reports about a significantly decreased vaccine effectiveness cannot be interpreted without further ado. Simply applying the same formulas from the controlled studies over the large, less structured field data can also lead to nonsense. Put another way: Numbers do not become truer if the sample (in this case, all vaccinated people) only becomes large enough. For risks and (side) effects, please ask your statistician.
Fraunhofer Institute for Industrial Mathematics ITWM