Universal BCG vaccination and protection against COVID-19: critique of an ecological study

This post was written by Lena Faust, Sophie Huddart, Emily MacLean and Anita Svadzian. The authors are PhD students in Epidemiology at the McGill International TB Centre, Montreal, Canada.

Like Comment

While there is no specific treatment or vaccine for COVID-19, a recent study has suggested the potential effectiveness of an existing vaccine, the Bacille Calmette-Guérin (BCG) vaccine, against COVID-19.

 The BCG vaccine is used in some countries for the prevention of tuberculosis (TB), a bacterial infectious disease caused by Mycobacterium tuberculosis (Mtb), which accounted for 1.5 million deaths globally in 2018. National BCG vaccination policies vary across the world, with universal BCG vaccination at birth still occurring in high TB-burden countries such as India and Ethiopia, while countries with lower overall TB incidence have largely discontinued universal BCG vaccination in favour of targeted vaccination of specific TB risk groups (such as children from families immigrating from high TB burden countries). The BCG World Atlas, created by the McGill International TB Centre, provides data on BCG policies and practices around the world.

The study

For the study by Miller et al, BCG vaccination policies across countries were collected from the BCG World Atlas. Data of COVID-19 cases and death per country were obtained on March 21, 2020, from https://google.org/crisisresponse/covid19-map. The ecological analysis found an association between COVID-19 death rates and universal BCG vaccination policies among middle and high income countries. The authors also observed an association between COVID-19 cases per capita and BCG vaccination policies, with countries with a universal BCG vaccination policy having lower cases per capita and death rates. Based on the analysis, the authors state “The correlation between the beginning of universal BCG vaccination and the protection against COVID-19 suggests that BCG might confer long-lasting protection against the current strain of coronavirus.”

Critique

Study Design

The main issue with this study lies in its design. Ecological studies often conflate population level exposures, such as country-wide policies, with exposures at the individual level, such as the effect of the BCG vaccine on a human body. What may hold at the aggregate level, will not necessarily be true when the heterogeneity of individuals is considered. In this study, low quality evidence observed at the population level is used to make sweeping inferences about BCG’s effectiveness on an individual level. Ecological analysis techniques simply cannot directly inform etiology of exposure/disease relationships; rather, they serve us well as strictly hypothesis-generating queries and should not be stretched beyond this purpose.

Timing matters

Since the analysis (COVID-19 data were downloaded on March 21, 2020), the pandemic has escalated in many countries, including low and middle income countries (LMICs) where BCG is given at birth. For example, COVID-19 cases in India have increased from 195 on March 21  to 1071 on March 31. In South Africa, cases have increased from 205 on March 21 to 1326 on March 31. Therefore, the paper’s findings could look very different as countries progress in their epidemic timelines (and expand diagnostic capacity). For example, once we see the full effects of community transmission in large, densely-populated countries such as India, where large numbers of day-labourers are now leaving cities after being put out of work amid the country-wide lockdown. This underlines that drawing premature conclusions from rapidly changing data in a pandemic is problematic.

Lack of testing

Accurate analysis is dependent on accurate data and we know that the current number of coronavirus cases is dramatically underestimated around the world due to shortages of diagnostic tests. The number of tests performed per capita varies wildly by country. Underestimation of the number of cases in lower income countries could entirely explain the authors’ observed results. India, for example, has one of the lowest COVID-19 testing rates in the world. Additionally, countries have each applied their own algorithm to determine who gets tested. Thus, the number and composition of known cases is heavily influenced by testing strategy. The authors of this paper acknowledge the concerns surrounding current estimates of COVID-19 cases and also use COVID-19 deaths as a measure of the severity of each countries’ outbreak. Unfortunately, even current COVID-19 death data are unreliable in some parts of the world. In wealthy countries, it is true that most deaths due to COVID-19 will occur in a hospital after having received a diagnosis of COVID-19. However, we know that in LMICs, even before this current pandemic, most deaths occurred at home without medical assistance. People dying at home in LMICs are unlikely to receive a confirmed diagnosis of COVID-19 and may not have their death accounted for in government records. Coronavirus deaths may also be missed in high-income countries that have been overwhelmed by the virus, like Italy. It is very difficult to draw conclusions about any potential impact of BCG vaccination when the data are so flawed.

Correlation is not causation

A critical flaw in the methodology of this study is apparent in the fact that a perceived relationship between an exposure and an outcome does not mean one causes the other - a concept often expressed through the well-known adage “correlation does not imply causation.” This can occur in situations where other underlying factors influence both the exposure and the outcome and thereby create a spurious association between the two (in epidemiology, this situation is referred to as confounding). Specifically, in the case of the relationship between universal BCG vaccination and the apparent lower COVID-19 case numbers and COVID-19-associated deaths, a multitude of underlying factors are at play. The crucial take-away here is that if these factors could be appropriately accounted for, we would very likely not see the association between BCG vaccination and protection against COVID-19 suggested by this paper. The concerns about data quality now become a potential source of confounding in the relationship between BCG vaccination and lower COVID-19 case numbers. It is likely that under-funded health systems in (LMICs) have low COVID-19 testing capacity, and at the same time, are more likely to have universal BCG vaccination policies still in place, given that TB burden is highest in LMICs. Therefore, the study actually does not demonstrate that populations in countries with universal BCG vaccination policies are somehow more protected against COVID-19, because lower case numbers and deaths may be more attributable to low testing capacity than to any supposed protectiveness of BCG. Therefore, statements in the paper that imply causality, such as the statement that “BCG vaccination also reduced the number of reported COVID-19 cases in a country” are dangerously misleading, particularly in a context where reported case numbers are in large part a function of testing capacity and testing strategy, rather than actual relative burden. 

A further important consideration is that it is difficult to draw causal conclusions across a broad range of countries with varying underlying demographic characteristics, such as differences in the age distribution of the populations of interest. This is particularly true of diseases for which severity differs by age. As shown in a recent study from Wuhan, China, older individuals are at higher risk for severe COVID-19, so differences in the age distribution of populations may significantly influence death rates observed across countries (for example, the WHO regions of South-East Asia and Africa have a median age of 27.0 and 18.7 years, respectively, whilst the population in the European region is much older (median 38.6 years), which may in part contribute to the higher death rates observed in Europe). In light of these considerations, given that no age-standardization method was used in the Miller et al. study, the strong limitations of the inferences made need to be emphasised.

Biological plausibility

Lastly, some of the “common sense” appeal of this paper’s claim is based on a bit of a false equivalence. The authors twice mention that BCG protects broadly against respiratory diseases. The idea that a vaccine for an infectious disease of the lungs could also protect against other infectious diseases of the lungs seems somewhat consistent and thus palatable. This hits a snag though, when we look at the type of TB against which BCG most effectively protects. Due to an unfortunate natural experiment wherein there was a global shortage of the BCG vaccine from 2013 to 2015, retrospective data show there was a surge in cases of TB meningitis, that is, TB of the central nervous system. The tidy idea of a lung disease vaccine protecting against another lung disease, i.e. COVID-19, is not quite so straight-forward, since BCG seems to offer most significant protection against TB meningitis. Additionally, it is well-established in the TB field that the BCG vaccine only confers protection against TB in young children, and that protection almost completely wanes by age 12. Recent studies have even examined whether applying a booster vaccination to older adolescents or adults would help extend this anti-TB protection. The important take-away is that BCG’s protective effect fades over time. So it is not clear how a vaccine that offers protection early in an individual’s life would protect individuals against COVID-19 once they are of an advanced age.

Conclusions

From the latest piece in Forbes to Bloomberg, the study’s findings are gaining more and more traction. There is danger in citing that there is evidence that a century-old vaccine may boost immunity in individuals, providing non-specific protection to other illnesses, and by extension protecting against COVID-19 or reducing severity of its presentation based on this analysis alone. Accepting these findings at face value has the potential for complacency in response to the pandemic, particularly in LMICs. One needs only to look at how this has been portrayed in news outlets of several LMICs already; the dangers of such portrayals misinforming the public should not be underestimated, for example, in  countries such as India, the wide BCG coverage offered by their universal vaccination policy may create  a false sense of security and lead to inaction. If claims such as these, based on insufficient evidence, interfere with pandemic response,  the most vulnerable populations will be those hardest hit by the consequences. 

With further research, it may emerge that the BCG vaccine does confer protection against COVID-19; however, with the current state of knowledge we cannot state this with any degree of certainty, and an ecological study does not provide sufficient evidence. More basic scientific research and appropriately designed trials are needed to parse out this relationship. An Australian trial is set to start imminently, aiming to investigate whether BCG vaccination protects against COVID-19 or reduces severity of COVID-19 in Australian healthcare workers. If, in a well-performed trial, we do in fact see  a benefit in re-purposing this vaccine, the evidence would be much more convincing.

Emily MacLean

PhD candidate, McGill University

I completed my MSc in Microbiology & Immunology at McGill, focusing on Leishmania. Currently I'm a doctoral candidate in epidemiology and am focusing on TB diagnostics at the McGill International TB Centre under the supervision of Dr Madhu Pai and Dr Nandini Dendukuri, but I still appreciate those posters with all the cell signalling pathways.

32 Comments

Go to the profile of Forhad Uddin Hasan Chowdhury

Fantastic write-up... well explained 

Go to the profile of Ben Johnson
Ben Johnson 7 months ago

Thanks for this excellent analysis! A friend shared the original study with me, and my response was similar to yours, but I have now shared your much more detailed response with them. I would also encourage you to write a review on MedrXiv if you haven't already. 

Go to the profile of J H Stebbings
J H Stebbings 7 months ago

This post nicely summarizes the weaknesses of ecological correlation studies, and to that I'd add only one point: it may suffer from "presentism."  If one is interested in mortality of the elderly, particularly males, immunization policies in the 1940s-1960s may be critical.  At age 80, I remember being immunized in the US against TB (leaves a big scar on shoulder!), and receiving booster shots in the military in the 1950s. I've no idea of the timetable for cessation of immunization thereafter.  

As to biological plausibility, the first internet posting I saw re BCG (https://www.sciencemag.org/news/2020/03/can-century-old-tb-vaccine-steel-immune-system-against-new-coronavirus?utm_source=Global+Health+NOW+Main+List&utm_campaign=18482039f6-EMAIL_CAMPAIGN_2020_03_23_12_56&utm_medium=email&utm_term=0_8d0d062dbd-18482039f6-871647 ) emphasized that BCG might also boost INNATE immunity in addition to inducing specific cellular-mediated immunity, which is what would boost covid9 resistance.  Apparently innate immunity has recently been found to have some memory, although  whether extending beyond the persistence of BCG organisms in the skin was obscure.

I consequently wondered whether a stiff tuberculin injection in those elderly vaccinated when young against TB might incidentally boost innate immunity and give some protection against current viral infections.

Take this with a grain of salt.  The writer is long retired, never worked on infectious diseases as part of his epidemiological research, and is working from memory!

Go to the profile of J H Stebbings
J H Stebbings 7 months ago

And on further thought, I realize memory is fallible, and I am having serious doubts as to my memory of the early history of immunization against TB (and don't feel like reviewing the old literature!) js

Go to the profile of Emily MacLean
Emily MacLean 7 months ago

Thanks for the comments! It definitely now does seem like there is an innate component to BCG-mediated immunity. And the recent studies about revaccinating young adults with BCG to boost anti TB immunity are v intriguing. Our point I guess is that from this ecological study alone, it is not possible to know whether it would extend to one particular other disease. Maybe it does! Maybe it doesn't! But we'll know with further research !

Go to the profile of Nikki Honigwachs
Nikki Honigwachs 7 months ago

Your article highlights the challenges of having done the research across countries and hence drawing conclusions across heterogenous groups. I would like to pose a possible solution where we could study infection and death rates in a fairly homogenous group where the study variables are similar with the exception of BCG vaccine administration. 

I saw an article that the majority of Covid-19 cases in Chicago are among African-Americans. Would it be possible to examine the place of birth of those African-American cases in Chicago and other cities in the USA? This community would be made up of individuals who were born in the USA but also more recent immigrants who were born in Africa, most likely Nigeria, Ghana, Kenya. If they were born in the USA, they would not have received the BCG vaccine. If they are immigrants to the USA but born in Africa, it is highly likely they received the vaccine. Is there is difference in infections and deaths between these two subgroups of the same community?


This will allow you to study the BCG vaccine effect stripping out variables such as weather, UV exposure, extent of social distancing, access to healthcare and (to a large extent) socioeconomic factors. I say to a large extent because it might be argued that recent immigrants have it tougher than homeborn African-Americans. You would not have the issues of studying BCG/Covid-19 across countries, or having to wait for the clinical trials running in Australia etc. This comparison could be conducted in any of the key cities in the USA where you have high Covid-19 cases and large groups of African-Americans and African immigrants living together. (I am aware of the pitfalls of this being viewed as a racial study - but it is not racial, it is seeking a "homogenous" group where the key variable is place of birth and hence vaccine exposure, and other variables are fairly constant.)

 

Go to the profile of Rob Karas
Rob Karas 5 months ago

 thank you for your hard work and insight

Go to the profile of Ioana Olaru
Ioana Olaru 7 months ago

Completely agree. This study actually reminded me about the study on chocolate consumption and winning the Nobel prize. Similar principle

Go to the profile of Michelle Kaminer
Michelle Kaminer 7 months ago

☺️👌

Go to the profile of Jane
Jane 7 months ago

This critique is deeply misguided and needlessly condescending in tone. The original paper clearly only claimed correlation and used that correlation to generate a hypothesis that the authors called for to be verified with randomized trials for causality. The trials are now underway. The article was nothing more, nothing less.

Meanwhile, this critique creates a strawman fallacy out of the original paper claiming that the original paper claimed causality and then proceeds to attack the strawman with kindergarden statistics concepts ("Correlation is not Causation"!).

A more useful response to the original paper would be A) randomized trial results or B) a more refined version of the analysis controlling for more confounders. Either of these could help cofirm/refute the hypothesis. Meanwhile, refuting imagined strawmans is not useful to anyone.

If instead this critique is intended as a response to the Forbes, Bloomberg and other clickbait media articles, perhaps this critique should be rewritten as a PSA and posted in those channels. The general public could use a lesson on correlation != causality, but to assume that your colleagues don't know this is condescending.

 

Go to the profile of Jun Sato
Jun Sato 7 months ago

I did an extensive research regarding COVID-19 and BCG vaccination correlation. I found that BCG strain is quite important. No BCG < BCG Denmark << BCG Japan/Russia.
https://www.jsatonotes.com/2020/03/if-i-were-north-americaneuropeanaustral.html

Go to the profile of Yutaka Akiyama
Yutaka Akiyama 7 months ago

Here is yet another "correlation" work did by us from Japan, showing relationship between the doubling time (DT) of total deaths toll and BCG vaccination of 47 countries. Distributions of DTs for 32 vaccinated and 15 non-vaccinated countries were significantly different (p=0.004). The analysis also showed that five countries using Tokyo 172-1 strain (Japan, Iraq, S. Korea, Malaysia, and Pakistan) had longer DTs (median 6.5 days) than other 27 vaccinated countries  (median 3.3 days). Difference of two distributions was significant (p=0.005). Yes, the correlation might be spurious and does not directly imply causality.

https://www.bi.cs.titech.ac.jp/COVID-19/Death_vs_BCGpolicy.html

Go to the profile of Nóra Máté
Nóra Máté 6 months ago

Your study is very interesting.

I also think there is a correlation.

There is a very surprising fact that may also be explained by this BCG hypothesis: Romanian workers in Spain and Italy

- very few of them died of COVID-19 abroad (cannot find the source in English) and

- despite their massive return to Romania - partly even before the quarantine measures - Romania belong to the less infected countries.

 

Go to the profile of Paola Rozada
Paola Rozada 7 months ago

Excellent finding and explanation 

Go to the profile of Klause
Klause 7 months ago

Wonderful

Go to the profile of Iwao
Iwao 7 months ago

Did you take strain type into consideration?

The BCG assumption doesn't state BCG make role to prevent infection, but to decrease heavy cases among infected.

Also, it suggest only old-type strain, mean Russian, Tokyo Bulgaria strain, is effective.

Please check death rate in old-type strain country against the other countries. Japanese NewYorkers must have died at least one person in 99.65%, but none yet.  

Go to the profile of Stefan Kirov
Stefan Kirov 7 months ago

Unlike Jane I found this blog to be very well written and very much on topic. I think the two studies linking BCG to COVID19 infection or mortality rates should have looked at some obvious confoundig factors.

I tried to do some of that: https://www.researchgate.net/publication/340463940_Association_Between_BCG_Policy_is_Significantly_Confounded_by_Age_and_is_Unlikely_to_Alter_Infection_or_Mortality_Rates

Not perfect but a start...

Go to the profile of Emily MacLean
Emily MacLean 7 months ago

Thanks for sharing your piece. Will read with interest

Go to the profile of Rob Karas
Rob Karas 5 months ago

Thanks miss Emily for  hard your work and insight. 

Go to the profile of Ray Sweha
Ray Sweha 7 months ago

This write-up is significantly more flawed than that study it criticizes. The authors of the original paper didn't claim causality, only correlation.  Two more correlation studies from Ireland and Japan were recently conducted as well https://www.medrxiv.org/content/10.1101/2020.03.30.20048165v1.full.pdf
https://www.irishtimes.com/news/health/coronavirus-more-striking-evidence-bcg-vaccine-might-protect-against-covid-19-1.4222110

These three studies point to one thing, death rate (per million) from covid-19 in countries with weak BCG vaccination history is TWINTY times that in countries with strong BCG vaccination history. Given the human and economical toll of this virus, a large number of research and public health institutions should be aware of this study and thinking how to validate it and activate it.  

My second qualm with this critique is that it takes issue with ecological studies in general. I'm not a medical scientist but I'm a data scientist with a PhD in computer science and a deep understanding of statistics. And the notion that a controlled study of a few hundred or thousand individuals should be trusted more than an vaccination data where 130 million individuals get vaccinated every year around the world and a history of using this vaccine for a hundred years is outrageous. We use sampling because it is very costly and unsafe to test a new drug on people, but if a census level data is available, it sure beats every sampling techniques regardless how immaculate the variables were controlled. Ecological studies shouldn't be discounted because they are not the norm, especially in a time of a pandemic where census data of morbidity and mortality is readily available. Just look at Japan, an over-crowded country with very high life expectancy, no social distancing measures, minimal testing and one of the first countries to report cases of covid-19, yet the suffering there is minimal. Other massive countries like India, Russia and Egypt are faring well even though you would think they should be crushed by this virus by now. At any rate, looking at  individual countries and trying to rationalize why, is not the proper way, but looking at the whole world, where many of these variables will even-out following the law of large numbers, is the proper method. And that is what these three studies did.  

 


My third qualm with this critique is that we shouldn't trust a causation study if we don't understand how the drug really work (etiology). This is very arrogant as it  assumes the human race have a good understanding of the natural world while in fact our body of knowledge is just skimming the surface of reality. The fact that we don't understand how  innate immune memory works, assuming it actually exists, shouldn't discourage us from embracing and using it for the good of humanity. Our attitude should be, "all models are wrong, some are useful"

I understand the author's concern that buying into this theory blindly can cause complacency in countries with strong BCG vaccine like India, but at the same time not taking this hypothesis seriously means thousands of lost lives in a country like the US where this hypothesis around BCG is not being studied yet. It is clear as day to me that certain countries are faring much better than others, and that is regardless of standard of living, race, government competency or testing capacity. Mass BCG vaccination  is the leading hypothesis now and we should expedite studied like the one in Australia (may be dive deeper into which strain is more effective) and we should also  understand BCG supply chain and how can we make it widely available if indeed it proved effective. 

Go to the profile of Stefan Kirov
Stefan Kirov 7 months ago

I am sorry, but if you have a degree related remotely to statistics you would kow better before trusting studies that do not explore confounders, associative or otherwise.

Go to the profile of Ray Sweha
Ray Sweha 7 months ago

Hi Stefan,
I read the analysis you added to the thread earlier. You raise a good point about having a country's life expectency as a bigger factor. This study from Japan cotrolls for life expectency. Hope you find it helpful 
https://www.medrxiv.org/content/10.1101/2020.03.30.20048165v1.full.pdf
Another question, I see you used the vaccination rate from WHO as a source. I could only find data dating back to 1980 but not older, can you share a source. Here is my source https://www.who.int/data/gho/data/indicators/indicator-details/GHO/bcg-immunization-coverage-among-1-year-olds-(-)
And thanks for putting your code and data source on github. 
Ray

Go to the profile of Stefan Kirov
Stefan Kirov 6 months ago

Ray,

For vaccination rates I have used only the most recent vaccination data.

As I said- my own study does not claim to solve an issue, but merely to expose it. One should really think of what other significant confounding factors are lukring in there.

Frankly, the fact that BCG policy predicts better cases/outcomes than actual vaccination rates tells me there is no causal link between BCG vaccination itself and COVID19 infections and most probably one of the many potential confounders is at work (genetic makeup, chronic diseases rate, etc.).

Finally- while there are hundreds of thousands for cases, in this analysis you have only a bit more than 100 data points. And the 2 interesting categories have only a handful.

I would rather see people spend their time and energy looking at ACE2 genetic varinats for example.

Go to the profile of Rob Karas
Rob Karas 5 months ago

I agree, we in America need to be all over this. When countries using the Tokyo BCG have 100 times less mortality than the more severely affected countries there should be a huge push to validate this and if amp up production of this vaccine.

Go to the profile of IQBAL AHMAD
IQBAL AHMAD 7 months ago

Very good explaination with right queries.Excellent article.I appreciate it.

Prof. Iqbal Ahmad

 

Go to the profile of Ben Johnson
Ben Johnson 7 months ago

It's great to see so much discussion on this post.

Please keep your comments respectful. Any comments in violation of our community policy will be deleted and the users blocked from commenting further.

Many thanks!

Go to the profile of Anne Moore
Anne Moore 6 months ago

Great work! Thank you!

Go to the profile of Balachandran Ravindran
Balachandran Ravindran 5 months ago

Mechanistically, BCG vaccination has been shown to re-program monocytes leading to functional changes in innate immunity that has been designated as ‘trained immunity’1. Not surprisingly clinical trials have already been initiated to investigate efficacy of BCG vaccine against Covid 192. One of the cardinal features of BCG vaccination is sensitivity of vaccinated subjects displaying skin reactivity (also called Mantoux test) to Mycobacterial tuberculin when tested using Purified Protein Derivative (PPD) from Mycobacterium tuberculosis. A large proportion of subjects living in tuberculosis endemic areas also develop skin sensitivity to PPD even when not vaccinated with BCG. Such skin testing has been used for several decades in tuberculosis endemic countries to detect exposure to M.tuberculosos and/or latent tuberculosis3. What is however not known is whether reactivity to PPD in Mantoux test is also associated with epigenetic re-programming of monocytes and induction of trained immunity. In the context of the current on-going pandemic, countries across the world could rapidly undertake PPD skin testing in their population to study correlation (if any) between susceptibility to Covid 19 infections and PPD skin reactivity. If a significant association between skin reactivity to PPD and resistance to Covid 19 infection (current as well as past infections) is demonstrated, it could validate the assumption that prior exposure to BCG or to Mycobacterium tuberculosis (in endemic countries) offers protection against Covid-19 infections. This could strengthen the case for undertaking clinical trials more widely to evaluate BCG vaccination for controlling the on-going global pandemic.