Nothing in this article is intended nor implied to constitute professional medical advice or endorsement. The views expressed in this article are those of the author and do not necessarily reflect the views/position of Doximity.
COVID-19 is a worldwide pandemic affecting millions, including the over 400,000 Americans who have died as a result of the virus. Ivermectin, an anti-parasitic drug, has been proposed as a potential effective treatment. In vitro (in cell cultures) and in animal models, ivermectin has shown anti-inflammatory and antiviral properties, including antiviral activity against SARS-CoV-2, the virus that causes COVID-19.
Of course, a safe and effective treatment for COVID-19 is desperately needed. The website ivmmeta.com claims to have completed a meta-analysis of outcomes for the use of ivermectin in COVID-19 and found significant positive benefits. What follows is an in-depth assessment of the results — it’s a little technical (but it has to be in order to understand the limitations of the analysis).
At the outset, the authors should be commended on their description and transparency of their work, which allows for this in-depth review. The main purpose of doing a systematic review and meta-analysis is to combine data from several small studies in an unbiased way to arrive at an “effect size,” or a single number for a treatment’s effectiveness. Observational data should not be included in any review evaluating the effectiveness of a treatment because, among other reasons, observational studies have a high risk of bias. Instead, only randomized trials (preferably double-blind, placebo-controlled trials) are appropriate for inclusion. Consider the hydroxychloroquine data from early in the pandemic, which was subsequently disproven in at least five large randomized trials. Or, consider observational studies that suggested steroids were associated with increased mortality, findings that were subsequently disproven in randomized trials that showed steroids actually decrease mortality in critically ill patients. The ivmmeta authors do exclude observational studies in favor of randomized controlled trials (RCTs), but only in some of their analyses. And yet, the authors conflate the two in many of their graphs, which is fairly misleading.
Every meta-analysis is vulnerable to publication bias. That is, researchers and editors are much more likely to publish positive or significant results than non-significant ones. Therefore, a meta-analysis may find a significant difference solely as a byproduct of the tendency to publish only positive trials. It is impossible to include the negative or nonsignificant results if they were never published. Researchers often account for this fact by including a “funnel plot” in their findings. Ivmmeta does not include a funnel plot and, as a result, it’s impossible to assess for publication bias in the data. Notably, if you include their RCT data on mortality, you will see an asymm etric funnel plot beginning to develop, suggesting that there are unpublished studies showing no benefit that were not included in the analysis. (See Figure 1)
Figure 1. Funnel plot of ivermectin randomized controlled trial data on mortality (not peer reviewed)
Another problem with many meta-analyses is heterogeneity; included studies should measure similar interventions, populations, and outcomes. Any meta-analysis of ivermectin data would have a heterogeneity flaw because the ivermectin studies have very different populations, doses, co-administered medications, and outcomes. The doses of ivermectin range from 6 milligrams once, to 12 milligrams every 12 hours for three doses, to 24 milligrams every 48 hours for two doses. Some doses were given with doxycycline, some with other therapeutic regimens, and some doses of ivermectin were given alone. The outcomes range from viral clearance, to symptoms, to mortality. In this context, any meta-analysis of ivermectin is unproductive, given the high amount of heterogeneity.
Further, the preferred way to analyze studies is to group similar studies and then analyze them. The authors of ivmmeta.com do attempt to look at mortality outcomes separately, but they group observational studies, unblinded RCTs, and double-blind, placebo-controlled RCTs in the same analysis. Additionally, the dosing and control groups in the included studies were highly variable — again, a meta-analysis on such highly variable heterogenous data is not scientifically sound.
Also, the ivmmeta authors chose to analyze only one outcome per study. They chose the most clinically relevant outcome from each study, and did so consistently. However, they also ignored many (often nonsignificant) results. They also ignored studies that reported no events in either group (which would technically be a study showing no effect for ivermectin). It would have been far more preferable to analyze all outcomes from all studies separately, and to use risk difference (instead of relative risk) to statistically account for studies that produced nonsignificant results (versus ignoring the studies entirely). There are, of course, limitations in each of the individual studies included in the meta-analysis. This is true in nearly every meta-analysis. As one example, a study by Hashim et al. was not double-blinded, and could therefore introduce bias on the part of either the patients or the investigators. The best meta-analyses either exclude these studies or account for their limitations.
Another significant issue with using this data is that many of the studies are not publicly available to review, and of those that are, very few of them have been peer-reviewed. For example, the study by Chala et al. supposedly has data available on clinicaltrials.gov ... but the site shows that nothing is posted. Ashgar et al. only has a YouTube video available (again, with nothing posted on the clinicaltrials.gov interface). It is therefore impossible in many cases to verify the ivmmeta authors’ interpretation of the data without the original data for reference. Replicability is a fundamental part of the scientific method, and it’s missing.
Finally, every drug has a cost, and risks that must be balanced against benefit. Anaphylaxis is a rare risk, for example, of most drugs. Specifically for ivermectin, most of the risks in these studies seemed to be symptoms that were well-tolerated (e.g., dizziness, nausea, vomiting). There were few reports of serious adverse events, though they did occur (e.g., serious gastrointestinal bleeding from gastritis). Many of the referenced studies, though, did not comment on adverse events or stated that investigators collected data but did not report it in the preprints. It is impossible to evaluate potential benefits without weighing them against the risks.
I have personally reviewed all of the studies. I also performed an independent search of PubMed and Medrxiv* to make sure there were no studies missing. I could not find any published studies that were not included. Based on my funnel plot analysis, there is likely publication bias in the ivmmeta data, meaning there are unpublished studies showing no benefit. Based on my review, I think ivermectin is a promising therapeutic for COVID-19, but the current data on its use is not convincing enough to outweigh its risks. There is a signal of benefit, yes, and in the near future there may be a well-done study that shows benefit in some patients. (Actually, there are several ongoing trials studying exactly this.) However, the data supporting ivermectin’s use published on ivmmeta.com is not robust enough to inform a practice change or suggest the drug should be prescribed for COVID-19 patients. There is certainly not enough convincing evidence to argue that a well-done RCT is unethical. In fact, the data strongly suggests that an RCT should be done. I encourage physicians and patients to participate in a randomized trial, so that we can better understand ivermectin’s real potential.
* The search term used was “ivermectin,” filtered by year 2020–2021 in PubMed (not filtered in Medrxiv). There were 582 hits in PubMed, of which five were relevant. There were 86 hits in Medrxiv, of which one additional article (not returned in the PubMed search) was relevant.
Dr. Joshua Davis is an emergency physician who completed his residency at Penn State Milton S. Hershey Medical Center. He completed medical school at Thomas Jefferson University and received his undergraduate degree from the University of Delaware. His research interests are broad and include emergency medicine topics along with patient safety, handoff communication, and diet and exercise. Dr. Davis is a 2020–2021 Doximity Research Review Fellow.
Dr. Davis has no conflicts of interest to disclose.
Click here to see more perspectives on COVID-19 from the Doximity network.
Click here for up-to-date news about COVID-19 on Doximity.
Image: Novikov Aleksey / shutterstock
