Recently, a very important pilot study was published by Stanford University. And it’s important you understand how the publication of this pilot trial is building momentum in the metabolic psychiatry movement.
They reported some amazing clinical outcomes that came from a total of just 10 medical visits over the course of 4 months. And this study was one of the ten clinical trials going on that I tweeted about in late March, 2024 on Twitter (X).
That means that as I am writing this article, 9 more are still happening!
Some of the clinical trials going on worldwide are actual Randomized Controlled Trials (RCTs), considered a methodological gold standard in medical research. By randomly assigning participants to either a treatment group or a control group, RCTs try to ensure that the only difference between these groups is the treatment itself, allowing for direct comparisons and hopefully clearer conclusions about a treatment’s efficacy. It’s considered a rigorous methodology designed to help reduce bias and is thought to enhance the trustworthiness of the results.
The results are used almost exclusively in shaping clinical guidelines and healthcare policies. Regulatory bodies like the FDA rely on the data from RCTs to approve new treatments and use them to evaluate whether the treatment is safe and effective before it is made available to the public.
Bear with me because we all know an entire article can be written discussing whether or not that goal stated by the FDA is the primary motivation or even remotely achieved. My point is that RCTs are required to bring ketogenic diets into the standard of practice medical care for those with mental illness.
What happens after RCTs?!
With RCT results coming to the forefront, it will only be a matter of time before systematic reviews are done. And as the general public, you are going to want to know what to look for in a quality systematic review.
So that’s what this article is about.
Getting you ready to evaluate and understand what a good systematic review looks like, what tools are used, and why it coming from a journal with an exemplary peer review process is so important.
Despite the recognized importance and influence of evidence syntheses in medical research and clinical practice, many systematic reviews suffer from methodological flaws, biased reporting, and a lack of rigorous peer review, leading to unreliable conclusions.
Many systematic reviews have been critiqued for methodological gaps, lack of precision, and failure to adhere to updated standards, thereby undermining their reliability.
This is a problem. Because these inadequacies significantly impact healthcare, as clinicians and policymakers might rely on flawed evidence when making critical decisions.
Think this is just me complaining? It’s not. It’s a whole thing talked about in scientific publishing.
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.
Kolaski, Kat1; Logan, Lynne Romeiser2; Ioannidis, John P.A.3. Guidance to best tools and practices for systematic reviews. JBI Evidence Synthesis 21(9):p 1699-1731, September 2023. | DOI: 10.11124/JBIES-23-00139
So, let’s talk about some of the components that make a good systematic review. And I will tell you, it always must begin with a pre-registration before it begins!
Why Pre-Registration of Systematic Reviews Matters
Pre-registering systematic reviews helps maintain the integrity and transparency of research methodologies. By registering their review protocols on databases like PROSPERO, researchers commit to a predefined research plan before they begin their systematic review. This process is important as it helps prevent potential biases in the review, such as selective reporting of outcomes based on the data found. It also facilitates accountability and enables other researchers and stakeholders to understand the scope and methods of the review beforehand. This makes reproducibility (a basic tenet of good science) more possible and, in turn, increases the credibility of the research. For the public and professionals relying on systematic reviews to make informed decisions in healthcare and policy-making, pre-registration adds a layer of trust and reliability to the findings presented.
The application of methodological and reporting standards varies widely, with many reviews failing to employ recognized tools like PRISMA or GRADE effectively, if at all. What are those? You will want to know so keep reading!
PRISMA is a widely recognized guideline used for reporting systematic reviews and meta-analyses. It provides a checklist of items that should be included in a review to ensure clear and transparent reporting. This includes the identification of information sources, search strategies, criteria for inclusion and exclusion of studies, methods of risk of bias assessment, and statistical analysis techniques. It is designed to make it easier for readers to assess the reliability and validity of the review’s findings.
Standardized reporting tools like PRISMA 2020 involve detailing every aspect of the systematic review process in the published report, ensuring that other researchers can replicate the study, assess its validity, and understand exactly how conclusions were reached.
GRADE is a systematic approach used to evaluate the quality of evidence in systematic reviews and to develop recommendations in guidelines. It assesses the quality of evidence across studies that contribute to data about specific interventions or treatments. GRADE considers factors like study limitations, consistency of effect, imprecision, indirectness (relevance), and publication bias to rate the quality of evidence as high, moderate, low, or very low. This tool was created to help healthcare providers make recommendations by balancing the benefits and harms, patient values, and costs.
You have probably seen a PRISMA diagram in research papers. It looks something like this!
Well, there is also a PRISMA checklist that authors complete and should be used as part of the peer-review process. Here are some of the questions in the PRISMA 2020 checklist for systematic reviews.
Methods
“Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses.”
“Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process.”
Discussion
“Provide a general interpretation of the results in the context of other evidence.”
“Discuss any limitations of the evidence included in the review.”
Other Information
“Provide registration information for the review, including register name and registration number, or state that the review was not registered.”
“Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.”
You get the idea! And you can see why this is important. You can find the complete checklist here. However, a well-filled-out PRISMA checklist is not the only useful tool that authors and peer reviewers have at their disposal. But still, even with these great tools, peer reviewers and journal editors often do not enforce stringent standards, contributing to the publication of evidence syntheses that do not meet high methodological quality.
So what does an exemplary peer review look like? At least for systematic reviews? Glad you asked. I am going to tell you. Keep reading 📖
Multilevel Peer Review
It should have a multilevel peer review process so that different reviewers with varying expertise assess a systematic review at various stages.
This could include initial reviews by methodological experts, followed by subject matter specialists, and potentially a final review by editorial boards. This ensures the quality and validity of the review through multiple lenses of expertise. During a multilevel peer review, many of the methodological and critical appraisal guidelines could be checked along the way, ensuring a better systematic review reported and published in the end.
Methodological Expectations of Cochrane Intervention Reviews (MECIR)
Methodological expectations of Cochrane Intervention Reviews (MECIR) should be used when appropriate because it helps to ensure the reliability and comprehensibility of the review outcomes.
What does MECIRcover? Glad you asked.
It gives guidance on protocol development to ensure the review question is clearly defined and that the methods planned are appropriate to answer it. It guides authors on how to do a systematic search, select studies, extract data, manage it, and analyze it! It provides detailed expectations for how findings should be presented, including handling bias, interpreting results, and framing conclusions in a context that is useful for healthcare decision-makers.
Critical appraisal tools such as AMSTAR-2 and ROBIS provide structured frameworks for assessing the quality and risk of bias in systematic reviews. AMSTAR-2 is used to scrutinize the methods behind systematic reviews, checking everything from the way studies are selected to how the data is analyzed and reported. While ROBIS focuses specifically on identifying any risk of bias in the review processes themselves, helping to ensure that the findings are reliable and trustworthy. Reviewers can use these checklists to evaluate whether systematic reviews have addressed potential biases in their search and selection processes, data collection, analysis, and reporting.
Emphasis on Education
Training for peer reviewers and authors to increase their understanding of how to apply methodological and reporting standards effectively to a systematic review is really a big deal. This type of training exists, and there is no reason not to make sure it is taken! For example, check out this free one from Wolters Kluwer. And advanced training options are not that expensive. This is the type of training required to help ensure that systematic reviews are not only conducted according to high standards but also critically evaluated for their methodological soundness and the credibility of their conclusions.
You might be curious to know that the criteria for non-systematic reviews, like narrative reviews, differ from those of systematic reviews. In those cases, peer reviewers prioritize comprehensive coverage, thematic organization, and insightful synthesis over strict methodological and reproducibility standards. Systematic reviews really are special in that they determine future standard of care practices for patients.
Feeling out of the Scientific Loop?
Hang in there because the research community acknowledges the need for public involvement in scientific research and is driving initiatives such as the Future of Open Science Policy project. This initiative, a collaboration among the Federation of American Scientists, the Center for Open Science, and the Wilson Center, works to democratize science by ensuring that federally funded research is accessible and accountable to the American public.
They are trying to integrate the experiences and needs of local communities directly into the scientific process, which benefits those who are most impacted by scientific advancements.
Keep Reading, There’s a Point
Why does this article about conducting this type of research and the peer review process of systematic reviews matter to you? I will tell you. ⬇️
Systematic reviews inform clinical guidelines and healthcare policies. When these reviews are reliable, they help ensure that the medical advice and treatments provided to patients are based on the best available evidence.
Flaws in systematic reviews can lead to misinformation or misinterpretation of medical data, potentially harming public health. It’s important you understand the mechanisms that underpin trustworthy medical research.
High-quality systematic reviews can prevent wastage of medical and research resources by accurately defining what treatments are effective. This efficiency matters in a healthcare context where resources are often limited and need to be allocated wisely.
The public has a vested interest in ensuring that healthcare funding is used on interventions that are proven effective. Not as a tool to sell poorly performing pharma and medical devices (for example).
How medical knowledge is advanced and validated is knowledge that can empower individuals to make informed decisions about their health and to engage more critically with medical news and research findings. And you know how I feel about you having the right to learn all the ways you can feel better!
And finally, informed citizens (like you!) can advocate for policies that prioritize scientific rigor and transparency in medical research. Public pressure can lead to more stringent publishing standards and better oversight of medical research, ultimately leading to improvements in healthcare quality for people in your community.
If you or someone you love is suffering from mental illness and would like to see ketogenic diets offered as an early treatment option that can be covered by health insurance, watching the research and publication process unfold and finding ways to get involved and advocate for good science publication can be an empowering stance.
Share this article with someone else watching the metabolic psychiatry movement unfold!
References
AMSTAR – Assessing the Methodological Quality of Systematic Reviews. (n.d.). Retrieved May 3, 2024, from https://amstar.ca/Amstar-2.php
Bristol, U. of. (n.d.). ROBIS tool. University of Bristol. Retrieved May 3, 2024, from https://www.bristol.ac.uk/population-health-sciences/projects/robis/robis-tool/
Guidance to best tools and practices for systematic reviews: JBI Evidence Synthesis. (n.d.). Retrieved May 3, 2024, from https://journals.lww.com/jbisrir/fulltext/2023/09000/guidance_to_best_tools_and_practices_for.2.aspx
MECIR Manual | Cochrane Community. (n.d.). Retrieved May 3, 2024, from https://community.cochrane.org/mecir-manual
Metabolic Mind (Director). (2024, April 8). Trailblazing Stanford Trial Shows Keto Improves Serious Mental Illness—With Dr. Shebani Sethi. https://www.youtube.com/watch?v=66TLG3Y7XPc
Opening Up Scientific Enterprise to Public Participation. (n.d.). Federation of American Scientists. Retrieved May 3, 2024, from https://fas.org/publication/opening-scientific-enterprise-to-public-participation/
OSF | Home. (n.d.). Retrieved May 3, 2024, from https://osf.io
Peer Reviewer Training Course [Basic + Advanced] – Wolters Kluwer. (n.d.). Retrieved May 3, 2024, from https://wkauthorservices.editage.com/peer-reviewer-training-course/
PRISMA 2020. (n.d.). PRISMA Statement. Retrieved May 3, 2024, from https://www.prisma-statement.org/prisma-2020
Schünemann, H. J., Brennan, S., Akl, E. A., Hultcrantz, M., Alonso-Coello, P., Xia, J., Davoli, M., Rojas, M. X., Meerpohl, J. J., Flottorp, S., Guyatt, G., Mustafa, R. A., Langendam, M., & Dahm, P. (2023). The development methods of official GRADE articles and requirements for claiming the use of GRADE – A statement by the GRADE guidance group. Journal of Clinical Epidemiology, 159, 79–84. https://doi.org/10.1016/j.jclinepi.2023.05.010
Sethi, S., Wakeham, D., Ketter, T., Hooshmand, F., Bjornstad, J., Richards, B., Westman, E., Krauss, R. M., & Saslow, L. (2024). Ketogenic Diet Intervention on Metabolic and Psychiatric Health in Bipolar and Schizophrenia: A Pilot Trial. Psychiatry Research, 335, 115866. https://doi.org/10.1016/j.psychres.2024.115866