Want to learn about scientific topics without needing a PhD? Check out the Science Simplified blog from TESS Research Foundation! Dr. Tanya Brown, PhD, works with researchers to make science accessible and empower rare disease community members with scientific knowledge. Dr. Brown has over a decade of experience in neurodevelopmental research and is currently the Scientific Director for TESS Research Foundation. Please reach out to her at [email protected] if you have questions or comments.
Thank you to Veronica Hood, PhD, the Scientific Director of the Dravet Syndrome Foundation, and Tanya Brown, PhD, for writing this article.
Click here to read Part Two
In our last two blog posts in this series we covered the structure of a research article (Part 1) and tips for critically reading a research article (Part 2). In part 3 of this series, we are focusing on a few last tips for evaluating research articles.
Are you reading a Research Article or a News Article?
One of the first things you should think critically about when you read about a new research study or scientific discovery is whether you are reading the primary research article or a news article.
News Articles:
- Provide a summary of the work highlighting just the most “exciting” points
- Provide an idea of the major impact the paper could have for human health
- May phrase findings to sound exciting or may overstate the interpretations from the study
- Are easier for broad audiences to understand
- A good news article will make sure to identify the original research article so that you can review the article yourself
Primary Research Articles:
- Contain details of the work, including:
- Rationale or background for performing the study
- Specific information about the methods used in experiments
- Results that include graphs and statistical results
- In-depth discussion of the study conclusions
- Are published in a scientific journal
- May be more challenging for non-scientists to interpret
If you find yourself reading a news article about a study that interests you, make sure you look at the original research article the news story is based on. This will allow you to see what the original study entailed, what it might really mean for advancing research, and how it might lead to better clinical outcomes or therapies. One important question to ask yourself is: Do you agree with the conclusions the news article made after reading the primary research?
Is the Research Article from a reputable scientific journal?
There are so many different scientific journals available, how do you know which ones are reputable? There are lots of different indicators that reveal whether a journal is a reliable source. One indicator of a reputable journal is if it is a peer-reviewed journal. The peer-review process is the process where scientists (the authors of an article), submit a scientific article to a journal, and then the journal editors send the article to 2-4 experts in the field to assess the science. The article is then reviewed by experts, who critically evaluate the article, make recommendations on how to improve the article, and state whether they think the article should be published or needs more work.
What is an Impact Factor? If you are familiar with the world of scientific journals, you may have heard the term impact factor. A high impact factor can reflect how much the journal “impacts” other research. The impact factor for a journal is a number that indicates how often the articles in a particular journal are cited by other research articles, and a higher impact factor indicates that the articles in a journal are cited more by other scientists. However, a highly specialized journal may have a low impact factor because the research topic is in a small field, but it still may be the most well-respected journal by experts in that area. Because of this we should avoid judging a journal solely by its impact factor.
There are many sources where you can find reputable journal articles. For biomedical research, this includes search engines such as PubMed or Web of Science.
There are lots of indicators that are found in reputable scientific journals, some of which we will point out here. A lot of this information can be found on a journal’s website. We will walk through an example of Aleisha Griffin’s paper used in our previous blog posts as an example. This article was published in the journal Communications Biology.
Some of the indicators that show that this is a reputable journal are:
- The journal has a clearly defined scope that is made public. Readers can find information on the journal website under “Journal Information.”
- This is a peer-reviewed journal. The peer-review process is clearly explained on their website.
- The journal Editors and Editorial Board are made up of experts in the field.
- Journal contact information is freely available.
- Published articles align with the journal scope.
- Publication fees are publicly available
Reminder: Scientists have to pay for each scientific article they publish. To keep their research programs running, scientists are under pressure to publish their work. Publishing scientific articles can help scientists gain recognition in the field, apply for more funding for their work, and share their findings. Scientists do not make money from the cost of readers buying the article.
For more information about learning about qualities of reputable journals, you can check out this information from the National Oceanic and Atmospheric Administration Central Library.
Be on the lookout for predatory journals. These are journals that publish false or misleading information, may not be rigorously peer-reviewed, lack transparency in their process, and may use aggressive or intimidation policies to publish articles. Agnes Grudniewicz et al. 2019 wrote a commentary explaining what predatory journals are, why they are dangerous, and how to identify predatory journals. Sometimes these articles look convincing! To identify a predatory journal, some of the things to look for are:
- A lack of transparency in the publishing process
- Deceptive and/or aggressive practices
- Misinformation: wrong contact information, fake journal information, undisclosed editorial boards
When was the article published?
In some regards, science moves incredibly slowly and other times quickly. While scientists still use information from many years ago, new scientific advances can change interpretations of older data. When the article was published is an important consideration. For some fields this might not matter, but for genetics and genomics, for example, the field is advancing so quickly that a paper published in 2005 may not be as relevant to our understanding in 2022.
If you aren’t sure about a paper, send an email to an expert and ask what they think of the article you found!
Is bioRxiv a trusted source for articles? bioRxiv (“bio-archive”) is a preprint server for the biological sciences. This widely-used resource helps to improve the accessibility of scientific articles by hosting an archive of unpublished papers that are submitted by researchers prior to publishing them in a journal. While it is a great resource, it’s important to remember that these articles are not yet formally peer-reviewed and so more caution is warranted when drawing conclusions from the article.
Who performed and funded the study?
It is also important to pay attention to what institution the work was performed at. All scientists have bias, so knowing where the research was completed and who funded the study can help identify potential bias. You can find information about this from the author affiliations. Was the work performed at a reputable university or institute? Was the work completed by a pharmaceutical or other for-profit company? Each type of institution has its own strengths, weaknesses, and biases that may influence how the data presented in an article is interpreted or presented.
Additionally, the funding sources that helped to pay for the research should also be listed in the article, usually at the very end before the list of references. Sometimes pharmaceutical or biotechnology companies will fund an academic researcher with the most relevant experience and expertise to perform a study with their product, and other times the companies may do the entire research study internally. You will be able to identify this by looking at the author affiliations and the source of funding for the study (usually listed at the very end of the article before the list of references).
To understand how bias may influence research, let’s take an example that has been in the news lately: concussions in football players. If the NFL funds a study investigating concussions, what might be some biases from the NFL? Different results may lead to big changes in NFL revenue, how fans view the NFL, and requirements for player safety. To remove some of these biases, research funded by the NFL may be performed in a blinded fashion or completed by scientists who are unaffiliated with the NFL.
What are blinded studies? Blinded studies, also known as masked studies, are investigations where the researchers are unaware of which group is a control group and which is receiving a treatment. The groups are only revealed after the analysis is complete. This helps remove bias from analysis. This is particularly important for subjective analysis such as pain or anxiety levels to determine the effect of a specific treatment in a clinical trial.
It is not inherently “bad” for the NFL to fund such a study (or for a pharmaceutical company to fund research into a potential therapeutic medication they want to develop). However, as a reader, you may want to pay closer attention to the way they collected the data and the conclusions they make from the interpretation of the data. What would be different if the research was completed by an NFL player? What about a coach or a fan? While this example is outside of epilepsy research, it provides a helpful way to understand how results may be different when interpreted from different perspectives.
Even articles from trusted sources can end up having mistakes.
Once you feel confident that the article is from a reputable source, and you have a handle on who the researchers were that did the study, it’s now time to dig in and read the paper (See Blog #2 in this series for tips on breaking down the information).
But…what happens when an article has a mistake? Occasionally, even articles in reputable journals from respected researchers may need to have an official correction or even be retracted. It could be due to a mistake that was made, the inability of the researchers to go back and replicate results, or the rare instance of someone falsifying data. Mistakes can happen to anyone, and including corrections is incredibly important to the scientific community as science builds upon itself. Any time this sort of correction or retraction occurs, the journal will include annotations on the original article. A correction may be referred to as a corrigendum (when the author made an error) or an erratum (when the publishing journal made an editorial error).
In addition to paying attention to indications of retractions or corrections, you can also do a Google search of the study to see if it is being discussed by other experts in the field. Are other experts skeptical of the data? Skepticism or adverse comments do not always mean something is wrong! Sometimes a major shift in understanding of a particular research topic can cause skepticism before it has been shown in multiple different ways and becomes widely accepted. Scientists want to be cautious in over-interpreting findings, and when a new discovery is made they want to feel confident in the results. Adversity or skepticism from other experts may be a flag to just take some additional caution in your interpretations of the study until more evidence is presented.
After reading this blog, you should have a better understanding of how to critically evaluate the source of the research you are interested in reading. If you have not already done so, be sure to check out the other blogs in this series that cover the various parts and sections of a scientific article (Part 1) and give tips on how best to approach reading and breaking down those sections (Part 2).
Still have questions about a scientific article? Send those over to [email protected] or [email protected]!
We want to hear from you! If you want to add to our list of topics for Science Simplified, please email Tanya Brown, PhD: [email protected].
