Sources and provenance

#The two-axis rubric used throughout this course

Most "how to spot fake news" advice collapses two separate questions into one gut feeling. This course keeps them apart, because they fail independently.

Evidence tier asks how good the underlying study or data actually is, regardless of who is repeating it.

1. A systematic review or meta-analysis of independent studies, or a large, pre-registered replication attempt.
2. A single peer-reviewed original study with a stated, checkable method.
3. A preprint, working paper, or institutional report whose method is visible but hasn't passed independent peer review.
4. A field's own stated standard, checklist, or declared consensus position — not a data-generating study, but a checkable methodology document that practitioners agreed to.
5. An anecdote, a single unreplicated case, or a number with no traceable original at all.

Source tier asks who is making the claim and what they have riding on the conclusion.

1. The primary document itself — the paper, the dataset, the registry entry — in its own words.
2. A working researcher in the same field, publicly engaging with the evidence rather than just citing it decoratively.
3. A specialist journalist or science writer, at a publication with an editorial and fact-checking process, with no stake in the conclusion.
4. An institution, company, or advocacy body describing its own work, product, or position.
5. Unattributed or unattributable content — no named, checkable author, and no visible editorial process at all.

A claim needs both numbers stated, not merged. A press release (source tier 4) about a peer-reviewed meta-analysis (evidence tier 1) is not the same credibility grade as the meta-analysis itself, and a working scientist's blog post (source tier 2) about their own unreplicated pilot finding (evidence tier 5) is not the same as a systematic review. Every source below carries both tags.

#Why "peer-reviewed" is a floor, not a finish line

John P. A. Ioannidis, "Why Most Published Research Findings Are False," PLoS Medicine 2(8): e124, 2005. Evidence tier 2 · Source tier 1. The single most-cited argument for why passing peer review doesn't make a finding true: Ioannidis shows, from first principles about sample size, effect size, and how many hypotheses get tested before one gets published, why a large share of published findings in fields with small studies, small effects, and flexible analysis choices should be expected to not replicate. It's a probabilistic argument, not an audit of any specific field's literature — treat it as the reason to check evidence tier at all, not as a number to quote about any particular claim.

Open Science Collaboration, "Estimating the Reproducibility of Psychological Science," Science 349(6251): aac4716, 2015. Evidence tier 1 · Source tier 1. The actual audit Ioannidis's argument predicted: a coordinated, pre-registered attempt to replicate 100 studies from three major psychology journals, using the original materials and adequately powered designs. 97% of the original studies had reported a statistically significant result; only 36% of the replications did, and the replicated effect sizes were, on average, about half the size originally reported. This is evidence tier 1 — a large, coordinated, pre-registered project — not a single lab's opinion.

John Bohannon, "Who's Afraid of Peer Review?" Science 342(6154): 60–65, 2013. Evidence tier 2 · Source tier 1. A deliberately flawed fake cancer-biology paper, submitted under a fictitious author and institution to 304 fee-charging open-access journals between January and August 2013. 157 accepted it — most without visible evidence that anyone had read past the abstract. It's one investigation, not a random sample of academic publishing, and its scope was scoped to open-access "pay to publish" journals specifically, not journals generally. What it establishes narrowly: a journal claiming "peer-reviewed" is a claim, not proof, and it is a claim worth checking.

Retraction Watch Database, retractionwatch.com — founded as a blog in 2010 by Ivan Oransky and Adam Marcus; the database launched in 2018 and was transferred to Crossref, the scholarly-infrastructure nonprofit, in 2023 as an open public resource. Evidence tier 1 · Source tier 1. A primary, searchable registry of tens of thousands of retracted papers with stated retraction reasons — the actual tool for checking whether a specific paper you're about to cite has since been pulled, rather than an opinion about publishing generally.

Jeffrey Beall's list of predatory publishers ("Beall's List"), maintained on the blog Scholarly Open Access from 2010 until Beall removed it in January 2017 under sustained legal and institutional pressure from listed publishers; at closure it named 1,163 publishers and 1,310 standalone journals. Evidence tier 4 · Source tier 2. Named here deliberately at a lower tier than the temptation to treat it as an authoritative registry: it was one credentialed academic librarian's own judgment calls, not a peer-reviewed dataset, and Beall himself was a contested, controversial source — some publishers he listed disputed the classification, and the pressure that ended the list is itself part of the record. Good for: a starting list of names to be suspicious of. Cannot support: a definitive verdict on any specific publisher not independently checked.

#Evaluating a source without becoming the mark

Sarah Blakeslee, "The CRAAP Test," LOEX Quarterly 31(3), 2004 — devised for a first-year workshop at the Meriam Library, California State University, Chico. Evidence tier 4 · Source tier 2. The checklist this course borrows its habit of "ask before you trust" from: Currency, Relevance, Authority, Accuracy, Purpose. It's a practitioner's teaching heuristic, not a tested intervention, and naming that plainly matters, because of what comes next.

Sam Wineburg and Sarah McGrew, "Lateral Reading: Reading Less and Learning More When Evaluating Digital Information," Stanford History Education Group working paper, 2017; and Wineburg, McGrew, Joel Breakstone, and Teresa Ortega, "Evaluating Information: The Cornerstone of Civic Online Reasoning," Stanford Digital Repository, 2016 (nearly 8,000 students, 7,804 assessed responses across 12 US states, January 2015–June 2016). Evidence tier 2–3 · Source tier 1. Both are empirical studies from an academic research group rather than journal-peer-reviewed articles, which is the honest reason they sit at tier 2–3 and not tier 1. What Wineburg and McGrew actually found matters directly for this course: when professional fact-checkers, historians, and undergraduates were given the same unfamiliar websites to evaluate, the fact-checkers were fastest and most accurate — not because they read the site more carefully, but because they left it immediately, opened new tabs, and checked what independent sources said about the publisher, the funding, and the author. Historians and students, taught closer reading of the page itself, did worse. This is a direct, sourced qualification on the CRAAP test named just above: a checklist applied to the page in front of you is exactly the kind of "deep, careful reading of a single source" this research found professional evaluators avoid in favor of checking laterally. Use CRAAP's five questions; don't stop at the page to answer them.

#The standards a field uses to police itself

Gordon Guyatt et al. (the GRADE Working Group), "GRADE: An Emerging Consensus on Rating Quality of Evidence and Strength of Recommendations," BMJ 336: 924–926, 2008. Evidence tier 4 · Source tier 1. The framework behind "systematic review beats single study beats anecdote" as a formal, field-adopted rating system — now used by Cochrane, the World Health Organization, and most major clinical-guideline bodies to grade evidence explicitly rather than by feel. It's a consensus methodology document, not a data-generating study, which is exactly why it sits at evidence tier 4: the value is the checkable, adopted standard, not a finding.

Matthew J. Page et al., "The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews," BMJ 372: n71, 2021. Evidence tier 4 · Source tier 1. The reporting checklist that makes a systematic review checkable at all — what search terms were used, how studies were included or excluded, whether the search was pre-registered. A systematic review that doesn't disclose these against a standard like PRISMA is harder to distinguish from a reading list dressed up as a review.

San Francisco Declaration on Research Assessment (DORA), drafted at the American Society for Cell Biology's December 2012 annual meeting and released publicly in May 2013; tens of thousands of individual and institutional signatories since. Evidence tier 4 · Source tier 1. A field-wide declaration that a journal's impact factor — a library metric for comparing journals in aggregate — should not be used as a stand-in for the quality of one specific article or researcher. Directly relevant to a common shortcut this course warns against: "it's in a high-impact journal" is a source-tier-4 fact about the venue, not an evidence-tier fact about the specific paper.

#Red flags this course names explicitly

• A "studies show" claim with no named study. Per Ioannidis and the Open Science Collaboration above, even a named, peer-reviewed single study carries real uncertainty — an unnamed one is unverifiable by construction, not just unverified.
• "Peer-reviewed" offered as the entire credibility case. Per Bohannon's sting and Beall's list, a journal's own claim to peer review is a claim, checkable against retraction and predatory-publisher records, not a stopping point.
• Careful, page-by-page evaluation of a single source, with no independent check. Per Wineburg and McGrew, this is the specific failure mode that beat trained historians and undergraduates against professional fact-checkers — thoroughness inside one tab is not the same skill as checking outside it.
• A journal's impact factor cited as if it graded the article. Per DORA, this substitutes a venue-level, source-tier fact for an article-level, evidence-tier one.

#Gaps

This list traces the primary literature behind this course's own teaching claims about evidence hierarchies, peer review's real limits, and source evaluation — the material this Reference lesson was built from directly, checked against primary sources rather than secondhand summaries of them. It does not yet carry out a line-by-line citation trace against every factual claim made in this course's other lessons page by page, because that trace has to run after those pages are in their finished form, not before. Until that pass runs, treat any specific number or claim elsewhere in this course that isn't visibly sourced to one of the primary documents above as UNVERIFIED rather than as an implicit extension of the sourcing done here — this list backs the framework and the sources named above; it does not silently vouch for every sentence around them.

#Single best entry point

Ioannidis (2005). Read it before anything else in this course: it's the argument for why source tiers and evidence tiers need to be tracked separately in the first place, and every other source in this list either measures the problem it predicts (Open Science Collaboration, Bohannon) or is part of the field's response to it (Retraction Watch, GRADE, PRISMA, DORA).