Source tiers

Two different questions get asked as if they were one. "How good is this evidence?" is about the claim itself — a controlled trial, a single observation, or just a plausible mechanism nobody has measured yet. "Who is telling me this, and what do they want from me?" is a separate question, about the messenger rather than the message. A well-run study reported by a source with a direct financial reason to shade the conclusion is still a well-run study — you now have two things to check instead of one, not a reason to discard it. Conflating those two questions is one of the most common failures in how people evaluate what they read: strong evidence from a source you should distrust doesn't become weak, and a confident claim from a source you'd normally trust doesn't become strong just because you trust the messenger. This module is only about the second question.

It isn't a classroom abstraction. It's the actual tagging system running this site's own research pages — every claim under /protocol on this domain is filed against the same six-value scale below, precisely so a reader can see whether a claim traces to a regulator's own ruling or to a company's marketing copy. What follows is that scale, worked through for the sources a student doing real research — a citation list, an EPQ, a personal statement claim — will actually run into.

#The six tiers, at a glance

#Primary — it already survived someone trying to break it

Peer review isn't a stamp of truth; it's an adversarial process. Before a serious journal publishes a paper, reviewers whose job is to find what's wrong with it — the underpowered sample, the uncontrolled confound, the statistic that doesn't support the headline — get to reject it or send it back. That's the mechanism behind primary tier: the claim survived people actively trying to kill it before you ever read it.

The same tier covers a regulator's own primary output — a statute, a ruling, an agency filing — because it's the actual record, not an account of it. POM Wonderful, LLC v. FTC is a clean example: the FTC found POM's pomegranate-juice marketing implied it treated heart disease, prostate cancer, and erectile dysfunction, and the D.C. Circuit upheld that any future disease-prevention claim needs at least one randomized, controlled human trial behind it — while separately ruling the FTC couldn't demand two as a blanket rule. Reading the opinion gives you the actual standard; reading a summary gives you whoever wrote the summary's idea of the interesting part.

Primary doesn't mean infallible. The Open Science Collaboration's 2015 replication of 100 psychology studies (Science) found only 36% reproduced a significant effect on retest, though 97% of the originals had reported one. Peer review means a claim survived the best pre-publication filter that exists — not that the filter is perfect.

#Researcher — real authority, with a hard edge where it stops

A working scientist explaining their own field — a professor summarizing their own published work, a specialist speaking on their actual specialty — carries real weight, because their reputation is staked on that subject in a way it isn't staked on anything else.

The edge matters: that confidence doesn't travel once they leave the field. A credentialed scientist's opinion outside their own specialty is borrowed authority — it sounds like expertise because the delivery and the credentials are unchanged, but the claim-to-verification relationship that makes this tier trustworthy is gone. The tell isn't how confident the delivery is; it's whether the subject of the sentence is the subject of their actual research.

#Operator — worth learning to check, because it's the easiest tier to fake

An operator has a real, checkable track record — not a credential, a result. The tier sits in an uncomfortable middle: like a company, an operator is usually selling something built on that record. Unlike a company, the record is meant to be independently checkable data, not a self-authored claim with nothing behind it. That combination — a real reason to persuade you, sitting on top of a record that's supposed to be verifiable — is exactly why this tier gets faked more than any other, and why it rewards checking rather than trusting the confidence of the pitch.

A record that clears the bar looks like this: a named person publishing specific, dated figures continuously, across good years and bad ones, checkable against something other than their own word. One real example: solo software founder Pieter Levels (levels.io) has published dated revenue figures for his projects continuously since the mid-2010s — modest five figures a year early on, a widely-reported $420,000-a-month spike for one product in 2024, fluctuating figures since — with no course or funnel behind the disclosure itself, and independent outlets like Indie Hackers have covered the numbers as they happened rather than only after the fact. That's what makes it operator tier rather than company tier: the figures are old enough to predate any reason to lie about them, specific enough to be wrong if invented, and corroborated by someone other than Levels himself.

Four checks turn a claimed track record into something you can actually evaluate:

• Is the number checkable against something the operator doesn't control? A payment-processor dashboard, a broker statement, a public leaderboard — not a screenshot they took themselves.
• Was it public before it needed to be believed? Search the Wayback Machine for the earliest archived version of the claim. A number that's existed unchanged for years is a different object from one that appeared the week enrollment opened.
• Does the record include a bad stretch, not just the best year? One good year proves far less than several years that include a flat or losing one.
• What's the denominator? A visible success is often one of many who tried the same thing; the ones who failed quietly don't publish anything, so the sample you see is never the whole sample.

This exact domain gives a live case. A Harvard Crimson Magazine investigation into college consulting (November 2025) found that large firms including Crimson Education and Empowerly advertise clients as three to eleven times more likely to gain admission than the average applicant. The piece does what operator verification is supposed to do: it presses on what the multiplier is actually measured against. A Harvard student interviewed for it, James Obasiolu, raised the obvious question — when a firm reports a high acceptance rate among its own paying clients, how much reflects the firm's work rather than the socio-economic advantages that clientele already had. A researcher quoted in the same piece, Tiffany Huang, made the companion point: without a comparison group of similar students who didn't use a consultant, it's genuinely hard to say what the intervention changed. Neither number is a lie exactly — but neither discloses its own denominator, and a number with no stated denominator is exactly what this tier exists to make you stop and check.

#Company — a direct reason to report favorably on itself

Company tier means the party funding the research, running it, and standing to profit from the result are the same party. That doesn't make the finding false. It does mean the study never faced the one thing that makes primary tier trustworthy — someone with authority to reject it and no stake in the outcome — and it means an unfavorable result can simply go unpublished, with no one outside the company ever finding out.

That's a measured pattern, not a hunch. A 2017 Cochrane systematic review by Lundh, Lexchin, Mintzes, Schroll, and Bero, pooling decades of comparisons, found consistent evidence that industry-sponsored drug studies report outcomes favorable to the sponsor's product more often than independently-funded studies of the same drug. That's the mechanism behind "a supplement company's own study" as a category: not that the science is necessarily wrong, but that it was designed, run, and selected for publication by the party with the most reason to want a particular answer.

POM Wonderful is the concrete case again: the company had funded its own research, and the FTC's finding — upheld on appeal — was that some of it fell short of the controlled, blinded standard the disease-prevention claims needed. Set that against an independent counter-example in the same broad category: the VITAL trial, NIH-funded, randomized, double-blind, placebo-controlled, in more than 25,000 US adults given daily omega-3 fish oil or placebo for a median of 5.3 years, published in the New England Journal of Medicine in 2019 (Manson et al.). It found no significant reduction in major cardiovascular events or cancer from omega-3 supplementation in the general population — a far more modest result than most fish-oil marketing implies. Same broad question, one study paid for by a party selling the answer and one that wasn't — and they didn't agree.

#Press — usually one step removed from the source it's reporting on

A news article summarizing a study is a claim about a claim: the journalist read the paper, or more often read the university's press release about it, and wrote something shorter and more attention-grabbing than either. The incentive isn't dishonesty — it's speed and engagement, working on a story that was often already simplified once before the journalist opened it.

That mechanism has been measured. Sumner et al. (BMJ, 2014) compared press releases against the underlying papers and the resulting news coverage, and found exaggeration overwhelmingly enters the chain at the press-release stage: when a release exaggerated a causal claim, news coverage repeated it 81% of the time; when it didn't, coverage did so only 18% of the time. The pattern held for exaggerated practical advice (58% versus 17%) and for over-generalizing animal results to humans (86% versus 10%). A journalist's summary and the paper itself are not small variations on the same information — by the time a claim reaches a news article, it has usually already passed through one earlier layer of simplification the article didn't add.

Worth naming honestly: journalism that does its own primary-document work — named sources, original interviews, the underlying filing rather than a summary of one — is doing something closer to primary-tier work, even published in a magazine. The Crimson piece above is exactly that. Press tier describes a typical position, not a permanent ceiling on what one piece of journalism can be.

#Surface — optimized to be found, not to be right

Surface content exists to rank in search results and convert a reader into a customer; accuracy isn't the thing being optimized for. The tells are mechanical once you look for them: no named, credentialed author; "studies show" with no link, date, or specific citation; a claim that reads nearly identically across a dozen domains, because it was written to match what already ranks rather than to describe what a source found.

The most dangerous version wears a primary-source costume. For years, librarian Jeffrey Beall maintained a public list of "predatory" journals — outlets that publish almost anything for a fee, with editorial boards that don't meaningfully review submissions. He took the list down in January 2017, citing pressure from his employer, and successors have continued the idea since. A paper published in a journal like that looks exactly like primary tier — formatting, citations, a DOI — while functioning exactly like surface tier: paid placement with no adversarial review behind it. Format is not evidence of tier; checking whether a journal is real is part of using this framework correctly, not an optional extra step.

#Tier is a starting prior, not a verdict

None of this replaces reading the actual claim. Tier tells you how skeptical to start, not the final answer — and it can move mid-claim. A researcher unimpeachable on their own published subject becomes company tier the moment they're describing the product they've started selling in that same field; the credentials didn't change, but the sentence did. The habit worth building, for any citation someone will actually check: name who is telling you this and what they'd gain if you believed it, before deciding how much of it to believe.