Deep Dive: Correlation Coefficient and Regression Calculation

#CODEX Deep-Dive — The Product Moment Correlation Coefficient: Calculation, Mechanism, and Real Exam Angles (WST01 spec 4.3)

Provenance, 2026-09-29 (revised same day on re-read): written after reading research/veridian/WST01-verified-facts.md in full (§1–§2b, §4 "Section 4", §7–§8, the per-lesson audit entry for wst01-correlation-coefficient-and-regression-calculation, and — on this revision — also the per-lesson audit entry for wst01-regression-gradient-interpretation-extrapolation.ts, which this document's first draft had grepped for PMCC content but not read in full), the existing lesson file content/veridian/curriculum/wst01/correlation-coefficient-and-regression-calculation.ts in full (including its own header note and closing flag block, which carry a documented correction to the facts bank — see §5 below), the four other existing technique-hub documents in this folder (codex-formula-and-method-bank.md §1.3/§2.4, forge-exam-technique-guide.md §3.2 and its trap table, signal-frequency-and-boundaries.md §1, and —on this revision — signal-topic-brief-correlation-coefficient-and-regression-calculation.md, which independently caught and corrected the same undercount §4 fixes below), and content/veridian/question-bank/wst01.ts (grepped for "correlation"/"PMCC"/"regression" — zero matches; this sub-topic has no question-bank entries at all). All new arithmetic below was computed independently with Python and is flagged as such at each point it appears; nothing below is asserted as a Pearson-published answer unless a specific document and question number is named for it.

#Why this document exists, and what it does not re-teach

The lesson file this topic is drawn from says plainly, in its own header, that spec item 4.3 — the product moment correlation coefficient — has "the thinnest evidence base of any WST01 lesson in this batch": exactly two primary-source anchors (one examiner report, one mark scheme), against five for its sibling topic (regression-gradient interpretation and extrapolation, spec 4.1/4.2). That lesson already teaches, in full, and this document does not repeat: the $S_{yy}$ recap, the $r$ formula itself, the unit-cancellation argument for why $r$ is bounded, a full algebraic derivation of coding invariance including the sign-flip edge case, a chain-drill on coding, four MCQs, and a seven-mark worked example (tutor revision hours vs. test score, $S_{xx}=30$, $b=3$, $S_{yy}=500$, $r=0.735$) with its own common-wrong-path. Reproducing any of that here would be restatement, not depth.

What follows instead is three things the reviewed archive supports but no existing WST01 document currently states: (1) a proof, rather than an assertion, of why $-1 \le r \le 1$ always holds; (2) an explicit derivation connecting the two printed forms of $S_{xx}$/$S_{yy}$/$S_{xy}$ (deviation and computational), which every WST01 document that uses them states without deriving; (3) a quantitative reconstruction — using real, already-published summary statistics from a genuine WST01 question — of exactly how large the documented $S_{pp}$ substitution error actually is, and what wrong final answer it would produce, neither of which the facts bank or any lesson currently computes. It closes with a synthesis of every real exam angle the reviewed archive has actually tested spec 4.3 from, with the honest denominator attached to each one.

#1. What the spec and formula booklet actually require

Spec item 4.3, verbatim (spec pp. 53–56, transcribed in the facts bank §1): "The product moment correlation coefficient, its use, interpretation and limitations. (Guidance: derivations and tests of significance will not be required.)" Three things are named as content in their own right — use, interpretation, limitations — not just the calculation, and "derivations... will not be required" is an explicit statement that a student never has to reproduce §3 of this document in an exam. It is included here as background understanding, in the same spirit as the lesson's own beyond-spec block on coding invariance.

The formula booklet (facts bank §2a, Issue 2, January 2021, pp. 13–17) prints, in full, both forms of $S_{xx}$, $S_{yy}$, $S_{xy}$, the PMCC formula $r = S_{xy}/\sqrt{S_{xx}S_{yy}}$, and the regression coefficient $b = S_{xy}/S_{xx}$. Nothing in this document's §2–§3 needs memorising for that reason — every formula used below is either printed in the booklet or derived from what is.

#2. The calculation model, in full

#2.1 Why the "deviation" and "computational" forms of $S_{xx}$ are the same formula

Every WST01 document that uses $S_{xx} = \sum(x-\bar x)^2 = \sum x^2 - \dfrac{(\sum x)^2}{n}$ states both forms side by side (booklet, both lessons, the CODEX formula bank) but none derives why they're equal — worth doing once, because the substitution trap covered in §4 below is exactly a failure to respect this equivalence. Expand the deviation form:

$$\sum(x_i-\bar x)^2 = \sum(x_i^2 - 2\bar x x_i + \bar x^2) = \sum x_i^2 - 2\bar x\sum x_i + n\bar x^2$$

Since $\bar x = \dfrac{\sum x_i}{n}$, the middle term is $2\bar x \cdot (n\bar x) = 2n\bar x^2$, so the last two terms combine to $-2n\bar x^2 + n\bar x^2 = -n\bar x^2$. That gives $S_{xx} = \sum x^2 - n\bar x^2$, and since $n\bar x^2 = n\left(\dfrac{\sum x}{n}\right)^2 = \dfrac{(\sum x)^2}{n}$, this is exactly $S_{xx} = \sum x^2 - \dfrac{(\sum x)^2}{n}$ — identical to the printed computational form. $S_{yy}$ follows by relabelling $x \to y$ throughout. $S_{xy}$ follows the same expansion with two variables: $\sum(x_i-\bar x)(y_i-\bar y) = \sum x_iy_i - \bar y\sum x_i - \bar x\sum y_i + n\bar x\bar y$, and substituting $\sum x_i = n\bar x$, $\sum y_i = n\bar y$ collapses the last three terms to $-n\bar x\bar y$, giving $S_{xy} = \sum xy - n\bar x\bar y = \sum xy - \dfrac{(\sum x)(\sum y)}{n}$ — again exactly the printed form. This is the reason the mean-squared substitution error (§4) is a different number, not a rounding variant of the correct one: $n\bar x^2$ (correct, appears once, already divided by $n$ inside $\bar x$ and then multiplied back by $n$) and $\bar x^2$ alone (wrong, missing that factor of $n$) are related by a factor of $n$ itself — for $n=12$, as in the real question below, a twelve-fold-scale error hiding inside a subtraction.

#2.2 The PMCC formula and what it measures

$$r = \frac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$$

$b = S_{xy}/S_{xx}$ answers "how much does $y$ change per unit of $x$"; $r$ answers "how tightly do the points cluster around a straight line, in both directions at once" — a comparison of the association ($S_{xy}$) against the product of both variables' own spreads, rather than $b$'s comparison against $x$'s spread alone. Both lessons already establish, correctly, that this is why $r$ is unit-free and $b$ is not; §2.3 below proves the boundedness that follows from it, rather than asserting it.

#2.3 Why $-1 \le r \le 1$ always holds — a real proof, not an assertion

Spec 4.3 explicitly waives derivations, and the existing lesson's mechanism block argues boundedness by unit-cancellation — a real and sufficient argument for why $r$ has no units, but not, on its own, a proof of the specific bound $\pm 1$. Here is that proof, standard for any dataset and worth having once in this course since nowhere else states it:

For any real number $\lambda$, and writing $x_i' = x_i - \bar x$, $y_i' = y_i - \bar y$:

$$\sum_{i=1}^n (x_i' - \lambda y_i')^2 \ge 0$$

— true for every $\lambda$, since it is a sum of squares. Expanding:

$$\sum x_i'^2 - 2\lambda\sum x_i'y_i' + \lambda^2 \sum y_i'^2 \ge 0 \quad\Longrightarrow\quad S_{xx} - 2\lambda S_{xy} + \lambda^2 S_{yy} \ge 0$$

This is a quadratic in $\lambda$ (coefficients $S_{yy}$, $-2S_{xy}$, $S_{xx}$) that is never negative for any real $\lambda$ — which is only possible if its discriminant is $\le 0$:

$$(-2S_{xy})^2 - 4S_{yy}S_{xx} \le 0 \quad\Longrightarrow\quad S_{xy}^2 \le S_{xx}S_{yy} \quad\Longrightarrow\quad |S_{xy}| \le \sqrt{S_{xx}S_{yy}}$$

Dividing both sides by $\sqrt{S_{xx}S_{yy}}$ (positive for any real dataset with more than one distinct $x$- and $y$-value) gives $|r| = \left|\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}\right| \le 1$ directly — $-1 \le r \le 1$ is not a convention or a coincidence of the formula's construction, it is a forced consequence of every real number squared being non-negative. Equality ($r = \pm 1$) holds exactly when the discriminant is zero, i.e. exactly one $\lambda$ makes the sum-of-squares zero, which happens only when every single point satisfies $x_i' = \lambda y_i'$ — the data lies exactly on a straight line through $(\bar x,\bar y)$. This is genuinely additional depth beyond what spec 4.3 requires (its own guidance waives derivations), included because it converts "$r$ is always between $-1$ and $1$" from a memorised fact into a checkable one.

#3. Micro worked examples, one per real distinct sub-case

Each dataset below is small, original to this document, and independently verified by Python before being written in; none reuses a dataset from either existing lesson.

3.1 — Computing $r$ from raw data. $x$: 1, 2, 3, 4, 5; $y$: 3, 5, 4, 8, 10. $n=5$, $\bar x = 3$, $\bar y = 6$. Computing the three sums directly: $S_{xx} = 10$, $S_{yy} = 34$, $S_{xy} = 17$. Then $r = \dfrac{17}{\sqrt{10 \times 34}} = \dfrac{17}{\sqrt{340}} = \dfrac{17}{18.439\ldots} = 0.9219\ldots \approx 0.922$ (3 s.f.) — a genuinely strong positive linear association, consistent with $y$ rising fairly steadily as $x$ does.

3.2 — The same data, linearly coded, confirming invariance numerically. Coding $u = (x-2)/3$ and $v = (y-1)/2$ (arbitrary positive constants, chosen only to be different from either existing lesson's coding examples) on the identical five points gives $S_{uu} = 1.111\ldots$, $S_{vv} = 8.5$, $S_{uv} = 2.833\ldots$, and $r_{uv} = 0.9219\ldots$ — identical to 3.1's $r$ to full precision, matching the mark-scheme-credited property "$r$ not affected by (linear) coding" (Jan 2024 MS, Q2(d)) on a dataset built specifically to check it, rather than merely citing it.

3.3 — Quantifying the real $S_{pp}$ substitution trap on its own real numbers. The facts bank's audit trail (its own per-lesson audit entry for wst01-regression-gradient-interpretation-extrapolation.ts) discloses the actual data line for the genuine question this trap comes from — Pearson Edexcel WST01/01, October 2021, Q2: $n=12$, $\sum x = 93$, $S_{xx} = 148.25$ (given), $\sum p = 273$, $\sum p^2 = 6602.72$, $\sum xp = 2347$ — and the confirmed real error, quoted directly from the examiner report: "the most common error made was using $(273/12)^2$ in the calculation of $S_{pp}$." Neither the facts bank nor either existing lesson computes what that substitution actually does to the answer. Doing so here, from these disclosed figures: the correct $S_{pp} = \sum p^2 - \dfrac{(\sum p)^2}{n} = 6602.72 - \dfrac{273^2}{12} = 6602.72 - 6210.75 = 391.97$; the wrong path substitutes $\bar p^2 = (273/12)^2 = 22.75^2 = 517.5625$ for the correction term directly, giving $S_{pp,\text{wrong}} = 6602.72 - 517.5625 = 6085.1575$ — over fifteen times larger than the correct value (ratio $\approx 15.5$), not a rounding-scale difference. Carrying this into the PMCC step it feeds (using the also-disclosed $S_{xp} = \sum xp - \dfrac{(\sum x)(\sum p)}{n} = 2347 - \dfrac{93\times273}{12} = 231.25$): the correctly-computed $r = 231.25/\sqrt{148.25\times391.97} \approx 0.959$, while the wrong-$S_{pp}$ path gives $r_{\text{wrong}} = 231.25/\sqrt{148.25\times6085.16} \approx 0.243$ — a wrong answer that still lands comfortably inside $[-1,1]$ and so trips no "$|r|>1$ means a mistake" self-check, the identical danger the existing lesson already documents for the separate sqrt-omission trap on a different real question (Jun 2024 Q4(b)). Caveat, stated plainly: these $r$-values are this document's own computation from the summary statistics the facts bank discloses for this question, not an independently verified transcription of a printed mark-scheme target for Oct 2021 Q2(b) — that exact printed figure was not confirmed in the source material this document read.

3.4 — Why "linear-only" is a real mathematical limitation, not a hedge. $x$: $-2,-1,0,1,2$; $y = x^2$: $4,1,0,1,4$ — a perfect, entirely deterministic relationship (every $y$-value is exactly predictable from $x$). Computing directly: $\bar x = 0$, $\bar y = 2$, $S_{xx} = 10$, $S_{yy} = 14$, and $S_{xy} = \sum x_i(y_i - 2) = (-2)(2)+(-1)(-1)+(0)(-2)+(1)(-1)+(2)(2) = -4+1+0-1+4 = 0$ exactly, so $r = 0/\sqrt{140} = 0$ exactly — the strongest possible relationship producing the weakest possible PMCC reading, because the relationship is symmetric about the line $x=0$ and curved rather than straight. This is the concrete arithmetic behind the teach block's own claim (that $r$ near 0 can coexist with a genuine, strong, non-linear relationship) and behind MCQ pmcc-mcq-4's U-shaped scenario in the existing lesson — this document supplies the worked numbers that claim was previously stated in words only.

#4. Synthesis: every real exam angle spec 4.3 has actually been tested from

Stated with the facts bank's own honest denominators throughout (§3 of the facts bank: 6 series with a fully-read examiner report, 14 series with a verified mark scheme).

What this table does not contain, on purpose: a percentage or fraction of series in which spec 4.3 specifically (as opposed to the wider 4.1–4.3 section) appears, even though the count is now pinned down more precisely than this document's first draft managed. Spec 4.3 itself — a PMCC calculation, interpretation, or named property, as distinct from the section's regression-line content — is confirmed present in 3 of the 6 fully-read ER series (Oct 2021, Jun 2022, Jun 2024) plus 1 mark-scheme-only citation (Jan 2024). The Jun 2022 instance is confirmed only by direct re-reading of the real question paper (recorded in the sibling lesson's own audit trail, and independently by signal-topic-brief-correlation-coefficient-and-regression-calculation.md in this folder) — no specific quoted wrong-answer mechanism for Jun 2022 Q2(a)–(b) survives in the facts bank's own written §4 prose, which is why this document's first draft, reading only that prose, undercounted it as 2 of 6. Stating a rate finer than "3 of 6, on this denominator, dated and named" — e.g. converting it to a year-over-year trend or a percentage of all 14 series — would still be exactly the kind of manufactured-precision this course's standing rules forbid, since only 6 of the 14 series have a fully-read examiner report at all.

#5. What remains flagged, not resolved, by this document

• The facts bank's own §4 "PMCC use and limitations (4.3)" paragraph, read as delivered, still describes the Jun 2024 Q4(c) accuracy quote as belonging to a "show that... then r" question. The correlation-coefficient lesson file's header documents, with specific re-verified numbers ($S_{dg}=12105.12$, $S_{dd}=16769.78$, $S_{gg}=9094.2$, gradient $0.7218\ldots\to 0.722$, intercept $-42.3$), that this is a mischaracterisation and that the quote belongs to the regression line's gradient step, not to $r$. This document follows the corrected account throughout §4 above; the facts bank text itself has not been re-edited to match, as of this reading.
• This document's own first draft undercounted spec 4.3's frequency as "2 of 6" fully-read ER series, having read this sub-topic's own audit entry in the facts bank but not the sibling lesson's — which records that its Jun 2022 Q2(a)–(b) tests spec 4.3 directly. Fixed on this same-day revision (§4 above now reads "3 of 6"); noted here as a correction actually made, not merely a risk flagged, since the wrong count had already been written once.
• §3.3's reconstructed $r$-values for Oct 2021 Q2(b) are this document's own arithmetic from disclosed summary statistics, not a confirmed transcription of a printed answer — stated plainly there and repeated here so it isn't missed.
• No standalone past-paper example testing $r$'s interpretation or limitations directly was located in the reviewed archive; §2.2's discussion and the existing lesson's teach block both rest on the spec's own wording, not on a quoted exam instance, and both say so.
• This document performed no new PDF fetches or searches; every cited fact traces to research/veridian/WST01-verified-facts.md, the existing lesson file's own header/flag block, or this document's own independently-verified arithmetic on numbers those two sources already disclose.