Our Model vs PELE

Silver Bulletin publishes PELE's match forecasts as a Datawrapper table (chart 3bTOr) and team ratings as another (pS7DN). Datawrapper keeps every published version at an immutable URL and serves each with an HTTP Last-Modified header. scripts/fetch_pele_forecasts.py walks version numbers until it hits a gap, and recovers 127 forecast-table versions and 133 ratings versions. It keeps only World Cup rows, maps each team code to our naming, and fails loudly on any unknown code rather than dropping the row.

For each match the benchmark uses PELE's last version published before a deadline: the exact kickoff when our fixture feed recorded it, otherwise 15:00 UTC on the earlier of PELE's US-Eastern date label and the venue-local match date. Every 2026 kickoff was at or after 16:00 UTC on its local date, so that is always pre-kickoff. The label-only version of this rule leaked one post-match forecast (see the audit above); it is fixed and pinned by a regression test in tests/test_pele_benchmark.py.

Live (21 matches). Our daily pipeline committed its forecasts to dashboard/data/upcoming_matches.json. The comparison reads every historical version with git log / git show and uses the newest commit made before kickoff. Local log files are deliberately ignored: they are gitignored, so their timestamps prove nothing to anyone else.

Replay (102 matches). Our Layer 1 model is re-run with the same code and features, but trained once, only on matches from 1992-01-01 to 2026-06-11 (production retrained daily). It is a scikit-learn stacking classifier over three members (XGBoost with 200 trees, depth 4, learning rate 0.1; a logistic regression on Elo difference and venue; a FIFA-ranking heuristic), blended by a logistic meta-learner with 5-fold stacking. Its six features (Elo difference, rolling goals for/against and win-rate differences, FIFA points difference, neutral venue) (including FIFA ranking points) are looked up strictly before each match date, so no match can see its own result. On the 21 live matches the replay lands within 1.1 probability points of what was actually published, which is the fidelity check for the replay.

Every forecast is a probability vector p = (home win, draw, away win) and every result is one of those three, so draws are a real outcome, not dropped or split. Knockout results use the recorded score including extra time; a shootout counts as a draw. The headline metric is the Ranked Probability Score, which respects the ordering (calling a home win a draw costs less than calling it an away win):

All three are strictly proper: a forecaster minimizes its expected score only by reporting its true beliefs.

Both forecasters are scored on the same matches, so every test works on per-match differences d = score(A) − score(B). The p-value comes from a two-sided sign-flip randomization test: under the null that A and B are interchangeable, each dᵢ is equally likely to be +dᵢ or −dᵢ. 10,000 random sign vectors give the null distribution of the mean, and p = (1 + #|flipped mean| ≥ |observed mean|) / 10,001. The 95% interval is a percentile bootstrap over matches (10,000 resamples). A paired t-test was rejected because upsets make the differences heavily skewed and some subsets have only 14 matches.

Caveats stated up front: no multiple-comparison correction is applied; matches share teams, so they are not fully independent; p-values are Monte Carlo estimates (about ±0.002 near p = 0.05).

dashboard/src/lib/pele/stats.ts is a dependency-free TypeScript port of src/benchmarks/scoring.py: the three scoring rules, the paired bootstrap and sign-flip test, decile reliability with expected calibration error, Spearman correlation with tie-averaged ranks, an inverse-normal (Acklam) for the power formula, and the grid-search / leave-one-out fitter used by the Model Lab. Randomness comes from a seeded Mulberry32 generator, so the same settings always show the same numbers.

Before any UI was built, the compiled engine was run against the same data as the Python benchmark. Mean score differences, ECE, and the leave-one-out blend and temperature fits matched to floating-point precision; bootstrap interval ends and p-values differed only by Monte Carlo error (for example p = 0.484 in the browser vs 0.479 in Python for the full-tournament RPS test), because the two languages use different random generators. One 10,000-resample test took roughly 50 ms on a laptop, so it runs synchronously on each control change without a Web Worker.

Forecast combination is a linear opinion pool, w·ours + (1−w)·PELE, over w ∈ {0, 0.05, …, 1}. Temperature scaling maps each probability to pᵢ^(1/T) and renormalizes, over T ∈ {0.50, 0.55, …, 1.50}. Both are fit by grid search on the selected metric.

Choosing a knob on the same 102 matches you then score is optimistic, so the lab also reports leave-one-out cross-validation: for each match i, the knob is chosen on the other 101 matches and match i is scored with it; the reported LOO score is the mean over all 102 held-out matches. The scores for every grid value are computed once, and each fold reuses them by subtracting the held-out match, so the whole LOO fit takes a few milliseconds. The lab also lists which values the folds chose, which shows how stable the optimum is.

The page is a Next.js 16 App Router route (src/app/pele/page.tsx), prerendered as static HTML at build time. It reuses the site's own Nav, SectionHeading and Footer, the existing design tokens and glass cards, Recharts for every chart and framer-motion for the same entrance animations the rest of the site uses. Sections are client components with local React state and memoized derivations; nothing is fetched at runtime.

The PELÉ tab sits next to the existing sport tabs but is deliberately not an entry in sports_config.json: PELE is a benchmark, not a sport, and that config also drives sport routing, the completed-sports showcase and a backend retrospective script. The tab is a separate link rendered after the config-driven sport tabs, with the same styling.

Charts follow a validated palette (forecaster colors and the home/away outcome pair pass a colorblind-separation check against the dark card), draws are dashed gray so identity never relies on color alone, low-confidence calibration bins (n < 10) are drawn hollow, and controls are native buttons, ranges and selects with ARIA roles.

Every PELE number on the page links to the exact Datawrapper CSV it came from, and every live forecast of ours links to the commit that published it. A one-page summary with the test behind each p-value is in docs/pele-benchmark-summary.md.

Limitations: one tournament, so no gap between the models is detectable; PELE's method column above paraphrases its published methodology; the replay is a faithful re-run of our model, not a record of what we published for the group stage; team-strength inputs come from results only.