Internal · read-only

SpOps admin — football module

Everything below is read-only: no destructive actions or write endpoints are exposed here, even where the backing infra supports them.

← back to the public site
1 · System

API / DB / storage health

Serving API (FastAPI)
Live client-side ping of https://wc26-serving.onrender.com/health
checking...
MLflow registry
Checked at last data export -- best-effort, degrades gracefully when the local docker-compose stack isn't up
unreachable
Supabase (Postgres)
Durable predictions/results mirror -- CSV/JSON logs remain the source of truth regardless
unreachable
Static data store
dashboard/data/*.json, committed to git -- this page (and the public site) render from these with zero live dependency
healthy

Generated at 2026-09-14 11:52UTC. Redis isn't part of this stack -- nothing here needs a cache (predictions are computed once daily; this page is static JSON) -- so it isn't faked in for the sake of matching a generic tech-stack checklist.

2 · MLOps

Pipeline runs & schedule

The real Airflow DAG task graph, not a diagram drawn by hand.

DAG wc26_daily_pipeline · schedule 0 6 * * * UTC
view DAG source →
(parallel branch)→
check_drift.py
(parallel branch)→
check_data_quality.py
Last successful data refresh
2026-09-14T11:52:55.658489+00:00
Last drift check
2026-09-14
Last data-quality check
2026-09-14

Runs in the project's docker-compose Airflow stack; timestamps come from the last successful run.

3 · Models

Production model & registry

Registered model
wc26-layer1-stacked-ensemble
mlflow unreachable at export time
Trained at
n/a
Training window
? → ?
Training rows
0
Ensemble member weights

How much each base model sways the final call (from the fitted meta-learner, not SHAP).

n/a
Feature schema (6)
elo_diffform_goals_for_diffform_goals_against_diffform_win_rate_diffrank_points_diffneutral

Precision/recall/F1/ROC-AUC aren't the natural fit here: Layer 1 outputs a 3-outcome probability distribution (win/draw/loss) each match, not a binary label, so this project tracks Brier score and log-loss instead (avg backtest log-loss: 1.341 across 2018/2022 checkpoints — see Backtest on the public site for the full breakdown against baseline).

4 · Training

Training history, tuning, feature importance

Recent training runs

No MLflow runs visible right now -- either the tracking server was unreachable at the last export, or none have been logged yet. Empty, not fabricated.

Hyperparameter tuning: scripts/tune_layer1.pyruns a real Optuna search over XGBoost, but honestly does not currently beat the default hyperparameters on the true backtest metric (see PROJECT_BRAIN.md) -- the pipeline runs on defaults, not tuned params. Not hiding that behind a fake "tuned" badge.

Feature importance (live)

Loading live feature importance...

This is a global, XGBoost-only importance (gain-based). For genuine per-match SHAP values, see below.

Explain a prediction (live SHAP)

Real per-match Shapley values from the deployed serving API (POST /explain) — genuine local interpretability (“why did the model say this, for this specific match”), not the global feature importance above. Bars below show each feature's real contribution to the home team's win probability for the matchup selected.

vs
5 · Monitoring

Latency, traffic, containers

Prediction latency

Click to take a real latency sample -- no history is persisted server-side yet, so this is a live reading, not a stored time series.

API traffic, CPU/memory, container status

Not yet implementedas persisted metrics -- Render's free tier doesn't expose container-level CPU/memory, and no traffic aggregation is stored yet (Tier 3: Prometheus/Grafana, see CLAUDE.md). What is real today: request-level X-Request-ID/X-Response-Time-Ms headers on every response (see the latency probe), and Docker healthchecks on mlflow/serving/airflow-webserver in the local docker-compose stack, verified live each session (see PROJECT_BRAIN.md).

6 · Data quality

Missing values, schema, duplicates, drift

Schema
valid

7 expected columns checked against the combined historical + live match data.

Missing values
0 cells across 7 columns

Zero missing values in the combined match dataset.

Historical / live overlap
74 of 104

Live-ingested 2026 matches already present in the periodically-refreshed historical CSV -- deduplicated by load_combined_matches() before training (see DECISIONS.md: a real double-counting bug this check exists to catch a regression of).

Duplicate keys (pre-dedup)

76 duplicate (date, home_team, away_team) key(s) across 49,588combined rows, before the training pipeline's own dedup runs.

dataset drift: not detectedas of 2026-09-14 -- reference 2016-01-01→2026-01-01 (19,282 rows) vs current 2026-01-01→2026-09-14 (838 rows)
elo_diff0.1072 (drift)
form_goals_against_diff0.0953
form_goals_for_diff0.0698
form_win_rate_diff0.0316
rank_points_diff0.0943

Wasserstein distance (normed)· checked daily by the pipeline

Drift score trend
7 · Predictions

History, confidence, bracket evolution

13
Matches graded
85%
Our AI accuracy
0.395
Our AI avg. Brier
77%
Bookmaker accuracy
Predicted Jul 14, 00:33 UTC✓ Correct
France0 - 2Spain
Our AI saidSpain 40%
World expectedFrance 40%
Predicted Jul 7, 21:40 UTC✓ Correct
Norway1 - 2England
Our AI saidEngland 51%
World expectedEngland 51%
Predicted Jul 7, 21:40 UTC✓ Correct
Spain2 - 1Belgium
Our AI saidSpain 60%
World expectedSpain 59%
Predicted Jul 7, 21:40 UTC✓ Correct
France2 - 0Morocco
Our AI saidFrance 50%
World expectedFrance 60%
Predicted Jul 7, 21:40 UTC✗ Missed
Switzerland0 - 0Colombia
Our AI saidColombia 43%
World expectedDraw 63%
Predicted Jul 5, 17:34 UTC✓ Correct
Argentina3 - 2Egypt
Our AI saidArgentina 73%
World expectedArgentina 69%
Predicted Jul 5, 17:34 UTC✓ Correct
United States1 - 4Belgium
Our AI saidBelgium 60%
World expectedUnited States 37%
Predicted Jul 5, 17:34 UTC✓ Correct
Portugal0 - 1Spain
Our AI saidSpain 50%
World expectedSpain 50%
Predicted Jul 5, 17:34 UTC✓ Correct
Mexico2 - 3England
Our AI saidEngland 42%
World expectedEngland 39%
Predicted Jul 5, 17:34 UTC✗ Missed
Brazil1 - 2Norway
Our AI saidBrazil 55%
World expectedBrazil 53%
Predicted Jul 4, 02:20 UTC✓ Correct
Paraguay0 - 1France
Our AI saidFrance 70%
World expectedFrance 81%
Predicted Jul 4, 02:20 UTC✓ Correct
Canada0 - 3Morocco
Our AI saidMorocco 51%
World expectedMorocco 51%
Predicted Jul 4, 02:20 UTC✓ Correct
Colombia1 - 0Ghana
Our AI saidColombia 79%
World expectedColombia 83%
Calibration -- when our AI says X%, does it happen X% of the time?
0-20%
n=0
20-40%
n=1
40-60%
n=9
60-80%
n=3
80-100%
n=0
Predicted confidence Actually correct
Replay (seed of Phase 6 — life after the tournament)
2026-09-14

Replaying 73 logged day(s) of real predictions (series: heuristic_l2_montecarlo_v1) -- not the full tournament, since it isn't over yet (final: 2026-07-19).

1. ESP Spain48.5%
2. ARG Argentina29.9%
3. ENG England19.7%
4. COL Colombia1.8%
5. SUI Switzerland0.1%
8 · Observability

Metrics, traces, deployments

Live serving metrics

reading /metrics-summary...

Serving API unreachable right now (Render free tier spins down after 15 min idle -- the first request cold-starts in ~10s). These are live production counters when it's up, not a static export.

Read client-side from https://wc26-serving.onrender.com/metrics-summary -- a JSON projection of the same Prometheus registry that /metrics exposes. Genuine round-trip numbers, not a fabricated dashboard.

Metrics pipeline: OpenTelemetry → Prometheus → Grafana

The serving API is instrumented with OpenTelemetry traces and a Prometheus /metrics endpoint (RED metrics -- request rate, errors, duration -- plus predictions-by-outcome and the loaded model version). A Prometheus container scrapes it every 15s and a provisioned Grafana dashboard renders it. Bring the whole stack up locally with docker compose -f docker-compose.airflow.yml up; Grafana is then at localhost:3001(admin/admin), dashboard “WC26 Serving API”.

Honest scope: the live counters above are real production numbers from the deployed API. The Grafana UI runs in the local/demo docker-compose stack, not on the Render free tier (which won't host a separate Grafana process) -- so it's a screenshot-and-run artifact, not a public hosted panel. Centralized log aggregation and alerting (Alertmanager) remain genuinely out of scope for this project and aren't faked in.