Validation

Is the data right, is the index stable, and does it measure what it claims to?

1. Cross-source checks

Every component is built from raw play-by-play by this project’s own code. To make sure that code reads the events correctly, the counts were compared with two independent sources.

Basketball Reference parses the same games with its own code. On 25 randomly sampled player-seasons (1,000+ minutes, fixed random seed):

Show code

xb[, .(Metric = metric, `Player-seasons` = n, Correlation = r, `Mean |difference|` = mean_abs_diff,
       `Within tolerance` = paste0(pct_within_tol, "% (±", tol, ")"))] |> small_gt()

Metric	Player-seasons	Correlation	Mean \|difference\|	Within tolerance
Shooting fouls drawn	25	0.9995	0.68	96% (±5)
And-ones	25	0.9997	0.08	100% (±3)
Offensive fouls	25	0.9980	0.24	100% (±3)
% of 2PM assisted	25	1.0000	0.03	100% (±2)
% of 3PM assisted	20	0.9997	0.07	100% (±2)

Show code

reactable(xbd, defaultPageSize = 8, compact = TRUE, striped = TRUE, theme = rt_theme,
          columns = list(player = colDef(name = "Player", minWidth = 140)))

pbpstats.com covers 2000-01 onward (earlier seasons in its database are missing games). For every player-season with 200+ FGA:

Show code

xp[era == "2000-01 to 2025-26", .(Metric = metric, `Player-seasons` = n, Correlation = r,
                                  `Within tolerance` = paste0(pct_within_tol, "% (±", tol, ")"))] |> small_gt()

Metric	Player-seasons	Correlation	Within tolerance
FGA (incl. heaves)	7965	1.0000	100% (±2)
FTA	7965	1.0000	100% (±2)
Turnovers	7965	1.0000	99.9% (±2)
Travels	7965	1.0000	100% (±1)
Offensive fouls	7965	1.0000	100% (±2)
3PT shooting fouls drawn (3-shot trips + 3PT and-ones)	7965	0.9990	99.6% (±1)
And-ones (2PT + 3PT)	7965	0.9999	100% (±2)
Unassisted 2PM	7965	1.0000	100% (±2)
Unassisted 3PM	7965	0.9999	99.8% (±1)

Against official box-score totals, field-goal attempts, free-throw attempts and turnovers match within ±2 for every qualifying-minutes player-season.

2. Tracking-data validation (Tier 2)

The design calls for checking the play-by-play proxies against NBA player-tracking data (2013-14+): for example, whether excess free-throw rate tracks free throws that did not come from drives, and whether shot-making above expected tracks efficiency on tightly-contested and pull-up shots. Tracking data was never an input to the index; it’s corroborating evidence only.

Show code

tv_file <- file.path(root, "data", "final", "tracking_validation.csv")
if (file.exists(tv_file)) {
  cat("**Results from tracking pages saved manually from nba.com:**\n\n")
  print(htmltools::tagList(small_gt(fread(file = tv_file)[, .(Check = check, r, `95% CI` = sprintf("[%.2f, %.2f]", ci_lo, ci_hi), `Player-seasons` = n)])))
} else {
  cat("*The NBA's stats server blocks automated requests, so this layer is pending (see `scripts/load_tracking_manual.R`). Sections 3–5 provide independent validation in the meantime.*")
}

The NBA’s stats server blocks automated requests, so this layer is pending (see scripts/load_tracking_manual.R). Sections 3–5 provide independent validation in the meantime.

3. Year-to-year stability

If the index measures a real playing style, a player’s rating this season should predict next season’s. Pure noise would correlate near zero.

Show code

stab[, .(Measure = fcase(measure == "ehi", "EHI", measure == "skill", "Skill Score", measure == "whistle", "Whistle Score",
                         default = COMP_LABELS[measure]),
         `r (season t → t+1)` = r, `95% CI` = sprintf("[%.3f, %.3f]", ci_lo, ci_hi), Pairs = pairs)] |> small_gt()

Measure	r (season t → t+1)	95% CI	Pairs
EHI	0.740	[0.722, 0.756]	2708
Skill Score	0.752	[0.736, 0.768]	2708
Whistle Score	0.748	[0.731, 0.764]	2708
Shot-making above expected	0.658	[0.636, 0.679]	2708
Shot difficulty	0.881	[0.873, 0.889]	2708
Self-creation	0.902	[0.894, 0.909]	2708
Clean footwork (travels + offensive fouls, reversed)	0.758	[0.742, 0.774]	2708
Excess free-throw rate	0.719	[0.700, 0.737]	2708
3PT shooting-foul rate	0.639	[0.616, 0.661]	2708
Perimeter and-one rate	0.483	[0.453, 0.511]	2708
Free-throw share of points	0.803	[0.790, 0.816]	2708

EHI is highly persistent (r = 0.74). Self-creation and shot difficulty, essentially a player’s role and shot diet, are the stickiest components. The perimeter and-one rate is the noisiest, which is why it carries the smallest weight.

4. Predictive validation: the 2021-22 rule change

The strongest test of a “whistle” measure is whether it predicts what happens when the league takes the whistle away. Players with higher pre-2021 Whistle Scores lost 0.38 points of true shooting per standard deviation after the crackdown (p = 0.049), while a placebo rule date shows no effect. Full analysis on the Rule Change page.

5. Face validity

Top and bottom 15 in four sample seasons. These were scanned for anything that looked like a data bug rather than a real playing style. Each bug found during development (and fixed) is logged in the project’s NOTES.md.

Show code

fv_wide <- function(s) {
  d <- fv[season == s]
  data.table(`#` = 1:15, `Most ethical` = d[end == "top"][order(rank), sprintf("%s (%.0f)", player, ehi)],
             `Most whistle-reliant` = d[end == "bottom"][order(rank), sprintf("%s (%.0f)", player, ehi)])
}

Show code

small_gt(fv_wide("2000-01"))

#	Most ethical	Most whistle-reliant
1	Glenn Robinson (82)	Dirk Nowitzki (4)
2	Gary Payton (79)	Paul Pierce (10)
3	Dana Barros (76)	David Robinson (10)
4	Mike Bibby (74)	Elden Campbell (14)
5	Antoine Walker (73)	Johnny Newman (24)
6	Terrell Brandon (73)	John Amaechi (26)
7	Chucky Atkins (71)	Derek Anderson (26)
8	Jason Terry (69)	Corey Maggette (28)
9	Tim Hardaway (69)	Larry Hughes (29)
10	Mookie Blaylock (68)	Joe Smith (30)
11	Isaiah Rider (68)	Austin Croshere (31)
12	Michael Doleac (68)	Pat Garrity (31)
13	Maurice Taylor (68)	Lamond Murray (34)
14	Peja Stojaković (66)	Kenyon Martin (35)
15	Damon Stoudamire (66)	Tyrone Nesby (35)

Show code

small_gt(fv_wide("2010-11"))

#	Most ethical	Most whistle-reliant
1	Tony Parker (77)	Kevin Martin (9)
2	Stephen Curry (76)	Dwight Howard (10)
3	Zach Randolph (76)	Corey Maggette (15)
4	Luis Scola (75)	Devin Harris (17)
5	Jameer Nelson (73)	Chauncey Billups (23)
6	Darren Collison (73)	Rodney Stuckey (23)
7	Kyle Korver (72)	Dwyane Wade (23)
8	Marcus Thornton (72)	Jamal Crawford (24)
9	Al Jefferson (70)	John Wall (24)
10	Raymond Felton (69)	Lou Williams (25)
11	Al Horford (69)	Blake Griffin (26)
12	Dirk Nowitzki (69)	Jarrett Jack (27)
13	Jason Terry (67)	Russell Westbrook (28)
14	Rudy Gay (67)	Kevin Durant (29)
15	Tayshaun Prince (67)	J.J. Hickson (29)

Show code

small_gt(fv_wide("2018-19"))

#	Most ethical	Most whistle-reliant
1	D'Angelo Russell (88)	DeMarre Carroll (10)
2	Nikola Vučević (85)	Jimmy Butler (11)
3	Kyrie Irving (84)	James Harden (16)
4	Quinn Cook (80)	Spencer Dinwiddie (17)
5	Buddy Hield (76)	Giannis Antetokounmpo (19)
6	Ish Smith (76)	Rondae Hollis-Jefferson (22)
7	Klay Thompson (74)	Jae Crowder (22)
8	Malik Beasley (74)	Alex Len (23)
9	Malcolm Brogdon (74)	Jusuf Nurkić (24)
10	E'Twaun Moore (74)	Alec Burks (25)
11	Jrue Holiday (73)	Lou Williams (27)
12	CJ McCollum (73)	Langston Galloway (27)
13	Al Horford (73)	Paul Millsap (29)
14	Monte Morris (72)	Joel Embiid (31)
15	Nikola Jokić (71)	Derrick Favors (31)

Show code

small_gt(fv_wide("2025-26"))

#	Most ethical	Most whistle-reliant
1	Bobby Portis (90)	James Harden (10)
2	Payton Pritchard (86)	Jerami Grant (12)
3	Jrue Holiday (84)	Ziaire Williams (16)
4	Ryan Rollins (83)	Keyonte George (17)
5	LaMelo Ball (80)	Karl-Anthony Towns (22)
6	Reed Sheppard (79)	Paolo Banchero (23)
7	Nikola Vučević (79)	Jimmy Butler (23)
8	Darius Garland (76)	Bennedict Mathurin (23)
9	Anfernee Simons (75)	Stephon Castle (23)
10	Jamal Murray (72)	Simone Fontecchio (25)
11	Caleb Love (71)	Deni Avdija (25)
12	Tre Johnson (71)	Norman Powell (26)
13	De'Aaron Fox (70)	Noah Clowney (26)
14	Klay Thompson (69)	Bilal Coulibaly (28)
15	Nikola Jokić (69)	Austin Reaves (31)

Bugs caught by these checks (now fixed):

Intentional fouls inflated the free-throw rates of elite free-throw shooters (fouled on purpose late in games) and poor ones (hack-a-big). Now only shooting-foul free throws count.
“And-ones on jumpers” as a share flagged pure spot-up shooters as foul-baiters. It’s now a rate per perimeter attempt.
Shrinkage priors were briefly distorted by tiny samples, producing |z| > 8 outliers. They’re now fit on reliable samples only.
Shot-chart joins, the 2012-13 missing games, and heave logging changes were all caught during the data audit, before any metric was built.