Study Results

Each cohort was collected under a different frozen version of the index. Neither cohort’s responses were used to build the original v1 index, so we rescored every schedule from both cohorts with v1 and analyzed all 100 sessions together. This analysis was decided after both cohorts closed. It is post hoc, not a pre-specified test, and not a validation result under our governance rules.

Across 993 graded comparisons, physicians chose the schedule v1 ranked higher 68% of the time (95% CI 65–71%). Agreement rose with the score gap, from 52% when schedules were within 10 points to 65% at 10–19 points and 76% at 30 points or more. It also rose with physicians’ own confidence, from 54% to 74%.

For reference, physicians agreed with their own earlier choice on repeated pairs 81% of the time, and with each other 65% of the time (free-marginal κ 0.30). The index agreed with physicians at κ 0.36. On pairs rated by at least three physicians, the index predicted each physician’s choice about as well as the consensus of the other physicians did (73% vs 71%, a difference within the margin of error). The remaining gap up to physicians’ own consistency reflects individual preference.

Applying the statistic from our locked analysis plan to the pooled data gives Z ≈ 4.3, with every score-gap stratum above chance. That exceeds the boundary the plan set for a 100-session look. Because the pooling was decided after the fact and the second cohort’s pairs were selected by a different index, we report it descriptively and do not treat it as a stopping decision.

Fifty practicing emergency physicians completed sessions of blinded head-to-head schedule comparisons — 626 graded forced-choice judgments in total. Participants never saw a score, ranking, or prediction.

Physician preference aligned with the schedule the index ranked higher on approximately 69% of graded pairs. The directional signal was statistically distinguishable from chance (Z ≈ 3.4), confirming that the index was measuring something physicians genuinely respond to.

The result did not cross the pre-specified early-stopping efficacy boundary, so v1 was treated exactly as pre-registered: as derivation evidence rather than a validation claim. Nothing on this page should be read as a validated performance estimate.

A pre-registered exploratory diagnostic on the same 626 comparisons identified which fatigue and circadian axes the index was under-weighting or missing entirely. The revisions were material enough that the v1 sessions were permanently converted to derivation data and excluded from validation statistics.

Under the governance rule set before data collection, revising the index converted the v1 sessions into derivation data. They were used to refit the scoring weights and are permanently excluded from every validation statistic. The revised index — CAI v2.1 — was frozen and validated on a fresh cohort; that cohort has since been closed under the same governance rule, and its evidence was used to derive the current frozen index, CAI v3. CAI v3 starts from the v1 engine and adds what the v2.1 cohort showed. The v3 cohort restarted on 2026-10-02 on the current frozen version; the one earlier v3 session was archived and is not counted.

Rank correlation between the v1 and v2.1 indices is high (ρ ≈ 0.95–0.97). Responses collected under one index can’t count toward validating a revised one. They can still be analyzed under the index that was frozen when they were collected, which is what the pooled section does.

The v2.1 validation cohort enrolled 50 emergency physicians who completed 500 graded head-to-head schedule comparisons before the cohort was closed under the pre-specified governance rules. (Graded comparisons exclude attention checks, close-score probes and pairs the index scored as tied.)

The central finding was encouraging: when two schedules differed widely in score (30+ points), physicians chose the higher-scored schedule 73% of the time (204 graded pairs). The index clearly separates genuinely good schedules from genuinely bad ones — the property that matters most for a scheduling tool.

The cohort also showed exactly where the index could be sharpened: physicians treat schedules scoring above the high-80s as similarly good, and night-pattern handling accounted for most of the remaining disagreement. Those findings were used — under the governance rule that converts a closed cohort into derivation data — to derive the current frozen index, CAI v3.

Rescored with the v1 engine, the same cohort agreed slightly more often (66.5% vs 66.2%). The difference is inside the noise, but it shaped v3.

One design lesson carried forward: physicians' choices show that a higher score meaningfully predicts preference, while the steepness of that relationship is modest: in the post hoc pooled analysis, agreement rose by about 4 percentage points for every 10 points of score gap. The v3 validation therefore tests the question physicians actually answered — "do physicians pick the higher-scored schedule?" — as its primary endpoint, with the steepness of the relationship reported descriptively rather than used as a pass/fail gate.

This page reports high-level findings from the completed v1 and v2.1 cohorts and a post hoc pooled analysis. It does not disclose the CAI scoring model, feature weights, normalization, formulas, or any other proprietary implementation details.

The live figures below describe the ongoing CAI v3 validation cohort: enrollment and data quality only. Agreement between physician preference and the index is sealed for this cohort until the pre-specified checkpoints are reached.