Skip to the report
TOP SECRET // EYES ONLY DECLASSIFIED — PUBLIC RELEASE
The Internal Bulletin
Published for field personnel of the Directorate of Measurement
VOL. XXIV — NO. 3TUESDAY, OCTOBER 12, 1965LANGLEY, VA.5¢ — INTERNAL DISTRIBUTION ONLY
DESIGN PROOF — FIGURES MARKED [SIM] ARE SIMULATED; PRODUCTION VALUES ARRIVE FROM THE LIVE WIRE [LIVE]
Extra
Methodology desk — declassification special
Approved
for release

Directorate subtracts every known cause; what remains is filed as the finding

The question
FILE NO. 00-1173-M/00

One chip, one day: quoted prices found in wide disagreement

On the day of this edition, field stations recorded 186 public quoted prices for one exactly-specified machine (the H100 SXM 80GB) every quote stated in dollars per GPU per hour. The cheapest asked $0.57; the dearest, $36.87, a spread of 64.8× for hardware that is, by specification, the same.

These are simultaneous quotes on a single UTC day, not prices moving over time. The disagreement is the subject of this bulletin: how much of it can be explained by observable causes, and how much cannot.

Exhibit A — The day's quotes, as receivedh100_sxm_80gb · TODAY · [SIM]
Each dot: one public quote, USD per GPU-hour, log scale. Vertical scatter is for legibility only.
Inter-office memorandum: scope of the comparison (technical annex)

1. UNIT. Price is USD per GPU per hour. Multi-GPU bundles are divided out before anything else happens.

2. ONE SKU AT A TIME. SXM, PCIe, and NVL are different SKUs; 40GB and 80GB are different SKUs. No cross-SKU pooling exists anywhere in the method.

3. ONE UTC DAY AT A TIME. The comparison is purely cross-sectional. THE MEASUREMENT IS DISAGREEMENT BETWEEN SIMULTANEOUS QUOTES, NOT PRICE VOLATILITY, which concerns movement over time.

4. The exhibit's data table: n = 186, min $0.57, p25 $2.02, median $4.28, p75 $16.08, max $36.87. [SIM]

Collection & custody
FILE NO. 00-1173-M/01

Field stations record every answer; originals sealed, never edited

Twice daily, at 08:00 and 20:00 hours UTC, stations operating query five providers and write down every answer exactly as given. Should a provider fail to answer, the cycle proceeds without it; the record shows holes, never fabrications.

Each original response is sealed in full into the central file, where it may never be edited or destroyed. Every later judgment (every translation, every calculation) is a disposable copy that can be re-derived from the sealed originals. This, officials note, is the answer to the question every reader should ask: how do we know you didn't tune this? Because the evidence cannot be touched, any opinion about it can be safely revised in public.

The public record counts only prices gathered by this regular twice-daily routine. Historical prices back-filled by other means exist in the archive but are kept out of the series, because mixing two collection mechanisms manufactures changes where none occurred. The series answers one question: what does the market look like, sampled the same way every day?

Exhibit B — Chain of custody, today's cycleh100_sxm_80gb · [SIM]
  1. STAGE 1
    COLLECT
    189 QUOTES [SIM]
    5 PROVIDERS · 08:00 & 20:00 UTC
  2. STAGE 2
    SEAL RAW
    189 ORIGINALS [SIM]
    WRITE-ONCE · FULL RESPONSE KEPT
  3. STAGE 3
    TRANSLATE
    186 FILED [SIM]
    3 SET ASIDE — NOT IN THE BOOK
  4. STAGE 4
    ADMIT TO PANEL
    186 ADMITTED [SIM]
    CRON-COLLECTED ONLY · BACKFILL HELD OUT
  5. STAGE 5
    MEASURE
    3 FIGURES + LEDGER
    MEDIAN · MIDDLE HALF · DECOMPOSITION
Inter-office memorandum: collection particulars (technical annex)

1. STATIONS. Five collectors: Vast.ai (REST), RunPod (GraphQL), AWS EC2 Spot (boto3), Azure retail catalog, GCP catalog. Timer fires 08:00 and 20:00 UTC, persistent, with a randomized delay. A collector that fails returns an empty list; one outage cannot fail a cycle.

2. THE DUAL QUERY. Vast is asked twice per run: once for on-demand, once for interruptible bids. The payload's is_bid field reads FALSE on every offer of BOTH queries (verified across 315,685 stored rows), so commitment type is derived from WHICH ENDPOINT ANSWERED, not from the data. A machine listed by both queries yields two observations at two prices: two distinct products, not duplicates.

3. THE CAP. Since 2026-06-23 unauthenticated Vast responses are silently capped at the 64 cheapest offers, structurally hiding the premium tier. Collection is authenticated.

4. CUSTODY. raw_observations stores the provider's complete response (~95 fields per Vast offer) as write-once JSONB plus five extracted columns. Derived tables (canonical_offers onward) are deleted and regenerated whenever the rules change.

5. POPULATION RULE (ADR-0007). Analytics reads only cron-collected rows; rows flagged as backfill are excluded. Precedent: a 90-day AWS backfill made pre-boundary days carry ~57 rows and post-boundary days ~30; pooled, this produced an apparent decline that was purely a change in who was in the sample. Caught before publication.

Nomenclature
FILE NO. 00-1173-M/02

Five dialects, one vocabulary; unmatched names set aside, never guessed

Every seller describes the same chip differently. Before prices can be compared, the names must be translated into one shared vocabulary, and the Bureau's rules are strict: an exact-match dictionary, hand-written entry by entry. A name not in the book is set aside and counted, never guessed at. No fuzzy matching, no tolerance for near-misses.

Where information is missing, it stays visibly missing: the label UNKNOWN enters the analysis as a real group of its own, so that no observation is quietly dropped and no gap is laundered into an answer. Meaningful hardware differences are never collapsed; different form factors and memory sizes remain different subjects, permanently.

Every translation keeps its paper trail. For any sealed original, the Bureau can replay exactly which rule fired, what the source value was, and what it became: a trail the write-path is structurally forbidden from touching.

Exhibit C — The translation book, specimen entries
As receivedFiled as
NVIDIA H100 80GB HBM3h100_sxm_80gb
H100 SXM5-80Gh100_sxm_80gb
H100-PCIE-80GBh100_pcie_80gb — A DIFFERENT SUBJECT
(region not reported)UNKNOWN — ITS OWN GROUP
GEFORCE RTX 4090 DSET ASIDE & COUNTED — NOT IN THE BOOK
Inter-office memorandum: the four rule modules (technical annex)

1. GPU SKU. Exact-string dictionary, ~190 hand-written entries → 96 canonical SKUs, format {model}_{form_factor}_{vram}gb. SXM / PCIe / NVL are different SKUs; 40GB / 80GB are different SKUs. Unmatched → skipped and counted (skipped_unknown_gpu). No fuzzy matching, no case folding.

2. COMMITMENT. Eight canonical values: on_demand, spot, reserved_1w/1m/3m/6m/1y/3y. Alternates map (interruptible, bid, preemptible → spot). Unknown non-null strings pass through unchanged as their own group. DISCLOSURE: a null commitment is coerced to on_demand, the one inference in the layer.

3. REGION. Per-provider lookup maps → (country, state, city). ONLY COUNTRY enters the model; state and city are stored, unused. RunPod reports no region → UNKNOWN.

4. BUNDLE. vCPU / RAM / storage extracted for marketplaces only (Vast, TensorDock); hyperscalers and RunPod contribute none → UNKNOWN. Bundle sits LAST in the fixed order, so its provider-shaped missingness is absorbed by the provider term before bundle takes its turn.

5. PROVENANCE. Each module carries a parallel explain_* trail (matched key, transformation, source value). An AST-level test fails the build if the write path references any explain identifier. Endpoint: /api/raw-observation/{id}/explain.

Dispersion
FILE NO. 00-1173-M/03

A day's quotes reduced to three numbers

For each day and each chip, the cloud of quotes is reduced to three figures: the middle price, and the range within which the middle half of all quotes falls. Averages are never used; one junk listing, priced by a seller who expects no taker, would drag an average to a price at which no buyer could actually transact.

Days too thin to summarize are not summarized: fewer than three quotes, and the Bureau declines to report rather than pretend. What is not known is not stated.

Exhibit D — Same quotes, now summarizedh100_sxm_80gb · TODAY · [SIM]
Observations 186 · Median $4.28 · Middle half $2.02$16.08 · Minimum 3 quotes or the day is not reported.
Inter-office memorandum: dispersion particulars (technical annex)

1. Grouped twice: per (date, sku) across all providers, and per (date, sku, provider). Outputs: observation_count, median_price, p25_price, p75_price.

2. MINIMUM 3 OBSERVATIONS per bucket, else the bucket is skipped entirely.

3. Percentiles, never mean ± sd: quote distributions are asymmetric, and a single mispriced line would average the price somewhere no buyer could transact.

The decomposition
FILE NO. 00-1173-M/04

Four causes removed in fixed order; the remainder refuses to be filed

Take all of the day's disagreement (one hundred percent of it) and subtract, one cause at a time, everything observable: where the machine is, how it is rented, who sells it, and what comes bundled. Each cause is credited only with what it explains after the causes before it have had their turn. Whatever is still standing at the end is the finding.

The order of subtraction changes each cause's share of the credit. It does not, cannot, change the remainder. Readers are invited to verify this themselves in Exhibit E: reorder the ledger and watch the shares move while the remainder stands still. That invariance is the entire reason the remainder, and not the causes, carries the headline.

Exhibit E: The ledger of accounts (operable)h100_sxm_80gb · MARKET-PRICED PANEL, n = 126 · [SIM]
OrderCause subtractedShare creditedMove
1ST WHERE IT IS — REGION3.2%
2ND HOW IT'S RENTED — COMMITMENT18.3%
3RD WHO SELLS IT — PROVIDER4.3%
4TH WHAT'S BUNDLED — BUNDLE8.4%
- THE REMAINDER: UNEXPLAINEDCannot be filed65.8%FIXED UNDER ANY ORDER
Current order: REGION → COMMITMENT → PROVIDER → BUNDLE · Reorder freely: the shares move; the remainder does not.
Inter-office memorandum: the calculation in full (technical annex)
Per (date, gpu_sku) with n ≥ 5:
    y        = ln(price_usd_per_hour)
    total_ss = Σ (yᵢ − ȳ)²
    for factor in (region, commitment, provider, bundle):  # fixed order
        keys = factor values, NULL → "UNKNOWN"
        if distinct(keys) < 2: attribute 0; continue
        ss_within = Σ (yᵢ − mean within CROSS-PRODUCT of prior keys)²
        attributed[factor] = max(0, previous_ss_within − ss_within)
    residual = max(0, total_ss − Σ attributed)
    # each component / (n − 1) to become a variance

SEVEN STANDING DECISIONS: (1) LOG PRICES: ratios, not dollar gaps. (2) SEQUENTIAL, NOT SIMULTANEOUS: conditioning is on the cross-product of all prior factors. (3) FIXED ORDER, most-exogenous → most-internal: region → commitment → provider → bundle. (4) ORDER CHANGES THE FACTORS, NEVER THE RESIDUAL. (5) NULL IS A GROUP: UNKNOWN participates as a real level. (6) A factor with fewer than 2 distinct values contributes exactly zero. (7) max(0,·) flooring: numerically safe.

The exhibit above runs this exact arithmetic, in the reader's browser, on the day's simulated panel. In production it runs on the live panel and every share can be walked back to the contributing offers.

The finding
FILE NO. 00-1173-M/05

Same method, same days, two answers: the population is part of the number

Two of the five providers publish administered list catalogs; one posted price per machine type, revised rarely. Such quotes are explainable by construction: where the machine is and how it is rented account for nearly everything. Pool them with marketplace quotes and the measured remainder collapses mechanically, with nothing in the market having changed.

Operate the switch in Exhibit F and perform the experiment yourself. The verdict, in the Bureau's words: a benchmark that quotes the pooled figure is reporting the composition of its sample, not the state of the market. The remainder is not a constant; it is a property of who is counted. That dependence is not a caveat on the finding. It is the finding.

Exhibit F — The unexplained share, 60 days (operable)h100_sxm_80gb · [SIM]
MARKET-PRICED PANEL — AZURE, GCP EXCLUDED
Solid line: MARKET-PRICED PANEL — AZURE, GCP EXCLUDED. Dashed: the other population, for comparison. Today [SIM]: market-priced 65.8% · pooled 15.2%.
Exhibit F as a table (for the record)
DayUnexplained share
DAY 669.9%
DAY 1268.5%
DAY 1866.7%
DAY 2464.9%
DAY 3062.0%
DAY 3665.7%
DAY 4263.7%
DAY 4863.6%
DAY 5466.1%
DAY 6065.8%
Wire brief: the counter-analysis

Rich model fails to beat four coarse factors, audit finds

To answer the objection that the remainder is merely the Bureau's own ignorance, a forty-five-feature model was trained on every observable a seller will disclose, under strict out-of-sample rules and a leakage guard that fails the run outright. Tested honestly, it explained less than the four factors. The gap is negative, and is published. Consult the counter-analysis file.

Wire brief: the anatomy of the leftover

Leftover sticks to specific machines: identity, not specification

In the one marketplace where machines can be tracked across days, over half of the within-day leftover attaches to the same specific hosts, day after day (ICC ≈ 0.55, published with sensitivity checks at three tenure thresholds). Whatever the remainder is, it is not random noise. See the host-identity file.

Inter-office memorandum: populations & the counter-analysis (technical annex)

1. MARKET-PRICED is defined as the panel excluding the administered catalogs (azure, gcp). The exclusion RECOMPUTES the decomposition through the same production function the nightly job uses; it does not filter a cached result. Endpoint: /api/basis/{sku}/timeseries?exclude_providers=azure,gcp.

2. THE BOUND. XGBoost on ~45 observable features, day-based expanding-window CV, a 10-day untouched holdout, a hard-failing permuted-target leakage guard, and day-demeaned R² for parity with the ANOVA. Result: negative gap: the rich model, tested out-of-sample, explains less than four coarse factors claimed in-sample.

3. HOST ICC ≈ 0.55, Vast on-demand panel only, ≥10-day tenure primary, recomputed at ≥5 and ≥20 with a 0.15 swing alarm. It says the effect is persistent; it does not say why.

Limitations: published in full
FILE NO. 00-1173-M/06

What the Bureau cannot see

Read with care
L1: Quotes, not transactions

Every figure rests on quoted list prices. Real transactions likely compress the disagreement; by how much is unobservable from public data. The remainder is therefore a lower-bound-style claim, not a settled market fact.

L2: Unlike generators

A marketplace of thousands of independent sellers and a catalog with a handful of posted lines are not like-for-like sources. Selection differs between them in ways the method does not model.

L3: Coverage

The record begins in April 2026 and runs on scheduled equipment (EC2 systemd timers). Days a provider failed to answer are holes, and stay holes; they are never back-filled into the public series.

L4: What is left in, by design

Reliability scores, interconnect, datacenter tier and other continuous qualities are deliberately not subtracted: normalizing them would require subjective mappings that are more opinion than measurement. They remain inside the remainder.

L5: No error bars

Every figure is a point estimate. A five-quote day and a five-hundred-quote day are stored and plotted identically; no confidence interval is published anywhere. This is the objection a hostile reviewer should lead with, and the Bureau prints it unprompted.

L6: One loud witness

One marketplace has supplied 68–75% of all filed offers. Under equal weighting, the remainder partly measures that single marketplace's internal heterogeneity wearing the market's name.

L7: Bundle is marketplace-only

The bundled vCPU / RAM / storage factor is measured for marketplace listings only; catalog providers contribute none and are filed UNKNOWN. Its position last in the order limits, but does not remove, the consequence.

And what the remainder is not

Not volatility. Not arbitrage. Not overpricing, nor a savings estimate, nor a claim that any two offers are perfectly substitutable. It is unexplained cross-sectional disagreement within one SKU, one day, one stated population; nothing more is asserted.

Letters to the editor

Readers ask; the desk replies

“Sirs: If the same chip is quoted at twice the price across town, has your Bureau not discovered free money?” - A READER,

The Desk replies: No. Unexplained is not exploitable. Two offers at different prices may not be substitutable for reasons the record cannot see: availability, latency, contract terms, trust. The record shows disagreement; it does not show a trade.

“Sirs: Is your ‘remainder’ not simply prices bouncing about, as prices do?”

The Desk replies: No. Every comparison is between quotes recorded on the same day. The remainder measures simultaneous disagreement, never movement over time, and it is neither an overpricing charge nor a savings estimate. It is the portion of same-day disagreement that survives every observable explanation.

THE INTERNAL BULLETIN · VOL. XXIV, NO. 3 · FILE 00-1173-MNOT A PRICE AGGREGATOR · NOT A DERIVATIVES ENGINEEVERY FIGURE: VALUE · POPULATION · DATE · SOURCE
DECLASSIFIED: AUTHORIZED FOR PUBLIC RELEASE