INT-NONEMPTYL1 Integrity
323,150 files checked · 323,149 pass1 fail
What: Every file is > 0 bytes.
How: Stat every tracked file and flag any whose size is 0 bytes.
Fix: Re-fetch from the source (re-run the fetch_*.py script), confirm a non-zero size, re-push. Investigate why the original write produced 0 bytes.
Flagged files (e.g.): host.json
SCH-RAGGEDL3 Schema
152,182 files checked · 152,062 pass120 fail
What: Every data row has exactly the header's column count.
How: Parse each CSV and compare every data row's field count to the header count; count the rows that differ.
Fix: Re-export with proper RFC-4180 quoting; verify column counts are uniform before push.
Flagged files (e.g.): 44541260.csv, 56286425.csv, 56286428.csv, 56286431.csv, 56286434.csv, ofac_consolidated.apisnapshot.csv.gz …
QC-ROWCOUNTL4 Content
152,182 files checked · 152,033 pass149 fail
What: Actual row count matches the catalog's declared row_count within tolerance.
How: Count data rows and compare to the catalog's declared row_count within tolerance; flag drift.
Fix: Re-fetch and compare; if the change is real, update row_count + CHANGELOG. Large unexplained drops are truncation — re-download.
Flagged files (e.g.): 59758862.csv, 59758868.csv, 59758877.csv, 59758880.csv, 59758889.csv, 59758892.csv …
QC-PKUNIQUEL4 Content
78,842 files checked · 78,309 pass533 fail
What: The primary/join key is unique (no duplicate keys).
How: Count distinct primary/join-key tuples versus total rows; duplicate rate = 1 − distinct/total (must be 0).
Fix: De-duplicate at the cleaning step or correct the key definition; document the canonical key in DATA_DICTIONARY.md.
Flagged files (e.g.): whispers_events_raw.csv.gz, 2qns-g7s7.csv.gz, f8kp-sfr3.csv.gz, k7a5-emiw.csv.gz, mg5r-efcm.csv.gz, qew5-gawh.csv.gz …
QC-NULLRATEL4 Content
152,048 files checked · 152,047 pass1 warn
What: Per-column null rate is below threshold (default 20%); a NULL in a key column is a BLOCKER.
How: Compute the null percentage of each column; flag any column above the threshold (default 20%); a null in a key column escalates to BLOCKER.
Fix: Investigate the source for suppression/coverage gaps; document expected nulls in the catalog notes, or re-fetch if the gap is an extraction error. Null key values must be resolved before use.
Flagged files (e.g.): cdc_fluview_who_nrevss_laboratories_US_v1.0.csv.gz
KEY-FIPSL5 Keys
1,965 files checked · 1,923 pass42 fail
What: state_fips / county_fips values are valid US FIPS codes.
How: Validate every state_fips/county_fips against the canonical US FIPS set; report the percentage of valid codes (must be 100%).
Fix: Correct the FIPS mapping via the committed county_name→FIPS crosswalk; re-run the FIPS-derivation step with the 100%-coverage hard check.
Flagged files (e.g.): StormEvents_details-ftp_v1.0_d1994_c20260323.csv.gz, StormEvents_details-ftp_v1.0_d1995_c20260323.csv.gz, StormEvents_details-ftp_v1.0_d1996_c20260323.csv.gz, StormEvents_details-ftp_v1.0_d1997_c20260323.csv.gz, StormEvents_details-ftp_v1.0_d1998_c20260323.csv.gz, StormEvents_details-ftp_v1.0_d1999_c20260323.csv.gz …
PROV-FIELDSL2 Provenance
5,737 datasets checked · 5,734 pass3 warn
What: Each active catalog entry has all required fields: description, source_url, source_tier, retrieved, update_cadence, geographic_granularity.
How: Parse each active CATALOG.md entry and test for the 6 required provenance fields; report which fields are missing per entry.
Fix: Edit the CATALOG.md entry to add the named missing field(s). Values must be transcribed from the real source — do not guess.
INT-CHECKSUML1 Integrity
5,737 files checked · 0 pass5,737 deferred — content checks deferred to a full run
What: The bytes the battery read hash to the file's recorded identity (git blob SHA for a tracked file; manifest sha256 for an object in S3).
How: While the walk streams a file's bytes, hash them (lib/content_digest.py): git blob SHA-1 for a tree read, SHA-256 for an S3 read. At the end of a COMPLETE read, compare to the recorded identity — the tree listing's blob SHA, or the manifest's sha256. Match = pass; mismatch = fail; a file not read, read in part, or with no recorded identity = not measured (reported, never counted as a pass).
Fix: Re-run the walk; if it recurs, fetch the file directly and compare its hash to the record. An S3 object that differs from its manifest was overwritten or restored from the wrong version — restore the recorded version, or re-deposit and re-record it.
INT-BINARYL1 Integrity
2,714 files checked · 2,599 pass115 deferred ✓ all clear
What: Every container opens: a ZIP/XLSX's end record and central directory are whole and agree; PARQUET carries PAR1 at both ends with a sane footer; a PDF has its %PDF- header and %%EOF trailer; a SHP header's length matches the file.
How: lib/archive_integrity.py, over every container tracked in git and every container object a pointer names in S3. Ranged reads only — for a ZIP the last 64 KiB (one more read if its directory is larger): find the end record (zip64 included), check the central directory ends exactly where the record says, every entry carries its signature, and the count matches. PARQUET: PAR1 at both ends and a footer length that fits. PDF: %PDF- and %%EOF. A tree verdict is carried to the next run only while the file's git blob SHA is unchanged (same SHA = same bytes); an unread container is NOT MEASURED, never a pass.
Fix: Re-download from the primary source and re-validate the archive opens before pushing.
QC-PKUNIQUE-KEYL4 Content
565 files checked · 544 pass21 fail
What: The dataset's OWN declared key (validation spec `key:`) has no duplicate tuples — independent of whether whole rows are byte-identical.
How: Resolve the dataset's declared `key:` columns from its validation spec (scripts/lib/spec_discovery.py), count distinct key tuples versus total rows; duplicate rate = 1 − distinct/total (must be 0). No declared key, or a declared column missing from the header, yields no measurement (None) rather than a false 0.
Fix: De-duplicate at the cleaning step, or correct the declared `key:` in the validation spec if it does not match the data's actual grain. A dataset with no declared key is UNMEASURED (None), not exempt — add a `key:` to measure it.
Flagged files (e.g.): fema_pa_applicants_national_v1.0.csv.gz, fema_hma_mitigated_properties_national_v1.0.csv.gz, grant_witness_terminations_grant_2025_v1.0.csv, samhsa_treatment_facility_locator_national_2025_v1.0.csv.gz, cms_aco_snf_affiliates_national_2026_v1.0.csv, billstatus_bills_117_v1.0.csv.gz …
QC-COVERAGEL4 Content
310 files checked · 0 pass310 warn
What: Every row-table file in the dataset carries content measurements — the fraction actually content-checked is published beside the verdict.
How: Classify every file in the dataset as a row table or not (lib/content_eligibility.content_candidate_file — candidacy, NOT eligibility, so a file excluded by a future size bound stays in its own denominator), then count how many of the row tables actually carry a content block. Reports checked/candidates and bytes_checked/bytes_candidate; flags any row table the walk did not open. A file the walk TRIED and could not read is SCH-PARSE's subject and is not double-counted here.
Fix: Re-run the content walk over this dataset (`validate_data.py --full`, or a domain-scoped run naming its domain). If the file legitimately cannot be measured, that is SCH-PARSE's subject — record the read failure rather than leaving the file silently unmeasured.
Flagged files (e.g.): measles_cases.parquet, measles_long_run.parquet, amendments.parquet, countries.parquet, datasets.parquet, elements.parquet …
