INT-NONEMPTYL1 Integrity
28,678 files checked · 28,678 pass ✓ all clear
What: Every file is > 0 bytes.
How: Stat every tracked file and flag any whose size is 0 bytes.
Fix: Re-fetch from the source (re-run the fetch_*.py script), confirm a non-zero size, re-push. Investigate why the original write produced 0 bytes.
SCH-PARSEL3 Schema
1 files checked · 0 pass1 fail
What: CSV/parseable files decode as UTF-8 and parse with a header row.
How: Attempt a UTF-8 decode and a CSV header parse of each parseable file; flag files that fail to decode or have no header.
Fix: Re-export with UTF-8 + a header row; fix the fetch script's write step.
SCH-RAGGEDL3 Schema
5,551 files checked · 5,551 pass ✓ all clear
What: Every data row has exactly the header's column count.
How: Parse each CSV and compare every data row's field count to the header count; count the rows that differ.
Fix: Re-export with proper RFC-4180 quoting; verify column counts are uniform before push.
QC-ROWCOUNTL4 Content
5,551 files checked · 5,551 pass ✓ all clear
What: Actual row count matches the catalog's declared row_count within tolerance.
How: Count data rows and compare to the catalog's declared row_count within tolerance; flag drift.
Fix: Re-fetch and compare; if the change is real, update row_count + CHANGELOG. Large unexplained drops are truncation — re-download.
QC-PKUNIQUEL4 Content
5,551 files checked · 5,467 pass84 fail
What: The primary/join key is unique (no duplicate keys).
How: Count distinct primary/join-key tuples versus total rows; duplicate rate = 1 − distinct/total (must be 0).
Fix: De-duplicate at the cleaning step or correct the key definition; document the canonical key in DATA_DICTIONARY.md.
Flagged files (e.g.): cde_arrest_agency_AL_v1.0.csv.gz, cde_arrest_agency_AR_v1.0.csv.gz, cde_arrest_agency_DC_v1.0.csv.gz, cde_arrest_agency_DE_v1.0.csv.gz, cde_arrest_agency_FL_v1.0.csv.gz, cde_arrest_agency_KS_v1.0.csv.gz …
QC-NULLRATEL4 Content
5,551 files checked · 4,941 pass610 warn
What: Per-column null rate is below threshold (default 20%); a NULL in a key column is a BLOCKER.
How: Compute the null percentage of each column; flag any column above the threshold (default 20%); a null in a key column escalates to BLOCKER.
Fix: Investigate the source for suppression/coverage gaps; document expected nulls in the catalog notes, or re-fetch if the gap is an extraction error. Null key values must be resolved before use.
Flagged files (e.g.): cde_arrest_agency_HI_v1.0.csv.gz, cde_arrest_agency_MS_v1.0.csv.gz, cde_arrest_national_v1.0.csv.gz, cde_arrest_state_v1.0.csv.gz, cde_pe_agency_AL_v1.0.csv.gz, cde_pe_agency_AR_v1.0.csv.gz …
KEY-FIPSL5 Keys
1,693 files checked · 1,693 pass ✓ all clear
What: state_fips / county_fips values are valid US FIPS codes.
How: Validate every state_fips/county_fips against the canonical US FIPS set; report the percentage of valid codes (must be 100%).
Fix: Correct the FIPS mapping via the committed county_name→FIPS crosswalk; re-run the FIPS-derivation step with the 100%-coverage hard check.
CAT-COVERL2 Provenance
934 datasets checked · 930 pass4 warn
What: Every data file in the repo is covered by a catalog entry (a file entry or a directory-summary entry).
How: Diff the repo's data files against catalog coverage (a file entry OR a directory-summary entry); list every data file with no coverage.
Fix: Add a CATALOG.md entry for the file, or extend the relevant directory-summary entry to cover its folder. Untracked data must not feed downstream projects.
PROV-FIELDSL2 Provenance
934 datasets checked · 921 pass13 warn
What: Each active catalog entry has all required fields: description, source_url, source_tier, retrieved, update_cadence, geographic_granularity.
How: Parse each active CATALOG.md entry and test for the 6 required provenance fields; report which fields are missing per entry.
Fix: Edit the CATALOG.md entry to add the named missing field(s). Values must be transcribed from the real source — do not guess.
QC-PKUNIQUE-KEYL4 Content
706 files checked · 631 pass75 fail
What: The dataset's OWN declared key (validation spec `key:`) has no duplicate tuples — independent of whether whole rows are byte-identical.
How: Resolve the dataset's declared `key:` columns from its validation spec (scripts/lib/spec_discovery.py), count distinct key tuples versus total rows; duplicate rate = 1 − distinct/total (must be 0). No declared key, or a declared column missing from the header, yields no measurement (None) rather than a false 0.
Fix: De-duplicate at the cleaning step, or correct the declared `key:` in the validation spec if it does not match the data's actual grain. A dataset with no declared key is UNMEASURED (None), not exempt — add a `key:` to measure it.
Flagged files (e.g.): oecd_stri_country_year_sector_v1.0.csv, medsl_precinct_2016g_AK_v1.0.csv, medsl_precinct_2024g_AK_v2.0.csv, medsl_precinct_2022g_AL_v1.0.csv.gz, medsl_precinct_2022g_AZ_v1.0.csv, vest_precinct_2016g_AZ_v1.0.csv.gz …
