The problem
A cleaned file alone hides decisions. Operators also need to know which rows failed, why they failed, which duplicate survived, and whether rerunning will create new copies.
Case study 03 · Python · data quality
Clean output plus evidence of what was rejected.
The pipeline validates a messy CSV, normalizes accepted records, applies an explicit duplicate-survivorship rule, loads idempotently, and generates a stakeholder-readable quality report.
Public demonstration using synthetic, mocked, or fictional data; not a paid-client result. Counts below describe the included sample run.
4/4focused tests
5sample rows loaded
2sample rows rejected with reasons
// try the browser demo
Messy rows in
The sample has mixed casing, three date formats, a currency string, duplicate order IDs, a missing email, and an invalid amount. Edit it. Break a row and it will tell you which line failed and why; change the later duplicate’s amount and watch which row survives.
Use synthetic sample data only. Do not enter confidential information, personal data, credentials, or customer records. Input stays in this page and is discarded on reload or close.
Nothing is silently dropped: every rejected row prints its line number and the rule it failed.
—input rows
—loaded
—rejected
Ready. Run the sample through the documented rules — or edit it first.
Public demonstration · synthetic order data · sample rules shown explicitly
// pipeline flow
Ingest→Validate→Normalize→Dedupe→Load→Report
Typed validation
Required identifiers, email shapes, dates, and numeric amounts are checked explicitly. Invalid rows are separated with their original line and named problems.
Normalization and dedupe
Dates, phone numbers, currency, email casing, and whitespace become consistent. Duplicate order IDs use a documented latest-date-wins rule that carries the whole winning row.
Repeatable delivery
Accepted records load idempotently into SQLite, and each run writes a human-readable report. The destination can be replaced without changing the validation contract.
// verified behaviors
8 input → 5 loaded + 2 rejected + 1 superseded duplicate → line-level quality reportThe test suite verifies missing/invalid-field reporting, type normalization, latest-record survivorship, and multiple date formats. Client work begins by agreeing the real rules on a non-sensitive sample.
Need rules applied to your files?
Send one representative non-sensitive sample, the row count, and the desired output format.
contact@metaworld.systems