Deterministic ingestion pipelines are a feature, not just plumbing

I have worked on imports where spreadsheets, file drops and API payloads arrived with different shapes and quality. Separate scripts and manual steps made it difficult to locate a failure or know whether a retry would repeat a write.

The design I worked towards gave those sources a common lifecycle, explicit validation and a useful record of each run. This account focuses on the approach; internal datasets, measurements and test results remain private.

Standardise the lifecycle

The source formats differ, but the pipeline can still follow the same sequence:

1. Accept the source and identify the run.
2. Normalise the input into a common shape.
3. Validate it against the relevant business rules.
4. Transform it into persistence-ready records.
5. Persist the validated result.

This makes source-specific mapping a contained concern. It also gives the team a consistent way to discuss where an import failed.

Validate before persistence

A failed run is easier to investigate when invalid rows have not already been mixed into shared tables.

I used validation before persistence to make the failure boundary explicit. Partial acceptance can be a valid product decision in other contexts, but it needs equally clear rules about which rows were accepted and what a retry should do.

Report an error someone can act on

A useful validation error identifies the source, file, row, field and failed rule. The operator should be able to locate the input and decide whether it needs correcting or whether the mapping or rule needs to change.

For illustration, a report might have this shape:

Source: supplier feed
File: example.csv
Row: 42
Field: episode_title
Rule: required canonical value
Result: empty after trimming

This is an illustrative format, not an extract from a production run.

Structured logs and correlation IDs connect that failure to the rest of the run. They reduce the need to reconstruct what happened from disconnected messages.

Separate configuration from shared behaviour

Per-source configuration can keep recurring imports understandable without duplicating the pipeline.

The boundary is useful when a new feed can reuse the lifecycle while retaining an obvious owner and its own mapping.

Define what a retry means

Idempotency requires a domain-specific answer to “is this the same work?” A run identifier helps, but the persistence rules also need to say what is new, already processed or intentionally replayed.

I want those decisions explicit before retries become part of an incident. The operator should be able to understand what rerunning the import will do.

The result is an import process that is easier to inspect: validation happens before writes, failures identify the relevant input, and replay behaviour is part of the design.