Data Engineering & Analytics

Repeated file processing can occur when arrival notifications are duplicated, a completion marker is missing, or a retry starts before the prior run is recorded. What to check Compare file identifiers, checksums, run IDs, object timestamps, and job retry history. Check whether a partial failure leaves the source file available for another trigger....

Reports may restate a prior month when a source system records a transaction after the period closed. Whether that is correct depends on the reporting rule and close process. What to check Check event time versus processing time, the report cutoff, late-arrival policy, refresh history, and whether corrections are included in prior periods or the...

Automated checks may validate that data is present and well-formed without confirming that it represents the intended business meaning. A technically valid value can still be mapped to the wrong category or period. What to check Review metric definitions, transformation logic, source ownership, joins, and expected business examples. Compare report...

Deduplication can remove legitimate records when its matching key is too broad or when the chosen survivor rule is unclear. Row counts alone may not reveal which information was lost. What to check Review the deduplication key, survivor logic, source IDs, and before-and-after samples. Check whether records with shared names or addresses were treated as...

A report may include cancelled orders if its filters use a status field that changes later, or if cancellation events are not reflected in the reporting model. What to check Compare the metric definition, order status history, cancellation timing, warehouse refresh, and filters in the report. Trace representative orders through source and analytics...

Exports can contain duplicate headers when joined tables share field names or a report aliases columns inconsistently. Downstream spreadsheets and integrations may then map values to the wrong field. What to check Inspect the export query, source schemas, aliases, and report templates. Compare the header row with the fields consumed by the receiving...

Identifiers may be unique only within one system or may be recycled over time. Joining records on the ID alone can combine unrelated people and distort customer history. What to check Confirm identifier scope and lifecycle, source system, tenant, and effective dates. Trace affected joins and compare with trusted attributes without exposing personal...

A report may show zero when a filter excludes the records, a refresh failed, or source values were mapped to a different category. The displayed number can look valid while hiding a pipeline issue. What to check Check report filters, date and timezone boundaries, refresh status, field mappings, and source-to-warehouse counts. Trace a few records...