Replacing a manual cross-system reconciliation process with an automated daily workflow.
Backend Engineer
2026
Go / Kafka / PostgreSQL
Production
Two independently-owned systems recorded the same transactions on different schedules. Any drift between them was caught, if at all, by a manual spreadsheet comparison when customer complains.
By the time a discrepancy surfaced, it had already reached a customer, company's reputation is at stake. The manual process took hours and depended on a DBA pulling ad-hoc queries.
No downtime on either source system was acceptable, and the reconciliation job could not add meaningful load during business hours.
Looked at streaming CDC versus a scheduled batch diff. Batch won: the discrepancy tolerance was measured in hours, not seconds, and batch was far simpler to reason about and operate.
A daily Go job reads both systems, computes a diff, and writes discrepancies to a queue for review — no DBA in the loop, no shared database access.
Kafka carries the extracted records; PostgreSQL stores the diff results and history. The job runs unattended and alerts only when it finds something.
Detected production data inconsistencies before they became customer-facing failures. Detection time dropped from six hours of manual work to about thirty seconds of automated comparison.
Most of the value came from making the failure visible quickly, not from anything clever in the diff algorithm itself.