Shopee · Python · Data quality
Shopee Seller Center Reconciliation
Shopee data from the platform API lands in Snowflake, but I still needed a repeatable way to check whether that warehouse data matched what the business could export from Seller Center. I built recon_engine.py so Python and pandas could perform the bulk comparison without consuming Claude or ChatGPT tokens, leaving AI for exception investigation instead of row-by-row reconciliation.
The problem: repeated comparisons were burning LLM tokens
To validate the Shopee pipeline, I needed to compare data that came from the platform API and landed in Snowflake against a Seller Center export. I initially used Claude and ChatGPT to help with those comparisons, but large row-level checks consumed too many tokens and were not a practical way to run the same reconciliation repeatedly.
The decision: move deterministic comparison into Python
The row-by-row comparison did not require an LLM. It required consistent normalization, matching rules, and repeatable classification. I built recon_engine.py with Python and pandas to handle that work locally.
| Order / SKU | Seller Center | Snowflake | Classification | Analyst action |
|---|---|---|---|---|
| SP-1042 / A12 | Paid | Paid | Match | None |
| SP-1043 / B07 | 10:14 | 10:17 | Timing difference | Explainable |
| SP-1044 / C31 | Complete | Missing | Needs review | Investigate source |
| SP-1045 / D09 | Present | Gap date | Source-coverage gap | Check coverage |
What the reconciliation is checking
The Seller Center extract acts as the business-facing reference point. The Snowflake side represents data ingested from the Shopee platform API. Comparing the two helps me check whether the API data is landing correctly in the warehouse and identify where differences need investigation.
Not every mismatch means the API data is wrong. Differences can come from timing, field transformations, missing source coverage, or a genuine data discrepancy. The script keeps those cases separate instead of returning one undifferentiated mismatch list.
The comparison: run the workflow at real scale
I ran the reconciliation across 107,160 rows so I could evaluate the normalization and classification logic at the scale of the actual comparison workload.
- Compared the same business keys across Seller Center and Snowflake.
- Normalized fields before evaluating equality.
- Kept timing differences, missing-record cases, and source-coverage gaps separate.
- Used the resulting exceptions as the smaller set for deeper investigation.
rows compared through a deterministic Python/pandas workflow instead of sending the full dataset through an LLM.
The result: use AI for exceptions, not bulk comparison
The script moved the repetitive, deterministic part of reconciliation into code. That means the full Seller Center versus Snowflake comparison can run without consuming AI tokens, while Claude or ChatGPT can still be useful later when a smaller set of unusual mismatches needs investigation.
The project became a reusable data-quality check for the Shopee ingestion path rather than a one-off comparison exercise.