Seller analytics report pipelines
Contributions to an AI-enabled marketplace seller analytics platform, centred on
Contributions to an AI-enabled marketplace seller analytics platform, centred on its multi-GB report ingestion.
- About 20x memory cut by reshaping to NDJSON before DuckDB
- Constant-memory streaming of 7 GB+ JSON into partitioned Parquet (ijson, pyarrow)
- Lambda handing oversized jobs to Fargate
- Load benchmarks and a written post-mortem
Stack
- TypeScript
- Python
- NestJS
- Next.js
- Turborepo
- Serverless Framework
- TimescaleDB