Marketplace report ingestion engine
Go on AWS, one Terraform apply per client
Lambda stages under Step Functions, report types defined as YAML mappings, a fake vendor API for tests, Parquet output, and one Terraform apply per client. Oversized jobs hand off from Lambda to Fargate, and multi-GB files were cut about 20× in memory before loading into DuckDB.
Stack
- Go
- AWS
- Terraform
- Parquet