Stack: PySpark
-
Why my Glue jobs cost 4× more than they should have
A partition that was never pruned, a read that scanned everything, and the bill that made it obvious.
-
AWS medallion data platform
End-to-end batch platform: raw CSV and JSONL to bronze, silver and gold in Postgres — 6 PySpark Glue jobs, 4 Airflow DAGs, fully Terraformed.