Detecting a squash ball that is 6 pixels wide

The bill arrived before the suspicion did. Three weeks of runs, a workload that had not grown, and a number that had — which is the shape of a problem that is nobody’s fault and everybody’s to find.

What the plan actually said

Reading the physical plan is the part people skip, because it is long and because the logical plan already looked right. The logical plan is a description of intent. The physical plan is a description of what happens.

df = spark.read.parquet(path)
df = df.filter(col("event_date") == target)  # after the read, not during

Two lines, in that order, and the partition column may as well not exist. The filter runs after everything has already been read, which means the cost is paid before the predicate is ever evaluated.

The fix, and what it did not fix

Pushing the predicate into the read path is a one-line change and it recovered most of the difference. It did not recover all of it, and the remainder turned out to be a second problem wearing the first one’s clothes — small files, thousands of them, each with its own open and close.

A cost regression is rarely one thing. It is usually one thing that made a second thing visible.

Compaction on write fixed the rest. The job now costs roughly a quarter of what it did, which is the same as saying it costs what it always should have. The maxRecordsPerFile setting is doing most of the work.1

---

Footnotes

1. Roughly 128MB per file on this dataset. Larger and the reads stop parallelising usefully. ↩