Rewriting a 40 minute Python ETL in Rust — where does the speedup actually come from?

I did the rewrite you are describing. Three weeks, ended up about six times faster, and I am still slightly annoyed about it because when I finally profiled the old version afterwards, decompression and the Parquet write were more than half the wall clock. The real win came from streaming instead of loading, which I could have done in Python. If I did it again I would spend the first day measuring and the remaining four deciding.

Try the one week option that is not a rewrite first. We had a very similar job, 6GB of gzipped CSV into Parquet with a dedupe, and moving from pandas to Polars with lazy scanning took an afternoon and cut it from 31 minutes to about 7. Most of that was not clever, it was that we stopped materialising the whole frame in memory and stopped doing the dedupe as a groupby over an object dtype column. If that gets you under your window, you have spent an afternoon instead of a week and you can still do the Rust version later with a known baseline.

Where Rust actually wins on this shape of job: you can stream records instead of building intermediate frames, you allocate far less because you can parse into borrowed slices rather than a String per field, and you can use every core without dancing around a global interpreter lock. What it does not win on is decompression, disk, or the Parquet writer, which are the same libraries underneath either way. So the ceiling is set by how much of your 40 minutes is actually spent in parsing and hashing rather than in IO.

Nobody has asked the only question that matters, which is where the 40 minutes goes. Run it with a profiler, or if you cannot, just time the stages by writing timestamps between them. I have seen exactly this job three times and twice the answer was that a single gzip stream was being decompressed on one core at about 100MB/s and no rewrite in any language was going to fix that. The fix in both cases was splitting the input into chunks that could be decompressed in parallel.

Measure the stages. Try Polars or DuckDB. Rewrite only if the profile says parsing and hashing dominate, and only after you have a baseline you can regression test against.