Measured 2,400 CI runs across 24 large projects: the median test workflow finishes in 2.2 minutes
Everyone talks about slow CI, so I measured what large open source projects actually run. The last 100 completed runs from 24 well-known repositories, split by workflow rather than lumped per repo.
24 repositories, 315 distinct workflows. That is thirteen each, and it was the first surprise: nobody has one pipeline any more.
The median test-shaped workflow takes 2.2 minutes. Automation workflows (labelling, stale bots, triage) take 0.2 minutes and are 14 percent of all runs.
The median is not the interesting part though. The tail is:
- deno
ci: 69.5 minutes - rust
CI: 67.6 - numpy sanitizer build: 30.5
- prometheus
CI: 28.2 - rollup
CI: 19.8
19 of 81 test workflows run over ten minutes. So most CI is fast and a fifth of it is where all the pain lives, which matches how people talk about it: nobody complains about the 2 minute job.
Retries, from run_attempt: 25 of 315 workflows contain at least one rerun. The concentration is what stands out: deno ci at 10 of 48 runs, tailwindcss Integration Tests at 9 of 43. Around a fifth of runs on those two, and zero on most others.
Caveat: this measures wall clock from run start to completion, so runner queue time is included. That is what a developer experiences, but it is not pure execution time.
@retry_button_addict · 2w ago · 2 replies
The retry concentration is the finding I would pull out. A fifth of runs on two projects and near zero elsewhere means flakiness is not a fact of life, it is a property of specific suites.
That is encouraging in a way people rarely frame it: if most large projects run at zero, the ones at 20 percent have a fixable problem rather than an inherent one.
Reply
Report
@counted_315 · 2w ago
Both of the high ones are integration suites hitting real services, which is the usual suspect and it is nice to see it fall out of the data.
Reply
Report