The trials and tribulations of a CI pipeline on Apache Infra
summary
Apache Cassandra runs ~40,000 tests per commit across two CI fleets: commercial CircleCI for pre-commit and an ASF-hosted Jenkins cluster ci-cassandra backed by CloudBees and ~50 donated agents. Builds push Docker test images to Docker Hub (using docker buildx for multi-arch) and use nightlies.apache.org to store logs and artifacts. Challenges include many flaky tests (often 100–200 occurrences), static-heavy code that resists unitization, heterogeneous black-box agents, Docker rate limits, and flaky ARM support. Continuous fixes and monitoring gradually improve green-run rates.
AI-written summary · about
chapters (15)
- 0:00Introduction to CI challenges
- 3:43Cassandra 4.0 changes and needs
- 7:25Testing in Cassandra project
- 13:18Challenges in distributed testing
- 15:01Codebase issues affecting testing
- 16:51Resource limitations in open source
- 18:58Comparing CI systems and flakiness
- 21:26Stable trunk development goals
- 22:51CircleCI cost and workflow issues
- 24:15ASF Infra and donated hardware
- 27:27Agent capacity and monitoring
- 28:55Human factors and trust in CI
- 30:39Nightlies and log management
- 32:27Community contributions and credits
- 33:50Future of Cassandra testing
related talks
26mFaster, Cheaper, Leaner: Horizontally Scaling a CI Pipeline
13mProviding Continuous Integration at CERN
15mDocker image testing in GitLab CI
01h 02mContinuous Integration: Tips & Tricks
01h 03mYet Another Continuous Delivery Talk
45mHighway to Continuous Integration
13mEvolution of containers in continuous integration
45mRunning Tests in Continuous Integration