Testing The Untestable
summary
A five‑year build pack starts with exactly zero tests and becomes a deployable, testable system by running real git repos against Heroku, measuring deploy logs, and tolerating flaky networks with retries and parallelism. It uses Hatchet to clone and deploy, a reple runner to drive interactive shells, Bundler retry features, Travis for CI, and RSpec plus rspec retry for nondeterministic failures. Result: 44 real end‑to‑end cases run in ~12 minutes, and targeted refactors save ~200ms per run.
AI-written summary · about
chapters (31)
- 0:00Intro
- 3:33What about the real world?
- 4:39What if the spreadsheet is wrong?
- 4:50Testing!
- 5:06Programmers are lucky
- 5:17Our tests are programs
- 5:42Outer Space is a lifeless vacuum
- 7:20Back To Software
- 7:40Task Force
- 8:38What is a buildpack?
- 9:074,573 lines of edge cases of
- 10:15Minimum Viable Patch
- 10:59the Fastest
- 11:17What's the cure for the MVP?
- 11:44Black box testing
- 12:32Real Ruby apps, real deploys
- 13:17clone repo create heroku app
- 16:08Process deadlock is no joke
- 18:50Your tests FAIL
- 21:41Failed assertions cause a test re-run
- 22:33code, use probability to approximate determinism
- 23:25Hatchet is test framework agnostic
- 23:38Run single tests
- 23:56RSpec plugin system is amazing
- 25:34First test: -5 minutes
- 26:34Parallel Rspec Runner
- 28:48Did testing make the buildpack faster?
- 29:21Unit test those components
- 30:40What does a black box Rails test look like?
- 30:57Mock for determinism
- 31:59Start with integration tests
related talks
42mCONTINUOUS DELIVERY FOR RUBYISTS
28mKeeping Developers Happy with a Fast CI
01h 00mTesting Heresies
49mRuby Software Continuously Delivered and Exhaustively Explained
40mNext Level Virtual Machine Maneuver!
54mBuilding and Managing a Ruby Infrastructure
34mShitlist-driven development and other tricks for working on large codebases
52mStory Driven Development: The Next Generation of Rails Functional Testing