Understanding the Fundamentals of Test Infrastructure

Build & CI/CD Systems

You need an automated hands-off way for code to be put through the paces to measure quality. Code must be built and tested by pipelines automatically when it hits key parts of your repo. There are strategies for when/how things are built. The key is there is a canary in the coalmine to tell the team when standards are not met. This runs in a shared area, the build system, not on a dev’s machine. This is how everyone’s work is checked.

Your job is to make sure this system provides a net value. There is cost in maintenance and set up and waiting for pipelines to run. Those must be outweighed by the confidence these tools provide. Different platforms will have different specific ways to build resources and orchestrate pipelines. You need to be able to figure out the best bang for your buck. Not only is there cost associated with time delays between merging code and it getting deployed, but there are monetary costs with runners. Don’t make a pipeline so convoluted that a whitespace change leads to an hour of waiting for pipelines that ultimately have to be re-run again because a rebase is needed before it’s finished. These suck. Make it so you can build and test things quickly with valuable feedback across different systems. Jenkins, GitHub Actions, CircleCI, Buildkite, Bitbucket Pipelines… these are where you orchestrate this.

Test Frameworks & Tooling

Hold my beer. There are different frameworks to work with: pytest, cypress, jest, and many more. These have their own peccadilloes, but one thing is universal: good testing is good testing. Test one thing. Test without environmental chaos. Tests should be quick and require minimal set up. Fast feedback is vital. Complexity is the enemy. Use factories to create fixtures. Tests should have the same result regardless of the order in which they’re run or the time or other effects. Hermetic fixtures always. Mock everything you aren’t testing (file system, clock, randomization, network). Good tests are good tests. How you get there depends on the framework, but they all have similar ideas. Mock, inject, tell don’t ask, small units, fast feedback. Use the right tool for the job, but also be creative when you can’t change tools. Write custom reporters, write plugins. If the tools are clunky, devs won’t use them. If testing is hard, devs won’t do it. If tests are brittle, they’ll be disabled or abandoned. This is the front lines of testing, TDD or integration. This is the hotness and you can show off all your tricks and strategies… but you are not the one who will write these tests. You are an advisor in this capacity. You need to provide the tools and frameworks and guidance and patterns that enable devs to write good tests. You can’t do it for them.

Infrastructure & Reliability

Confidence is why we test. Flaky tests undermine trust, which erodes confidence. In addition to flaky tests, there may be flaky pipelines caused by infrastructure issues. Maybe you don’t have enough runners to handle all your pipelines. Maybe there’s contention competing for a shared resource like a test db. Ironing out these issues to avoid failures based on infra instead of “bad code” is part of the job. Pipelines and test architecture must be robust and dependable. In addition to consistency and reliability, performance can be impacted by infra decisions. Measure runs to keep timing history, shard tests and eagerly bin-pack to distribute the load for fast parallel runs across clusters. Provide docker images for reproducing runs locally the same way they’re handled in CI/CD pipelines (avoid works on my machine, make it easy to repro exactly). Provide consistent mocks/test data. Make sure you can count on your pipelines and tests to be deterministic and un-flaky.

You need to be able to monitor these. Logging and metrics and alerts give you feedback on pipeline health. Use them well with a mindfulness on signal/noise ratio. Life is not fair and sometimes you need to retry. Self-healing pipelines is OK sometimes, but don’t use it as a catchall for delicate tests. Finally, we have constraints. You need to work with ops to figure out how many resources are available to you. Use them wisely (budget, runners, etc).

Developer Experience

Devs need things to fit into their system. Things should be quick. Things should be easy. Things should be clear. Provide patterns. Provide boilerplate. Provide bespoke tooling (reporters, linters, etc). Make it quick to run a single test. Make it easy to TDD. Make feedback clear and concise. A test failure should have everything you need: What failed, what the parameters were, how to reproduce (completely), what was expected, what was received. Make everything dummy proof. Cognition should not be wasted on the tools. The tools should feel comfortable and natural and transparent, allowing devs to focus on the code. Make reporting from pipelines clear. Provide dashboards on pipeline health, historical data for test results, highlight and auto-quarantine flaky tests, show timings, provide all the data needed for devs to test easily with high value results. Ingratiate yourself and the entire test stack into their environments without being onerous or dogmatic.

Listen to your devs.

Culture & Strategy

Lots of devs hate tests because their only experience has been with bad tests. When done well, devs will love the confidence they provide. Don’t dictate, demonstrate. Make a compelling case for the value you bring. Build a culture where testing is desired, not endured. If you tell the story of how tests will provide confidence and show that it can be done without strangling devs with slow build times and red herrings and chasing coverage they will want tests. This is where you do a lunch and learn and a tip of the month and a “check this out” thing. Make it cool and nerd out. Proselytize and spread the word. Some people will get it.

Importantly, speak upward, too. Don’t just hype up the ideals of testing, gather data and show it to the organization that pays you. Identify metrics used to measure success, gather data, and show how things change with your ideas and tools. Tell the story of how the test infrastructure makes a more reliable product and how it catches things early. Don’t test for the sake of testing. Show people why we are testing and align with their goals. Honestly, if it’s a small org with a high tolerance for slop, you don’t have a reason to be there.