Production Oriented Development

1. Engineers should operate their code

Engineers are the subject matter experts for the code they write and should be responsible for operating it in production. In this context, “operating” means deploying, instrumenting, and monitoring code as well as helping to resolve incidents related to or impacting that code. 

2. Buy Almost Always Beats Build

If you can avoid building something, you should. Code is the most expensive way to solve a problem that isn’t addressing a core area of your business. For most small to mid-sized companies, there are open source or better yet, hosted solutions that solve a wide range of common problems.

3. Make Deploys Easy

Deploying should be a frequent and unexciting activity. Engineers should be able to deploy with minimal manual steps and it should be easy to see if the deploy is successful (this requires instrumenting your code for observability, which - tada - is covered above), and it should be easy to roll back a deploy if something doesn’t go well. Deploying frequently implies that deploys are smaller, and smaller deploys are generally easier, faster and safer.

4. Trust the People Closest to the Knives

The people who work with a system are the ones who understand it best. This applies to any part of the socio-technical systems within which we all work. In the case of software systems, the engineers who deploy every day and are on-call for critical services understand the level of risk they operate in.

5. QA Gates Make Quality Worse

Many teams have a manual QA step that gets performed before deploys. The idea, I guess, is to have someone run automated or manual tests to verify that a set of changes are ready to be released. This sounds like a comforting idea - having a human being (or team of human beings) “verify” a release before it goes out - but it falls victim to several false assumptions and creates some misalignments that do more harm than good.

Luckily, solving this one is easy. Instead of having a dedicated QA team work on creating manual and automated test cases that run in a fictitious QA environment, reassign that team to work on continuous testing in production. Instead of being a gate for deploys, a QA team could continuously verify that production is working as expected. QA teams are also well situated to lead Chaos Engineering initiatives, where faults are intentionally injected in production. QA engineers could also work on making the CI/CD pipeline more reliable, so that deploys are no longer a nightmare.

6. Boring Technology is Great.

With thanks to Dan McKinley, always strive for boring tech when possible. Systems are inherently unpredictable, and you want a wide area of expertise to fall back on when shit goes sideways. There are also routine operations that you’ll have to do (deploys, database migrations, etc) and it’s Very Nice to have widely used and tested tooling for this stuff. I think of databases most often when I think about this belief. MySQL is a database with many, many quirks, but it is so widely used, that you should still just use it most of the time.

Very few organizations have the bandwidth to debug unique problems. You don’t want unique problems, especially when performing routine operations - i.e. storing bytes on disk, choosing a new leader in a cluster, garbage collecting objects, querying time-series data, etc. Having unique problems will kill a small to medium size team. It will sap you of your creative energy, which is better used creating value for customers who want to pay you monies for your software. Use your innovation tokens wisely!

7. Simple Always Wins

I don’t have much to say about this, but we’re all writing YAML and JSON instead of XML and we’re all using HTTP instead of CORBA, RMI, DCOM, XPCOM, etc. Right? In that same spirit, I’d rather debug problems in a LAMP stack than a Microservices architecture any day.

Quick sidebar on Microservices: as with so many trends in tech, they are often sold as a panacea. Let me be clear: Microservices, designed well, solve some specific problems and as with most solutions to complex problems, involve several trade-offs.

8. Non-Production Environments Have Diminishing Returns

A more direct heading for this section would be “Non-Production Environments are Bullshit”. Environments like staging or pre-prod are a fucking lie. 

9. Things Will Always Break

It’s impossible, even undesirable, to avoid failure. Lean into the fact that failure is inevitable, and focus on how you respond to it.