Systems design explains the world: volume 1

• As a Novice going for Junior, you had to prove you could fix bugs without too much supervision;
• Going for Senior, you had to prove you could implement a whole design with little supervision;
• Going for Staff, you had to show you could produce designs based on business problems with basically no management;
• Going for Senior Staff, you had to solve bigger and bigger business problems; and so on.

People who are naturally excellent at glue work often stall out early in the prescribed engineering pipeline, even when they’d be great in later stages (staff engineers, directors, and executives) that traditional engineers struggle at. In fact, it’s well documented that an executive in a tech company requires almost a totally different skill set than a programmer, and rising through the ranks doesn’t prepare you for that job at all. Many big tech companies hire executives from outside the company, and sometimes even from outside their own industry, for that reason.

In systems design, there is rarely a single right answer that applies everywhere. But with centralized vs distributed systems, my rule of thumb is to do exactly what Jo Freeman suggested: at least make sure the control structure is explicit. When it’s explicit, you can debug it.

Chicken-egg problems

Video game console makers (Nintendo, Sony, Microsoft) have become skilled at this; they’re the only ones I know who do it on purpose every few years. Some tricks they use are:

• Subsidizing the cost of early console sales.
• Backward compatibility, so people who buy can use older games even before there’s much native content.
• Games that are “mostly the same” but “look better” on the new console.
• Compatible gamepads between generations, so developers can port old games more easily.
• “Exclusive launch titles”: co-marketing that ensures there’s value up front for consumers (new games!) and for content producers (subsidies, free advertising, higher prices).

The next level up is a two-sided market, such as Uber or Ebay. Nobody can get a ride from Uber unless there are drivers; but drivers don’t want to work for Uber unless they can get work. Uber has to attract both kinds of users (and worse: in the same geographic region! at the same time of day!) before either kind gets anything from the deal. 

The most difficult level I know is a three-sided market. For example, UberEats connects consumers, drivers, and restaurants. Getting a three-sided market rolling is insanely complicated, expensive, and failure-prone.

Second-system effect

Second system effect arises through the following steps:

• An initial product starts small and is built incrementally, starting with a low budget and a few users.
• Over time, the product gains popularity and becomes profitable.
• The system evolves, getting more and more hacks on top, and early design tradeoffs start to be a bottleneck.
• The engineers figure out a new design that would fix all the mistakes we know about, plus more! (And they’re probably right.)
• Since the product is already popular, it’s easy to justify spending the time to “do it right this time” and “build a strong platform for the next 10 years.” So a project is launched to rewrite everything from scratch. It’s expected to take several months, maybe a couple of years, and a big engineering team.

The results are remarkably consistent:

• The project takes longer than expected to reach feature parity.
• The new design often does solve the architectural problems in the original; however, it unexpectedly creates new architectural problems that weren’t in the original.
• Development time is split (or different developers are assigned) between maintaining the old system and launching the new system.
• As the project gets increasingly overdue, project managers are increasingly likely to shut down the old system to force users to switch to the new one, even though users still prefer the old one.

Innovator’s dilemmas

A summary of the Innovator’s Dilemma is as follows:

• You (Intel in this case) make an awesome product in a highly profitable industry.
• Some crappy startup appears (ARM in this case) and makes a crappy competing product with crappy specs. The only thing they seem to have going for them is they can make some low-end garbage for cheap.
• As a big successful company, your whole business is optimized for improving profits and margins. Your hard-working employees realize that if they cede the ultra-low-end garbage portion of the market to this competitor, they’ll have more time to spend on high-valued customers. As a bonus, your average margin goes up!Genius.
• The next year, your competitor’s product gets just a little bit better, and you give up the new bottom of your market, and your margins and profits further improve. This cycle repeats, year after year. (We call this “retreating upmarket.”)
• The crappy competitor has some kind of structural technical advantage that allows their performance (however you define performance; something relevant to your market) to improve, year over year, at a higher percentage rate than your product can. And/or their product can do something yours can’t do at all (in ARM’s case: power efficiency).
• Eventually, one year, the crappy competitor’s product finally exceeds the performance metrics of your own product, and promptly blows your entire fucking company instantly to smithereens.

The dilemma comes from the fact that all large companies are heavily optimized to discard ideas that aren’t as profitable as their existing core business. Any company that doesn’t optimize like this fails; by definition their profitability would go down. So thousands of worker bees propose thousands of low-margin and high-margin projects, and the company discards the former and invests heavily in the latter (this is called “sustaining innovation” in the book), and they keep making more and more money, and all is well.

Fancy expensive databases were the only way to get high transaction throughput, until SSDs came along and made any dumb database fast enough for most jobs.

Complicated database indexes and schemas were great until AWS came along and let everyone just brute force mapreduce everything using short-term rental VMs.

JITs were mostly untenable until memory was so much slower than CPU that compiling was not the expensive part. Software-based network packet processing on a CPU was slower than custom silicon until generic CPUs got fast enough relative to RAM. And so on.

But in the book, it had a meaning. There are two kinds of innovations: sustaining and disruptive. Sustaining is the kind that big companies are great at. If you want to make the fastest x86 processor, nobody does it better than Intel (with AMD occasionally nipping at their heels). Intel has every incentive to keep making their x86 processors better. They also charge the highest margins, which means the greatest profits, which means the most money available to pour into more sustaining innovation. There is no dilemma; they dump money and engineers and time into that, and they mostly deliver, and it pays off.

A “disruptive” innovation was meant to refer to specifically the kind you see in that plot up above: the kind where an entirely new thing sucks for a very long time, and then suddenly and instantly blows you away. This is the kind that creates the dilemma.

If you’re a startup and you think you have a truly disruptive innovation, then that’s great news for you. It’s a perfect answer to that awkward investor question, “What if [big company] decides to do this too?” because the honest truth is “their own politics will tear that initiative apart from the inside.”