Engineering Management - Tools Are Top Priority
Share
This is the fourth part of a series of notes regarding my thoughts during my time in engineering management at Facebook. Read the intro here.
Tools are top priority
The core premise of any technology-driven company is that it is providing a tool that makes human life better, faster, more effective. Driving the effectiveness of that company and especially its own technology operations is the effectiveness of its tools.
Over and over, the successive waves of new startups with structurally more advanced operations (think Google with the massive compute power of its datacenters, or startups which can build an entire website with just a small team) are enabled due to an advance in tools. Computerized tools, unlike physical tools, can stack their leverage arbitrarily upon each other, compounding their leverage to enormously high levels.
Hence, your operating efficiency, and thus the number of people you need to hire, and therefore your costs, are directly impacted by the ingenuity of your internal tools. This means that your tools teams should not be a back-office, after-thought function staffed with second-string players. Your most talented engineers should be working on your tools, and your culture must reflect this priority. Writing great tools and continuing to improve and replace them is more important than the next shiny feature.
Example 1: At Facebook, there was a time (2005 thru mid-2006) when we hired customer service people at a constant ratio with user growth. When we had 10M users, we had less than 20 people in customer service. As we climbed towards our first 100M users, it was clear we could not just staff this up by 10x, so we charged our internal tools team with working closely with our customer service analysts to build ever more innovative tools and user interfaces to improve the efficiency of our customer service department by orders of magnitude. Today, we have only ~3x more customer service people (i.e. around 60-70), serving well over 325M users. There is no external company, off-the-shelf product, or management consulting strategy that could have yielded an order-of-magnitude gain in personnel effectiveness; it was the work of a small internal tools team that analyzed the work being done and created custom tools to streamline it by automating the parts that a computer could do quickly while optimizing the experience so that the human analyst could concentrate on what humans were best at.
Example 2: Well past the time when we had hundreds of database machines (and tens of thousands of database instances spread across them), instead of hiring DBAs, we had a single database engineer who administered all of those machines by writing ever more effective tools. Today Facebook continues to manage thousands of databases with only a handful of database-oriented personnel. The original "second DBA" was even only hired because eventually our first database engineer decided she wanted someone else to cover the night shift.
The quality of your tools and your ability to continue to evolve them will allow you to suppress the need to hire for operational roles, allowing each front-line individual to do more, which simultaneously improves overall coordination (fewer people means coordination is easier) and keeps costs down. Today, the results of Facebook's engineering leverage ratio means that there is one engineer for every 1.2 million users and despite our blistering user growth, the ratio is growing. While some of Facebook's best engineers work directly on front-line features, a large percentage of them work on internal abstractions and tool frameworks that end users never see, but which vastly magnify the effectiveness and power of their fellow engineers.
If your culture reflects this priority, you'll have no problem getting your best engineers to work on improving tools. The best engineers tend to go where they have the greatest impact.
Today, a mere decade after the first dot-com boom, the web division of the New York Times uses more advanced technology and gets more done with fewer people than many of the original titans of Web 1.0. If your technology company today is a leader in anything, it is because it was likely started with the latest in tool technology. You will not retain this position without a culture that values an ongoing first-class investment in your internal tools, because if you don't, not only will your competitors who do pass you by, so will the entire market.