The Engineering Manager

My laptop is going out of the window

That email.

It’s been watching you all day. Lurking.

You’ve skirted around it, and you’ve turned your attention to other things: to Slack conversations, to pull requests, and even to writing that API documentation that’s been dangling at the bottom of your to-do list for weeks.

But it’s still there.

And now you have nothing else to distract you.

You open it. A wall of text appears on your screen. It’s even got proper formatting and numbered lists. What on Earth is this gigantic essay all about?

You feel your energy escape from your soul via your eyes, sucked into the event horizon of wordy dialogue that must have taken the author hours to write.

You sigh and begin reading through it.

Time passes.

By the time you get to the end, you’ve lost your train of thought and have forgotten the points that you wanted to write in your reply. You scroll right up to the beginning of the email and start reading it again.

Like before, you reach the bottom of the text, but – surprise, surprise – it’s taken so long to read that you’ve forgotten what you wanted to say.

You sigh again, but for longer this time.

Frustrated, you try a different technique.

You hit the “Reply” button so that you can write your response in sequence as you read through it. The in-line reply window opens, and now you can’t fit your response and the original email on the screen at the same time: no matter which way you resize it, all of the text and boxes dance around like a marionette.

Your brow furrows and you scratch the back of your head.

Feeling inspired, you open a new window, side by side with the original window, so you can fit the email on your screen and concurrently compose your reply at the same time.

So far so good.

You slowly read through the wall of text, making bullet point notes on it as you go. There are about six main talking points that you’ve extracted, and you expend some effort in polishing them up to form a solid narrative. You read, then re-read what you’ve written.

It looks reasonable. You sound intelligent. That makes a change.

You move your mouse cursor towards the “Send” button. As you click, you notice a message popping up at the bottom of the screen.

2 new message(s) in thread. Click here to show.

Ah, damn it.

You click to show the messages. Two more walls of text. You read them both.

The first reply has said everything you’d already written. Why did you bother? At least your opinion has been validated.

The second reply is quite confused. You don’t think they’ve read the original email properly, and they’ve tailed off into the realms of the bizarre, almost as if they’ve translated the text into Spanish, then into French, then into Klingon, then back into English.

What are they on about?

A blip sound.

1 new message(s) in thread. Click here to show.

You click.

It’s the original author’s out of office email.

A blip sound.

1 new message(s) in thread. Click here to show.

Oh, wonderful: your colleague who was very slow to write their initial reply has just chimed in on some points in the first email, taking things in a completely disparate direction.

You begin scanning the content to understand why they didn’t quite get the point.

1 new message(s) in thread. Click here to show.

Click.

Out of office again. Sigh.

You rest your head in the palm of your hand.

This conversation makes no sense any more, and you’ve been sitting here reading, writing and peeling apart these replies for 20 minutes.

You consider what your employer’s insurance is like on their equipment, and whether it covers your laptop “accidentally” launching itself out of the window in a bid for freedom.

Smash.

There’s a time and place for email

Let me begin by stating that email is brilliant. I love email.

It’s archival, the threading system works well, and since Ray Tomlinson sent the first ever message in 1971 (to himself, as a test – allegedly it may have read “QWERTYUIOP”), I would argue that email has done a fantastic job of bringing the world even closer together. The asynchronicity bridges timezones. Whole businesses are run via email communication.

Yet, there are times that email isn’t as good as other forms of communication. But let’s stay positive and look at what it’s good at first:

• Archival notices. Since email hangs around forever, and since that it is easily searchable, email is perfect for making timestamped announcements that everyone will see and refer back to.
• Newsletters. In my experience, whatever may feel like oversharing rarely is received that way. Sending out regular newsletters to your team, department or company is an excellent use of email to inform and increase your visibility.
• Conversations with a narrow focus. Emails that cover one concise topic can allow people globally to contribute, assuming that the purpose of the message is to solicit opinion.
• Follow ups to ratify decisions. After having a meeting or a decision point, a follow up email is a perfect way of putting in writing what has just happened so everyone is aligned.

So that’s the good stuff. But what’s email bad for?

• Conversations with many active authors. The little story above is obviously an exaggeration, but “hot” email threads with lots of active participants begin to feel like a series of sliding doors. Everything gets confusing, communication is poor, effort is wasted, and nobody gets anything done. Consider a flurry of email thread activity as a signal to jump in a Slack channel, or do a video call, or start a shared document.
• Anything requiring a quick response. Email isn’t like a DM, and comes with no guarantee of timeliness of response. People have very different approaches to their email. Some batch, some practice inbox zero, some simply get so much that they forget to reply. If you want a quick response, send a DM or make a call.
• Topics that have many layers of context. Extremely complex subjects with many sub-contexts become extremely difficult to reply to. The email format doesn’t support levels of nesting without being fairly creative. Maybe another medium is better. If you feel like you need a deep breath after reading something complex, suggest another medium to discuss it further.

If you’re going to write an email for the reasons above, don’t. Please save others the pain!

Help your reader out

As well as using email for the right purpose, there are some ways that you can be courteous of the fact that when your recipients open your message, they are giving up their time to you.

For non-trivial email content, you can specify the actions that you want the readers to take, even if those actions are just to read it and do nothing else. The recipient, upon digesting the first couple of lines, can rest assured that even if a big block of text is coming up, they need only understand it, and not compose a short essay in response. They’ll thank you for it.

Additionally, for longer emails you can provide a short summary at the top to ease the reader in gently, or let them make the decision as to whether they want to read the whole thing or not. You may even find that as you write the summary, you can delete large parts of the proceeding text as they’re not needed after all.

And, if after all, it isn’t something that is best suited to being communicated via email, you could maybe try some alternatives:

• Having a discussion in a private meeting room
• Walking over to someone’s desk to ask them a question
• Creating a Slack channel or group DM
• Writing your thoughts in a shared document and soliciting comments
• Going for a walk around the block to chat about it
• Chatting about it over lunch
• Just getting on with something using your best judgement

There’s plenty of alternatives that you could be doing rather than firing up Gmail.

So, in summary

Be a good email citizen. Otherwise this laptop gets it.

Mnemosyne

Recently, we redesigned the main storage architecture of one of our products.

We did so by writing a distributed bitmap index of our own that was exactly suited to the type of searches and aggregations that users perform frequently. My colleague, Phil Messenger, who served as the lead engineer on the project, gave a technical talk at Devoxx 2018 about how the system was conceived and designed.

While this project was running, and despite my absolute trust in the team’s technical ability to deliver it, I couldn’t help feel that familiar nag in the back on my mind: are we definitely sure that this hasn’t been built elsewhere? We were convinced that it hadn’t, but it is always hard to be one hundred percent sure.

Since it was a new distributed system, development and production rollout was challenging and proceeded at the cost of developing less customer-facing features. This reality was a harder sell to our product team, as our application, at the time, was still playing catch-up in the market.

In retrospect, the storage system has been a resounding success. It is exponentially faster than our previous Elasticsearch index, scales linearly with usage, and has future-proofed the next few years of growth for this particular product.

But building something from scratch, rather than composing it from existing open source technologies, is a risky strategy. What if we’d missed an existing technology when we did our research? What if it had turned out much harder than originally expected? What if we couldn’t get buy-in from management and manage expectations accordingly?

How did we know we hadn’t we fallen under the spell of Not Invented Here syndrome?

Not Invented Here

Not Invented Here, or NIH for short, is a mindset that embodies a reluctance to reuse or purchase existing products or technologies because they originate from the “outside”. The NIH mindset rejects perfectly capable solutions that lie beyond the boundary of a company, department or team.

But why?

In technology specifically, NIH highlights situations where engineers side with reinventing the wheel due to a belief that building technology in-house will be better, quicker and cheaper than using an existing solution. In many cases, and especially with the current wealth of excellent open source software available for use, this is an anti-pattern and a big, bad smell to be picked up by management.

Assuming a suitable technology exists, why not use it, even if it does require a bit of modification? After all, isn’t software reuse the biggest lever that great engineers can use for getting the job done quickly and reliably?

Well, not always.

NIH as a differentiator

In 2001, Joel Spolsky wrote an article outlining his experience at Microsoft where the Excel team would reinvent the wheel in order to reduce the external dependencies in their software. They even had their own C compiler. The result? Excellent cross-platform compatibility and a small executable. He argued that when working with an outstanding team, reinventing the wheel can be a differentiator.

He concluded the article with a recommendation:

If it’s a core business function – do it yourself, no matter what.

Sound advice.

After all, as Joel states, if you’re a Web accounting company, then you should probably write your own accounting software. If you buy someone else’s accounting engine and re-skin it, you’re not really adding much value, and if you’re a technology company without valuable technology, what are you?

Your decisions are often smaller

So far, so good. You’ll want to make sure that you build the technology that supports the core business.

But as a manager, or as an engineer, you’re going to be thinking about whether to reuse or build components at a much more granular level. Yes, you may be writing your own Web accountancy software in general, but of which components should it be composed?

Should you use this storage system, or that one, or should you write our own? Should you use this CSS pattern library for your UI, or should you start writing one yourselves?

When Joel’s article was written 17 years ago, the open source community had not seen the surge in popularity and progress enabled by centralization around communities like Github. There’s more code to reuse than ever before, and it is much easier to find.

I would assume that reuse discussions in 2001 had a slightly different focus: a decision to possibly choose between implementing an established project from Apache or GNU, or between purchasing a system from another company, or building it from scratch.

I can see how many decisions to build rather than reuse could have stemmed from high licensing prices, since SaaS was in its infancy – Salesforce was just 2 years old – or from lack of faith in open source projects, because they moved slower.

Paralysis of choice

Although now we have the luxury of being able to find lots of software to reuse, there is another problem: which should we choose? There is a huge search space: one can choose from many open source implementations at differing levels of maturity, or from numerous paid-for SaaS offerings with a matrix of pricing options, to on-premise installations, or, of course, one could build their own. Argh!

Additionally, companies that operate with a closed-source model such as Google and Facebook also publish academic articles and white papers detailing how their technologies work. The algorithmic challenge or architectural quandary that is a team is facing may have also been solved before. Some papers are reimplemented as open source projects in their own right, such as Apache HBase which is based on Bigtable, but for those that aren’t, what search keywords do we use to find the paper containing our answer?

Despite the difficult problem of choosing, assuming you can find it, it’s highly likely that there is software to reuse that will solve your problem, albeit with a learning curve and possible modification to that software. For the business, this is great: it means things get done faster.

The correlation with staff happiness

I am a backend engineer by training, and the landscape of storage technologies has never been more diverse: hundreds of databases, each with unique design properties, exist in relatively mature states, actively maintained and often associated with notable technology companies that happened to be the first to solve this problem in the open. Unless you are working in an industry that is wrapped in red tape, such as finance, it’s likely that there’s always going to be something out there that does the job for you.

But what sort of effect does this have on the job satisfaction of developers? Karl L. Hughes argued that the bulk of software engineering in 2018 is just plumbing; that most production engineers are responsible for composing multiple pieces of existing technology in order to solve a business problem.

This, of course, means that the number of unique problems that one gets to solve from scratch, such as the challenges set in Computer Science coursework, or the fundamentals that were worked on in the halcyon days of computing, is greatly reduced. For the most creative and talented engineers, this can lead to boredom. Yet another CRUD API attached to Postgres to drive some JavaScript widget? Yawn.

Some NIH will give your best staff something to really get their teeth into. A chance to create. A chance to contribute something original. An incentive to remain at your company. Under which conditions should we allow original and innovative projects to occur?

Why we build from scratch

There are some areas of the computing industry working up against the cutting edge on tough, innovative problems. Consider the current race to produce the best hardware to support the needs of machine learning at scale, and the software frameworks being written to best utilize that compute power.

This new paradigm of computing has a lot of low level, challenging and exciting problems to solve. Take Google’s TPUs (specialized hardware for machine learning) and TensorFlow (a software framework for machine learning):

• They are solving a problem that has clear market value and customer need. The current generation of data scientists and machine learning practitioners need the software frameworks and performant compute power in order to do their job more effectively. Their ability to do so drives innovation in most, if not all areas, of our industry, creating value everywhere.
• They are attempting to solve a problem at scale. Demand goes far beyond what can be serviced by existing, non-specialized hardware and software. A scalable solution requires rethinking the problem and building something from scratch.
• Demonstrable innovation encourages further buy-in. Continuing to develop the best hardware and software for doing machine learning is a large factor in encouraging engineers to choose GCP as a cloud provider, increasing its stickiness in organizations, making Google even more money.

The case for building from the ground up here is clear, and that’s all well and good: but you may be reading this and coming to the conclusion that your company is doing nothing remotely as interesting as solving the future machine learning needs of the whole industry (and yes, that is probably true; not many companies are working on problems at this scale outside of the FAANG group).

However, these principles can guide your approach to knowing the right inflection point to build something from the ground up.

When you should build something yourself

As mentioned previously, it’s been 17 years since Joel’s original post on NIH. The industry has changed: more technology is out there to reuse than ever before, and it is more mature and better supported. Hardware is also cheaper, meaning we can get away with less optimized implementations, because that slightly more powerful instance on AWS only costs us a little bit extra.

One could argue that the problems that many software creators face require less innovative solutions: many venture-backed companies are centered around solving a unique disruptive business problem with technology, rather than solving a unique technology problem.

In order to unlock the door to being able to build innovative technology yourself, it is important first to prove that your product has clear product-market fit and that you’re reaching the scale point at which existing solutions no longer work. Before this point, it is extremely hard to justify the time and money spent reinventing that wheel into something more specialized.

In my experience, the path travelled before reaching a NIH inflection point looks something like this:

1. While proving product-market fit, reuse as much as possible. If the product hasn’t proved itself, reuse, reuse, reuse. Time is of the essence. Even if you know that Postgres deployment won’t scale, you haven’t earned the customers yet to deserve the opportunity to build something more innovative. It just isn’t worth the investment.
2. After proving product-market fit, creatively scale your existing components as much as possible. During an initial growth phase, see how you can stretch the existing technologies. Even if you still know that singular Postgres deployment won’t scale, running a heavily sharded set up with your own load balancing is much less development effort than building your own storage system.

From that point onwards the choice to build your own technology will stem from one of two situations:

1. A proposed high-value feature cannot be achieved by off-the-shelf components. Here, the future of the product simply cannot be supported by any existing reused solution. You have no choice but to roll your own.
2. Your scale begins to push up against what feels like the limit for a given technology. Here, for example, predictions for the next 18 months of usage will highlight the current system not being able to run fast enough, and you can’t find any examples of other installations coping at that level of scale. Here it’s easier to bet on yourself, as you have superior domain knowledge of what your customers are trying to do, compared to adapting an existing generalized system even further to undocumented boundaries in production.

Going it on your own

If you find yourself in one of the two situations described above, then this is an exciting moment in time. You’re pushing up against a problem at a scale that requires real innovation. You’ve also got a product that’s performing well enough to be worth the investment of effort. You’ve earned the ticket to go it alone.

This, however, is where it starts to get hard:

• Progress will be much slower than reusing existing software.
• Progress will be much less predictable as you repeatedly hit tough challenges and unknowns.
• Progress will be much less transparent and understandable by non-technical people in the business.

But, at last, you’re working on something truly innovative.

Good luck.