The Engineering Manager

git merge

In the last four weeks, I’ve made a transition from having my line manager based in the same office, which has been a situation I’ve been used to for all of my professional life, to having them be remote. In my case this has happened because of the merger of Brandwatch and Crimson Hexagon. The CTO of the combined company is now based in Boston, and I’m in Brighton, England.

I have a VP Engineering role, which, silly job title aside, means that I have a division of the Engineering department reporting to me, focussed around building our Analytics and Audiences applications. We have other divisions of Engineering focussed around our infrastructure and compute, our data platform and the Vizia product. At the time of writing, I have 38 people in my division.

I’ve been fortunate to have always had the CTO in the same office over the recent years. As the company has continued to grow at a fairly fast pace, I’ve had local support. Ideas, thoughts, gripes: they’ve been there in the same place or on the same timezone.

There have been a number of benefits to having the leader of the department co-located:

• My staff have been able to get to know him easily. We’re all just around most days. This makes them feel connected all of the way up the chain with minimal effort.
• The general narrative of what’s going on, such as happiness, morale, stress levels, has been observable by both myself and my manager.
• If there’s ever a crisis – of people or of production systems – then, most of the time, 35 steps is all I’ve needed to get some counsel or a second opinion.

However, things are now quite different.

After our companies merged, the CTO role was given to the Engineering department leader in the other company, putting myself in an interesting position:

• I now have a manager who is not in the same physical location, so I lose out on all of the informal in-person contact that I had before.
• My manager is now 5 hours behind me, meaning I have less times of the day in which to speak to him.
• The new CTO doesn’t initially know me or any of my people; only what we’re responsible for. The rest is a black box.

Letters across the pond

Over the last few weeks, as was expected by the merger, we’ve both been very busy, both with logistics and with traveling. Our weekly hourly 1 to 1s often end before we’ve managed to cover everything off, and then we’re sliding into another meeting before clearing all items on our agenda, which has been frustrating.

Because these weekly catch ups didn’t seem like enough time, and because email chains typically devolve into stasis, I started writing a weekly digest which I send each Friday afternoon. The idea was that I could take some time to properly summarize everything that was going on in my world and flag anything that I needed help with. 

This has been working really well. 

I write it in a Google Doc, which means that a lot of the smaller items can get covered off asynchronously via the comments. Larger items that are worth spending some more time on become the focus of our conversation in our 1 to 1, and that more precious face to face time is spent on the meat of the main issues, rather than on the periphery. Both of us enjoy written communication too, so this works very well. It also gives us an ideal chance to poke fun at our Britishisms and Americanisms.

Here’s roughly what I cover in the weekly document. It takes me about 30 minutes to write:

• Any interesting developments in any of the ongoing work streams, such as new links to demos, updates on estimates, or anything particularly good or bad that’s unfolding.
• The latest on what’s next in the project pipeline from conversations with Product.
• The general feel within the teams, such as happiness and morale. Are any of them overworked, or, on the contrary, spinning the wheels while waiting for a decision on the next thing? Are the teams right sized and is this looking true for the coming months?
• An in-depth look at anything that’s front of mind right now, such as hiring, or thoughts about backend architecture and scaling, or contemplations over cool ideas we could pitch to the Product team.
• A list of “documents of interest”, such as designs for upcoming features or architecture, or the fortnightly product and engineering updates that get sent out. I don’t expect any of these to be read in detail, but they’re there to satisfy any curiosity.
• Occasionally a light sprinkling of GIFs. Because life’s too short to not use that one of Kermit furiously slapping the typewriter.
  

Soap opera rather than novel

I’ve been trying to open up my black box as much as possible to give my new manager a view into the decisions that I make on a day to day, and to allow my thought processes to be observed and discussed. However, the style of writing was challenging at first: how do I make the digest interesting and not a labour?

Given that my new manager was taking the role of the reader and I was the author, I didn’t really know where to start or how to collate my thoughts. But then I came to realize that it wasn’t my job to be the creator of a novel, thoroughly documenting everything that happened. Instead I needed to take the position of a screenwriter of a soap opera: an inventor of a regular rolling feed of narrative that is easy to soak in, letting the reader learn the characters and plot lines gradually by osmosis.

Tuning into The Wire halfway during Season 3 can leave you feeling a little lost and overwhelmed by the detail, but switching on Eastenders a couple of times during the week allows you to (assuming you want to…) follow along pretty easily. I decided to be more Eastenders, except with less arguing and fighting in the Queen Vic.

I scatter the document with parts prefixed with “Your thoughts please…” where I’d like to get some input. We usually chat on the comments around these parts.

Getting comfortable with async await

Although I thought that the experience may be more jarring at first, I think that I am getting better with a predominantly asynchronous relationship. 

There can be some benefits to having a remote manager, after all:

• Because our face to face time is more valuable, we prepare more for when we do talk, meaning that conversations are rewarding.
• We do a lot of written communication, which allows us to think more deeply about what we’re saying and how we’re saying it before presenting it to one another.
• We have to continually operate from a place of trust, since we cannot easily insert ourselves into each other’s worlds to observe and come to our own conclusions. I like this.
• I feel like I have to step up and represent my people more, in terms of my personal accountability and in promoting their cause, which can only be a good thing.
• The introduction of even more extreme timezone differences across the now global Engineering department means we need to get better at being a company that supports flexible remote working, fast. I would like to think that being forced to break our predominantly European timezone habits will make it easier for us, in time, to hire people remotely all over the world.

But, still, a quick chat in the kitchen is nice, and is missed.

Have you ever had a colleague complain about there being too much process? Or pointed out, powerlessly, that an existing process is pointless? I would posit that you have. In fact, it may have been you doing the complaining. 

I’ve certainly been there. 

I’ve been there in times where my jaw has dropped when opening up the description of the deployment process for an older part of the system, and in times where I’ve torn my hair out as a simple approval for a flight has gone round and round in endless repeating circles.

“Surely it must be easier than this,” I hear you cry and hear myself cry.

Yes, processes can be extremely annoying. But they don’t have to all be annoying. And, in fact, they should do more help than harm.

What is a process anyway?

Process is a very abstract term: in the dictionary, which I have just opened, it is defined as a series of actions or steps taken in order to achieve a particular end. In business in general, this could be the process in order to get some budget signed off or to hire a new person. 

If you’re an engineer reading this post, then the word process may make you think about the steps that your JIRA tickets take from being created to being complete, or the way that deployments of your live application are done, or even the forms you need to fill in for your annual review.

Regardless of whatever the end is that the process is there for, the definition of that process is meant to make that end result more replicable, predictable and consistent regardless of the individual(s) that execute it. 

Process as code

You could think of a process as the code that the company has written about how to do a particular thing. Everyone else following that process is executing it to achieve the same desired, consistent result.

But, as all programmers know, code evolves over time. Old code gets stale and misses out on the latest updates and features. The original authors of that code have also since learned more and become better programmers since they wrote it.

Have you ever gone back to code you wrote years ago and wondered what on earth you were doing at the time? The same is true of processes. Companies can grow, shrink, mature or completely reform: you can’t expect the processes to stay the same as a result.

Processes, like code, shouldn’t be set in stone. They should be revisited, tweaked, refactored, rewritten and deleted. 

Arguing against processes

If processes don’t change despite being inefficient, then you’ll need to work on getting them changed. But why do processes get left alone and not continually updated?

Sometimes a business won’t want to change processes because those in charge are lazy: they’d rather not have to invest any effort into something that works well enough, regardless of how inefficient it is in practice. Tut.

Additionally, a process can often become orphaned because the creation of that process is to ensure that knowledge can be handed over (e.g. the process of deploying the live application) or to ensure the consistency expected by a particular individual (e.g. the annual review process as defined by the head of HR). As time passes, the relevance of those processes fades and need updating. But those who defined them may no longer care as deeply about them, since their attention is now elsewhere.

Still, in the same way that the management of a business would expect their employees to be efficient and diligent with their work, that same management should also expect to continually update the collective ways of working that they have codified in order to demonstrate that they are being efficient and diligent also. 

A process should fundamentally serve those that are continually applying it. If those closest to the ground want to make it better, then they should absolutely be allowed to change that process for themselves. It’s likely they know more than the original authors about the latest and greatest way to get that work done.

Let chaos reign, rein in chaos

Still, even the best processes don’t have their place. There is also a good argument for process to not exist.

When a team is embarking on innovative or unknown work, process seems only get in the way, constraining ways of working and thinking. Chaos often breeds innovation, so why should we stifle it? But, on the other hand, if there’s no process, aren’t we all going to hurtle into the sea in a fiery ball?

There’s a neat quote that suggests a way of dealing with these situations:

Let chaos reign, then rein in chaos.

Andy Grove

Often during innovative or chaotic periods, it is best to let go of wanting to control things and let them unfold in a natural way, even if it seems like bedlam.

Chaos often gives birth to a new way of working; an emergent practice which can be the new best way of tackling a problem. After people converge on this new way naturally, then the emergent practice can be codified into the new process. Chaos is transitory, but necessary. Things will stabilize eventually.

A practical example is running R&D projects. Often truly innovative developments need space and lack of constraints for those that are working on them to explore all aspects of the problem. Too many meetings, checkpoints, reporting upwards and real-world engineering constraints can kill creativity.  

Although it can be deeply uncomfortable, especially for directive-driven managers, to think that an investment into R&D is “not being managed”, letting go of the situation and trusting staff to do things the way that they want to can, paradoxically, get better results.

As the R&D project begins to move into a prototype and requires more input from designers and engineers, then chaos can be reined in. 

But not before.

So, in summary

Process is good when it creates predictable results. However, like code, processes must be continually updated and revised to stay relevant, and the updating should be done by those closest to the process: the staff actually executing it. 

Creative and innovative projects can benefit from having no process to let chaos become an emergent behavior which will eventually become the new way of doing things. Let chaos reign, then rein in chaos.

Now, according to my own process, this is where I hit “publish”.