The Engineering Manager

Expansion: the aftermath

The company did brilliantly two years ago. But now they’re saying you’re slow.

Last year you managed to open up nearly 40 headcount in Engineering and you managed to fill those roles with talented people, almost doubling the size of the department. Yet, another year on from there, others in the business are left wondering what effect that had.

Shouldn’t have you been able to dramatically ship more software with all of these extra people? Where are all of the features that you were previously unable to prioritize because you didn’t have the staff to do so? Why is it that a small competitor is seemingly moving faster than you?

If you have managed a department through a period of rapid growth, and especially if you have aspirations of doing so in the future, you will experience this scrutiny. You know that simply adding people doesn’t make everything faster, but what can you actually do about it? How can you get others in the business to understand?

Internal and external scrutiny

Let’s divide the source of the pressure into internal and external scrutiny.

• Internal scrutiny is probing from inside the business as to what has happened to your output after your department expanded: after all, even adding just tens of people, depending on their seniority, can represent a yearly investment from the business of over £1M. They’d like to know where that money is going and what it is being used for.
• External scrutiny is the perception of the business in the marketplace after it has seen you grow. Maybe there was a PR piece explaining how your company is continuing to expand after a stellar year, or maybe you raised VC capital. Those that track the marketplace and your competitors will observing your business to see proof of what that growth has delivered. No visible progress may suggest to the outside observer that there are problems. You don’t want to look weak to your market.

The unfortunate truth is that even if you are shipping often, you will always face questions about your output, both internally and externally. As a leader, you have two weapons against the dark forces of criticism for being slow.

The first is making sure that you’re building things as quickly and efficiently as you can, given your current circumstances and resources. That’s a given. The second is making sure that you are communicating transparently and often about what you’re doing. Engineers are often very good at building awesome things the right way, but often not as good at communicating what they’re achieving, hence the scrutiny.

Before we look at some techniques you can use, why is it that you might be naturally getting slower as you get bigger?

It’s not people, it’s productivity

Let’s clear something up first. Arguments around perceived speed can become ad hominem very quickly, and that’s a route that you do not want to go down. While it is true that bigger departments can mean that performance issues can go unnoticed more easily, it isn’t a small handful of individuals that are making your whole department slow. It’s about being a victim of your own success, having to deal with legacy systems, and the increased communication overhead of having ever more people. The more accurate way to phrase your perceived slowness is that the productivity per head is decreasing, and this is the problem that needs addressing.

So what makes this productivity drop? Let’s revisit those three areas.

1. Being a victim of your own success

As brilliant as it is to do well, it also creates more work. You will have more customers expecting a higher level of service. You’ll have to create or expand your support team. You will have to tighten up the security of your system. You’ll have to improve your monitoring, build up your out-of-hours rota, and so on. Additionally, you will be continually storing more and more data which requires continual expansion and upgrades of storage systems. You may need to rewrite whole parts of your application estate so that they scale. An ever-expanding array of work requires attention just to keep things running the way that they currently are.

2. Legacy systems

Being a victim of your own success also means that you’ve created a lot of code that is fast becoming legacy. It is impossible to predict future needs, so it is therefore impossible to create infinitely extensible systems. A new product direction, new set of features or complete pivot can highlight that your current codebase is going to need dirty great hacks (bad) or a complete rewrite (also bad) to survive.

Even if you are able to continue extending your current codebase, the ever-increasing size means that it takes longer for your engineers to understand where best to fit new features, longer to write tests that actually cover all end-to-end scenarios (if that’s even possible) and longer for new changes to be peer reviewed. There’s the added pressure that having a well-established paying (with money and/or data) customer base means downtime really isn’t acceptable any more, increasing the collective fear of making sweeping changes quickly.

3. Communication and process overhead

When you’re a small start-up, the entire business can make unilateral decisions at speed as they can sit in the same room. When you’re bigger, global, and have clearly defined departments, roles, and responsibilities, decisions can converge towards a glacial pace. This isn’t limited to business decisions: technical decisions can also suffer. A decision on how to overhaul the main codebase could affect hundreds of engineers. How do you make sure that enough people are consulted? This can mean more processes and meetings are required for consensus to be reached. This can get slow, but it’s unavoidable.

Without clear processes, meetings and steps for how decisions are made, life can get very messy. The Mythical Man Month famously outlines the effect of increasing numbers of staff on potential lines of communication. When n people need to communicate amongst themselves, there are n(n-1)/2 channels*. For example, a 400 person company has 79800 communication channels(!) We need meetings and processes to tame this, but not too few and not too many. Getting the balance right is hard.

So what can you do?

We’ve painted a pretty dire situation. What can you do about it?

There’s no silver bullet, but as a leader you only really have two tools at your disposal for ensuring that you’re going quickly enough: ensuring that you’re all building software in the best way that you currently can, and making sure that your progress and decisions are communicated as transparently as possible.

1. Develop software pragmatically

You’ll need to make sure that your engineers are being as pragmatic as possible when it comes to building and maintaining your application estate. As the company grows, so will the codebase, and the risk of it getting gnarly increases dramatically.

To begin with, coach a culture of refactoring as you go along. I like the boy scout metaphor, where each engineer should try and leave an area of the codebase they are working on cleaner than when they arrived. The key in this approach is the scope: just improve the area that you are working on. You wouldn’t expect the boy scouts to improve the entire forest after camping there. Refactor a clear path and move on through.

Actively encourage technical debt being tackled. Perhaps reserve 10% of the time in your sprints to working on technical debt issues, and as a leader, champion this decision. Encourage your staff to all advocate how important technical debt removal is for the long-term success of the department. Related to this: never be afraid of deleting features. Did it not ship fully? Delete it. Did interactions with that area of the product wane significantly over time? Make the hard call and get rid of it. It will only slow you down in future.

2. Have small purposeful teams

Zooming out from the code, you’ll also want to make sure that your teams are right-sized. They need to be big enough and with the right balance of skills so that each team can make progress on their project or feature with as little outside support as possible, yet small enough so that decisions happen quickly. A team that gets too big can become impossible to plan for, both in the quantity of work and in sticking to a theme that unites them.

Each team should have a clear purpose and reason to exist. Perhaps they build and maintain a particular product or feature (i.e. their success is customer-facing) or they are purely building infrastructure (i.e. their success is serving the department). Either way, teams with a clear purpose will have an intrinsic motivation to go fast because they will be self-motivated. They will require less hands-on management as a result.

3. Communicate transparently and often

As a leader in the department, you should not expect that if you are having a stellar quarter that the rest of the business is going to know about it. Part of your role is ensuring that everyone else is informed on your progress. You owe it to your staff to share the great things that they are doing.

There are a number of techniques that you can employ here:

• Broadcast. You could publish a regular newsletter, written for a non-technical audience (e.g. the commercial mailing list), describing the progress in your team, division or department. It sounds simple, but in my experience, people really engage with these communications and enjoy reading them. Especially so if they are light-hearted and entertaining.
• Invite outsiders in. Always give other departments the chance to engage. Invite them to be stakeholders in your projects and demos. They will then better understand the pace of progress and the difficulty of technical challenges.
• Be social. Hold regular catch-ups with those in your network to discuss everything that you’ve been up to. Ask their advice candidly as a peer and friend.
• Offer Q&A. Ask to drop in on existing non-Engineering meetings where required as a guest. You could even set up your own regular session with the rest of the business.

The idea here is that as the company grows, the noisier the company gets. You need to ensure that even if you are doing great work that people know that you are. No matter where you work, you will probably face accusations of being slow, so you need to make sure that the rest of the business is making that judgement based on the facts that you present rather than just assumptions about what you’re up to.

In summary

It’s not that people get slower as a company gets bigger; it’s that productivity per head decreases when those staff are part of a larger organization. As a leader, it’s important for you to make sure that your teams are developing software in the most pragmatic way possible and that their progress and achievements are transparent to the rest of the business. If you are open, accessible and just plain amicable as a leader, you’ll find that the internal scrutiny will develop into camaraderie as you are able to prove that you’re doing the best that you can with what you have.

The knock-on effect is that those who question your speed can engage with you in conversations that criticize the right things (i.e. resources, scope, prioritization) rather than the wrong things (i.e. not “working hard enough”, performance of individuals), as the right things can be collaboratively discussed and altered where necessary, without judgement.

*This is the same number of edges in a full mesh topology. Humans and computers have more similarities than you think!

Work harder! Work faster!

Has anyone ever asked that of you and your team? How did it make you feel? Did it motivate or demotivate? Cause inspiration or decline?

Software engineering, like many other forms of technological and scientific work, can be opaque in technique to those that are not skilled in it. Unlike watching a stonemason carve a statue, it can be difficult to observe how work is progressing. Just what exactly are all these people doing all day? They look pretty relaxed, right? Can’t they just work harder and get this over the line quicker?

Crunch periods are well-documented in the video game industry. They create tales of heroics, but more often, woe and burnout. Crunch is that time on a project when you are staring at the ever-decreasing gap between the current uncompleted work and the deadline, and the team realizes that the only way to get there is to stress out, sleep in the office, and neglect life outside of work and their health in order to get the damn thing done. Based on that description, you can probably tell that I don’t like crunch. It breeds the wrong kind of culture. However, a crunch situation is well-defined. The deadline is set and it cannot be moved, and you’ve gotta get there. You’ll know when you’re there because you can ship it and stop.

But what about when there is no particular deadline and there is a feeling from others that your team isn’t delivering fast enough?

The spotlight

Let’s imagine that the CEO pulls you aside and tells you that she thinks your team just aren’t working hard enough. She rarely sees any of them staying as late as everyone else, nor getting in before them. They spend a lot of time playing table tennis and visibly having fun together, but she feels they aren’t producing enough output to really move the needle for the business.

The spotlight is on you now, and it’s really uncomfortable. As their leader, you will inevitably be expected to employ the stick in order to save the team’s reputation and your own. If you are new to the game, then you may be wondering why this is being asked of you and what you’re going to do about it. If you can’t use the stick effectively with your team, then that same stick might just drag you off the stage.

Let’s explore.

The mismatch

Be objective and consider the other side of the argument: why were you and your team asked to work harder or work faster? What is it exactly that the outside observer feels that your team are lacking?

Here are some of the things that I’ve heard throughout my career about why a team isn’t meeting expectations:

• Lack of visible output: The team hasn’t delivered anything that the observer has seen as worthwhile for a given period of time. This could be because the team has had a poorly defined project, has been subject to unrealistic expectations, bad luck, actually not working hard enough, poor prioritization over other things that are actually more important, and so on. It could be because they are doing a lot of behind-the-scenes work with infrastructure. The mismatch here is that a level of perceived visible output has not been met.
• Lack of hustle: There may be observations that your team is not getting in until after 9:30 and is going home way before 5:30. They might be supposedly spending a lot of time goofing off rather than working. They might just not look that busy. Either way, there is a mismatch on the expected hustle that a team is expected to be showing.
• Lack of purpose: The team are observed as not showing enough passion for their work. They may have been observed as saying the project is stupid or pointless, and that if they could make the decision, they wouldn’t bother doing it. They might just be really quiet and look uninspired by what they’re doing. There is a mismatch between the importance of the work in the eyes of the team and the observer.

Looking at some of these perceived behaviors, you may decide that the best course of action is to get the team in a room and have a good shout at them for not being good enough, or fast enough, or passionate enough. But that version of applying the stick just doesn’t work. You’re a clever person and you should be smarter than that.

Bad stick

We can make some observations about software engineering which are true of many creative and scientific professions.

Firstly, the people that are on your team are working in an industry where demand for talent dramatically outstrips supply. If any of your engineers quit, they could easily have multiple job offers within days. Given that this is the case, you cannot scare someone into working harder through making them fear for their job: they could just get another one. On top of this, applying the stick too harshly will just make them leave. They could easily find a job elsewhere that is less stressful.

Engineers are also self-motivated. They are not doing this job because they have to. They are doing this job because they want to. What exactly motivates them can vary greatly from person to person: some enjoy optimizations to make things faster, some enjoy building customer-facing features, some just love problem-solving. But the reasoning is all the same: the many years of difficult education and training to become a good engineer was not done through gritted teeth: it was done through a genuine curiosity and passion for the craft. Getting too aggressive with the stick will get in the way of them doing a good job, which is what they are motivated to do, so they will inevitably leave.

Viewed through the lens of the in-demand, self-motivated engineer, you can see how when the stick is applied badly, with poorly defined logic, it inevitably causes problems for you as a manager. Setting fake deadlines is a terrible idea: they’ll see through your deception. Telling them to work harder and faster with no clear reason or purpose will make you look stupid.

Good stick

True leadership to increase throughput comes through fostering purpose and passion in your team. When your engineers are clear on their purpose in the organization and how they can move the dial for the company, they will intrinsically perform better. When they are passionate about their work, they will work harder and faster and later because they enjoy doing it. Coaching a team towards this is much more difficult than just telling them to work harder and faster whilst being angry about it. It requires emotional intelligence, understanding of how they work and what motivates them, and the confidence and oratory skills to deliver that message so that it inspires.

Some techniques to motivate a team about the importance of their work include:

• Making the benefit clear: Most engineers that I have worked with are self-motivated to do work that benefits others. As their leader, you have the duty to frame the benefit of what you are building and why. Which users will benefit? Is it all, or is it some? How much money do those users bring into the business? Are your customers actually other engineers because you are overhauling a part of the infrastructure that will make future work faster, easier and more efficient?
• Creating clear feedback loops: Decide on KPIs to measure work against, whether that is driven purely by quantitative (i.e. clicks, load time, average throughput) or qualitative (i.e. NPS surveys, user interviews) data. The quicker that this feedback can get to the team, the quicker that they can see that their work is making a difference and be inspired to do more of it.
• Collectively deciding on trade-offs: If you are in a position where a tricky, late project is putting pressure on a team to deliver, then be open and honest with them about it. You typically work with three main factors: scope, resources and time. These are your levers that you can tweak in order to get a project over the line more quickly. Instead of analyzing the situation and making a decision on your own, why not involve the team so that they fully understand the current situation and feel empowered to work with you to do something about it? It’s more effort, but it has a greater long-term benefit.

If your team is aligned and have clarity on how they can make a sizable impact, you probably won’t find yourself in the situation described earlier in the article where their output is being questioned. But if you are, you should be able to articulate the benefit of what the team are working on clearly to the observer, and then work with the team to motivate them to put in that bit of extra effort: reiterate the impact that their work is having and press how vital they are in making it so. Work with them on the means to measure this if necessary (e.g. by using HEART metrics or by rallying inspirational stakeholders around the team).

In summary

Figuratively beating engineers with the stick to tell them to go faster and harder without reason is going to compromise the respect that you have earned from those that are working for you. If it was possible to simply “go faster”, then your smart engineers would have already worked out how to do so. The skill in leadership is to run your team(s) so that they have a clear purpose and that they understand how to move the dial for the business. They will then be self-motivated to drive a project towards completion in the best way that they can.

You may need to involve yourself in discussions with them about how to effectively pull the levers of scope, resources and time in order to keep everything on the right track, but you should primarily concern yourself with how your team can have an intrinsic motivation to get the job done whilst giving them autonomy, mastery, and purpose. Anything less than that makes you no smarter than a program that you could write to send your team an email once a day telling them to hurry up.

You’re better than that, right?