My engineering newsletters, or how I share context in a distributed team

One idea I took from Turn the Ship Around!, a book about leading people, was to give them context. People should always have context. Otherwise, they’ll make decisions that optimize only their own corner of the system. Here’s a concrete example to make that less abstract.

We were having trouble keeping our test environments stable. The team knew there was a budget constraint, expressed as the blanket rule “Save money and don’t spend unnecessarily.” They proposed several solutions, most of which meant at least a week of development and further tuning. Reserving hardware in advance for a few dozen dollars solved it permanently in a few clicks. Without context, the team couldn’t see a way past the rule. Of course we should save money, but spending less in one place can drive up costs somewhere else.

There are plenty of ways to share context, depending on how the team is set up and where people work. Our team of more than 35 people works across several cities and time zones. Since I like writing, I send a newsletter regularly on weekends. Here’s what I try to do:

• Summarize the past week and highlight what matters in the week ahead.
• When there’s a problem, such as a release mishap, explain how we’re addressing it at the system level.
• Thank people by name when they go above and beyond.
• When I call something out, I keep it impersonal and focus on the systemic problem and how we’ll respond. It’s also a chance to reinforce what matters in our culture by showing how I’d like us to handle specific situations.
• Make room for small talk, book and article recommendations, and reminders about meetups or conferences where we’ll be speaking or which are worth attending.

Rather than keeping it all in my head, I take notes during the week. At the weekend I turn them into a newsletter and send it to the engineering team.

[The newsletter excerpts have been adapted for private review. Names, internal links and personal updates have been omitted.]

Hi everyone,

I’ll start by thanking a few people.

– Thanks to the teammate who picked up both production and non-production issues from the developer blockers channel. We now have a new dashboard for microservices as well.

– Thanks to the teammate who sped up the build and the unit and integration tests. From 45 to 5 minutes in the best case is nearly an Olympic gold. There’ll be a talk on how to avoid breaking it again. It’s in all our interests to keep from slipping back.

– Thanks to the teammates putting in evenings to move the monolith to Spring Boot.

– Thanks for the Grafana dashboards showing test environment availability and health.

– Thanks to the teammate who decided to speak at a conference.

Now for a bit of explanation.

On Thursdays we rehearse the release in staging to make sure deployment goes smoothly. We introduced this after problems deploying parameter encryption. That means you can’t tack something onto the release guide at five on Thursday afternoon. If it didn’t go through rehearsal, it doesn’t go into the release.

And a few things you might not know.

– A teammate is prototyping for mobile.

– We’re working on showing what’s deployed in each environment on the dashboard.

– A write-up about an application rewrite is nearly done.

– We’re hosting a meetup in the office.

– Teammates are giving talks about an API gateway and mobile development.

– There’s a meetup coming up on front-end development.

Take care!

Hi everyone. I’ll start unusually, with the praise our engineering team got at a group meeting. We can pat ourselves on the back for a moment. Now to the problems we should all be concerned about.

The health check is green. I’d like us to take any other result seriously and act on it. The team handling operational issues should take the lead.

We have three incident reviews, one for an outage and two for post-release issues. [Internal links omitted.]

We need to be consistent about reviewing every release. Our process isn’t reliable. If we don’t record the problems and get them onto the right teams’ backlogs to fix, every release will be a gamble. Differences between environments seem to cause the most trouble. We’ve updated the Thursday rehearsal procedure to address them. Please merge and test the changes as early as possible.

Repeated regressions concern me even more. One prevented people from opening a user tab on Friday morning. The fix took until the evening because the tests or build had been failing for half a day. When a broken branch or build stops us fixing other problems, the blockers channel should be buzzing. If broken continuous integration is holding up 25 developers, fixing it is the highest priority unless production is on fire. Someone needs clear ownership, ideally from the team handling operational issues.

On Friday we dealt with a spike in traffic at the API gateway. Seeing the monolith’s error rate made me shudder. How do you separate signal from noise in that? It showed how little visibility we had into operations. We want one developer to rotate into the team handling operational issues and improve observability.

Let me know if you have any comments.

[A third party’s personal update has been omitted.]

Here’s to a better week. Let’s have another go and do it better.

Take care and keep smiling.

Hi everyone,

– I hope you enjoyed the offsite and found something interesting, whether in a technical talk or in the session on how our business works and why we do things this way.

– The bug-fixing day is under way. We have 635 points in progress. We’ve added more back-end tasks, so I hope everyone finds something and helps us reach 1,001 points. Tasks shipped via the fast track count double.

– The first part of the mobile app is nearly ready to release. People with the relevant devices can help test it.

– Conference season is approaching. Have a look at what you’d like to attend. One teammate has already picked an event.

– We still have open engineering positions and the referral bonus is available. There’s also still a chance to move to a different area of development.

– A teammate managed to open-source part of our infrastructure code. The next big target is our tests. There’s also a recording of his talk on how the technology works under the hood.

– Here are a few interesting things I’ve read lately.

– Articles on product ownership, software components, a regulator’s approach to cryptocurrencies, and how programming errors affect phone battery life.

Take care and keep smiling.