I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."
I do think his choice of words was a bit suboptimal. But, it got the point across.
> Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
The best leaders know what they need to pay attention to. You cannot know everything. You can know any one thing (or perhaps several), but there are more things to know than you will have time to learn in your lifetime. The question is what is important for you to know, and what you leave to somebody else.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
> It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done.
That makes sense - I don't know if that's what the blog post was implying though. Ironically, this blog post could probably benefit from a few more details about what the meeting was supposed to achieve :)
A lot of times, people just need a gentle nudge, or an official blessing from an authority figure, to go ahead and make the change that is necessary. You can tell people to manage up, to fully own their work and their process, etc. But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line. A worker on the assembly line might know the best way to make the line run more smoothly but it’s easy to think that’s “not my job” or “above my pay grade”. Sometimes all that’s needed is to tell them, “Yeah, go ahead and do it.”
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
> But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
Unless the organization really truly rewards that behavior. Which is rare.
The senior-most people (VPs, founders) often say they want it. But it’s the middle managers who will stab you in the front in revenge, while staring you in the eye. And if the VP doesn’t step in, they are endorsing the outcome.
In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, or hearing about the ground level operations, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case. At a certain point that becomes somebody else's job that is much closer to the team at hand.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
> In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
This is a very strong argument that no organization should be allowed to get that big.
You can trust someone to know the details and reasons for why something happened, and still want to provide the specific direction of "what are you doing to make sure it doesn't happen again". The organizational default is often to explain a fault (or worse, focus on blame) without specifically working to find a process improvement to prevent it from happening again.
I don't read the article this way. It's pretty sane for an exec to draw a line where the team's autonomy meets leadership's responsibility for structure and incentives around which said team needs to organize.
"You did the right things and met the goals we set, but we hit a problem and we all agree it shouldn't happen again. Let's talk about what needs to change to address this."
“Trust” is usually bounded. I trust my elementary aged son to get dressed for school without supervision. That doesn’t mean I trust him to start a campfire without supervision. A chef might trust that their new employee is a hard worker and well intentioned. That doesn’t mean they also trust them to create and serve original dishes.
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details. I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
If execs demand a big change every time something goes wrong, they will demoralize everyone working for them, and probably waste a lot of money.
Most of the time, operational failures are either the result of a new feature being sent off without enough testing, or a dozen small failures lining up to remove resilience. Big changes without well-thought out plans primarily serve to stoke executive egos.
> If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details.
I should have been clearer.
When I said that good leaders pay attention to things from the bottom-up, that doesn't mean they always know everything needed for any discussion. It just means they are able and keen to understand the low-level details when they matter, because they routinely look at such details in the projects that they're involved in.
Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
> I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
> Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
I don't think it's necessarily misleading; it's just reminding folks that senior leaders have a different perspective than those below them and tailoring your message to the audience is important.
In this case of this, it was a reminder that while it's easy to believe the SVP is being dismissive, that's not the case at all. Instead, they want systems-oriented decisioning, not detail-oriented.
If anything, this was the SVP helping to grow the leaders under him by forcing them to think at the levels required to interact w/the rest of the organization.
> How can you know if their "big change" is appropriate and good without knowing the details of the problem the change is addressing?
I'm a Senior Director leading three teams of engineering and science resources. I hired smart and capable people. Why should I, as a SrD be a technical bottleneck?
> How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
That’s the trust thing. Maybe the SVP trusts the people in his organization are capable of making the appropriate change without his supervision and what he’s doing is coaching them on how to do an incident postmortem properly.
Of course as you called out, the SVP might already have all the necessary details to do more than that. But simply pushing people in the right direction is a big piece of leadership’s job, at least when they have people they trust.
If I trust you know what happened, and trust you to know what to do so it doesn’t happen again, why do I need to know? And how will I know (and endorse) your decision if I don’t know what happened?
They trust your ability to do a root cause analysis. They don’t trust that you feel empowered to do anything about it, so they are verifying that you actually do.
Not wanting to know the details is a red flag. Sometimes the process of expressing the details can reveal things that were unknown by one or both parties, leading to improved processes. The only acceptable exception would be "Let me explain. No, there is too much. Let me sum up."
Also, not every IC or EM has the same skills and experience. There may be certain categories of work that you can be 100% hands-off and trust them to just do it, and others where you really need to understand bottom-up. Part of being an eng leader is to know when to insert yourself and when to "trust the process".
Leadership still holds engineers accountable. And for that, they need to know the plan. Leaders are also held accountable, so again, they need to know the plan.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
I work in a competent team and leadership is required.
Yup, several things are wrong with it even though it may be a good idea as a good shortcut on the part of the SVP
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
There's still some problem with that approach. OK, I will tell what I'm going change to prevent recurrence of the issue. How does that makes sense to the audience? Unless they just want hear that "some" change will be there and don't care about how that change would make any sense.
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
What you describe is exactly what a 'leader' is supposed to 'see'. When to know when details are needed, and when to trust their delegate. The trust is also balanced: if what you say does not align with what the leader is 'seeing', then trust falls apart.
Everyone has some measure of the 'big picture' and the person at top is supposed to see the biggest, final picture. Where everyone and their processes fit, within the big picture. Delegation could be perfectly aligned (and trust would be easy) if not for quality:
Quality determines how well the relationship works between leader and delegate.
The amount of quality is determined by the amount of need, and urgency available. The relationship (and communication) evolves over time as quality is determined by leaders.
Regardless, there's always going to be difficulty because there's never an ideal system where quality is perfectly measured.
OP is already at the director level. The further you go up the management chain, the less people are involved in process or technical decision making -- they are primarily concerned with organizational decision making. It sounds like the issue was something that was fully within the management scope of lower levels of management here:
> Something had gone wrong that shouldn't have.
The SVP is trying to determine what change in management is taking place, not what technical change is taking place.
One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
> To drive change in your organization, don't ask "why did this happen?".
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
I don't get it neither. It's important for context to document "why something happened" alongside action items to prevent it from happening again. This is part of history that will be extremely valuable as company knowledge on why processes exists in the first place.
Otherwise, what will be the discussion when it's time to question it later?
I think the problem is when the execs aren’t looking at change-for-the-better. Instead they’re looking to appoint blame. Moving to a position of assumed competency in your staff and taking the perspective of how-to-improve-next-time is a real step forward in maturity. My guess is it’s rare and blame is easier: most management doesn’t reach this level, instead it’s immature.
Action items need an a forcing feature (like an SLO) to insure they are picked up. And someone needs to follow up when the item isn't handled timely. It should show up in team performance metrics and guide changes to processes to insure they get done.
Actionable items go onto the backlog. The backlog grows from the top.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
The important part is the action items are assigned an owner and executed.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
An SVP of engineering should care about the details, this is the essence of leadership. When you don’t care about the details you are just a manager and worthless i.e. you should be replaced with a leader.
> [The Manager] continued:
>
>> I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
>>
>> Then it'll happen again.
>>
>> So I don't want the details. I want to know what we're changing.
Even if the manager's approach was correct, this is just a hurtful way to frame it to your team. It's great that the author could reverse-engineer what the manager was doing, but it shouldn't come to that.
Why couldn't the manager say something like:
> You are all great engineers, so I already know that everyone made the best decision available with the information they had at the time. What are we going to do differently moving forward?
I'm still not sure I agree with the strategy, because I don't see how you can understand what happens next without understanding what went wrong. But I can see how the framing might help steer the discussion.
I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
Interesting. At our org we always do both. Root cause, what are we changing. I assumed that was standard practice.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
If the person who is charged with resolving the issue is capable, there's no point in a senior manager knowing the details outside of professional curiosity.
This is similar to the approach in healthcare, at least my experience in pharmacy.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
Organizations don’t choose whether or not to ask questions. People do. The behavior of an organization is the sum of the actions of the people in it. If you are a member of an organization, and you ask ‘how do we prevent this from happening again?’ Guess what? The organization just learned how to ask that question.
Now it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
This is a really really smart SVP. The answer to “how do we prevent this in the future” is to identify decisions that reduce that problem from happening again. This doesn’t have to be perfect. You can say “we are going to do X next time” and also say “but we don’t know how far X will work”. When the failure happens again, you can retire process X and move on to Y. The aim is to make a series of decisions that eventually get to the heart of the issue.
Everything else is a conversation that takes up space on a post-mortem or runbook.
If I recall correctly, in The Pragmatic Engineer, this was described simply as "provide options, not excuses" and is a critical way of building credibility.
Love this idea; won't go over well at all with the other accountants I work with. MY CFO doesn't like graphs because they "hide too many details' and insists on everything presented to her being in a table. MY VP pulls out a calculator and re-does the math on tables in decks by hand to help him 'understand'.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
We call this incident postmortem, part of a company process. After serious problems and firefighting, when the patches and hacks and people working 24 hour and it admins wake on a Sunday morning to do work, we sit down and try to figure out how this will _never_ happen again.
SVP seems like such a useless appendage in the story.
He doesn’t want the details, he just wants someone else to tell him what they’re gonna do. But if he doesn’t know the details, he won’t understand if the plan will make things better or worse.
So… what exactly is this presumably valuable person there to do? Just to gage the confidence in the room? Presumably everyone was confident before the mistake happened.
What does this person who just wants a solution add? Why isn’t he fostering a discussion and adding his insight? Why is he in the room at all?
How about explain the problem and solution in 3 to 5 levels of details in a top-down hierarchical manner sort of like a decision tree. Let people decide how far they want to drill down.
If you don’t know what happened how would you be able to tell if the changes are addressing the problem? If you don’t care about the problem being addressed why jump in a call in the first place?
Learned this doing incident reviews. If the exec asks for details it usually means they don't believe you yet, so "skip to what changes" is actually the good outcome.
> At first I thought "I don't want the details" sounded dismissive.
Well that is because it was delivered by someone who was actually being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix.
But otherwise be very wary of this pattern of communication where "we" might mean "you". Anyone with any sense of risk management should know not to do this.
If you have not seen it, watch Margin Call. When you get to the boardroom scene [0] you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 feet level.
Beyond the drama of the way the scene ends, it's actually a very good model of how that interaction should play out for everyone — making it clear that everyone in the room shares in the "we" of it, including the decisionmaker.
I have been in rooms where it played out at least something like that, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.
And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.
[0] said scene, but really, really, watch the film if you never have:
I can understand the inclination, but actually I /do/ want the details, because without the details I cannot consider and measure if the change proposed is a reasonable pathway to help manage the class of problem. Many times, people in organizations while well-meaning are siloed and focused specifically in their own niche and therefore will propose changes that cannot be coherent in any holistic manner across all the parts of the organization that would need to be affected by the change. That means you /do/ need to understand the details, and also have a big picture view.
The executive is there to help with the big picture, but you can't see the big picture if the canvas is blank.
If I'm to explain what needs to change, that will by definition address processes or lack thereof that led to the problem, no? Which is then explaining what happened and why in an indirect manner, so many of the proposed changes may not make sense without more context.
I'm giving you the details because I'm going to give you options for solutions and I need priority information about those solutions, and you need to be informed about the trade-offs and costs of each option because you have better knowledge of our overarching objectives than I do.
If our overarching objectives don't impact our on-the-ground decisions, what are you even for?
I would not have said that. I would have asked for the details and tried to find the gaps because, almost always, something could have been done. Then the question is how much risk reduction are you willing to pay for.
Presumably the SVP joined the meeting because the incident was significant. The team can present the options but the executive may need to make a judgement call, and he can't do that responsibly if turns his brain off. For instance, what if it happened due to things falling in the cracks between teams, possibly indicating poor org design, or to understaffing? Such things are the responsibility of management.
The quote box at the start of the article is the essence of it, the rest is AI-speak dressing it up.
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”
I do think his choice of words was a bit suboptimal. But, it got the point across.
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
That makes sense - I don't know if that's what the blog post was implying though. Ironically, this blog post could probably benefit from a few more details about what the meeting was supposed to achieve :)
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
The senior-most people (VPs, founders) often say they want it. But it’s the middle managers who will stab you in the front in revenge, while staring you in the eye. And if the VP doesn’t step in, they are endorsing the outcome.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, or hearing about the ground level operations, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case. At a certain point that becomes somebody else's job that is much closer to the team at hand.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
This is a very strong argument that no organization should be allowed to get that big.
- We’ll just fire joe - Ok cool
I agree with the parent comment. You need SOME level of detail, otherwise your decision might as well be a guess.
"You did the right things and met the goals we set, but we hit a problem and we all agree it shouldn't happen again. Let's talk about what needs to change to address this."
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
If I have a handle on things from bottom-up to top-down, I don't need them to rehash weed-level technical and specific details. I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
Most of the time, operational failures are either the result of a new feature being sent off without enough testing, or a dozen small failures lining up to remove resilience. Big changes without well-thought out plans primarily serve to stoke executive egos.
I should have been clearer.
When I said that good leaders pay attention to things from the bottom-up, that doesn't mean they always know everything needed for any discussion. It just means they are able and keen to understand the low-level details when they matter, because they routinely look at such details in the projects that they're involved in.
Of course it's possible that this story is just told badly and the SVP already knew the technical details, in which case the blog post is a bit misleading.
> I need them to tell me what we're going to change, and it needs to be a BIG systemic change, not minutiae, which is where many engineers thrive.
How can one know if the engineers' proposed "BIG systemic change" is appropriate and complete, without knowing the details of the problem it is addressing?
I don't think it's necessarily misleading; it's just reminding folks that senior leaders have a different perspective than those below them and tailoring your message to the audience is important.
In this case of this, it was a reminder that while it's easy to believe the SVP is being dismissive, that's not the case at all. Instead, they want systems-oriented decisioning, not detail-oriented.
If anything, this was the SVP helping to grow the leaders under him by forcing them to think at the levels required to interact w/the rest of the organization.
> How can you know if their "big change" is appropriate and good without knowing the details of the problem the change is addressing?
I'm a Senior Director leading three teams of engineering and science resources. I hired smart and capable people. Why should I, as a SrD be a technical bottleneck?
That’s the trust thing. Maybe the SVP trusts the people in his organization are capable of making the appropriate change without his supervision and what he’s doing is coaching them on how to do an incident postmortem properly.
Of course as you called out, the SVP might already have all the necessary details to do more than that. But simply pushing people in the right direction is a big piece of leadership’s job, at least when they have people they trust.
If I trust you know what happened, and trust you to know what to do so it doesn’t happen again, why do I need to know? And how will I know (and endorse) your decision if I don’t know what happened?
It seems weird.
I work in a competent team and leadership is required.
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
Everyone has some measure of the 'big picture' and the person at top is supposed to see the biggest, final picture. Where everyone and their processes fit, within the big picture. Delegation could be perfectly aligned (and trust would be easy) if not for quality:
Quality determines how well the relationship works between leader and delegate.
The amount of quality is determined by the amount of need, and urgency available. The relationship (and communication) evolves over time as quality is determined by leaders.
Regardless, there's always going to be difficulty because there's never an ideal system where quality is perfectly measured.
> Something had gone wrong that shouldn't have.
The SVP is trying to determine what change in management is taking place, not what technical change is taking place.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
Otherwise, what will be the discussion when it's time to question it later?
This should not be exclusive.
I usually do want the details, but I also want to communicate my expectations up top and unambiguously.
Good article write-up, I didn't need to read more than one paragraph to get all the information from the whole page.
1. timeline of events
2. impact
3. 5 whys — the details of how and why things happend
4. actionable items
I've never seen a post-mortem without actionable items.
Action items need an a forcing feature (like an SLO) to insure they are picked up. And someone needs to follow up when the item isn't handled timely. It should show up in team performance metrics and guide changes to processes to insure they get done.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
That line is absolute gold
People need to tell their stories. They need to be heard and have their work and its difficulty respected.
That's what went wrong. Everything else is an abuse victim rationalizing their abuse.
Even if the manager's approach was correct, this is just a hurtful way to frame it to your team. It's great that the author could reverse-engineer what the manager was doing, but it shouldn't come to that.
Why couldn't the manager say something like:
> You are all great engineers, so I already know that everyone made the best decision available with the information they had at the time. What are we going to do differently moving forward?
I'm still not sure I agree with the strategy, because I don't see how you can understand what happens next without understanding what went wrong. But I can see how the framing might help steer the discussion.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
Now it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
Maybe I'm a bit of a pollyana, but most places I worked had folks who cared and wanted to improve things.
Where are these places? It feels like they are hard to come by, and their hiring requirements can be highly competitive
Everything else is a conversation that takes up space on a post-mortem or runbook.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
He doesn’t want the details, he just wants someone else to tell him what they’re gonna do. But if he doesn’t know the details, he won’t understand if the plan will make things better or worse.
So… what exactly is this presumably valuable person there to do? Just to gage the confidence in the room? Presumably everyone was confident before the mistake happened.
What does this person who just wants a solution add? Why isn’t he fostering a discussion and adding his insight? Why is he in the room at all?
Well that is because it was delivered by someone who was actually being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix.
But otherwise be very wary of this pattern of communication where "we" might mean "you". Anyone with any sense of risk management should know not to do this.
If you have not seen it, watch Margin Call. When you get to the boardroom scene [0] you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 feet level.
Beyond the drama of the way the scene ends, it's actually a very good model of how that interaction should play out for everyone — making it clear that everyone in the room shares in the "we" of it, including the decisionmaker.
I have been in rooms where it played out at least something like that, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.
And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.
[0] said scene, but really, really, watch the film if you never have:
https://www.americanrhetoric.com/MovieSpeeches/moviespeechma...
I'm not so sure this means "I trust you."
The executive is there to help with the big picture, but you can't see the big picture if the canvas is blank.
If you ask them, of course they will make up a bs story with their fake answer. The fact is that they fail the ablation test.
If our overarching objectives don't impact our on-the-ground decisions, what are you even for?
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”