Analytic Story

how a detection team went from level two to level five without anyone working harder

0. The gift that turned into a problem

In January the board funded the thing Lakeshore Mutual had never actually had: detection engineering as a job, rather than a thing people did between shifts when the alert queue allowed. Nine hundred detections already existed. Nobody had ever owned the question of what they covered.

Maya Reyes had made the ask honestly — not we are overwhelmed but here is the specific list of techniques nothing in our environment is looking for — and Hendricks had approved it in eleven minutes. By March, Priya Raghavan had moved off the SOC floor — where a year earlier she had been the analyst who finally escalated the intrusion — and had two engineers, a Jira board, nine hundred inherited detections, and the following sentence written on a whiteboard where she could see it from her desk:

Get to risk-based alerting inside a year.

Everyone agreed on the destination. Instead of a thousand separate alerts screaming individually, every detection would contribute a risk score to the entity it touched — this user, this host — and the SOC would work the entities whose accumulated risk crossed a line. Fewer, better, contextual. The vendor called it RBA. The literature called it obvious. The pilot needed sixty analytics that met a real bar: mapped to a technique, carrying a risk score and a confidence rating, validated against the data model, tested with attack data, with a measured false-positive rate.

Those nine hundred inherited detections were the same nine hundred someone had once cited to a director as evidence of coverage. When Priya went through them against the bar, she got to forty.

At their current rate — about one new analytic a week, on the good weeks — sixty more would take fifteen months, and that was assuming nothing else ever landed on them, which was not a thing that had ever been true.

1. Level two, and honest about it

She wrote out where they actually were, using the maturity vocabulary because the board already spoke it.

They were level two. Not level one — level one is chaos, and this wasn't chaos. They had a backlog. They had a repeatable-ish way of working. Two engineers who were genuinely good. Things got done, mostly, when someone pushed.

But: prioritisation was whoever asked loudest. The definition of "done" lived in Sam's head and was slightly different from the definition in Marcus's head. Nobody could say how long an analytic took to build, because nobody had ever measured it — when Maya asked, the answer was "depends," which is the level-two answer to every question. Coverage was reported as nine hundred rules, which is inventory, not coverage. And when work stalled, it stalled invisibly: no one could tell you which of the eleven things in flight was stuck, or on whom.

The team was busy. The board was patient. Nothing shipped.

"Depends on what?" Maya asked.

"Depends on the data source, mostly." Priya heard herself. "I don't actually know. I've never timed one."

2. Mapping the line

Walt Otani, still coaching third base, still retired from thirty years of running plants in Michigan, said the thing he always said when Maya passed him a problem on the bleachers.

"Don't tell me it's slow. Show me the line. Every station, in order, and two numbers on each: how long somebody's hands are on it, and how long it sits."

So they mapped it — not the incident chain from last year, but the factory that makes detections. Eleven steps, from a threat-intel note or an analyst request all the way to a tuned analytic running in production with a measured false-positive rate. Three of them sat down with the worksheet and the last quarter's tickets and filled in every step with real timestamps.

Twenty-two hours of human work. Sixty-six days of pure waiting — and once each step's failure rate was folded in, the median analytic took seventy-eight business days end to end. Process cycle efficiency: one percent.

Priya stared at it for a while. "One percent."

"Typical for creative work is about five," Walt said. "World-class is twenty-five. You're not broken and you're not lazy — you're below average, and nobody in this building knew, because nobody had ever put the two numbers side by side."

And there was one more number, which they nearly missed because nobody had ever counted it: forty percent of analytics came back. They reached step six, failed testing, and went back to step three or four to be rebuilt. Four in ten. Every one of those was capacity spent twice.

3. The wait that was manufacturing the defect

This is the part Priya later described as the moment the method earned its keep, and it took about twenty minutes.

They asked why analytics failed testing. The answer: the data wasn't shaped the way the engineer assumed — a field missing, a source not normalised to the data model, a log type that only covered half the estate.

Why did the engineer assume? Because step 3, data validation, was supposed to catch exactly that.

Why didn't it? Because step 3 meant filing a request with the platform team and waiting — median twenty-one days — for someone to confirm field availability and coverage.

Why did they skip it? Because it was twenty-one days. An engineer with a backlog and a manager asking for velocity does not stop for three weeks to confirm a field exists. They make a reasonable assumption and build. Sometimes they're right.

And there it was, in a form Walt recognised instantly from the shop floor: the queue was manufacturing the defect. Not carelessness, not skill. A wait so long that the rational move for a competent person was to skip the step designed to prevent rework — so the rework rate was a symptom of the queue, not an independent problem. Add up the loop and the wait and step 3 was costing them far more than twenty-one days.

Two more root causes fell out of the same session. Peer review's eighteen days was a single senior engineer as the sole reviewer, batching his reviews to Fridays. Prioritisation's six days was the absence of any rule at all: with no criterion, every request needed a conversation.

Three waits. Three causes: a queue that induced defects, a batching bottleneck, and a missing decision rule. None of them was a person working slowly, and none of them cost money to fix.

4. Level three: writing down what "done" means

Level three is where a process stops living in people's heads.

They wrote a definition of done for an analytic story, one page, non-negotiable: mapped to a technique; a risk score and confidence rating; data model validated with evidence attached; test cases with attack data; a tuning plan; and a named owner for the FP review thirty days out. It was the same bar the RBA pilot needed, which was not a coincidence — Priya had simply stopped treating the pilot's bar as extra work and made it the standard.

They put a single intake in front of everything, with a two-line prioritisation rule anyone could apply without a meeting: technique on the prioritised list, weighted by whether the data exists today.

And they set a WIP limit of three analytics in flight per engineer, which was the least popular decision of the quarter and the one that mattered most. Eleven things in flight had felt like productivity. It was mostly context-switching wearing productivity's clothes.

Then Priya went to Dae-ho Kim, who owned the platform team and the twenty-one-day queue, with the map and no accusation, and asked the only useful question: what would you need for this not to be twenty-one days?

It turned out Dae-ho had wanted to kill that queue for a year. Nearly every request was the same three questions about field availability and coverage — questions his team answered by running a query anyone could run, if anyone had known it existed. He didn't want to be a service desk for a lookup. In four weeks they built a data availability catalogue: source, normalisation status, coverage percentage of the estate, last verified, self-service.

Step 3 went from twenty-one days to about twenty minutes.

Peer review got a second reviewer and a rule against batching. Nothing waits for Friday.

5. Level four: the numbers become the management system

Level three is standard work. Level four is standard work you can predict, and predictability is a different animal from speed.

They started running the decomposition every month, on the same eleven steps, and putting four numbers on the wall:

• Lead time, median and spread
• PCE, per step and overall
• Rework rate, as a first-class metric rather than an embarrassment
• Coverage, expressed as the percentage of prioritised techniques with a validated analytic in production — never again as a rule count

By the end of the second quarter:

Priya was careful about the throughput line, because Walt had made her be.

"Same two engineers, same hours," she told Maya. "We didn't get faster — nobody is typing quicker. Two things happened, and only one of them changes the count. The waiting between steps got shorter, which is why each analytic takes eleven days instead of seventy-eight — that is speed to market and it changes nothing about capacity. Separately, we stopped building four in ten of them twice, so the labour we spend per finished analytic fell from about twenty-five hours to about fifteen. That is the only reason the count went up, and it explains why the first big lead-time win barely moved output at all: forty-eight to fifty-one. If anyone asks whether we're producing more because the team is working harder, the answer is no, and I can show them the rework chart."

The spread mattered as much as the median. Seventy-eight days had been seventy-eight on average, with a tail out past four months, which meant every commitment Priya had ever made to the SOC was a guess. Eleven days with a tight spread meant she could say the analytic for that technique lands the week of the fourteenth, and be believed, twice in a row, which is how a security team accidentally becomes a partner instead of a cost.

6. Level five: the process starts improving itself

Level five is not "very good at level four." It's when improvement stops being a project somebody has to champion and becomes something the process does on its own, because the machinery generates the improvements.

Three things tipped them over.

Defect prevention, not defect detection. The rework rate stopped being a number they reported and became a trigger: any analytic that bounced spawned an automatic five-minute review of why, with the answer written into the definition of done. Two of those reviews produced permanent changes — a required schema check, and a rule that any analytic touching a source under 80% estate coverage gets flagged before design, not after testing.

Production telemetry feeding the line. Every deployed analytic reported its own false-positive rate, and anything crossing a threshold opened a tuning ticket automatically — a real intake item with a place in the queue, rather than a Slack complaint from a tired analyst at 2 a.m. The process began consuming its own output as input.

Retirement as a first-class step. Step 11 had always said keep / tune / retire, and for years the retire branch had never once been taken, because deleting a detection feels like reducing security. Now, with FP rates and true-positive counts per analytic, they could retire with evidence. They killed a hundred and forty detections in one quarter — noisy, redundant, or covering techniques two better analytics already covered. The SOC's alert volume dropped eleven percent before RBA had shipped a single thing.

And then they did the thing that actually marks level five, which is that they experimented. They ran two tuning strategies side by side on similar analytics for six weeks and kept the one with the better true-positive density — a genuine A/B test in a discipline that usually runs on conviction.

7. What it was all for

The RBA pilot went live in the second week of October, eight months after the whiteboard.

It needed sixty analytics at the new bar. They had eighty-four, because a line that turns out seventy-nine a year at a four percent rework rate produces surplus, and surplus is what lets you be selective. Every one carried a risk score, a confidence rating, and a technique mapping — not because RBA required extra paperwork, but because the definition of done had required it since March.

The results the SOC cared about: alert volume down by roughly two-thirds, and the entities that surfaced were ones an analyst would have wanted to look at. The results Priya cared about were duller and more durable. Coverage of prioritised techniques: sixty-one percent, measured monthly, trending. Time from we should detect that to we detect that: eleven days, predictable. And a line she could defend in any room, which was that not one of those improvements had come from working harder or spending more.

Then Maya did the thing that mattered most, which was to take the same eleven-step decomposition and point it at the next process over: threat intelligence intake. Then the analyst tuning-request path. Then exception handling.

The same shape showed up every time. A queue nobody owned. A step whose delay was quietly causing a defect two steps downstream. A batching habit. A missing decision rule. It turned out the security programme wasn't slow in a hundred original ways; it was slow in about four ways, repeated across a landscape nobody had ever mapped.

8. The board, again

Hendricks read the one-page summary and asked the question he always asks.

"How much of this do you actually know, versus believe?"

"The eleven days, the rework rate, and the coverage percentage come out of the systems and you can audit all three," Priya said. "The alert-volume drop is measured against the same period last year. The claim I can't verify is the important one: whether better coverage means we'd catch the next intruder sooner. That's a hypothesis with evidence behind it rather than a promise. I've written down what I expect and dated it."

"And the maturity level? You keep using the word."

"Nobody appraised us. There's no certificate." She had thought about this. "What I can tell you is that eight months ago we couldn't answer how long anything took, and today we run the same measurement every month, act on what it says, and the process changes what it does without me having to push it. If that isn't level five, it's close enough that arguing about the label would be the least useful conversation in this room."

Hendricks made a note.

"The thing I keep coming back to," he said, "is that you didn't ask me for anything this year."

"We asked for the detection engineering team last year. This year we found out where its time was going." Priya shrugged. "One percent of it was work. The rest was waiting, and waiting is free to fix. It just isn't free to find."

On a rec-league field across town, a third-base coach was explaining to a nine-year-old that everybody wants to talk about how hard you swing, and almost nobody wants to talk about where you stand before the pitch.

The napkin, for a recurring process:

1. Map the line — every step, in order.
2. Two numbers on each step: hands-on hours, and days it sits.
3. Divide: how much of the elapsed time is actually work? (Around 5% is typical. Under that, you are below average and nobody has noticed.)
4. Find the biggest wait — then ask what defect it is causing downstream.
5. Fix the wait. Write down what "done" means. Limit what's in flight.
6. Measure the same steps again next month. Every month. That's the whole difference between a fix and a process.