A KPI "Collapse" Traced to Order Mix, Not a Tracking Bug: Before the Team Chased the Wrong Problem
The chart said the sky was falling. The register said one giant order had walked in the front door.
What was getting in the way.
A trusted ratio, ad-attributed revenue as a share of total online revenue, drops from its long-running norm into territory that looks like a tracking disaster. The instinctive response costs real time: engineers audit tags, analysts re-pull reports, agencies get asked hard questions, and everyone operates on the assumption that measurement is broken. If the metric itself is the problem, if its denominator quietly absorbed something the numerator can never see, all of that effort chases a bug that doesn't exist.
Context
Universal: the pattern applies to any ratio KPI that mixes attributable and non-attributable events: revenue share, lead share, conversion rate against mixed traffic, cost per anything with a blended denominator.
How the work runs.
- 01
Reproduce the metric before explaining it
The agent rebuilds the ratio from the warehouse, ad-platform revenue across four platforms over storefront revenue, and matches the reported number within a point. Now the conversation is about one agreed-upon number, not dueling dashboards.
- 02
Decompose numerator and denominator separately
The drop has to live somewhere: either attributed revenue fell, or total revenue rose in a way ads can't claim. The decomposition points at the denominator.
- 03
Find the mix shift
Stripping large staff-assisted trade orders, phone-and-rep transactions no ad click could ever be credited for, the ratio reads essentially flat across both months. One single staff-edited order accounted for a seventh of the current month's revenue by itself. The "collapse" was order mix, not measurement.
- 04
Check the metric's own lag while you're in there
A second-order check shows the current month's ads figure is still meaningfully under-reported against its own conversion-lag baseline: recent months always look worse before attribution catches up, which had been amplifying the panic.
- 05
Report the real problem found underneath
The audit still surfaces a genuine, separate issue: channel spend down sharply year-to-date with efficiency decaying over several months, concentrated in campaigns that didn't earn their budget back. The team gets a true problem to work instead of a false one, and a recommendation to redefine the KPI with trade orders excluded so it can't cry wolf again.
Evidence from the workflow.
Each system has a role.
intake (order records and ad-platform revenue, joined)
Data warehouse
signal (the numerator, reproduced independently)
Ad platforms (×4)
signal (the denominator, decomposed by order type)
Storefront / order system
action (verdict, the real finding, and the KPI redefinition)
Analysis report
Why this is Coworker.
Takes assignments and reports back. You hand it work, it prepares and returns a result.
- 01
The reported number is reproduced from source data before any explanation is offered, no root-causing a figure the agent can't independently generate. Exclusions are justified by order provenance (transactions ads could never claim), stated in the report, and proposed as a standing metric definition for a human to approve. Retired conclusions are retired explicitly.
Impact / Outcomes
A tracking-bug hunt that would have burned days across multiple people is retired in minutes with arithmetic: the ratio was flat once untrackable trade orders were excluded.
The metric itself gets fixed, trade orders excluded from the denominator, so the same false alarm can't fire again.
A real decline hiding under the false one (spend and efficiency decay in specific campaigns) is surfaced with evidence, so the team's attention lands on a problem that actually exists.
Find where a workflow like this fits.
Start with the systems, work, constraints, and authority already present in your operation.