How we deliver Lighthouse Watch.

The previous "Observatory Retainer" framing positioned the post-Sprint engagement as discretionary, ad-hoc, and bespoke per client. That framing produced three failure modes: (a) low conversion (clients defaulted to "we'll handle it ourselves" and silently let systems drift), (b) non-uniform delivery (each client got a different version of the work, which is impossible to scale past 3–4 accounts), and (c) margin compression (we kept saying yes to scope creep because the contract didn't draw a perimeter).

Watch fixes all three by being productized: a fixed shape, a fixed cadence, three priced tiers, a perimeter that's written down. The trade-off is less per-client customization in exchange for the ability to deliver consistently across many clients while building toward the pattern-library moat.

This means: do not customize Watch deliverables for individual clients beyond what's allowed in the scope ladder below. If a client asks for something outside Watch, propose a Beacon Workshop or a follow-on Sprint, then escalate the request to the principal so we can decide whether it should become part of Watch v1.1.

Each Watch account has three roles. On Pro tier, the lead engineer is named publicly and committed; on Standard the lead engineer is consistent but not contractually named.

A single lead engineer can carry 4 Standard or 2 Pro Watch clients simultaneously without quality slipping. Pro tier consumes more capacity because of the named-engineer commitment + advisory hours + tighter SLA.

Mixed loads convert at 1 Pro = 2 Standard. So a "full" engineer is 4 Standard, or 2 Pro, or 1 Pro + 2 Standard.

Hard ceiling: when an engineer hits capacity, the next sale either triggers a hire or is queued. Do not stack a 5th account. The quality of the +1 improvement is the first thing to slip and it's also the part clients renew for.

Every Watch month follows the same shape regardless of tier. The Pro tier adds 4 hrs of advisory available on demand inside the month; the Essentials tier removes Stream C (the +1).

Stream A is the boring part. The point is that the client never has to think about it. Four sub-tasks, all instrumented and largely automated.

Trigger: any major foundation-model release (Claude N+1, GPT N+1, Gemini N+1) within 7 business days of GA. Procedure:

1. Subscribe the studio to the relevant changelog feeds. We monitor: Anthropic, OpenAI, Google, Mistral, Meta. #studio-foundation-watch Slack channel posts on RSS hit.
2. For each agent in the client's deployment, generate a side-by-side eval run: incumbent model vs. candidate. Eval set = the production eval suite. Run via the model-swap harness (tools/model-swap/).
3. Score: pass-rate delta, cost delta, latency delta. Produce a 1-page swap brief per agent. File in client folder under watch/swap-briefs/YYYY-MM/.
4. Recommendation: stay, swap, or side-by-side (run both for one cycle to gather more data). Default to stay unless candidate model wins on at least two of the three axes by >5%.
5. Surface the recommendation in the next health report. If the recommendation is swap and it impacts evals or cost by >15%, schedule a 30-min call before the report goes out.

Trigger: scheduled, monthly, week 03. Procedure:

1. Run the production eval suite for every agent in deployment. Capture pass-rate per check and aggregate per agent.
2. Compare to last month's results. Threshold: a regression of >3 points pass-rate on any check is automatic P1 ticket.
3. Capture results in watch/evals/YYYY-MM.json and surface the trend in the health report.
4. If any check has been failing for 3 consecutive months, propose an eval-set update (the check might be obsolete) or an agent change (the check might be revealing real drift).

This catches subtle output regressions that pass evals but feel different to reviewers. Procedure:

1. Sample 50 agent outputs from the last 30 days, randomly weighted by reviewer-flagged frequency.
2. Score against the original baseline samples (captured at Sprint handoff) using the drift harness. Output: drift score 0–1.
3. Threshold: drift score >0.25 triggers a root-cause review. Common causes: model version silently auto-updated, prompt-template change in shared library, upstream data-shape shift.
4. Surface drift score in health report regardless of threshold.

Most outages of agent systems aren't model failures — they're providers deprecating endpoints with insufficient notice. Procedure:

1. Maintain a per-client dependency manifest (watch/deps/<client>.yml): every external API the agents call, version pinned where possible.
2. Watch the deprecation pages of each provider weekly. Auto-poll where APIs are available.
3. When a deprecation date is announced for a dependency in scope, immediately open a P1 ticket and schedule the migration within the deprecation window minus 30 days. Never finish a migration in the last 2 weeks before deprecation.

Stream B is what keeps the system trustworthy to the humans who sign off on its outputs. It's also the part that compounds: every governance decision becomes a runbook line, every runbook line becomes a candidate pattern.

The Governance Runbook (delivered at Sprint handoff, lives in the client's repo) is a living document. It updates whenever:

• An incident happens — output that should have been blocked got through, or output that should have shipped was incorrectly held.
• A policy changes — client legal or compliance team issues a new constraint that touches agent behavior.
• A regulatory ask arrives — auditor, regulator, internal audit asks for a control we should formalize.

Revision process: PR against the runbook, reviewer (the client's GC or designee) signs off, runbook merged, all gated agents re-tested against the new check, reviewers notified to re-read.

Every gated agent has a target acceptance rate of ≥80%. Acceptance = reviewer approves the agent's output without modification. Below 80% means the agent is producing more friction than value, even if its evals pass.

1. Capture acceptance rate per agent, per week, in watch/metrics/acceptance.csv.
2. If 4-week rolling avg drops below 80%, open a P1 ticket: schedule a 30-min review with the client's reviewers to identify the friction.
3. Common causes: prompt outdated to the use case, evals are easier than the real review, reviewer UI mismatch, scope drift in the agent's tool selection.
4. Resolution may itself become a +1 improvement. If so, scope it explicitly with the client at the next monthly review.

The Reviewer UI logs every reviewer action: approve, modify, reject, escalate. Modifications and rejections are signal — not for the agent's eval suite directly, but for the eval-set composition.

1. Monthly: extract the last 30 days of reviewer modifications and rejections. Cluster by failure mode.
2. For any cluster with ≥5 instances, propose adding a new check to the eval suite that catches that failure mode.
3. If the new check fails on the current agent, the next +1 improvement is the fix.
4. Rejection categories that recur across multiple clients become candidates for the canonical Studio eval set, contributing to the library moat.

Every third month: 90-min call with client CMO/CTO sponsor + Studio principal + lead engineer. Agenda:

• Outcome metrics — has the agent moved the original Sprint metric?
• Pattern-library updates available — what could be deployed in the next quarter?
• Architecture concerns — any structural change warranted given new client priorities or new infra?
• Forward roadmap — next quarter's anchor improvements, in writing.

Output: 1-page recommendation memo, filed in client folder, referenced at the next renewal conversation.

The +1 improvement is the differentiator and the renewal hook. The client signs Watch · Standard or Pro because of this stream. Treating it casually destroys the entire product.

• A new pattern from @umbra/patterns deployed into the client's existing agent system. Example: replacing a hand-rolled gating primitive with the canonical governance.human-in-the-loop-gate pattern.
• A workflow optimization that cuts a measured friction. Example: collapsing a two-step approval into a single-step approval with a deferred audit trail, where reviewers have already signaled the second step is rubber-stamping.
• An eval-set expansion that catches a real-world failure mode that recurred in the reviewer feedback loop. Counts only if it produces a measurable improvement in acceptance rate.

• A new agent or new workflow — that's a Sprint.
• A bug fix — that's maintenance (Stream A or B).
• A purely cosmetic change — UI polish, copy tweaks, log formatting. Real mejoras affect outputs, gates, or trust.
• An untestable change. If we can't write an eval that distinguishes the +1 from the baseline, the +1 doesn't ship.

1. Month-minus-1 review. In the previous month's review call, propose 1–3 candidate +1's with rough effort and target metric. Client picks one. Default candidate sources: reviewer feedback loop, library updates, quarterly architecture recommendations.
2. Week 01 confirm. First week of the new month: confirm scope of the chosen +1 in writing (Linear ticket + Slack message to client sponsor).
3. Week 02 build. Engineer implements. Eval check expanded to cover the +1's specific behavior.
4. Week 03 ship-behind-gate. Deploy with feature flag or behind a gate. Reviewers see both old and new behavior in shadow mode.
5. Week 04 promote-or-kick. If eval passes the threshold AND reviewer feedback signals improvement, promote. If either fails, kick to next month with a written diagnosis.
6. Library contribution check. If the +1 looks like a recurring pattern (already seen in 2+ clients), tag it for promotion to @umbra/patterns. Three matches = canonical.

The health report is the only deliverable many client stakeholders ever read. It is also the artifact procurement uses to justify the renewal. It must be sharp, scannable, and signed.

3–4 pages. PDF generated from the canonical template (lighthouse-watch-monthly-report.html). Sections, in order:

• Header — client name, month, version, signature line for lead engineer.
• This month at a glance — 4 KPIs (acceptance avg, evals pass-rate, +1 status, incidents). Single line each.
• Model status — versions in production, last evaluation, recommended action (per agent).
• Acceptance & eval trends — 90-day chart per agent, brief commentary on movements.
• Incidents this month — 1 line each, link to full RCA if applicable.
• The +1 improvement delivered — what we shipped, what changed, before/after metric.
• Pattern-library updates available — list of new patterns since last report, with one-line recommendations.
• Next month preview — the +1 on deck, any model migrations expected, any quarterly review scheduled.

Direct, dry, no fluff. The client's CFO might read this. No marketing language, no "we're excited to announce." If something didn't ship, say so on the first line of the relevant section. Examples:

• Yes: "+1 improvement missed eval threshold (passing 81% vs. target 90%); deferred to next cycle. Diagnosis: eval set under-covers edge case X; expanding."
• No: "We're working hard on enhancing the system and look forward to delivering even more value next month."

• Draft by Week 04, signed off by Studio principal.
• Delivered via email to client sponsor on the 1st of the month, before 9am client-local time. PDF attached + permanent link in client folder.
• Acknowledged in Slack within 48 hours. If no acknowledgment, ping. If still no acknowledgment after 5 days, principal calls — silent disengagement is a renewal-risk signal.

1. Slack channel (shared, named #watch-<client-shortname>) — first line. All routine traffic.
2. Email to lead engineer — escalation if Slack unread >SLA.
3. Phone (Pro tier only) — direct line to lead engineer. P0 reserved.
4. Principal escalation — when SLA is at risk, when a request is outside Watch scope, when client is signaling churn.

One month before contract end. Studio principal + client sponsor. Agenda:

• Review the year's outcomes against the original Sprint metric.
• Show the +1 improvement ledger — every one delivered, before/after metric.
• Show the patterns inherited from the library during the year — value the client got "for free" via Watch.
• Propose the renewal: same tier, step-up to Pro, step-down to Essentials, or a new Sprint.

Default outcome target: 85% of clients renew at or above the same tier. Track in CRM.

This is signal, not failure (sometimes). Diagnosis questions:

• Did we deliver the cadence reliably? (If not — that's our miss.)
• Did the +1 improvements actually move metrics? (If not — that's a scope or selection problem.)
• Is the agent system itself still in production, or did the client retire the use case? (If retired — Watch was successful but the underlying need ended.)
• Did the client hire internal capacity to take over? (If yes — that's an outcome we should celebrate even though revenue ends.)

Document the answer. If the cause is a Studio failure, post-mortem in #studio-watch-ops within 14 days.

1. Final health report covers the full final month.
2. Repository handoff: client gets the canonical eval suite, the runbook, the swap briefs, all health reports, and the pattern-library deployment manifest. No ongoing access to the canonical @umbra/patterns updates after offboard date — those are a Watch benefit.
3. Slack channel archived 14 days after final report. Email contact preserved for emergency questions for 90 days.
4. Client added to win-back outreach 90 days post-offboard with a check-in note from the principal.

This section exists so that engineers running Watch accounts understand the economic constraints behind the pricing and don't quietly let scope expand to commercially unsustainable levels.

Hire a new lead engineer when sustained Watch revenue clears $60k/mo for 3 consecutive months (≈ 2.5 Standard accounts). The new engineer is brought into the rotation by shadowing the principal on 1 client for 1 quarter, then taking it over.

Do not hire ahead of the curve. Watch capacity is one of the few places we can absorb churn — under-hire by 1 client and use principal time to fill the gap.

Watch is delivered against a small, sharp toolset. Adding new tools requires a write-up and principal sign-off — every tool added is a tool we have to maintain across all clients.

• No client-facing dashboard. The health report is the dashboard. A live dashboard creates expectation of real-time monitoring that Watch doesn't promise.
• No bespoke per-client eval frameworks. Everything goes through the Studio harness. Bespoke = unscalable.
• No multi-channel client comms. Slack + scheduled email + scheduled call. Anything else is creep.