Skip to main content

​
Commercial Intelligence Chat — Rollout Report (P6.1)

Status as of this report: shadow-ready, GA not activated (per P6.1 scope: “no activar GA”). This document is the artifact required by P6.1’s acceptance (“reporte de precision/costo/latencia y riesgos residuales disponible”) — it is a snapshot compiled from the test suites and per-card reports of P2.3–P6.1, not a live production dashboard.

​
What shipped (P2.3 → P6.1)

​
Precision (trajectory evals)

Trajectory-level invariants are proven end-to-end through the real ChatService pipeline in apps/api/src/chat/chat-evals.spec.ts (describe block chat evals (P6.1) — commercial trajectory, 7 new tests) plus the per-feature specs each acceptance line traces back to: Suite-wide result at time of this report: 2263 tests passing, 4 skipped, 0 failing (npx vitest run from apps/api).

​
Cost

Real per-call pricing for Exa/Apollo/Parallel does not exist yet — that ledger is the separate “Managed Model Gateway” initiative (moved out of this project after an unauthorized cross-project card injection; tracked at project 49b86c65-006d-4d86-9d6a-327aaeeacf6a). Every unit cost referenced inside exa.adapter.ts, apollo.adapter.ts, and exhaustive-discovery-policy.ts is a clearly-named placeholder constant, documented as such in each file’s header comment. The shape of cost enforcement is real and tested:
  • Hard budget constraints are enforced in research-planner.service.ts (a stage is dropped entirely, never partially executed, if it would exceed max_budget_usd).
  • Apollo enforces a maxCostUsd cap per batch, reporting uncovered fields rather than exceeding it (apollo.adapter.spec.ts).
  • Parallel requires an approval whose quoted cap covers the actual quote before runApproved() will proceed (exhaustive-discovery-policy.spec.ts).
Residual risk: until real pricing lands, budget enforcement is correct in shape but not in magnitude — a plan that “fits the budget” against placeholder unit costs may not fit a real invoice. This is the single largest blocker to GA.

​
Latency

No production latency measurement exists (no PERF_DB_URL/TEST_DATABASE_URL configured in this environment — scripts/harness/check.sh’s performance gate reports SKIP: no perf DB). P4.3’s research-query.service.spec.ts includes a synthetic scale smoke (500 refs × 10 runs, p95 < 200ms) but explicitly documents that it measures in-process compute with mocked repositories, not real Postgres latency under load. A real p95 measurement against a representative dataset in staging is required before GA — this is a residual gap, not a solved problem.

​
Feature flags & kill switches (this card)

  • apps/api/src/config/commercial-feature-flags.ts — four independent, default-ON flags (exa, apollo, parallel, commercial_compiler), each read from its own env var (COMMERCIAL_FEATURE_*_DISABLED), consistent with the existing broker-rollout.ts pattern (pure functions, no DB round-trip).
  • commercial_compiler is wired into chat.service.ts: COMMERCIAL_FEATURE_COMPILER_DISABLED=true degrades every commercial turn to ungrounded, exactly like a compile() failure — never throws, never half-executes.
  • Exa/Apollo already had per-provider kill switches (EXA_KILL_SWITCH, APOLLO_KILL_SWITCH, P4.5/P4.6) — commercialEmergencyStatus() is the single call that reports all four feature flags AND both provider kill switches together, so an incident runbook has one place to check “is anything commercial still running?” instead of five files to grep.
  • Not wired: the exa/apollo/parallel feature flags exist as functions but are not yet consulted from inside the adapters/planner themselves (only commercial_compiler is wired end-to-end into chat.service.ts). Wiring the other three is mechanical (if (!isCommercialFeatureEnabled('exa')) throw/return unsupported) but was not done in this pass — flagged honestly rather than claimed complete.

​
Shadow mode

apps/api/src/radar/planning/shadow-plan-recorder.ts persists a ResearchPlan (P4.4’s pure buildResearchPlan() output) as an agent_runs row (agent: 'shadow_planner') for later comparison — no new schema, reuses the existing table. It performs exactly one start() + one finish(), sets no model (the planner never calls an LLM), and is proven to add zero I/O beyond that single write (shadow-plan-recorder.spec.ts, 3 tests). Not done: an actual “compare shadow plan vs. what a human chose” reconciliation job/report does not exist yet — the recorder is the write-side primitive P6.1 asked for; the comparison/analysis layer is a follow-up, not built in this pass.

​
Harness result

Run from repo root:
The one check.sh failure is a pre-existing false positive, not a regression from this card: apps/web/.env.local and apps/dashboard/.env.local contain Vercel-CLI-generated OIDC JWTs. Both files are correctly .gitignore’d (git check-ignore -v confirms), never committed, and existed before this session began. The guardrails script scans the filesystem rather than git diff, so it flags them regardless of what this card changed. Recorded here rather than “fixed” by deleting a legitimate local dev artifact.

​
Residual risks (ranked)

  1. P0 — no real pricing. Budget/cap enforcement is structurally correct but validated against placeholder unit costs. Do not enable real paid calls in production until the Managed Model Gateway project’s price book (or an equivalent for Exa/Apollo/Parallel specifically) lands.
  2. P1 — no real latency measurement. The p95 in P4.3’s tests is a mocked-repository smoke, not a Postgres-under-load number. Run scripts/harness/perf-check.sh against a staging dataset before GA.
  3. P1 — three of four feature flags aren’t wired into their adapters yet. exa/apollo/parallel flags exist and are tested in isolation but the adapters don’t consult them — only commercial_compiler is fully wired. An operator flipping COMMERCIAL_FEATURE_EXA_DISABLED=true today would have no effect until this wiring is finished.
  4. P2 — no shadow-plan comparison job. The write-side (ShadowPlanRecorder) exists; nothing reads the recorded shadow plans back for analysis yet.
  5. P2 — stale Driftless topics. driftless context get --diff shows 4 drifted topics referencing files P4.9 deleted (LeadsPanel.tsx, WarehousePanel.tsx, now ResearchArtifactPanel.tsx). Housekeeping, not a functional risk.
  6. P2 — TypeORM advisory-lock leak documented, not fixed. chat-turn-lifecycle-invariants (existing Driftless topic) already documents a known bug: pg_advisory_unlock failure doesn’t discard the poisoned connection, wedging a thread’s single-flight lock. Out of P6.1’s scope to fix, flagged because it directly affects the reliability of every commercial turn.

​
Rollback

Every P6.1 change is additive:
  • commercial-feature-flags.ts/.spec.ts — new files, delete to revert.
  • chat.service.ts — one added import + one added && isCommercialFeatureEnabled(...) clause in an existing if; reverting is a one-line diff.
  • shadow-plan-recorder.ts/.spec.ts — new files, delete to revert; radar.module.ts — 2 added lines (import + provider registration), revert by removing them.
  • chat-evals.spec.ts — new describe block + a 3-argument extension to the existing makeHarness() signature (opts.commercialContext, backward compatible since it’s optional).
  • No migrations. No data written to any table this report didn’t already document as reusing an existing column/table.