Skip to main content

​
A0 — Chat shell decision: trajectory evals + assistant-ui spike

Project: 70160c56-cf2f-4d94-81a0-ee9000eee76b (Commercial Intelligence — Agentic Workbench) Card: A0. Congelar trayectorias y decidir shell de Chat Status: in review (never done — the reviewer decides) Verified against: staging HEAD 26087499 (fix(api): bound database connection pools) Scope: evaluation + a reversible, non-production spike. No production behavior changed.

​
0. Correction notice

An earlier pass of this card was built inside a worktree rooted at fc038f17 — several commits behind real staging (git merge-base HEAD 26087499 = fc038f17, i.e. the worktree was a direct ancestor, not staging itself). That pass claimed ResearchArtifactPanel.tsx did not exist and that no paid-approval/suspend-resume mechanism existed anywhere in Chat. Both claims were false — they were true of the stale commit, not of staging. The reviewer caught this, the worktree was rebased onto 26087499, and every file map, eval, and claim below was re-derived from scratch by reading the current code on this exact commit. Two things from the first pass were also flagged and are fixed here: an adapter test that asserted on its own input instead of the adapter’s output (a tautology), and a spike file that was five disconnected unit tests instead of one integrated trajectory.

​
1. What this covers

  1. A hermetic, red-by-design trajectory eval suite pinning 10 concrete gaps between the playbook’s generic Workbench model and the real, substantially-built chat.service.ts — verified against the actual code, not assumed absent.
  2. A reversible spike: an ExternalStoreRuntime-shaped adapter with a fixed tool-activity tautology, exercised through ONE integrated trajectory (persisted history → deltas → tool activity → cancel → an artifact patch across a real suspend→resume cycle).
  3. Real, executable type-compatibility evidence against the actual @assistant-ui/core package — run in an isolated scratch install, zero dependency left in this repo.
  4. A keep/adapt/replace recommendation with a compatibility matrix, a migration-cost estimate, and a rollback plan.
Owner invariant (2026-08-04): compras-jalisco / licitaciones / any single geography or provider is ONE vertical fixture among many a future warehouse will serve. Test 9 below is not hypothetical — the real system’s own capability-catalog fixture (CAPABILITY_BUNDLE_FIXTURE) ships exactly ONE capability today (public_procurement_new_tender), and the real query the service builds addresses exactly one capability_id per turn. The gap this invariant names is the CURRENT shape of the code, not a future risk.

​
2. What actually exists on staging HEAD 26087499 (read in full before writing anything)

The system is considerably more built than the playbook’s “Chat v0, read-only” framing suggests. It already implements a full vertical — under a different, capability-specific vocabulary than the playbook’s proposed generic one (WorkContract/ExecutionPlan/HumanCheckpoint) — for market opportunities: None of this is claimed to be missing below. The evals pin the SPECIFIC, verified gap between this real, substantial implementation and the playbook’s generalized (multi-capability, structurally-checkpointed) model.

​
3. Trajectory evals (hermetic, red by design, 10/10 confirmed)

File: apps/api/src/chat/agentic-workbench.trajectory.spec.ts Config: apps/api/vitest.trajectory.config.ts, scoped to this ONE filename (not a *.trajectory.spec.ts glob) — the repo already has opportunity-trajectory.trajectory.spec.ts on that naming convention as a real, unrelated, GREEN regression suite; a glob would have silently swept it into this red-by-design config and dropped it out of CI’s normal gate. apps/api/vitest.config.ts’s exclude is scoped the same way (already fixed by the reviewer during the rebase). Every assertion targets something concretely verified in the code above — a field absent from a real DTO, a route the real classifier never returns, a ledger status the real AgentRunsService never writes, a query field that is a string not a string[] — never a substring match on assistant prose. Each is written as the DESIRED end-state (so a fix makes it turn green automatically, never requiring a rewritten assertion) and confirmed red against the real harness. Reproduce: pnpm turbo run build --filter="./libs/*" once (workspace libs need their dist/ built for Vite/vitest resolution — pre-existing, unrelated to this card), then pnpm --filter @driftless/api exec vitest run --config vitest.trajectory.config.ts. Confirmed: 10/10 red, each for the reason in the table above — none crash on setup, none fail on an unrelated typo. Do not “fix” these by loosening an assertion. A green result means the behavior shipped in chat.service.ts/chat-stream.hub.ts/chat-tools.ts/intent-preflight.ts — that is A1+ work, not A0.

​
4. Spike: an ExternalStoreRuntime-shaped adapter, one integrated trajectory, real type-compat evidence

​
4.1 The tautology fix

The first pass’s “surfaces tool activity” test asserted facts about liveEvents (the function’s OWN input) while buildExternalStoreAdapter silently dropped tool.activity/tool.result — the assertion would have passed regardless of whether the adapter did anything with them. apps/dashboard/src/redesign/spike/chatExternalStoreAdapter.ts now folds the REAL ChatStreamEvent union (imported from ../../api, not reinvented) into extras.toolSteps — a typed, deduplicated-by-id list — and the spec asserts on adapter.extras.toolSteps (the adapter’s OUTPUT), never on the raw input log.

​
4.2 One integrated trajectory, not five disconnected tests

apps/dashboard/src/redesign/spike/chatExternalStoreAdapter.spike.spec.ts is now ONE it() that walks a single simulated thread through every phase the acceptance criterion names, each phase building on the previous one’s state:
  1. Persisted history — two prior rows seed the thread (the real source of truth; never re-derived).
  2. onNew forwards to the real send() — no drafted state lives in the adapter.
  3. Incremental deltas — the assistant row grows token by token; the persisted rows are never duplicated.
  4. Tool activity fires and resolves — asserted on extras.toolSteps (the tautology fix).
  5. Cancel — onCancel() forwards to the real stop callback.
  6. Artifact patch across a real suspend→resume cycle — using the REAL OpportunityWorkflowRunResult type (imported from ../chatThreads, not an invented artifact.* event vocabulary): a suspended flow with one row and a quote, then a success flow with the SAME run_id and a second row added — the original row survives, the artifact’s identity does not change (patched, not replaced).
Reproduce: pnpm --filter @driftless/dashboard exec vitest run chatExternalStoreAdapter — 1 test file, 1 test, green (it tests OUR mapping code, which is supposed to work).

​
4.3 Real type-compatibility evidence, zero workspace footprint

The reviewer asked for either real evidence against the actual assistant-ui runtime/API, or a documented reason it isn’t possible — without leaving a dependency or lockfile churn in this repo (the first pass’s pnpm add caused a 2,272-line lockfile diff across unrelated packages before being reverted). What was done: an isolated npm project, entirely outside the pnpm workspace ($CLAUDE_JOB_DIR/tmp/assistant-ui-typecheck — not inside this repository, no pnpm-lock.yaml/package.json touched):
A probe.ts there assigns a literal value shaped field-for-field like buildExternalStoreAdapter()’s real return type (messages, isRunning, extras.toolSteps, extras.opportunity, onNew, onCancel, convertMessage) to the REAL, shipped ExternalStoreAdapter<T> generic type imported from @assistant-ui/core:
A clean compile against the actual shipped .d.ts is real, executable evidence of structural compatibility — not an inference from documentation. git status --short pnpm-lock.yaml apps/dashboard/package.json apps/api/package.json in this repo shows nothing: zero footprint. Version chosen (0.2.23, and @assistant-ui/react@0.14.28 if the wrapping React package is installed later) was confirmed 8+ days old, clearing the workspace’s 7-day minimumReleaseAge supply-chain gate — the gate was respected, not bypassed, when the first pass hit it. What was NOT done: rendering the adapter through useExternalStoreRuntime + <Thread /> in an actual browser/jsdom pass. That needs the real npm package installed in the workspace (a pnpm add, correctly scoped this time to a version ≥7 days old) — a small, low-risk, well-understood follow-up for whoever picks up the next card, not claimed as done here. A secondary, real signal from reading the shipped source during this probe (not new exploration — carried over from the prior pass, now correctly framed): useExternalStoreRuntime/useAssistantTransportRuntime in @assistant-ui/react@0.14.28 live under an internal legacy-runtime/ path, re-exported from the still-independently-versioned @assistant-ui/core@0.2.23. Both are live, public, and exported from the package’s root entry — “supported, re-exported” is not “deprecated” — but it is worth carrying as a roadmap-risk note.

​
5. Compatibility matrix

​
6. Decision: ADAPT, staged — keep ChatThreadView as the production shell now; build the ExternalStoreRuntime bridge alongside, cut over only at parity

Unchanged from the first pass’s conclusion, now on firmer, corrected evidence:
  • Keep ChatThreadView.tsx (1356 lines of working behavior: streaming, tool chips, grouped citations, feedback, regenerate, model picker, context-pressure nudge, the ContextPill/ResearchArtifactPanel suspend-approve flow) as the production surface through A1+. There is currently nothing new to render that the current shell can’t already show — the trajectory evals (§3) pin gaps in what the BACKEND emits, not in what the shell can display.
  • Adopt as the target bridge ExternalStoreRuntime (never AssistantTransport — it would hand wire-protocol ownership to a third party, directly against playbook §13’s “no debe convertirse en un segundo dueño del workflow”). The spike (§4) now proves the mapping is clean, holds no state of its own, correctly patches a real multi-row artifact by identity, and type-checks against the actual shipped package.
  • This operationalizes the playbook’s own §14 line — “ChatThreadView custom: SPIKE; conservar como fallback temporal hasta paridad del adapter” — rather than relitigating it.
Rollback: the spike is two files under apps/dashboard/src/redesign/spike/, imported by nothing — rm -rf apps/dashboard/src/redesign/spike fully reverts it. The type-compat probe lives entirely outside this repository ($CLAUDE_JOB_DIR/tmp) and leaves nothing to revert. The two vitest.config.ts/vitest.trajectory.config.ts edits are one-line reverts each. No production file’s behavior changed; no dependency was left installed. No second owner of state: the adapter re-derives messages/extras from its inputs on every call (proven structurally, §4.2); Mastra + ChatService + ChatStreamHub remain the only writers of run/thread/message state. Enrichment / provider-neutral port: confirmed to already exist (entity-enrichment-provider.port.ts, implemented by apollo.adapter.ts) — out of scope for A0, unaffected by this shell decision either way, since ExternalStoreRuntime has no opinion on backend tool architecture.

​
7. What stays red by design

All 10 trajectory evals (§3) are expected to stay red until the corresponding backend behavior ships, each naming the exact file and missing primitive (checkpoint, plan, an artifact.* stream event, a suspended ledger status, a capabilityId on the route result, a monitor route, a replay buffer, resolveCheckpoint, N-capability query construction). Whoever picks up A1 should start here: the gap between “what chat.service.ts does today” and “what the playbook’s generic model requires” is now executable and verified against real staging, not prose.