Skip to main content

Staging Blackbox QA — CLI/MCP/API

Date: 2026-06-28 Branch: claude/driftless-staging-blackbox-qa-31y5rx Scope: Staging — CLI v0.26.0, MCP tool-registry, REST API surface Method: Source-level blackbox (staging API blocked by proxy policy from this environment; CLI unit tests + deep code inspection)

Environment


Test Suite Results


Issues Found


BUG-01 · getGitRemote() silently returns null for proxy-format git remotes

Severity: HIGH Surface: CLI Exact command: Any workspace-dependent CLI command run in a Claude Code remote environment Git remote URL in this environment: http://local_proxy@127.0.0.1:41729/git/joseluistello/Driftless Root cause: getGitRemote() (apps/cli/src/git.ts) branches on whether the URL contains @. The proxy URL has local_proxy@127.0.0.1, so it hits the SSH parsing path. The SSH path splits on : and checks that exactly 2 segments remain — but 41729/git/joseluistello/Driftless has 4, so the function returns null. Expected: Either getGitRemote() handles standard HTTP remotes with port numbers (these are common in managed CI/cloud environments), or workspace resolution falls through cleanly to a helpful error rather than “no git remote found.” Actual:
Evidence: Unit test src/git.spec.ts:71 fails in this environment:
The test calls the live getGitRemote() without mocking — it exposes the production bug. Suggested validation:
  1. git config --get remote.origin.url in a Claude Code remote env → confirm proxy URL
  2. Call getGitRemote() directly → confirm it returns null
  3. Fix: add a HTTPS-with-port path: URL.hostname/URL.pathname parsing that ignores the port

BUG-02 · All workspace-dependent commands fail with misleading “no git remote found” instead of proper errors

Severity: HIGH Surface: CLI Commands affected: context search, context list, context get, area list, collection list, collection records, broker connections, broker invoke, project list, project get Exact command:
Expected: Argument validation errors specific to each command. Actual: All print Error: no git remote found. — the workspace resolver fires before argument checking. Root cause: resolveWorkspaceSlug() is called at the top of every command handler before argument validation. When the API is unreachable and no workspace is cached, it exits with the git remote error. Argument validation that would produce a useful message never runs. Suggested validation: Run each command above with no arguments; observe whether the error is workspace-related or argument-related. Fix: validate required positional args before calling resolveWorkspaceSlug().

BUG-03 · Unit test BUG-01 repro: git.spec.ts calls live getGitRemote() without mocking

Severity: MEDIUM Surface: CLI test suite File: apps/cli/src/git.spec.ts:71 Test:
Expected: Test passes regardless of the git remote URL format in the test environment. Actual: Fails when git remote is in proxy format (http://user@host:port/path/org/repo).
Suggested validation: Mock execSync with known URL strings; test the parsing logic, not the live environment.

BUG-04 · context search limit default differs between CLI (server’s 50) and MCP (5)

Severity: MEDIUM Surface: CLI help text vs MCP tool schema CLI help says:
MCP tool schema:
Expected: Consistent default and maximum across CLI and MCP. Actual: An agent using the CLI gets up to 50 results; an agent using MCP gets 5 results by default (max 20). Worse — the CLI help mentions server hard-caps at 50, which is now a lie for MCP users (hard cap is 20 via MCP). Suggested validation:
  1. driftless context search "auth" --json | jq length → observe count
  2. tools/call driftless_context_search { query: "auth" } → observe count
  3. Compare. Fix: align defaults, or document the difference explicitly in MCP tool description.

BUG-05 · driftless_broker MCP tool is hidden by default with no discoverability signal

Severity: MEDIUM Surface: MCP tools/list Exact request:
Expected: Either driftless_broker appears in tools/list, or the response carries a hint: “broker available — set DRIFTLESS_BROKER_ENABLED=true”. Actual: driftless_broker is silently absent from tools/list unless the server env has DRIFTLESS_BROKER_ENABLED=true. The CLI driftless broker --help surfaces it; MCP does not. An agent parsing MCP tools cannot know the broker exists. Code:
Suggested validation: tools/list on staging with broker flag off → confirm broker absent, no hint. Fix candidates: (a) Always include driftless_broker in tools/list but return a 503 on call; (b) Add a next_action hint in the driftless_areas or driftless_workspaces tool result pointing to broker when the flag is off.

BUG-06 · collection records --status X without --limit fetches unbounded (legacy endpoint)

Severity: MEDIUM Surface: CLI Exact command:
CLI help claims: bounded server-side; follow nextCursor to page Actual: When only --status is given (no --limit or --cursor), the code sends to the legacy endpoint that returns a bare array with no pagination:
A collection with 1,000 records filtered to status=in_progress returns all of them unbounded. No nextCursor is emitted, so the agent doesn’t know it’s an unbounded response. Expected: Help text should say “bounded when —limit is given; —status alone is unbounded”. Fix candidate: Send a default server-side limit even when only --status is set, or remove the legacy path and always use the keyset envelope. Suggested validation:
  1. Create a collection with 100+ records in in_progress status
  2. driftless collection records <id> --status in_progress → observe record count, check for nextCursor

BUG-07 · broker connections --help and all broker subcommand --help show parent help

Severity: LOW Surface: CLI Exact command:
Expected: Help specific to the connections (or operations, invoke, etc.) subcommand. Actual: Shows the full broker --help text. The --help flag is consumed at the top of brokerCommand() before subcommand dispatch. Code:
This means there is no per-subcommand help for broker. An agent running broker operations --help expecting to learn about the --refresh flag gets the full reference dump instead. Suggested validation: Run each subcommand with --help and confirm output is identical. Not a crash — just poor UX.

BUG-08 · MCP driftless_project action:‘list’ has no limit parameter; large workspace returns all

Severity: MEDIUM Surface: MCP Exact call:
Expected: Paginated response with has_more/cursor hints; or at minimum a documented bound. Actual: The implementation:
No limit param sent; no has_more field in response. A workspace with 200 active projects returns them all, potentially flooding the agent context. Suggested validation: Create 50+ projects on staging; call driftless_project action:'list' via MCP → observe if all are returned. Fix: Add limit/offset params to schema; emit { items, count, has_more } envelope.

BUG-09 · MCP driftless_collection action:‘list’ also has no limit

Severity: LOW-MEDIUM Surface: MCP Same pattern as BUG-08 — action:'list' for collections:
No limit. A workspace with many collections returns them all. The MCP driftless_areas action:‘list’ uses a hard 40-item cap client-side; driftless_collection does not.

BUG-10 · Smoke test uses non-workspace-scoped /topics/retrieve endpoint

Severity: LOW Surface: REST API smoke test File: scripts/harness/smoke.sh:line ~175
MCP code uses:
The smoke test tests POST /api/v1/topics/retrieve (no workspace in path), while the actual MCP-called endpoint is workspace-scoped. If the server resolves workspace from the auth key for the bare endpoint, the smoke test passes but doesn’t verify the actual MCP path. If the server requires the workspace slug, the smoke test passes on a different code path than MCP uses in production. Suggested validation: Compare response of both endpoints with the same payload.

BUG-11 · context list --suggested and --all fetch unbounded from CLI (large workspaces)

Severity: LOW-MEDIUM Surface: CLI Code:
When --suggested or --all is used, the server request has no limit parameter. A workspace with 10,000 topics triggers an unbounded fetch, then a client-side sort and slice. This is a memory/latency risk for large workspaces. Suggested validation: On a staging workspace with 200+ topics:
  1. driftless context list --suggested --json | jq length → observe count
  2. driftless context list --all --json | jq length → observe count

Summary

Highest-severity staging blockers:
  • BUG-01 and BUG-02 together mean any agent operating in a Claude Code remote environment (proxy git remote, no cached workspace) cannot use ANY CLI command that resolves a workspace. This affects onboarding, first-use after rotating a key, and CI that sets DRIFTLESS_API_KEY but has no prior login state.
  • BUG-04 means agents using CLI and agents using MCP see systematically different result sets for the same search query.
  • BUG-05 means the broker surface is invisible to MCP agents unless they know to look at the CLI help for feature flag hints.
Bugs found: 11 Duplicate cards skipped: 0 (first QA run; no prior cards to deduplicate against) New cards to create: 11