tx

tx trace

Execution tracing for debugging agent run failures

Purpose

tx trace provides execution tracing for debugging agent run failures. It combines three primitives:

  1. IO Capture: File paths to transcripts, stderr, and stdout stored on run records
  2. Metrics Events: Operational spans and metrics written to the events table
  3. Run Heartbeat: Stall detection and reaping based on transcript or log progress samples

Traces help answer "why did this run fail?" by correlating internal service timing with agent tool calls.

At A Glance

Failure Triage

Rendering diagram…
A run leaves breadcrumbs in files and events. `tx trace` pulls them back together during debugging.

Stall Detection

Rendering diagram…
Heartbeat and transcript growth determine whether a run is healthy, stalled, or ready to reap.

List Recent Runs

tx trace list [options]

Options

OptionDescription
--hours <n>Time window in hours (default: 24)
--limit <n>Maximum number of results (default: 20)
--jsonOutput as JSON

Examples

# List recent runs
tx trace list

# Output:
# Recent Runs (last 24h)
# ───────────────────────────────────────────────────────────────────────────
# ID              Agent            Task            Status      Spans      Time
# run-abc12345    tx-implementer   tx-6407952c     failed         23      2h ago
# run-def67890    tx-implementer   tx-8a4be920     success        45      3h ago

# Show last 48 hours
tx trace list --hours 48

# JSON output
tx trace list --json
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

const recent = await tx.runs.list({
  limit: 20,
  status: ['failed', 'running'],
})

console.log(recent.runs.map((run) => ({
  id: run.id,
  status: run.status,
  agent: run.agent,
})))

Run listing is available via CLI. The MCP server exposes heartbeat, stalled, and reap tools (see below).

tx trace list --json
tx trace show run-abc12345 --json
GET /api/runs?limit=20&status=failed

Query Parameters

ParameterDescription
limitMax results (default: 20)
cursorPagination cursor
statusFilter by status (comma-separated)
taskIdFilter by task ID
agentFilter by agent name

Example

curl http://localhost:3456/api/runs?limit=10&status=failed

Response

{
  "runs": [
    {
      "id": "run-abc12345",
      "taskId": "tx-6407952c",
      "agent": "tx-implementer",
      "status": "failed",
      "startedAt": "2025-01-15T14:00:00.000Z",
      "endedAt": "2025-01-15T14:27:00.000Z",
      "exitCode": 1,
      "errorMessage": "ValidationError: Cannot mark blocked task as done"
    }
  ],
  "nextCursor": null,
  "hasMore": false,
  "total": 1
}

Run Heartbeat and Stall Detection

# Record a heartbeat sample for a running run
tx trace heartbeat run-abc12345 --transcript-bytes 2048 --delta-bytes 256

# List stalled runs (default transcript idle threshold: 300s)
tx trace stalled

# Reap stalled runs
tx trace stalled --reap --transcript-idle-seconds 300
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

await tx.runs.heartbeat('run-abc12345', {
  transcriptBytes: 2048,
  deltaBytes: 256,
})

const stalled = await tx.runs.stalled({ transcriptIdleSeconds: 300 })
const reaped = await tx.runs.reap({ transcriptIdleSeconds: 300, dryRun: false })
{
  "tool": "tx_run_heartbeat",
  "arguments": {
    "runId": "run-abc12345",
    "transcriptBytes": 2048,
    "deltaBytes": 256
  }
}
{
  "tool": "tx_run_stalled",
  "arguments": {
    "transcriptIdleSeconds": 300
  }
}
{
  "tool": "tx_run_reap",
  "arguments": {
    "transcriptIdleSeconds": 300
  }
}
POST /api/runs/:id/heartbeat
GET /api/runs/stalled?transcriptIdleSeconds=300
POST /api/runs/stalled/reap
curl -X POST http://localhost:3456/api/runs/run-abc12345/heartbeat \
  -H "Content-Type: application/json" \
  -d '{"transcriptBytes":2048,"deltaBytes":256}'

Show Run Details

tx trace show <run-id> [options]

Arguments

ArgumentRequiredDescription
<run-id>YesRun ID (e.g., run-abc12345)

Options

OptionDescription
--fullCombined timeline with tool calls from transcript
--jsonOutput as JSON

Examples

# Show metrics events for a run
tx trace show run-abc12345

# Output:
# Run: run-abc12345
# Agent: tx-implementer
# Task: tx-6407952c
# Status: failed
# Transcript: runs/run-abc12345.jsonl
# Stderr: runs/run-abc12345.stderr
#
# Metrics Events:
# ───────────────────────────────────────────────────────────────────────────
# 14:23:45  [span]  TaskService.show                    12ms  ok
# 14:23:46  [span]  ReadyService.getReady               45ms  ok
# 14:24:12  [span]  TaskService.update                   8ms  ok
# 14:27:09  [span]  TaskService.done                   156ms  error
#           └─ ValidationError: Cannot mark blocked task as done

# Combined timeline (metrics + transcript tool calls)
tx trace show run-abc12345 --full

# Output:
# Combined Timeline:
# ───────────────────────────────────────────────────────────────────────────
# 14:23:45.100  [span]    TaskService.show              12ms ok
# 14:23:45.200  [tool]    Bash: tx show tx-6407952c
# 14:23:46.000  [span]    ReadyService.getReady         45ms ok
# 14:23:46.100  [tool]    Read: /path/to/file.ts
# 14:27:09.000  [span]    TaskService.done              156ms error
#               └─ ValidationError: Cannot mark blocked task as done
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

const detail = await tx.runs.get('run-abc12345')

console.log(detail.run.status)
console.log(detail.messages.length)
console.log(detail.logs.stderr)

Run detail inspection is available via CLI. The MCP server exposes heartbeat, stalled, and reap tools.

tx trace show run-abc12345 --full --json
GET /api/runs/:id

Example

curl http://localhost:3456/api/runs/run-abc12345

Response

{
  "run": {
    "id": "run-abc12345",
    "taskId": "tx-6407952c",
    "agent": "tx-implementer",
    "status": "failed",
    "startedAt": "2025-01-15T14:00:00.000Z",
    "endedAt": "2025-01-15T14:27:00.000Z",
    "transcriptPath": "runs/run-abc12345.jsonl",
    "stderrPath": "runs/run-abc12345.stderr",
    "exitCode": 1,
    "errorMessage": "ValidationError: Cannot mark blocked task as done"
  },
  "messages": [
    {
      "role": "assistant",
      "content": "I'll read the task details...",
      "type": "text",
      "timestamp": "2025-01-15T14:23:45.000Z"
    }
  ]
}

View Transcript

tx trace transcript <run-id>

Outputs raw JSONL content from the transcript file. Designed to be piped to jq for filtering:

# View full transcript
tx trace transcript run-abc12345

# Filter to tool calls only
tx trace transcript run-abc12345 | jq 'select(.type == "tool_use")'

# Filter to assistant messages
tx trace transcript run-abc12345 | jq 'select(.type == "assistant")'
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

const transcript = await tx.runs.transcript('run-abc12345')
const toolCalls = transcript.filter((message) => message.type === 'tool_use')

Use the CLI to access transcripts:

tx trace transcript run-abc12345

Transcript messages are returned as part of the run detail endpoint:

GET /api/runs/:id

The messages array in the response contains parsed transcript entries.

View Stderr

tx trace stderr <run-id>

Outputs raw stderr content. Useful for debugging crashes and runtime errors:

tx trace stderr run-abc12345

# Output:
# Error: SQLITE_BUSY: database is locked
#     at Database.exec (/path/to/better-sqlite3.js:...)
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

const stderr = await tx.runs.stderr('run-abc12345', { tail: 50 })
console.log(stderr.content)

Use the CLI:

tx trace stderr run-abc12345
GET /api/runs/:id/stderr

Query Parameters

ParameterDescription
tailNumber of lines from the end (default: all)

Example

curl http://localhost:3456/api/runs/run-abc12345/stderr?tail=50

Response

{
  "content": "Error: SQLITE_BUSY: database is locked\n    at Database.exec ...",
  "truncated": false
}

Show Recent Errors

tx trace errors [options]

Options

OptionDescription
--hours <n>Time window in hours (default: 24)
--limit <n>Maximum number of results (default: 20)
--jsonOutput as JSON

Example

tx trace errors

# Output:
# Recent Errors (last 24h)
# ────────────────────────────────────────────────────────────────────────────────
# 14:27:09  [span]   run-abc12345    tx-implementer
#           Name: TaskService.done
#           Error: ValidationError: Cannot mark blocked task as done
#           Task: tx-6407952c
#           Duration: 156ms
#
# 13:15:22  [run]    run-xyz98765    tx-planner
#           Name: Run failed
#           Error: Process exited with code 1

Aggregates errors from three sources:

  • Failed runs: runs with status = 'failed'
  • Error spans: spans with status = 'error' in metadata
  • Error events: events with event_type = 'error'
import { TxClient } from '@jamesaphoenix/tx-agent-sdk'

const tx = new TxClient({ dbPath: '.tx/tasks.db' })

const errors = await tx.runs.errors({ hours: 24, limit: 10 })
console.log(errors.map((entry) => ({
  source: entry.source,
  runId: entry.runId,
  error: entry.error,
})))

Use the CLI:

tx trace errors --json
GET /api/runs/errors?hours=24&limit=10

TracingService

The TracingService provides programmatic tracing within Effect-TS service code.

withSpan

Wraps an Effect with a named span that records duration and status to the events table:

import { TracingService } from '@jamesaphoenix/tx'
import { Effect } from 'effect'

const program = Effect.gen(function* () {
  const tracing = yield* TracingService

  // Wrap an operation with a span
  const result = yield* tracing.withSpan(
    'MyService.processTask',
    { attributes: { taskId: 'tx-abc123' } },
    Effect.gen(function* () {
      // ... your operation here
      return 'done'
    })
  )
  // Records: event_type='span', content='MyService.processTask',
  //          duration_ms=<elapsed>, metadata={status:'ok', attributes:{...}}
})

recordMetric

Records a point-in-time metric value:

const program = Effect.gen(function* () {
  const tracing = yield* TracingService

  yield* tracing.recordMetric('queue_depth', 42, { agent: 'tx-implementer' })
  // Records: event_type='metric', content='queue_depth',
  //          duration_ms=42, metadata={agent:'tx-implementer'}
})

withRunContext

Scopes a run ID to all nested spans using Effect's FiberRef:

const program = Effect.gen(function* () {
  const tracing = yield* TracingService

  yield* tracing.withRunContext('run-abc12345',
    Effect.gen(function* () {
      // All spans within this scope will have run_id = 'run-abc12345'
      yield* tracing.withSpan('nested.operation', {},
        Effect.succeed('value')
      )
    })
  )
})

Noop Implementation

When tracing is disabled, TracingServiceNoop is used. It passes effects through unchanged with zero overhead:

import { TracingServiceNoop } from '@jamesaphoenix/tx'

// Zero-overhead: withSpan returns the effect unchanged
// recordMetric returns Effect.void
// withRunContext returns the effect unchanged

IO Capture Architecture

tx stores file paths, not file contents. The orchestrator decides how and where to capture IO:

.tx/
├── tasks.db
└── runs/
    ├── run-abc12345.jsonl    # Claude transcript (stream-json output)
    ├── run-abc12345.stderr   # Stderr capture
    ├── run-abc12345.stdout   # Stdout capture (optional)
    └── run-def67890.jsonl

Run Record Columns

ColumnDescription
transcript_pathPath to JSONL transcript file
stderr_pathPath to stderr capture file
stdout_pathPath to stdout capture file

Orchestrator Example

#!/bin/bash
RUN_ID="run-$(openssl rand -hex 4)"
RUN_DIR=".tx/runs"
mkdir -p "$RUN_DIR"

# Create run record
curl -X POST http://localhost:3456/api/runs \
  -H "Content-Type: application/json" \
  -d "{\"agent\": \"tx-implementer\", \"taskId\": \"$TASK_ID\"}"

# Run agent with IO capture
claude --print --output-format stream-json "$PROMPT" \
  > "$RUN_DIR/$RUN_ID.jsonl" \
  2> "$RUN_DIR/$RUN_ID.stderr"

# Update run record with paths and status
curl -X PATCH "http://localhost:3456/api/runs/$RUN_ID" \
  -H "Content-Type: application/json" \
  -d "{\"status\": \"completed\", \"transcriptPath\": \"runs/$RUN_ID.jsonl\"}"

Transcript Adapters

Different LLM tools produce different transcript formats. tx includes adapters for parsing:

  • ClaudeCodeAdapter: Parses Claude's --output-format stream-json JSONL
  • GenericJSONLAdapter: Fallback for other JSONL-producing tools

The adapter is selected automatically based on the agent type stored in the run record.

  • tx ready: List tasks available to work on
  • tx done: Complete a task and end the logical work unit
  • tx sync: Export and import append-only stream events

tx msg send / tx msg inbox

Channel-based agent-to-agent messaging

Headful Experience

Dashboard UX for tasks, docs, runs, and cycles

On this page

PurposeAt A GlanceList Recent RunsOptionsExamplesQuery ParametersExampleResponseRun Heartbeat and Stall DetectionShow Run DetailsArgumentsOptionsExamplesExampleResponseView TranscriptView StderrQuery ParametersExampleResponseShow Recent ErrorsOptionsExampleTracingServicewithSpanrecordMetricwithRunContextNoop ImplementationIO Capture ArchitectureRun Record ColumnsOrchestrator ExampleTranscript AdaptersRelated Commands