In this field report
A hands-on NVIDIA NeMo Relay and Arize Phoenix field report: trace execution, verify the outcome, and treat the recording itself as sensitive data.
The useful failure
Our first test looked almost too simple: ask an agent to execute a tiny script and return exactly VALUE=42. The terminal tool succeeded. The final model answer did not. That split is the reason to keep a flight recorder: a successful tool invocation is not the same as a successful task.
We used NVIDIA’s Hermes tracing tutorial in an isolated Ubuntu virtual machine, with a pinned Hermes 0.21.1 runtime and NeMo Relay 0.8.3. The model was NVIDIA Nemotron 3.5 Lightning through the NVIDIA Build service—not inference running on one of our local RTX cards. NVIDIA’s tutorial pairs exact-result checks with traces, rather than using a green tool status as the final verdict. NVIDIA’s tracing walkthrough.
What each component does
NeMo Relay is NVIDIA’s runtime instrumentation layer; Hermes is the agent harness; Arize Phoenix is the separate trace-inspection backend. They are not three names for one NVIDIA application. Relay can export lifecycle events, structured trajectories, and OpenTelemetry spans. Phoenix receives the span representation rather than reading the local trajectory files. NVIDIA’s companion guide; Arize Phoenix documentation.
Relay does not choose an agent’s plan or replace its harness, and it is not a model server or hosted tracing service. This article is about understanding execution, not claiming that recording makes an agent autonomous, accurate, or safe by itself. NVIDIA NeMo Relay FAQ (v0.8.4).
Our retained local results
Tool succeeded; the final answer did not match.
2 model calls · 7,274 tokensTool succeeded; the exact answer was returned.
3 model calls · 10,945 tokensOne failure and one pass, not a reliability rate. Token counts are reported totals, not a speed or cost comparison.
| Synthetic task | Verified outcome | Model calls | Tool calls | Reported tokens |
|---|---|---|---|---|
| Exact-answer smoke, first run | Fail: tool succeeded; final answer malformed | 2 | 1 | 7,274 |
| Same smoke, one controlled repeat | Pass: exact answer returned | 3 | 1 | 10,945 |
| Conference research example | Pass: expected conference and report verified | 5 | 4 | 34,857 |
| Fabricated queue audit | Pass: stale note rejected; exact counts returned | Not asserted from this receipt | 2 read calls | 7,125 |
These are our local receipts, not the numerical example shown in NVIDIA’s documentation. The tasks differ. This is not a model ranking, token-efficiency contest, latency benchmark, or reliability-rate estimate. The two smoke runs establish one failure and one pass—not a robust success percentage.
The saved-response comparison localized the first failure to the model’s recorded answer, not a display-only glitch. The trace and verifier answered different questions: the former showed the command had run; the latter showed the requested exact answer had not been delivered.
The recording needs its own privacy test
We first found fabricated note text in a saved trace even with full-payload capture disabled. A narrow redaction canary subsequently removed one unique fake marker from the tested trace surfaces while retaining call relationships, durations, and token totals.
The expanded test did not pass. Some direct-event paths never reached the exporters, so their redaction was not tested. Fabricated values also remained in Hermes session-storage and log surfaces. That is a failed expanded-coverage gate, not evidence that every real payload is safe. NVIDIA separately cautions that traces can contain prompts, model answers, tool arguments, outputs, and file paths. NVIDIA’s data-sharing warning.
We therefore have not enabled blanket native Relay capture across the shared live agent gateway. Our later production observability path uses separate allowlisted metadata exporters. It is not equivalent to a native Relay recording of every chat, and it is not permission to publish raw traces.
A point-in-time health check
At 4:32 p.m. Central on October 26, our private Phoenix container was running and its readiness check returned HTTP 200. It still matched the image previously verified as Phoenix 20.19.0. The four existing metadata collectors were active with zero restart counts and no consecutive transport failures; the scheduled attention report was fresh.
Healthy capture does not mean everything is fine. That report recorded 26 tool-error events in the preceding 24 hours, with per-tool attribution unavailable, and 20 review events still marked unreviewed. Those are event counts, not 26 failed render jobs or 20 distinct drafts. The historical failed visual-checker canary also remains unresolved, even after it ages out of a rolling query window. Zero recent checker records is not a successful quality evaluation.
The takeaway
A flight recorder makes an agent’s work inspectable; it does not replace a task verifier, privacy policy, or human acceptance. Start with a small synthetic task, pin the runtime, preserve every outcome, and check the result before interpreting the trace. If you want to claim that a harness change helps, use repeated matched tests with one deliberate change—not a single faster-looking trace.