1
00:00:00,100 --> 00:00:04,000
So, uh, if you've been running the alpha
builds of the Codex CLI lately,

2
00:00:04,560 --> 00:00:10,020
you-you-you probably noticed things got a
little, well, messy if you were trying to

3
00:00:10,060 --> 00:00:12,860
debug complex multi-agent runs.

4
00:00:12,960 --> 00:00:18,140
Oh, and before I forget, massive thanks to
Jellypod to help make this daily show a

5
00:00:18,160 --> 00:00:19,300
reality.

6
00:00:19,320 --> 00:00:24,340
But yeah, we-we're talking about rollout
tracing, which is that opt-in diagnostic

7
00:00:24,380 --> 00:00:30,800
system that basically logs every raw event
locally so you can compile them offline.

8
00:00:30,840 --> 00:00:34,750
And in the recent
zero-dot-one-forty-three-dot-zero alpha builds,

9
00:00:35,240 --> 00:00:39,200
if you tried to run the codex debug
trace-reduce command on those logs,

10
00:00:39,700 --> 00:00:41,660
it would just completely crash.

11
00:00:41,720 --> 00:00:45,360
Like, boom, stack trace, game over.

12
00:00:47,038 --> 00:00:49,828
Yeah, it was-it was pretty spectacular.

13
00:00:50,398 --> 00:00:55,918
And the culprit was actually this tiny,
seemingly harmless metadata passthrough

14
00:00:55,958 --> 00:01:01,698
field, specifically turn_id inside the
model-visible tool calls.

15
00:01:01,818 --> 00:01:05,878
The compiler, uh, it-it just didn't know
how to handle it when it was trying to hash

16
00:01:05,918 --> 00:01:07,558
and compare the JSON bodies.

17
00:01:08,394 --> 00:01:12,754
Right, because the trace-reduce compiler
is trying to look at these massive JSON

18
00:01:12,894 --> 00:01:17,154
payloads and dedup or group them by
hashing the body.

19
00:01:17,174 --> 00:01:21,744
But when you have fluctuating metadata
like turn_id injected into the tool calls,

20
00:01:22,284 --> 00:01:26,114
the hashes don't match even if the
semantic tool call is identical.

21
00:01:26,654 --> 00:01:29,154
The compiler was just choking on that
variance.

22
00:01:30,085 --> 00:01:30,685
Exactly.

23
00:01:31,135 --> 00:01:35,905
So, the fix they just shipped in the
latest zero-dot-one-forty-three-dot-zero-alpha

24
00:01:35,965 --> 00:01:40,405
releases is actually pretty
straightforward but crucial.

25
00:01:40,465 --> 00:01:44,955
They, uh, they literally just strip out
those non-identity metadata fields right

26
00:01:44,985 --> 00:01:46,715
before the hashing and comparison stage.

27
00:01:47,225 --> 00:01:51,635
So now, the compiler ignores turn_id and
other transient fields,

28
00:01:52,105 --> 00:01:57,655
hashes the actual structural JSON, and
boom, your offline diagnostic compilation is

29
00:01:57,745 --> 00:01:59,045
stable again.

30
00:01:59,093 --> 00:02:03,603
It's honestly a lifesaver if you're
dealing with deep nested loops where agents are

31
00:02:03,653 --> 00:02:05,753
calling tools back and forth.

32
00:02:05,813 --> 00:02:10,673
If you can't reduce those traces, you're
basically staring at a multi-gigabyte text

33
00:02:10,693 --> 00:02:13,873
file of raw JSON events, which is...

34
00:02:14,493 --> 00:02:15,793
yeah, not fun.

35
00:02:16,499 --> 00:02:17,958
Oh, absolutely.

36
00:02:18,539 --> 00:02:23,819
Staring at raw JSON is-is a special kind
of developer purgatory.

37
00:02:23,859 --> 00:02:28,439
But-but here's the thing, setting this up
is actually a bit hidden.

38
00:02:28,479 --> 00:02:33,819
To even turn on rollout tracing, you have
to set this specific environment variable:

39
00:02:34,319 --> 00:02:39,799
CODEX_ROLLOUT_TRACE_ROOT and point it to a
local directory.

40
00:02:39,819 --> 00:02:44,499
And once the raw events write there,
that's when you run that hidden CLI tool,

41
00:02:45,039 --> 00:02:51,779
codex debug trace-reduce --input <path>
--output <path> to

42
00:02:51,819 --> 00:02:54,219
compile it all into a readable graph.

43
00:02:55,026 --> 00:02:56,956
Wait, so it's writing everything to disk?

44
00:02:57,316 --> 00:02:59,076
Like, everything-everything?

45
00:03:01,044 --> 00:03:03,653
Yes, and that is a massive security
gotcha.

46
00:03:03,693 --> 00:03:08,203
Because rollout traces write
*everything*-we're talking raw source code,

47
00:03:08,593 --> 00:03:13,853
active API credentials that might be in
your environment, and raw prompts-straight

48
00:03:13,913 --> 00:03:16,193
to your local disk in plain text.

49
00:03:16,813 --> 00:03:20,793
If you don't add that trace directory to
your global .gitignore,

50
00:03:20,813 --> 00:03:23,074
you are one lazy git add .

51
00:03:23,373 --> 00:03:28,033
away from pushing your entire local secret
key vault to a public repo.

52
00:03:29,292 --> 00:03:29,982
Oh, yikes.

53
00:03:30,652 --> 00:03:33,502
Yeah, that-that is a recipe for an
absolute disaster.

54
00:03:34,112 --> 00:03:37,652
Seriously, if you are listening to this
right now and using the alpha,

55
00:03:38,292 --> 00:03:42,762
go add your trace root directory to your
.gitignore immediately.

56
00:03:43,392 --> 00:03:45,152
Like, pause the podcast and do it.

57
00:03:46,148 --> 00:03:48,228
Seriously, do it now.

58
00:03:48,788 --> 00:03:53,848
And-and while you're messing with the
alpha channel, there's another really weird UX

59
00:03:53,888 --> 00:03:57,468
bug in the terminal user interface, the
TUI.

60
00:03:57,528 --> 00:04:02,428
If you use the /feedback command to report
an issue, and you type your feedback

61
00:04:02,488 --> 00:04:04,448
right after the command on the same
line...

62
00:04:05,168 --> 00:04:07,548
it-it doesn't actually go to the feedback
pipeline.

63
00:04:08,330 --> 00:04:08,859
Wait, what?

64
00:04:09,310 --> 00:04:10,090
Where does it go then?

65
00:04:11,687 --> 00:04:12,856
It gets misrouted.

66
00:04:13,546 --> 00:04:19,446
The CLI treats any trailing arguments
after /feedback as a standard chat prompt.

67
00:04:20,016 --> 00:04:24,616
So instead of filing a bug report, you're
literally just sending your feedback text

68
00:04:24,656 --> 00:04:26,996
to the LLM agent as your next chat turn.

69
00:04:27,536 --> 00:04:28,896
It's-it's wild.

70
00:04:30,282 --> 00:04:31,142
That is...

71
00:04:31,182 --> 00:04:32,002
wow.

72
00:04:32,102 --> 00:04:35,502
So the agent is just going to reply to you
complaining about the bug,

73
00:04:35,982 --> 00:04:37,942
instead of actually filing it.

74
00:04:38,022 --> 00:04:42,822
Okay, so the workaround there is to just
type /feedback, hit enter,

75
00:04:43,302 --> 00:04:44,942
and wait for the dedicated prompt.

76
00:04:45,622 --> 00:04:45,982
Got it.

77
00:04:46,562 --> 00:04:49,722
Well, that's it for our quick look at the
Codex diagnostics.

78
00:04:50,302 --> 00:04:53,562
Keep those trace roots out of Git, and
we'll catch you next time.