1
00:00:00,000 --> 00:00:01,601
Welcome to the show.

2
00:00:01,601 --> 00:00:05,444
James, imagine this: you ask Claude for a
code fix,

3
00:00:05,444 --> 00:00:07,443
and instead of a neat little patch,

4
00:00:07,443 --> 00:00:11,607
it starts sprinkling random Thai or
Chinese characters into an English

5
00:00:11,607 --> 00:00:15,443
reply, drops a syntax error into the
generated code,

6
00:00:15,443 --> 00:00:16,724
and then just...

7
00:00:16,724 --> 00:00:19,763
cuts the answer off halfway with NO error
message.

8
00:00:19,763 --> 00:00:21,681
That was not one-off weirdness.

9
00:00:21,681 --> 00:00:24,961
In Anthropic's September 2025 postmortem,

10
00:00:24,961 --> 00:00:27,526
they said three separate infrastructure
bugs,

11
00:00:27,526 --> 00:00:32,719
one after another, degraded Claude
responses from August through

12
00:00:27,526 --> 00:00:32,719
mid-September,

13
00:00:32,719 --> 00:00:37,491
and roughly 30% of Claude Code users saw
at least one bad message in

14
00:00:37,491 --> 00:00:41,678
that window.Thirty percent is
the number that sticks.

15
00:00:41,678 --> 00:00:45,316
Not "a few unlucky edge cases" -- THIRTY.

16
00:00:45,316 --> 00:00:49,013
That's basically one out of three Claude
Code users getting at least

17
00:00:49,013 --> 00:00:52,987
one poisoned response between August and
mid-September.

18
00:00:52,987 --> 00:00:56,822
And the no-error-surfaced part is the bit
that freaks me out,

19
00:00:56,822 --> 00:00:59,062
because if the model crashes loudly,

20
00:00:59,062 --> 00:01:00,983
fine, you know you've got a problem.

21
00:01:00,983 --> 00:01:04,022
If it answers confidently with corrupted
output,

22
00:01:04,022 --> 00:01:06,743
now the developer starts doubting their
own benchmark,

23
00:01:06,743 --> 00:01:10,719
their own tests,
their own eyes.Exactly.

24
00:01:10,719 --> 00:01:13,118
It's like your trail bike makes a funny
noise,

25
00:01:13,118 --> 00:01:17,039
but only every sixth ride,
and only when you're already late.

26
00:01:17,039 --> 00:01:21,120
You don't know if the bike's broken or if
you've gone a bit spare.

27
00:01:21,120 --> 00:01:23,518
Anthropic laid out a timeline,

28
00:01:23,518 --> 00:01:28,925
and the first bug starts on August 5:
some requests were routed to servers

29
00:01:28,925 --> 00:01:32,485
configured for an upcoming 1M-token
context window.

30
00:01:32,485 --> 00:01:37,967
Not the general setup people expected --
this upcoming configuration.

31
00:01:37,967 --> 00:01:40,722
So traffic gets silently misrouted,

32
00:01:40,722 --> 00:01:44,043
and right there you've got weird behavior
before anyone's even talking

33
00:01:44,043 --> 00:01:47,802
about model quality in the
abstract.Wait -- August

34
00:01:47,802 --> 00:01:48,603
5 is early.

35
00:01:48,603 --> 00:01:54,524
So this wasn't "one bad deploy on one
day." This was the start of a chain.

36
00:01:54,524 --> 00:01:58,368
And that 1M-token context window server
config,

37
00:01:58,368 --> 00:02:02,204
even hearing that phrase, you can feel
how easy it would be to dismiss.

38
00:02:02,204 --> 00:02:07,505
Like, "oh, infra stuff." But misrouting
to the wrong server class means

39
00:02:07,505 --> 00:02:10,925
you're not even testing the model you
THINK you're testing,

40
00:02:10,925 --> 00:02:13,118
right?That's the tension.

41
00:02:13,118 --> 00:02:18,595
People say "the model got weird," but one
of these failures wasn't the

42
00:02:18,595 --> 00:02:22,552
model getting moody -- it was traffic
going somewhere it shouldn't.

43
00:02:22,552 --> 00:02:28,056
Then August 25 brought a second issue:
output corruption from a runtime

44
00:02:28,056 --> 00:02:29,835
optimization misconfiguration.

45
00:02:29,835 --> 00:02:34,581
That's a nasty one, because now it's not
just where the request lands;

46
00:02:34,581 --> 00:02:37,194
it's the actual generation path getting
mangled.

47
00:02:37,194 --> 00:02:39,837
That's where those garbled characters,

48
00:02:39,837 --> 00:02:44,297
truncated responses, and incoherent
outputs fit the picture much more

49
00:02:44,297 --> 00:02:46,204
directly.August 25
twice,

50
00:02:46,204 --> 00:02:51,657
though -- because there was ALSO an
XLA/TPU compiler bug that surfaced

51
00:02:51,657 --> 00:02:54,596
the same day,
tied to mixed-precision arithmetic.

52
00:02:54,596 --> 00:02:55,958
And that's a different beast.

53
00:02:55,958 --> 00:02:58,912
Misconfiguration is one category of
problem.

54
00:02:58,912 --> 00:03:03,071
A compiler bug in XLA on TPUs with mixed
precision...

55
00:03:03,071 --> 00:03:06,905
that's the sort of sentence that makes
every ML engineer sit up straight,

56
00:03:06,905 --> 00:03:10,574
because now the system can choose subtly
wrong tokens in a way that's

57
00:03:10,574 --> 00:03:12,400
hard to reproduce.Yeah,

58
00:03:12,400 --> 00:03:16,933
and "subtly wrong tokens" sounds small
until you think about what code

59
00:03:16,933 --> 00:03:18,232
generation IS.

60
00:03:18,232 --> 00:03:22,559
One token off can turn valid code into
broken code.

61
00:03:22,559 --> 00:03:25,995
One token off in prose can be invisible.

62
00:03:25,995 --> 00:03:30,395
One token off in a command or import path
can waste an afternoon.

63
00:03:30,395 --> 00:03:35,195
So when Anthropic says there were three
separate infrastructure bugs,

64
00:03:35,195 --> 00:03:39,519
that matters because these weren't three
copies of the same failure.

65
00:03:39,519 --> 00:03:42,319
One silently misrouted traffic.

66
00:03:42,319 --> 00:03:47,255
Another corrupted output probabilities
through runtime optimization gone

67
00:03:47,255 --> 00:03:47,760
wrong.

68
00:03:47,760 --> 00:03:53,710
And the XLA/TPU mixed-precision issue
nudged token selection off in ways

69
00:03:53,710 --> 00:03:58,710
that were slippery and hard to pin
down.The phrase I'd use is

70
00:03:58,710 --> 00:04:00,565
epistemic damage.

71
00:04:00,565 --> 00:04:04,045
Not just software damage -- trust damage.

72
00:04:04,045 --> 00:04:07,166
If generated code acquires syntax errors,

73
00:04:07,166 --> 00:04:08,927
okay, a linter catches some of that.

74
00:04:08,927 --> 00:04:12,772
If English replies suddenly spit out Thai
or Chinese characters,

75
00:04:12,772 --> 00:04:14,211
you know something is off.

76
00:04:14,211 --> 00:04:17,890
But if the model is merely choosing the
wrong token occasionally,

77
00:04:17,890 --> 00:04:22,435
and it still looks plausible,
now your A/B test might pass one hour and

78
00:04:22,435 --> 00:04:23,491
fail the next.

79
00:04:23,491 --> 00:04:27,710
Your eval suite might say the prompt is
the problem when the real issue

80
00:04:27,710 --> 00:04:31,843
is lower in the stack.That's
the bit I keep coming back to.

81
00:04:31,843 --> 00:04:35,043
Developers love to assume the bug is in
their prompt,

82
00:04:35,043 --> 00:04:39,515
or their parser, or some midnight change
they pushed with a cup of terrible

83
00:04:39,515 --> 00:04:41,115
shed coffee in hand.

84
00:04:41,115 --> 00:04:41,913
Been there.

85
00:04:41,913 --> 00:04:47,165
But this postmortem is a good reminder
that infrastructure bugs can masquerade

86
00:04:47,165 --> 00:04:48,467
as application bugs.

87
00:04:48,467 --> 00:04:52,220
If a response truncates with no surfaced
error,

88
00:04:52,220 --> 00:04:55,501
you might waste hours rewriting perfectly
decent code.

89
00:04:55,501 --> 00:04:58,775
If corruption sneaks into probabilities,

90
00:04:58,775 --> 00:05:04,101
you can end up "fixing" the wrong
thingAnd Anthropic publishing the timeline

91
00:05:04,101 --> 00:05:07,044
matters for exactly that reason.

92
00:05:07,044 --> 00:05:09,279
August 5 for the routing issue.

93
00:05:09,279 --> 00:05:13,608
August 25 for the runtime optimization
misconfiguration.

94
00:05:13,608 --> 00:05:20,488
August 25 again for the XLA/TPU compiler
bug with mixed-precision arithmetic.

95
00:05:20,488 --> 00:05:24,712
Those dates give developers something
concrete to line up against their

96
00:05:24,712 --> 00:05:25,372
own logs.

97
00:05:25,372 --> 00:05:30,599
Because otherwise "Claude felt off for a
while" really does sound like

98
00:05:30,599 --> 00:05:31,574
vibes.

99
00:05:31,574 --> 00:05:33,614
This was NOT vibes.

100
00:05:33,614 --> 00:05:35,929
It was three specific failures,

101
00:05:35,929 --> 00:05:42,067
over a specific window, affecting a
specifically large slice of

102
00:05:35,929 --> 00:05:42,067
users.

103
00:05:42,067 --> 00:05:46,319
And then you get the other kind of AI
pain -- not silent degradation,

104
00:05:46,319 --> 00:05:47,997
but hard breakage.

105
00:05:47,997 --> 00:05:52,076
Opus 4.

106
00:05:47,997 --> 00:05:52,076
7 shipped on April 16, 2026,

107
00:05:52,076 --> 00:05:54,237
and if your old config passes temperature,

108
00:05:54,237 --> 00:05:59,514
top_p, or top_k with any non-default
value in the Messages API,

109
00:05:59,514 --> 00:06:00,714
you now get a 400.

110
00:06:00,714 --> 00:06:05,767
So something as innocent-looking as
temperature: 0.

111
00:06:00,714 --> 00:06:05,767
7 -- a setting teams

112
00:06:05,767 --> 00:06:10,715
have copied around for years -- can stop
production COLD.Temperature

113
00:06:10,715 --> 00:06:11,675
0.

114
00:06:10,715 --> 00:06:11,675
7.

115
00:06:11,675 --> 00:06:16,686
That's the sort of line sitting in config
files like an old garden gnome

116
00:06:16,686 --> 00:06:18,873
-- nobody remembers who put it there,

117
00:06:18,873 --> 00:06:20,711
but everybody's afraid to move it.

118
00:06:20,711 --> 00:06:24,711
And now Opus 4.

119
00:06:20,711 --> 00:06:24,711
7 says, nah mate,

120
00:06:24,711 --> 00:06:25,432
400.

121
00:06:25,432 --> 00:06:33,680
Not "we'll ignore it." Not "deprecated
warning." Just a hard API failure.

122
00:06:25,432 --> 00:06:33,680
Exactly.

123
00:06:33,680 --> 00:06:44,044
And there's a second breaking change
right next to it: thinking.

124
00:06:33,680 --> 00:06:44,044
budget_tokens

125
00:06:44,044 --> 00:06:44,044
is gone too.

126
00:06:44,044 --> 00:06:45,350
Extended thinking budget control is
removed from the API surface for

127
00:06:45,350 --> 00:06:49,157
this model, and passing that setting also
returns a 400.

128
00:06:49,157 --> 00:06:52,716
So if you upgraded by swapping the model
name and left the rest of your

129
00:06:52,716 --> 00:06:56,654
request body untouched, you could break
on TWO fronts before the model

130
00:06:56,654 --> 00:06:59,521
even starts generating.Let me
try to explain it back.

131
00:06:59,521 --> 00:07:03,848
Old habit says, "new model,
same knobs." You keep temperature,

132
00:07:03,848 --> 00:07:17,946
maybe top_p, maybe top_k, maybe thinking.

133
00:07:03,848 --> 00:07:17,946
budget_tokens if you've been

134
00:07:17,946 --> 00:07:18,244
tuning that.

135
00:07:18,244 --> 00:07:22,683
In Opus 4.

136
00:07:18,244 --> 00:07:22,683
7, those familiar knobs aren't just
ignored -- some of them

137
00:07:22,683 --> 00:07:24,566
are basically wired to the trapdoor.

138
00:07:24,566 --> 00:07:27,640
Is that fair?

139
00:07:24,566 --> 00:07:27,640
That's fair.

140
00:07:27,640 --> 00:07:30,282
And the trapdoor number is 400,

141
00:07:30,282 --> 00:07:34,755
which developers hate because it's
immediate and very unromantic.

142
00:07:34,755 --> 00:07:36,035
Your app just fails.

143
00:07:36,035 --> 00:07:39,160
Then there's the sneakier problem: cost.

144
00:07:39,160 --> 00:07:42,200
Opus 4.

145
00:07:39,160 --> 00:07:42,200
7 uses a new tokenizer,

146
00:07:42,200 --> 00:07:47,676
and Anthropic says it can use anywhere
from 1x to 1.

147
00:07:42,200 --> 00:07:47,676
35x more tokens than

148
00:07:47,676 --> 00:07:49,796
earlier models depending on content type.

149
00:07:49,796 --> 00:07:54,037
So your prompt and output can look
basically unchanged to a human,

150
00:07:54,037 --> 00:07:59,151
but the bill goes up because the
tokenizer slices the text

151
00:07:54,037 --> 00:07:59,151
differently.[questioning

152
00:07:59,151 --> 00:08:02,040
tone] That 1.

153
00:07:59,151 --> 00:08:02,040
35x is the sticky one for me.

154
00:08:02,040 --> 00:08:06,800
Thirty-five percent more tokens without
changing what the user THINKS

155
00:08:06,800 --> 00:08:10,569
they said or got back -- that's like your
servo at the petrol station

156
00:08:10,569 --> 00:08:14,836
charging extra because it decided your
jerry can has more edges today.

157
00:08:14,836 --> 00:08:16,994
Same fuel, bigger bill.

158
00:08:16,994 --> 00:08:21,056
And because it's "depending on content
type," you can't just assume one

159
00:08:21,056 --> 00:08:24,320
neat multiplier across everything.

160
00:08:21,056 --> 00:08:24,320
Right,

161
00:08:24,320 --> 00:08:28,776
and for agentic apps there's another
wrinkle: reports of regressions

162
00:08:28,776 --> 00:08:33,840
in long-context retrieval and agentic
search at default effort levels.

163
00:08:33,840 --> 00:08:37,887
The workaround people have pointed to is
using high or xhigh effort in

164
00:08:37,887 --> 00:08:39,909
Claude Code for agentic workloads.

165
00:08:39,909 --> 00:08:43,698
Which means the default may not be good
enough for the exact class of

166
00:08:43,698 --> 00:08:47,032
tasks a lot of teams care about --
multi-step searching,

167
00:08:47,032 --> 00:08:52,485
tool use, long-context
reasoning.And that's where

168
00:08:47,032 --> 00:08:52,485
the "just bump the version"

169
00:08:52,485 --> 00:08:54,245
habit absolutely bites you.

170
00:08:54,245 --> 00:08:56,885
Because now the upgrade isn't one thing.

171
00:08:56,885 --> 00:09:00,812
It's API compatibility, removed controls,

172
00:09:00,812 --> 00:09:05,934
tokenization economics,
and behavior changes in agentic search.

173
00:09:05,934 --> 00:09:10,166
If your team treats model upgrades like
changing a logo in the header

174
00:09:10,166 --> 00:09:12,967
-- easy peasy -- you're in strife.

175
00:09:12,967 --> 00:09:15,530
This is dependency-upgrade territory.

176
00:09:15,530 --> 00:09:18,328
Read the changelog, diff the payloads,

177
00:09:18,328 --> 00:09:24,600
run evals, watch costs, the whole box and
dice.I actually

178
00:09:24,600 --> 00:09:27,917
think that's the deeper lesson connecting
both stories.

179
00:09:27,917 --> 00:09:30,795
In the 2025 Claude degradation case,

180
00:09:30,795 --> 00:09:35,505
the danger was invisible system behavior
making developers mistrust themselves.

181
00:09:35,505 --> 00:09:41,344
In Opus 4.

182
00:09:35,505 --> 00:09:41,344
7,
the danger is visible if you look -- 400s,

183
00:09:41,344 --> 00:09:45,715
removed fields, token shifts -- but teams
often don't look closely enough

184
00:09:45,715 --> 00:09:48,848
because they treat model swaps as a quick
version bump.

185
00:09:48,848 --> 00:09:52,683
Same emotional outcome: the engineer on
call at 1 a.

186
00:09:48,848 --> 00:09:52,683
m.

187
00:09:52,683 --> 00:09:54,519
is staring at a dashboard thinking,

188
00:09:54,519 --> 00:09:59,481
"Did I break this, or did the platform
move under me?"Yeah.

189
00:09:59,481 --> 00:10:04,715
And maybe that's the new default mindset:
don't treat a model name like

190
00:10:04,715 --> 00:10:05,956
a cosmetic string.

191
00:10:05,956 --> 00:10:08,674
Treat it like upgrading a database engine,

192
00:10:08,674 --> 00:10:11,073
or a compiler, or a payment SDK.

193
00:10:11,073 --> 00:10:16,969
Because if a harmless old temperature: 0.

194
00:10:11,073 --> 00:10:16,969
7 can 400 your production app

195
00:10:16,969 --> 00:10:22,357
on April 16, 2026,
the real bug isn't just in the release.

196
00:10:22,357 --> 00:10:27,059
It's in the habit of assuming
intelligence APIs behave like

197
00:10:22,357 --> 00:10:27,059
plug-and-play

198
00:10:27,059 --> 00:10:27,709
widgets.

199
00:10:27,709 --> 00:10:29,829
They don't.

200
00:10:29,829 --> 00:10:34,271
Anyway -- go audit your
config before it audits you.

201
00:10:34,271 --> 00:10:37,995
See ya.That's a solid place to
leave it.

202
00:10:37,995 --> 00:10:38,875
See you next time.