1 00:00:00,140 --> 00:00:06,980 So, I was looking at my smartctl report yesterday, and my secondary NVMe drive-- the 2 00:00:07,160 --> 00:00:12,140 2 terabyte one I bought literally eight months ago-- is already showing six percent 3 00:00:12,180 --> 00:00:12,420 wear. 4 00:00:13,160 --> 00:00:14,940 Six percent. 5 00:00:15,000 --> 00:00:15,840 That is... 6 00:00:15,900 --> 00:00:21,500 that is INSANE for a drive that mostly just holds some code repositories and some 7 00:00:21,540 --> 00:00:22,280 local tools. 8 00:00:22,940 --> 00:00:25,870 And then I remembered I had Jellypod running in the background, 9 00:00:26,400 --> 00:00:29,520 which led me down this rabbit hole of looking at Codex. 10 00:00:31,457 --> 00:00:34,187 Wait, six percent in eight months? 11 00:00:34,737 --> 00:00:39,207 That- that's like what, over a hundred terabytes of writes already? 12 00:00:39,257 --> 00:00:41,617 What is Codex doing to that drive? 13 00:00:42,610 --> 00:00:43,430 Exactly. 14 00:00:43,510 --> 00:00:43,910 It is. 15 00:00:44,350 --> 00:00:48,310 It's the Codex CLI version zero point one four two point zero, 16 00:00:49,010 --> 00:00:53,190 and the Desktop version twenty-six point six one six. 17 00:00:53,250 --> 00:00:55,190 They are absolutely shredding storage. 18 00:00:55,870 --> 00:01:00,490 If you run iotop or look at Activity Monitor, you see this sustained, 19 00:01:00,930 --> 00:01:02,490 non-stop write pressure. 20 00:01:03,110 --> 00:01:04,530 It's hum-- hum... 21 00:01:04,930 --> 00:01:07,810 it's hovered around four to five megabytes per second. 22 00:01:08,490 --> 00:01:10,470 Every single second it is running. 23 00:01:11,150 --> 00:01:15,450 Do the math on five megabytes per second over a full day, and you are looking at 24 00:01:15,490 --> 00:01:18,380 nearly four hundred and thirty gigabytes of writes a day. 25 00:01:19,189 --> 00:01:21,529 Four hundred gigabytes a day? 26 00:01:21,589 --> 00:01:22,209 That is... 27 00:01:22,269 --> 00:01:24,569 that's a death sentence for a consumer SSD. 28 00:01:25,249 --> 00:01:27,909 Most of those drives only have a Terabytes Written... 29 00:01:27,949 --> 00:01:32,949 uh, TBW rating of maybe six hundred terabytes total. 30 00:01:33,009 --> 00:01:37,509 You'd burn through the entire warranty in what, a year and a half? 31 00:01:37,591 --> 00:01:40,171 Less than a year if you leave it running twenty-four seven. 32 00:01:40,681 --> 00:01:43,431 And the culprit is actually incredibly specific. 33 00:01:44,091 --> 00:01:45,591 It's the logging subsystem. 34 00:01:46,251 --> 00:01:53,131 Specifically, there's a rust module-- codex_api::sse::responses-- that is dumping 35 00:01:53,271 --> 00:01:58,951 raw Server-Sent Events websocket frames directly into a SQLite database. 36 00:01:59,671 --> 00:02:06,091 It's writing to a file called logs_2.sqlite in your local application data 37 00:02:06,151 --> 00:02:06,651 directory. 38 00:02:07,331 --> 00:02:12,271 And the developers forgot to put a throttle on the TRACE level logs for these SSE 39 00:02:12,341 --> 00:02:12,811 streams. 40 00:02:13,571 --> 00:02:19,151 Every single heartbeat, every workspace update, every tiny cursor movement event 41 00:02:19,731 --> 00:02:21,471 gets written to disk instantly. 42 00:02:22,145 --> 00:02:27,955 Wait, why is it writing raw websocket frames to a local SQLite db anyway? 43 00:02:27,975 --> 00:02:32,355 Is that just poor design or are they trying to build some kind of local audit log? 44 00:02:32,875 --> 00:02:37,255 Because SQLite, by default, is going to do a full write-ahead log write, 45 00:02:37,635 --> 00:02:42,275 a WAL flush, for almost every single transaction unless they've configured it 46 00:02:42,335 --> 00:02:44,315 specifically to hold things in memory. 47 00:02:45,153 --> 00:02:49,493 It seems to be a debug feature that made it into production build profiles by 48 00:02:49,593 --> 00:02:50,713 accident. 49 00:02:50,753 --> 00:02:55,473 They are using SQLite in WAL mode, yes, which means you have two files: 50 00:02:56,153 --> 00:03:02,833 logs_2.sqlite and logs_2.sqlite-wal. 51 00:03:03,503 --> 00:03:07,602 And because the SSE connection is constantly screaming with data from the server, 52 00:03:08,183 --> 00:03:12,602 the WAL file is just getting hammered with these tiny, sequential, 53 00:03:12,633 --> 00:03:13,493 synchronous writes. 54 00:03:14,153 --> 00:03:18,763 It is the absolute worst-case scenario for flash memory wear amplification. 55 00:03:19,377 --> 00:03:24,377 Okay, so until they push a hotfix-- because who knows how long that will take-- we 56 00:03:24,417 --> 00:03:25,737 need a way to stop this. 57 00:03:25,977 --> 00:03:28,377 I mean, we can't just let our drives burn. 58 00:03:28,417 --> 00:03:30,197 Can we just intercept the write? 59 00:03:30,857 --> 00:03:34,507 What happens if we write-protect the file, or is there a cleaner way? 60 00:03:35,476 --> 00:03:41,376 If you just make logs_2.sqlite read-only, the client actually crashes on startup. 61 00:03:42,076 --> 00:03:44,936 It expects to be able to initialize the database schema. 62 00:03:45,736 --> 00:03:50,136 But there's a really elegant workaround using SQLite triggers. 63 00:03:50,176 --> 00:03:54,696 If you open the database directly using the sqlite3 command-line tool, 64 00:03:55,256 --> 00:03:59,756 you can create a trigger that intercepts the inserts before they actually touch the 65 00:03:59,796 --> 00:04:01,476 disk and just discards them. 66 00:04:02,683 --> 00:04:03,753 Ah, interesting! 67 00:04:03,873 --> 00:04:08,943 So you let the client think it's writing, but the database engine itself just drops 68 00:04:08,973 --> 00:04:09,973 the payload? 69 00:04:10,133 --> 00:04:11,733 How do you actually write that query? 70 00:04:12,708 --> 00:04:12,888 Right. 71 00:04:13,428 --> 00:04:15,328 So first, you have to find the database. 72 00:04:15,988 --> 00:04:21,048 On macOS, it's usually under ~/Library/Application Support/Codex/logs_2.sqlite. 73 00:04:21,127 --> 00:04:24,688 You run sqlite3 on that file. 74 00:04:25,368 --> 00:04:32,268 Then you run a DDL command: CREATE TRIGGER block_sse_spam BEFORE INSERT 75 00:04:32,328 --> 00:04:36,708 ON logs BEGIN SELECT RAISE (IGNORE) ; END;. 76 00:04:37,548 --> 00:04:41,938 What this does is, every time the Codex client tries to run an INSERT INTO logs 77 00:04:42,028 --> 00:04:48,448 query, the trigger runs first, executes RAISE (IGNORE) , which silently aborts the 78 00:04:48,488 --> 00:04:51,788 insertion without returning an error to the host application. 79 00:04:52,448 --> 00:04:56,488 The client thinks the write succeeded, but nothing gets written to the WAL file. 80 00:04:57,083 --> 00:04:59,083 That is incredibly clever. 81 00:04:59,763 --> 00:05:00,163 But... 82 00:05:00,483 --> 00:05:03,123 wait, what happens when they release an update? 83 00:05:03,663 --> 00:05:08,333 If the update runs a database migration, isn't it going to drop or fail because of 84 00:05:08,363 --> 00:05:09,512 that trigger? 85 00:05:09,543 --> 00:05:11,453 Or what if they change the table structure? 86 00:05:12,339 --> 00:05:13,828 That is the big caveat. 87 00:05:14,528 --> 00:05:19,588 If the next version of Codex tries to run a migration that drops or alters the logs 88 00:05:19,628 --> 00:05:24,828 table, the migration might fail, or it'll just drop our trigger and start writing to 89 00:05:24,848 --> 00:05:26,008 disk again. 90 00:05:26,068 --> 00:05:27,598 So it's a temporary patch. 91 00:05:28,288 --> 00:05:32,048 If you want a more robust solution that doesn't touch the database schema, 92 00:05:32,648 --> 00:05:34,128 you have to use a RAM disk. 93 00:05:35,749 --> 00:05:39,149 Right, offload the entire log directory to volatile memory. 94 00:05:39,879 --> 00:05:45,679 On Linux, you could just symlink the log directory to /tmp or use tmpfs. 95 00:05:45,779 --> 00:05:52,119 On macOS, you'd have to create a small RAM disk using hdid and format it as HFS+, 96 00:05:52,779 --> 00:05:55,429 then symlink the Codex log folder to that mount point. 97 00:05:56,239 --> 00:05:59,779 If it writes five megabytes per second to RAM, who cares? 98 00:06:00,259 --> 00:06:02,119 RAM doesn't have a TBW limit. 99 00:06:03,159 --> 00:06:03,799 Exactly. 100 00:06:04,279 --> 00:06:09,779 You just run a startup script that creates, say, a 512-megabyte RAM disk, 101 00:06:10,279 --> 00:06:12,619 mount it, and let Codex go wild. 102 00:06:13,349 --> 00:06:17,779 The only downside there is that your logs are wiped every time you reboot. 103 00:06:17,799 --> 00:06:23,079 But honestly, who actually needs historical TRACE logs of their editor's websocket 104 00:06:23,119 --> 00:06:23,899 connection anyway? 105 00:06:24,499 --> 00:06:27,919 It's completely useless data for ninety-nine percent of users. 106 00:06:29,387 --> 00:06:34,427 Yeah, I think I can live without my cursor-movement telemetry surviving a system 107 00:06:34,527 --> 00:06:34,907 crash. 108 00:06:35,567 --> 00:06:39,717 I'm going to set up that SQLite trigger right now before my drive loses another 109 00:06:39,727 --> 00:06:40,747 percent of its life. 110 00:06:41,327 --> 00:06:42,487 Good catch on this one, Ethan. 111 00:06:43,283 --> 00:06:44,363 Definitely. 112 00:06:44,443 --> 00:06:48,543 Let's keep an eye on the next few release notes to see if they finally throttle this 113 00:06:48,583 --> 00:06:48,803 thing. 114 00:06:49,503 --> 00:06:50,933 Alright, talk to you next time.