Post

Replies

Boosts

Views

Activity

Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Thanks Hixie, and everyone else adding reports. Good to see another analysis arriving at the same explanation. One wrinkle with the watchdog: on October 20 I had high CPU and delayed notifications with the cleanup cursor stuck at 0xfffffe00, but two table doublings still available. IIUC, waiting for the "event table has grown too large" warning would miss that case. Restarting fseventsd cleared it; my monitor watches CPU and notification latency too. Also +1 to the request for diagnostics that work with SIP enabled. What should people capture before restarting? And can Apple link FB25090184, FB25224901 and FB25346707?
Topic: App & System Services SubTopic: Core OS Tags:
3d
Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Thanks Kevin. I already have a workaround: restart fseventsd when it goes crazy. So "fix now" isn't that urgent. I'd like to understand the cleanup failure. Most build output, caches, and sandboxes are already on a separate APFS filesystem. Source checkouts and one temporary output base are still internal, so that isn't a complete test of your suggestion... but it is close. The part I dont understand is why zero-reference records accumulate once the cursor stops at 0xfffffe00, then get reclaimed after a natural resize without restarting the daemon. IIUC those records should already be eligible for cleanup. Could client configuration explain that even when the reference count is zero? Also, one correction to my earlier description: A runaway on October 20 happened with two table doublings still available. Reaching the size cap isn't necessary for the CPU and notification delays.
Topic: App & System Services SubTopic: Core OS Tags:
2w
Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Hi Kevin. I'm debugging this as the user of the Mac. Finder and other existing applications are FSEvents clients, and the CPU/memory problems predate the diagnostic monitoring. In fact, they were the reason I started doing it. The monitor I added samples fseventsd once a minute. A notification test briefly starts fswatch on a small private probe directory, creates a file, measures the notification delay, and stops the listener. That listener isn't watching the Bazel trees. Then it does a cursor/record inspection by reading the daemon's memory without suspending it or writing. Probe only. I dont have a reliable files/sec or FSEvents/sec figure for the build trees. There is substantial file creation and removal during builds; the fs_usage counts I have are syscall counts, so they wouldn't establish that rate. I haven't isolated which existing subscribers need to watch those trees or whether their watches can be narrowed. That's up to them, and I dont have any direct control there. But heavy use shouldn't really break things. What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load. I'm still treating the Desktop symptom separately. My concern with the cleanup code is that, once the cursor stops, zero-reference records seem to accumulate until a resize resets it. At the maximum table size, that recovery route is no longer available and fseventsd begins accumulating memory and CPU like a runaway snowball.
Topic: App & System Services SubTopic: Core OS Tags:
3w
Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Thanks Hixie, and everyone else adding reports. Good to see another analysis arriving at the same explanation. One wrinkle with the watchdog: on October 20 I had high CPU and delayed notifications with the cleanup cursor stuck at 0xfffffe00, but two table doublings still available. IIUC, waiting for the "event table has grown too large" warning would miss that case. Restarting fseventsd cleared it; my monitor watches CPU and notification latency too. Also +1 to the request for diagnostics that work with SIP enabled. What should people capture before restarting? And can Apple link FB25090184, FB25224901 and FB25346707?
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
3d
Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Thanks Kevin. I already have a workaround: restart fseventsd when it goes crazy. So "fix now" isn't that urgent. I'd like to understand the cleanup failure. Most build output, caches, and sandboxes are already on a separate APFS filesystem. Source checkouts and one temporary output base are still internal, so that isn't a complete test of your suggestion... but it is close. The part I dont understand is why zero-reference records accumulate once the cursor stops at 0xfffffe00, then get reclaimed after a natural resize without restarting the daemon. IIUC those records should already be eligible for cleanup. Could client configuration explain that even when the reference count is zero? Also, one correction to my earlier description: A runaway on October 20 happened with two table doublings still available. Reaching the size cap isn't necessary for the CPU and notification delays.
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
2w
Reply to fseventsd cleanup cursor exhaustion and notification stalls on macOS 26.7.1
Hi Kevin. I'm debugging this as the user of the Mac. Finder and other existing applications are FSEvents clients, and the CPU/memory problems predate the diagnostic monitoring. In fact, they were the reason I started doing it. The monitor I added samples fseventsd once a minute. A notification test briefly starts fswatch on a small private probe directory, creates a file, measures the notification delay, and stops the listener. That listener isn't watching the Bazel trees. Then it does a cursor/record inspection by reading the daemon's memory without suspending it or writing. Probe only. I dont have a reliable files/sec or FSEvents/sec figure for the build trees. There is substantial file creation and removal during builds; the fs_usage counts I have are syscall counts, so they wouldn't establish that rate. I haven't isolated which existing subscribers need to watch those trees or whether their watches can be narrowed. That's up to them, and I dont have any direct control there. But heavy use shouldn't really break things. What I need is for normal file-change notifications to keep working, including Finder/Desktop updates, without the sudden memory bloat and CPU load. I'm still treating the Desktop symptom separately. My concern with the cleanup code is that, once the cursor stops, zero-reference records seem to accumulate until a resize resets it. At the maximum table size, that recovery route is no longer available and fseventsd begins accumulating memory and CPU like a runaway snowball.
Topic: App & System Services SubTopic: Core OS Tags:
Replies
Boosts
Views
Activity
3w