Post

Replies

Boosts

Views

Activity

Metal 4: MTL4Compiler crashes when many threads capture binaries into one MTL4PipelineDataSetSerializer
Hey all, hoping someone on the Metal team can take a look at this one. It's a crash inside Metal 4's binary capture, and as far as I can tell there's no way to fix it from the app side. Filed as FB25478288 (partial repro is link is there too). Why I think it matters The setup that crashes is basically the recommended Metal 4 workflow: one MTL4Compiler with a MTL4PipelineDataSetSerializer (CaptureBinaries) attached, called from maximumConcurrentCompilationTaskCount threads, to harvest pipelines for an offline archive. Lots of pipelines share a vertex function, which I'd guess is true for most real renderers. The crash is 100% inside Metal.framework, with no app code on the crashing thread. It's easy to hit: 3 out of 3 harvest runs crashed for us, one of them 2 s in. The serializer is attached to the compiler implicitly, so the app can't put a lock around whatever it does internally. MTL4Compiler is Sendable, and the docs say it handles multithreading by default, so I'd expect capture to be thread-safe too. Environment MacBook Pro M2 Max, macOS 27.2, Xcode 27.2, Metal toolchain 32023.922.4 Also reported on an M3 Pro ~33k render pipelines over ~16k vertex functions, up to ~1,000 pipelines in a row sharing one vertex function MTL4RenderPipelineDescriptor with MTL4LibraryFunctionDescriptor stages (sometimes wrapped in MTL4SpecializedFunctionDescriptor), all colour attachments Unspecialized The serializer gets written out with serializeAsArchiveAndFlushToURL: every so often, then we start a fresh compiler + serializer Crash EXC_BAD_ACCESS (SIGSEGV), "possible pointer authentication failure": objc_msgSend MultiLevelAirntBinaryCache::getElementAtDestination(MTLBinaryKey*, ...) -[MTLCompiler compileFunctionRequestInternal:frameworkLinking:linkDataSize:reflectionOnly:compilerTask:completionHandler:] ... AGX::Compiler::compileProgram<AGX::VertexProgramKey, ...>It looks like a use-after-free: two compiles capturing the same vertex program at the same time, and an entry in the destination archive gets freed while the other one is still using it. It only happens with capture on, and it goes away if we never let two capture compiles that share a stage function run at once. Workaround we ship (not great) A lock per stage function around each capture compile, plus sending runs of pipelines that share a vertex shader to the same thread. That stopped the crashes, but we had to put a timeout on the lock, because the compiler occasionally never returns, so in theory it can still race. Second, related crash If serializeAsArchiveAndFlushToURL: runs while other threads are still compiling into that serializer, we get a different crash (null deref at 0x30) in dispatch_data_create_map ← -[_MTLBinaryArchive newArchiverIdWithBinaryKey:entry:srcArchiverId:functionType:]. Our standalone repro hits this one reliably. The getElementAtDestination crash only shows up with our real workload so far. Repro + details https://github.com/Aeonitis/metal4-capture-race It's a small Objective-C repro (Metal + Foundation only) with a bunch of modes, the results so far, and a comparison with how Vulkan (VkPipelineCache) and D3D12 spell out the thread-safety rules for this same pattern. Questions Is capturing from multiple threads into one MTL4PipelineDataSetSerializer meant to be supported? If not, could the docs say so? Is one compiler + one serializer per thread a safe workaround, or is there shared state between compilers in the same process that would still race? Is calling serializeAsArchiveAndFlushToURL: while compiles are still in flight on that serializer allowed? Is this already fixed in a seed
0
0
22
3h
Metal 4: MTL4Compiler crashes when many threads capture binaries into one MTL4PipelineDataSetSerializer
Hey all, hoping someone on the Metal team can take a look at this one. It's a crash inside Metal 4's binary capture, and as far as I can tell there's no way to fix it from the app side. Filed as FB25478288 (partial repro is link is there too). Why I think it matters The setup that crashes is basically the recommended Metal 4 workflow: one MTL4Compiler with a MTL4PipelineDataSetSerializer (CaptureBinaries) attached, called from maximumConcurrentCompilationTaskCount threads, to harvest pipelines for an offline archive. Lots of pipelines share a vertex function, which I'd guess is true for most real renderers. The crash is 100% inside Metal.framework, with no app code on the crashing thread. It's easy to hit: 3 out of 3 harvest runs crashed for us, one of them 2 s in. The serializer is attached to the compiler implicitly, so the app can't put a lock around whatever it does internally. MTL4Compiler is Sendable, and the docs say it handles multithreading by default, so I'd expect capture to be thread-safe too. Environment MacBook Pro M2 Max, macOS 27.2, Xcode 27.2, Metal toolchain 32023.922.4 Also reported on an M3 Pro ~33k render pipelines over ~16k vertex functions, up to ~1,000 pipelines in a row sharing one vertex function MTL4RenderPipelineDescriptor with MTL4LibraryFunctionDescriptor stages (sometimes wrapped in MTL4SpecializedFunctionDescriptor), all colour attachments Unspecialized The serializer gets written out with serializeAsArchiveAndFlushToURL: every so often, then we start a fresh compiler + serializer Crash EXC_BAD_ACCESS (SIGSEGV), "possible pointer authentication failure": objc_msgSend MultiLevelAirntBinaryCache::getElementAtDestination(MTLBinaryKey*, ...) -[MTLCompiler compileFunctionRequestInternal:frameworkLinking:linkDataSize:reflectionOnly:compilerTask:completionHandler:] ... AGX::Compiler::compileProgram<AGX::VertexProgramKey, ...>It looks like a use-after-free: two compiles capturing the same vertex program at the same time, and an entry in the destination archive gets freed while the other one is still using it. It only happens with capture on, and it goes away if we never let two capture compiles that share a stage function run at once. Workaround we ship (not great) A lock per stage function around each capture compile, plus sending runs of pipelines that share a vertex shader to the same thread. That stopped the crashes, but we had to put a timeout on the lock, because the compiler occasionally never returns, so in theory it can still race. Second, related crash If serializeAsArchiveAndFlushToURL: runs while other threads are still compiling into that serializer, we get a different crash (null deref at 0x30) in dispatch_data_create_map ← -[_MTLBinaryArchive newArchiverIdWithBinaryKey:entry:srcArchiverId:functionType:]. Our standalone repro hits this one reliably. The getElementAtDestination crash only shows up with our real workload so far. Repro + details https://github.com/Aeonitis/metal4-capture-race It's a small Objective-C repro (Metal + Foundation only) with a bunch of modes, the results so far, and a comparison with how Vulkan (VkPipelineCache) and D3D12 spell out the thread-safety rules for this same pattern. Questions Is capturing from multiple threads into one MTL4PipelineDataSetSerializer meant to be supported? If not, could the docs say so? Is one compiler + one serializer per thread a safe workaround, or is there shared state between compilers in the same process that would still race? Is calling serializeAsArchiveAndFlushToURL: while compiles are still in flight on that serializer allowed? Is this already fixed in a seed
Replies
0
Boosts
0
Views
22
Activity
3h