Hi Kevin, Thank you for the deep dive into DART (IOMMU) mapping overhead. To address your question regarding the "typical I/O sizes" the system dispatches, we performed an A/B comparison to observe the behavior of the macOS Storage Stack under different kIOMaximumSegmentByteCount configurations. Testing Methodology: We instrumented UserProcessBundledParallelTasks_Impl to log the exact fRequestedTransferCount for every incoming task. While the documentation refers to this field as a "count," the values received (e.g., 16,384) confirm it represents the total byte count. To stimulate the kernel's merging logic, we used fio with a mixed workload ranging from 4KB to 64KB (bsrange=4k-64k). The following distribution was captured during a 10-second high-load test: I/O Size (Bytes) | Case A (16K Limit) | Case A % | Case B (64K Limit) | Case B % ------------------|--------------------|----------|--------------------|--------- 16,384 (16K) | 3,174 | 81.2% | 2,906 | 82.9% 524,288 (512K) | 526 | 13.5% | 430 | 12.3% 262,144 (256K) | 210 | 5.3% | 167 | 4.8% 4,096 (4K) | 0 | 0% | 2 | 0.06% (Note: The 512K and 256K requests are large composite tasks consisting of multiple segments, respecting our 512KB hardware limit set via HBA constraints.) Observations and Conclusions: Validation of Intuition: The data perfectly aligns with your intuition. Even in Case B, where we allowed up to 64KB segments, over 82% of the requests remained exactly 16KB. macOS on Apple Silicon clearly has an inherent preference for 16KB alignment, likely matching the native page size. Merging Efficiency: Tiny 4KB fragments were almost entirely non-existent. Even though fio issued 4KB requests, the Storage Stack merged them into 16KB blocks before they reached the DEXT. The 2.8x Performance Secret: This explains why the 16KB segment limit provided such a massive boost. Since 16KB is already the "natural" size the system prefers to send, explicitly locking kIOMaximumSegmentByteCount to 16KB removes the calculation cost for the DART to handle cross-page boundaries without forcing the system to send smaller or more fragmented I/O. By setting the driver's constraints to match the kernel's native behavior, we’ve effectively put the I/O path on the "fast track" by simplifying the mapping logic. Best regards, Charles

Hi Kevin, Thank you for the detailed explanation regarding page size alignment. We followed your advice and recalibrated our HBA constraints during the Start() sequence to match the native 16KB page size of Apple Silicon. Our specific settings are as follows: kIOMaximumSegmentByteCount: 16,384 (16 KB) kIOMaximumSegmentCount: 33 (maintaining our 512 KB hardware limit) Mode: Bundled Mode enabled (256 slots) Testing on our new RAID 5 hardware (4 HDDs) showed a dramatic reduction in system overhead by aligning segments to the 16KB page boundaries and leveraging the batching capabilities of Bundled Mode. Key findings from our benchmarks (Bundled vs. Legacy): Random Reads (16K - 64K): We observed a massive performance leap. At 16K (single-page I/O), Bundled Mode outperformed Legacy by 162%. At 32K and 64K, the gains increased to between 220% and 288%. Mixed Workloads (70% Read / 30% Write): Across various block sizes, Bundled Mode consistently delivered a throughput increase ranging from 40% to 290%. System Responsiveness: The system feels significantly more responsive during concurrent small-file or metadata-heavy operations. This confirms that reducing context switches and simplifying IOMMU (DART) mapping complexity is indeed the key to performance on the Apple Silicon platform. One interesting observation: in 4K Random Read tests, Legacy Mode occasionally showed slightly higher numbers, which we suspect is due to specific cache-hit behaviors. However, as soon as the I/O size reached or exceeded the 16KB page size, Bundled Mode became the undisputed winner. Your insight was spot on: at the DriverKit level, page size alignment is more critical than RAID stripe alignment. By streamlining the DART mapping process, we’ve effectively unblocked the I/O pipeline. Best regards, Charles

Hi Kevin, Thank you for all the help here! I’ve moved the discussion regarding HBA constraints and alignment optimization to a new thread as suggested: Optimizing SCSI HBA Constraints and Alignment for DriverKit on Apple Silicon Best Regards, Charles

Hi Kevin, The Bundled Mode implementation is stable now, and we’ve been running some benchmarks based on your suggestions about 16KB page sizes and real-world performance. We analyzed the count parameter in UserProcessBundledParallelTasks specifically during the 16K Random I/O test session. Across 10,147 processed batches, about 15.6% of the calls had multiple requests bundled together, with a peak of 6 slots in a single RPC. While the majority (84.4%) were still dispatched with count=1 due to the high responsiveness of the DEXT, the batching definitely kicks in as the load increases. Regarding the throughput—we noticed that the results fluctuate based on the system's cache state and the physical limits of the 4-HDD RAID 5 setup. Since 4K and 16K random writes are smaller than the RAID stripe size, they often trigger Read-Modify-Write (RMW) cycles, making mechanical seek time the primary bottleneck. Even with these hardware constraints, Bundled Mode is a clear winner for random reads. Here is a summary of the numbers we observed under normal system cache conditions: Test Case Bundled Mode Legacy Mode Difference 4K Random Read 131 MiB/s 94.7 MiB/s +38.4% 4K Random Write 35.5 MiB/s 33.1 MiB/s +7.3% 16K Random Read 222 MiB/s 198 MiB/s +12.1% 16K Random Write 2098 MiB/s 2160 MiB/s Comparable (Cache Hit) The 38.4% gain in 4K Random Reads is particularly impressive. It shows that by reducing IPC overhead and context switches, we can get commands to the hardware queue much faster, which significantly improves responsiveness even before the physical disk heads move. Overall, Bundled Mode consistently handles high-frequency random requests better than Legacy Mode. Best regards, Charles

Hi Kevin, Thank you for your guidance on the UserProcessBundledParallelTasks architecture and IPC optimization. After several adjustments, our DEXT driver is now running stably. We have completed a comprehensive performance comparison between Bundled Mode and Legacy Mode. Below is a summary of our findings. Following your suggestion, we analyzed the console logs during 4K Random I/O stress tests (iodepth=32). By analyzing approximately 6,363 calls, we confirmed that the kernel actively performs command batching: Results: Slots per Call (Count) Frequency Observation 1 Slot 5,654 Standard/Low load; kernel dispatches immediately. 2 Slots 628 Active Batching initiated under increased pressure. 3 - 4 Slots 75 Sustained high-concurrency batching. 5+ Slots 6 High-load peaks (Max observed: 11 slots per call). This data confirms that the batching mechanism effectively reduces the number of RPC/IPC calls to the DEXT when the system is under pressure. We conducted benchmarks using fio on RAID 5 array (4 HDDs) connected via Thunderbolt 3 (4K Random R/W, Buffered I/O, 4 Jobs, Queue Depth 32). Key Performance: Mixed Random (70% Read / 30% Write): Read Throughput: 278 MiB/s (Legacy) → 536 MiB/s (Bundled) [~1.93x improvement] Write Throughput: 120 MiB/s (Legacy) → 231 MiB/s (Bundled) [~1.92x improvement] Random Read (4K): Throughput: 121 MiB/s (Legacy) → 142 MiB/s (Bundled) [~17% improvement] Random Write (4K): Performance was comparable (~35-36 MiB/s), limited by the physical seek characteristics of the HDDs. As you anticipated, the advantages of Bundled Mode are most significant in mixed random workloads, where we observed nearly a 2x increase in throughput. This validates that reducing IPC overhead by utilizing shared command/response buffers significantly boosts performance when the system is saturated with a large volume of small I/O requests. Thank you again for your expert assistance and professional advice throughout this process. Best regards, Charles

Hi Kevin, Thank you for your previous guidance regarding memory ownership and OSAction lifecycle management. The Bundled Mode implementation is now stable, and the driver operates without crashes. We have conducted performance stress tests using an LSI 3108 RAID controller via Thunderbolt 3, configured with 4 HDDs in RAID 5. The results from AJA System Test Lite (4GB, 10bit RGB) and fio (BS=1M, iodepth=32, Direct=1) show that I/O throughput in Bundled Mode remains at approximately 800 MB/s, which is identical to the performance measured in Legacy Mode. Analysis of the driver logs during these tests reveals that the count parameter in UserProcessBundledParallelTasks is consistently 1, even under high queue depth. The batching mechanism intended to reduce RPC and context switch overhead does not appear to be active. Our implementation for reporting device capacity and task limits follows the standard IOUserSCSIParallelInterfaceController specifications. UserReportHighestSupportedDeviceID returns an ID of 63, and UserReportMaximumTaskCount is set to 64 as shown below: kern_return_t IMPL(DRV_MAIN_CLASS_NAME, UserReportHighestSupportedDeviceID) { *id = 63; return kIOReturnSuccess; } kern_return_t IMPL(DRV_MAIN_CLASS_NAME, UserReportMaximumTaskCount) { *count = 64; return kIOReturnSuccess; } The processing entry point consistently reports only a single slot per call: void DRV_MAIN_CLASS_NAME::UserProcessBundledParallelTasks_Impl( const uint16_t indices[32], uint16_t count, OSAction * completion) { // Log consistently shows 'count' is 1 uLog("[BUNDLED] Mode enabled, processing %u slots", count); if (!ivars->useBundledMode || !ivars->fCommandBuffers || !ivars->fResponseBuffers) { return; } for (uint16_t i = 0; i < count; i++) { uint16_t slotIndex = indices[i]; // Dispatch logic... } } Example of the console log output during fio testing with iodepth=32: default 20:45:33.098031+0800 kernel [UserProcessBundledParallelTasks_Impl] [BUNDLED] Mode enabled, processing 1 slots Given these observations, we would like to understand the specific conditions or thresholds required for the kernel's block layer to batch multiple SCSI commands into a single RPC call. We are interested in whether the kernel's bundling logic prioritizes high IOPS (Random I/O) over high bandwidth (Sequential I/O), and if the 1MB block size used in our tests might cause the kernel to dispatch commands immediately to minimize latency instead of waiting to bundle them. Furthermore, we would like to verify if there are other reported parameters beyond UserReportMaximumTaskCount that influence the frequency of batching, or if there is a recommended method to determine if the macOS Block Layer is attempting to bundle commands before they reach the DEXT layer. Thank you for your expertise and time. Best Regards, Charles

Hi Kevin, Thank you for the detailed guidance. As you suggested, I have filed a bug report via Feedback Assistant regarding the migration path from legacy KEXT power management to DriverKit. I have included our legacy initialization logic and the specific technical contradictions we encountered during the transition. Feedback ID: FB22320055 Regarding your comment on calling super: You were absolutely correct. Failing to invoke super::SetPowerState(powerFlags, SUPERDISPATCH) was indeed the cause of the missed RPC ACK, which triggered the 21-second watchdog panic. Implementing that call as the final step in my SetPowerState_Impl has resolved the immediate panic issue. We have since stabilized the driver's lifecycle and deactivation logic. We would greatly appreciate it if you could follow up on the Feedback report to help clarify the canonical resource management boundaries for complex storage drivers in DriverKit. Best Regards, Charles

Hi Kevin, Thank you for your feedback and the feedback from the engineering team. We have integrated all suggestions from your forum posts (ID: 875288022, 875587022) and the Bug Report, and have conducted a full round of testing. Below is our current status. We now perform a memset on SCSIUserParallelResponse in the ISR and correctly populate the version, fControllerTaskIdentifier, and fBytesTransferred fields. The ISR now differentiates between Bundled/Legacy modes, calling BundledParallelTaskCompletion (without release()) and ParallelTaskCompletion (with release()) accordingly. The aforementioned fixes have resolved all 0x92000006 Panics and DEXT Corpse crashes. Unplugging the hardware or deactivating the DEXT while the driver is in a hung state no longer triggers a panic. We are in a logical deadlock. The kernel dispatches a probe command before UserCreateTargetForID returns, and both of our methods for handling this command result in a permanent hang of the registration process: Scenario A (Reporting SUCCESS): We mark the target as "Ready" during the UserInitializeTargetForID phase, allowing the command to be processed by the hardware and completed as SUCCESS by the ISR (with a fully initialized data structure). Result: UserCreateTargetForID hangs indefinitely. Logs confirm the ISR successfully calls BundledParallelTaskCompletion, but the registration call never returns. Scenario B (Reporting Selection Timeout): We mark the target as "Ready" only after UserCreateTargetForID returns and intercept the preemptive command in UserProcess... to report SERVICE_DELIVERY_FAILURE. Result: UserCreateTargetForID also hangs indefinitely. In your latest reply, you suggested the deadlock might be caused by our queue configuration. We have reviewed our architecture: Our UserCreateTargetForID runs on a dedicated serial queue (AuxiliaryQueue). All I/O completions (BundledParallelTaskCompletion) occur on the InterruptQueue or DefaultQueue. There are no shared IOLocks between these execution paths. Based on this, a queue deadlock within the DEXT seems unlikely. In both Scenarios A and B, we observed the exact same behavior: the hung UserCreateTargetForID call returns success immediately only when we manually deactivate the driver, triggering the Stop() sequence. It appears the kernel's registration thread is waiting for a signal that the DEXT has not yet sent, and this signal is only triggered when the driver terminates. We have now ruled out data structure corruption, API misuse, and DEXT-level queue deadlocks. Is there anything else we should be aware of that we might have missed? Best Regards, Charles

Hi Kevin, Following up on my previous update regarding the registration hang and SAM layer panic. We performed further experiments using the Selection Timeout (SERVICE_DELIVERY_FAILURE) approach as you suggested. Below are the results: 1. Selection Timeout Experiment Results We modified the DEXT to report SERVICE_DELIVERY_FAILURE immediately for Bundled commands arriving before the registration returns. We confirmed fControllerTaskIdentifier matches the request. Stability Improvement: With this change, any attempt to unplug the hardware or deactivate the DEXT no longer triggers a Kernel Panic. Resource lifecycle management (retaining/releasing descriptors) is now functioning correctly. Persistent Hang: Despite reporting the timeout, UserCreateTargetForID remains hung indefinitely and never returns on its own. 2. Log Evidence: The "Unlock" Mechanism The logs show that the kernel registration thread is blocked until a termination signal is received. Log A: Hang after Selection Timeout default 14:00:07.773080 kernel [AsyncCreateTargetForID_Impl] Calling UserCreateTargetForID for LUN 0 default 14:00:07.773519 kernel [UserInitializeTargetForID_Impl] Target 0 callback success. // DEXT reports Selection Timeout via Bundled API default 14:00:07.774781 kernel [UserProcessBundledParallelTasks_Impl] [Guard] Target 0 not ready. Reporting Selection Timeout. default 14:00:07.774789 kernel [UserProcessBundledParallelTasks_Impl] // --- } UserProcess Return // DEADLOCK: No further output. AsyncCreateTargetForID_Impl execution is interrupted. Log B: Stop Sequence Unblocking the Queue Upon issuing the Deactivate command, the original UserCreateTargetForID for LUN 0 returns success immediately after Stop() starts. The serial queue then proceeds to subsequent LUNs. default 14:02:33.557678 kernel [Stop_Impl] // { --- default 14:02:33.558307 kernel [Stop_Impl_block_invoke] All cancels finished. Calling super::Stop. // Registration for LUN 0 finally returns success default 14:02:33.558402 kernel [AsyncCreateTargetForID_Impl] Target 0 fully registered. default 14:02:33.558406 kernel [AsyncCreateTargetForID_Impl] // --- } LUN 0 Exit // Serial queue proceeds; LUN 1 and LUN 2 fail with kIOReturnAborted during termination default 14:02:33.558460 kernel [AsyncCreateTargetForID_Impl] Calling UserCreateTargetForID for LUN 1 default 14:02:33.558484 kernel [AsyncCreateTargetForID_Impl] Target 1 registration failed: 0xe00002bc default 14:02:33.558559 kernel [AsyncCreateTargetForID_Impl] Calling UserCreateTargetForID for LUN 2 default 14:02:33.558634 kernel [AsyncCreateTargetForID_Impl] Target 2 registration failed: 0xe00002bc 3. Request for Guidance Our data confirms that reporting either SUCCESS or FAILURE during registration does not allow UserCreateTargetForID to return under normal conditions. It only returns when the DEXT enters its termination phase. Is there a specific signal, barrier, or status required in Bundled mode to notify the SAM layer that the probe is complete and allow UserCreateTargetForID to return normally? Or should we handle these initial commands differently to avoid this deadlock? Best Regards, Charles

Hi Kevin, Thank you for the guidance. Based on your suggestions, we have implemented architectural modifications. Below is the status of implementation and the current issue. We store and retain IOMemoryDescriptor and IOMemoryMap objects in ivars. The ISR accesses the shared buffer address, which resolved the issue where GetAddress() returned NULL. The ISR differentiates between command sources. In Bundled mode, the DEXT calls BundledParallelTaskCompletion without calling release(). In Legacy mode, it calls ParallelTaskCompletion followed by release(). These changes eliminated the 0x92000006 Kernel Panic and DEXT Corpse crashes. The kernel dispatches Bundled commands immediately after UserInitializeTargetForID returns, but before UserCreateTargetForID completes. We found that reporting command completion while UserCreateTargetForID is still executing causes the UserCreateTargetForID call to hang. Based on this behavior, we infer a re-entrancy deadlock. The registration thread waits for the SAM target object to be fully instantiated, while the DEXT simultaneously reports an I/O completion for that same target. We believe this causes a locking conflict within the kernel state machine. Since UserCreateTargetForID does not return, the serial discovery queue is blocked, stopping all subsequent target registrations. Is our understanding correct? How should we resolve the issue we are currently facing? Best Regards, Charles

Hi Kevin, During the implementation of batch task buffer mapping, we observed an inconsistency between API return values and actual memory addresses, which leads to an immediate system Panic (Kernel Data Abort). Inside the implementation of UserMapBundledParallelTaskCommandAndResponseBuffers, we performed the following operations on the Response Buffer: We invoked CreateMapping(0, 0, 0, 0, 0, &ivars->fResponseMap) on the provided parallelResponseIOMemoryDescriptor. CreateMapping returns kIOReturnSuccess, and the resulting ivars->fResponseMap object pointer is non-null. However, a subsequent call to ivars->fResponseMap->GetAddress() returns NULL (0x0). After the DEXT returns from UserMapBundledParallelTaskCommandAndResponseBuffers with this NULL address, the system immediately triggers a Panic when the kernel attempts to process subsequent discovery commands (e.g., Inquiry). Panic Type: Kernel data abort Fault Address (FAR): 0x0000000000000000 Exception Class (ESR): 0x96000006 (Translation Fault, Level 0) Faulting Module: com.apple.iokit.IOSCSIArchitectureModelFamily Impact: The system enters a reboot loop until the hardware link is physically disconnected. Questions Recommended Handling for Success with NULL Address: What is the correct error-handling mechanism when CreateMapping succeeds but GetAddress() returns NULL? Does this indicate a specific alignment restriction, memory fragmentation, or a resource conflict on Apple Silicon? Synchronization for Memory Visibility: To ensure that the virtual address mapping established in UserMap... is globally visible across all CPU cores and execution contexts before the kernel issues the first Bundled Task, are there recommended synchronization primitives or Memory Barrier calls? OSAction Reference Counting in Bundled Mode: Since multiple task slots share a single OSAction in Bundled Mode, should the DEXT perform manual retain/release operations on this shared action to align with the framework's lifecycle management? Implementation Strategy for Discovery Transition: Is there a recommended pattern for transitioning from the Discovery Phase (Inquiry/Report LUNs) to Bundled Mode? If the DEXT returns an error during UserMap... due to addresses not being ready, what is the impact on the possibility of re-entering Bundled Mode later? Best Regards, Charles Attached Supporting Data Please find the complete Panic Log attached for your review. Attachment: FB21636775_Panic_Report_Verbose_Mode.txt FB21636775_Panic_Report_Verbose_Mode.txt

Hi Kevin, Thank you for your deep insight. Your theory matches our observed facts perfectly! I have just updated FB21636775 with the latest symbolicated crash log captured today (Jan 19). 1. Evidence of Concurrent Execution Conflict From the symbolicated backtrace, it is clear that at the moment of the crash: Thread 1 (AuxiliaryQueue): Is in the middle of executing UserCreateTargetForID (Frame 12). This RPC call has not yet returned to our DEXT. Thread 4 (Crashed Thread): Has already received the hardware interrupt for the first command (TEST UNIT READY) and is attempting to invoke the completion API. 2. Crash Characteristic Analysis The system encountered an Address size fault (Null dereference) at address 0x0000000000000008. This confirms your deduction: because UserCreateTargetForID is still pending on the AuxiliaryQueue and has not returned, the target-related objects or OSAction metadata in the kernel are not yet fully initialized. Attempting to invoke the action from an asynchronous thread leads to invalid memory access. This also explains why our "synchronous test" succeeded: within the scope of UserProcessBundledParallelTasks, the system ensures that the target is already fully established and the action pointer is valid. Questions: In the DriverKit framework, what is the recommended best practice for handling this specific race where hardware completion arrives before the target creation RPC returns? Should the kernel allow probe commands to be dispatched to hardware before UserCreateTargetForID has officially returned success to the DEXT? We look forward to your guidance, as resolving this is critical for the system stability of our high-performance path. Best Regards, Charles

Hi Kevin, Thank you for your candid feedback. You are absolutely correct—the crash log I previously shared was from an experimental attempt where we mistakenly used the legacy ParallelTaskCompletion API for bundled commands. I apologize for the confusion. I have now strictly followed your guidance, and here is the latest update: 1. Corrected Bug Report & Logs I have updated the Bug Report (FB21636775) with a new symbolicated crash log. This log definitively captures the failure while invoking the correct BundledParallelTaskCompletion API (as shown in Frame 8 of the trace). Even with the correct API, the system still triggers an __assert_rtn followed by an Address size fault (ESR: 0x56000080). 2. Over-retaining Experiment Results (Hard Reset) I attempted the "over-retaining" experiment you suggested (adding an extra retain() at the start of the loop and omitting release() in the ISR). The result was critical: it triggered an immediate Kernel Panic / Hard Reset every time. Because the system resets so rapidly, no .ips crash log is generated for this specific test case, and NVRAM is cleared. This is a much more severe failure compared to the DEXT-only crash we saw previously. 3. The "Sync vs. Async" A key observation remains: if we invoke BundledParallelTaskCompletion synchronously within the UserProcessBundledParallelTasks submission loop (intercepting the TUR command), it works perfectly and the kernel immediately issues the next command. The failure only occurs when the completion is triggered from the asynchronous interrupt handler, despite the fact that our queues are now serialized on the Default Dispatch Queue. We have reverted to the Legacy path for now to maintain stability for other development tasks while we wait for your further guidance on these findings. Best Regards, Charles

Hi Kevin, Thank you for the suggestion. We have performed the "over-retaining" experiment by adding an extra retain() to the OSAction at the start of the UserProcessBundledParallelTasks loop. Unfortunately, the result remains the same: the DEXT still triggers a Corpse crash immediately upon calling the completion signal in the asynchronous path. As the legacy mode (UserProcessParallelTask) remains 100% stable under our newly serialized queue configuration, we have decided to revert to the legacy path for now to continue our product development while we wait for investigation on the bug report (FB21636775). Any further insights into why the asynchronous completion fails specifically in Bundled mode would be greatly appreciated. Best Regards, Charles

Hi Kevin, Thank you for your detailed feedback and for spending time looking at your internal driver implementation. I have followed your suggestions and filed a formal bug report. 1. Bug Report Filed I have filed a bug report via Feedback Assistant. Feedback ID: FB21636775 I have attached the original and symbolicated crash logs, along with reduced code snippets showing our implementation. 2. OSAction Pointer Confirmation I have verified the value of the completion pointer received in UserProcessBundledParallelTasks. As you suspected, the pointer address is identical for all commands within a single bundle. 3. Crash Log Insights The symbolicated crash log confirms that the panic is triggered by an __assert_rtn inside OSMetaClassBase::QueueForObject during the call to completion. Specifically, when we attempted the "Unified Path" (using legacy ParallelTaskCompletion for bundled commands) as a stability test, it triggered an immediate panic, which confirms your point that we MUST use BundledParallelTaskCompletion for the bundled path. Crashed Thread Backtrace (Summary): Thread 4 Crashed:: Dispatch queue: DriverKitAcxxx-Default ... 4 DriverKit 0x19e255fb8 __assert_rtn + 88 5 DriverKit 0x19e25629c OSMetaClassBase::QueueForObject(...) 6 DriverKit 0x19e2267bc OSMetaClassBase::QueueForObject(...) 7 DriverKit 0x19e226fac OSMetaClassBase::Invoke(IORPC) + 476 8 SCSIControllerDriverKit 0x19e373364 IOUserSCSIParallelInterfaceController::ParallelTaskCompletion(...) 9 <BundleID> 0x1003156a4 MyDriver::InterruptHandler(...) Next Steps: I am now proceeding with the "over-retaining" experiment you suggested (adding an extra retain at the start of the bundled submission loop) to see if it resolves the asynchronous crash when using BundledParallelTaskCompletion. I will update you with those results shortly. Best Regards, Charles