Hi Kevin,
You were right about the DMA issue, but also slightly wrong about how far we could get with it. No worries, only slightly. ;)
Intel has made the story a little more entertaining. There is something that looks suspiciously like a working bypass, which we initially summed up in two words: “ignoring facts.”
Your recommendation is clear: keep maxTransferSize at one page. That is the supported way to avoid depending on favourable mapping conditions. Our Intel configuration goes beyond that limit. Reducing it to one page would cost us the performance we are currently seeing.
The interesting part is what happens before the request reaches our DEXT. On Intel, we traced the family’s DMA preparation through AppleVTD::iovmMapMemory, followed by a single DMA segment covering the full request. Once that succeeds, we access the CPU-visible buffer through the descriptor API. We are not treating a device DMA address as a CPU pointer.
So the original dependency is still there. Intel’s mapping path lets these requests through anyway. Call that a feature, a loophole or Intel being rather forgiving. Strictly speaking, the Intel path gets this wrong, but its laxness is currently a stroke of luck for us. It does not make Intel the better CPU or Apple Silicon the worse one. The framework still expects a device-style DMA mapping for a controller whose data movement is done by the CPU.
Tadaaa!!! As usual: Do it wrong to get it right. ;)
That is what I meant by “ignoring facts” at the start. Yes, the underlying issue exists on Intel too, but it has had no observable effect in our tests. I probably don’t need to tell you that relying on this isn’t exactly a brilliant long-term strategy, but in our case it has paid off so far. We have now run the Intel implementation for several days without observing write, read or verify errors. With those results, I think we can live with this temporarily, while keeping the software explicitly in beta until the framework issue is fixed.
We still account for hw.pagesize internally; that is separate from limiting maxTransferSize to one page. Page alignment is not required by our tested configuration.
We could explain the broader behaviour, but this is a public forum where everyone can read along. We’d quite like to keep a little of our technical head start. ;)
We can share the broader test results, though. Both Intel and M5 completed our RAW I/O tests successfully. These covered for example:
Page reordering: A-B → B-A.
Page duplication: A-B → A-A.
Physical memory fragmentation.
Virtually contiguous buffers with non-contiguous physical backing.
Misaligned buffers and transfers crossing page boundaries.
Randomized prefaulting combined with memory churn.
These were RAW tests, not file operations through a mounted filesystem. We tried pretty much everything and really tortured the machines, but nothing killed the kernel. We have not observed read/write/verify errors in the completed runs either. We are keeping these results separate from your UBC explanation. Successful RAW runs do not establish that the mounted-filesystem path will behave the same way, nor do they invalidate your warning about memory pressure and allocation history.
The M5 shows exactly that. Its kernel panicked only when we switched from RAW access to the OS filesystem by formatting or mounting the target. No matter what we tried, even an already formatted target refused to mount, which blocks all filesystem tests there for now. So this is where the DMA address issue becomes visible: only in the filesystem path and only on the M5.
RAW-only support would technically be an option, but I suspect “you can access your storage, just please don’t mount it” would be a slightly harder sell to users. ;)
Seriously, though:
Another big thank you to you and the kernel team!!!!
From what our software and measurements are showing us, we sometimes think you give your own kernel too little credit. We appreciate how quickly you have helped us investigate this and that the storage team is taking it seriously. I’m very curious to see what we can get out of this driver once it no longer has to work with one hand tied behind its back.
With all that explained, here is what we would like to do, if you’re comfortable with it: start a public beta for Intel Macs.
The Intel implementation is now running at a little over 20 Gbit/s and works well enough to be useful. Reducing the transfer limit to one page would cost us that performance. We understand why you recommend it. Our successful tests do not remove the uncertainty you described.
We would make that limitation clear in the beta notes and keep the software explicitly in beta until your fix is released, while gathering feedback from more Intel configurations and continuing our work on Apple Silicon.
I know this goes against your recommendation and Apple’s, but I think it’s a practical compromise we could both be happy with. And no worries if you explicitly advise against it. Given your role at DTS, I’d completely understand and respect that position.
Best greetings,
Torsten
