How to get file size before and after write operations using Endpoint Security?

One of the customer's requirements is that write event reports must provide information about the file size both before and after the write operation. The challenge is that the ES_EVENT_TYPE_NOTIFY_WRITE event does not provide information about the changes, and in the stat structure, I see the file's stat prior to the write event.

So, I think you might need to rethink this a bit. To start with, the event volume is obviously enormous and you're basically talking about issue an extra stat call per write. That's a lot of stat calls. However, it's even worse that than because the additional latency inherent in the notify path means that you're going to lag behind the modifying process, so now the stat calls you're issuing aren't even guaranteed to sync up with the write().

Finally, keep in mind that mapped I/O isn't going to generate any event, so you're always going to be missing "something".

Is there any way within the Endpoint Security Framework to guarantee the sequence of NOTIFY events?

No.

Are there any other supported (public) APIs or methods besides Endpoint Security to track file system changes?

Note at the file level. FSEvents will monitor at the directory level, but it's about tracking directory changes overall, not individual file changes. It also isn't tied to individual syscalls in the way ES does.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Well, unfortunately, as I understand it, there is currently no reliable way to track file size immediately before and after each write operation on macOS using public APIs.

Thank you very much for the detailed clarification!

Well, unfortunately, as I understand it, there is currently no reliable way to track file size immediately before and after each write operation on macOS using public APIs.

Correct. More specifically, the combination of factors like:

Mapped file I/O occurring "outside" of syscall visibility.
Multiple processes modifying the same file simultaneously.
The latency of ES notifications vs. write speed.

...mean that you can't reliably correlate any particular file change with a SPECIFIC write call, at least not in the general case.

I'd also add a clarification on this point:

on macOS using public APIs.

One thing I want to emphasize here is that the issues here are about the system's larger architecture and design, NOT about API access. In other words, this isn't a case where we could support this if we simply gave you access to Some Special Private API™. Multiple points within the system directly interfere with what you're trying to do so this isn't a question the system can reliably answer, even with "full" system access.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you very much. One more thing to clarify!

I would like to confirm one specific point regarding the ordering guarantees of Endpoint Security notifications.

Does Endpoint Security guarantee that the order in which NOTIFY events are delivered to a single es_client_t matches the actual chronological order in which the corresponding file operations occurred in the kernel?

For example, if a process performs open() followed by close() on the same file descriptor, is it guaranteed that ES_EVENT_TYPE_NOTIFY_OPEN will be delivered before ES_EVENT_TYPE_NOTIFY_CLOSE, assuming both events are generated, neither is dropped, and the client is subscribed to both event types?

The es_new_client documentation guarantees delivery in the order the system delivers messages, while the WWDC20 session "Build an Endpoint Security app" states that ES delivers messages in the same order as the events occurred on the system (video 32:50).

I would appreciate confirmation of whether this guarantee applies to the actual chronological order of file operations in the kernel, rather than just the order in which messages are queued or delivered to the client.

Actually, I want to jump back to something you said here:

I tried to implement a mechanism based on tracking the sequence of events (open -> write -> close), but it seems macOS does not guarantee such a sequence, as the agent receives ES_EVENT_TYPE_NOTIFY_CLOSE earlier than ES_EVENT_TYPE_NOTIFY_OPEN.

Have you specifically proved that this is ACTUALLY what's going on, ideally using a basic test app that you control? Or is this just something you're inferring based on what you've seen looking at logs from other processes?

The issue here is that, in general, the ES system does guarantee that a basic sequence like:

{
	open(...)
	write(...)
	close(...)
}

Should generate this sequence in order:

1. ES_EVENT_TYPE_NOTIFY_OPEN
2. ES_EVENT_TYPE_NOTIFY_WRITE
3. ES_EVENT_TYPE_NOTIFY_CLOSE

However, what makes this confusing/complicated is when this gets involved:

"...something you're inferring based on what you've seen looking at logs from other processes?"

The problem here is that real-world usage patterns make all of this much, much more complicated.

For example, a properly written document-based app may not EVER actually open() or write() on the document it's editing, and, if it does call open(), it's not calling it for the reason you might expect. Our longstanding recommendation is that document apps implement "safe saving" instead of directly modifying its target file, so it's perfectly reasonable for a document-based app to:

1. Use clonefile (man clonefile) to duplicate the original file to some "private" location.

2. Open and modify that duplicate however it wants.

3. Save its changes to the original file by using renamex_np (man rename) to atomically replace its target file.

Adding to the complexity, the more robust/well-written the app is, the LESS likely it is to directly access that original target file. For example, a well-written app should be using things like file coordination to monitor for changes from other sources. The naive approach would then be to open and read the file to determine what changed. However, blocking file coordination like that isn't great, and the longer you hold the file open, the more likely it is that some other process will directly modify the file, further complicating your own analysis. So, the other (often better) approach would be to clone the modified file to a private location and then do your comparison against the clone instead of the original.

As another example, it's very common for apps to do things which look crazy from the syscall/EndpointSecurity perspective. Most apps are written as a collection of weakly connected components, which means those components can end up repeating "work" in ways that seem nonsensical.

For example, if a process performs open() followed by close() on the same file descriptor, is it guaranteed that ES_EVENT_TYPE_NOTIFY_OPEN will be delivered before ES_EVENT_TYPE_NOTIFY_CLOSE, assuming both events are generated, neither is dropped, and the client is subscribed to both event types?

Yes, however, the critical words here are:

...on the same file descriptor…

…as that's forcing some degree of serialization— the "open" call is what generated the fd being passed to "close", at which point the ES system's ordering guarantee applies. HOWEVER, the problem here is that your ES client can't actually know that two syscalls are actually using the same fd, even if they target the same file AND no other open call has targeted that file.

The problem is that it's possible for an app to call close on a file that it didn't actually open. File descriptors can be transferred through mach/XPC and the receiver is responsible for closing them. And, anticipating your question, no, I don't think this case is really detectable or blockable within the context of the target app. You could block the connection itself, but you can't really see or control what's transferred across that connection.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Thank you very much for the clear-cut explanation!