C#•13mo ago

✅ FileSystemWatcher - avoiding buffer overflow

I have a file-based communication protocol - insane, I know. It is used to communicate many messages a second. The FileSystemWatcher has a default buffer size of 8KB, which allows up to 15 notifications, assuming the filenames are 260 characters long. Let's say I increase the buffer size 2x. Is there anything else I can do to avoid the notification buffer from overflowing? I have also configured the following:

      _watcher = new FileSystemWatcher
      {
        Path = _path,
        InternalBufferSize = 16_384, // 16KB
        IncludeSubdirectories = false,
        Filter = "*_*_*_*_*" + Constants.FILE_EXTENSION,
        NotifyFilter = NotifyFilters.CreationTime
      };
      _watcher.Created += OnCreated;

      _watcher = new FileSystemWatcher
      {
        Path = _path,
        InternalBufferSize = 16_384, // 16KB
        IncludeSubdirectories = false,
        Filter = "*_*_*_*_*" + Constants.FILE_EXTENSION,
        NotifyFilter = NotifyFilters.CreationTime
      };
      _watcher.Created += OnCreated;

I'm also thinking about invoking the body of the OnCreated handler in a background thread. Is that a bad idea?

private void OnCreated(object source, FileSystemEventArgs e)
{
  ThreadPool.QueueUserWorkItem(_ =>
  {
    var fullName = Path.GetFileNameWithoutExtension(e.FullPath);
    var isMatch = Regex.IsMatch(fullName, pattern);

    if (!isMatch)
      return;

    if (DisableFileQueue)
      NewMessage?.Invoke(this, e.FullPath);
    else
      FileQueue.Add(e.FullPath);
  });
}

private void OnCreated(object source, FileSystemEventArgs e)
{
  ThreadPool.QueueUserWorkItem(_ =>
  {
    var fullName = Path.GetFileNameWithoutExtension(e.FullPath);
    var isMatch = Regex.IsMatch(fullName, pattern);

    if (!isMatch)
      return;

    if (DisableFileQueue)
      NewMessage?.Invoke(this, e.FullPath);
    else
      FileQueue.Add(e.FullPath);
  });
}

12 Replies

Denis•13mo ago

I chose ThreadPool over Task.Run to avoid running on the same thread as the OnCreated. Afaik, I must exit OnCreated as fast as I can, to prevent the buffer from overloading

Zendist•13mo ago

What is your problem exactly?

Denis•13mo ago

The buffer overflows, and I am no longer receiving notifications via the FSW

Zendist•13mo ago

Gotcha. Consider reading all of https://learn.microsoft.com/en-us/dotnet/api/system.io.filesystemwatcher.internalbuffersize?view=net-7.0

FileSystemWatcher.InternalBufferSize Property (System.IO)

Gets or sets the size (in bytes) of the internal buffer.

Zendist•13mo ago

There are some important details if you want to increase the buffer size.

Denis•13mo ago

And what about running the body of the OnCreated in a worker thread? WIll that help? Assuming that the Regex operations are slow and cannot be optimized

Zendist•13mo ago

Sure--you could minimize that operation by placing the work in a blocking collection and return immediately, passing along the heavier work to some always-running thread to take care of.

Denis•13mo ago

And... can it be done the way I've written? Using the ThreadPool?

Zendist•13mo ago

I wouldn't personally do that, I would use BlockingCollection, but sure 😊.

Zendist•13mo ago

https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.blockingcollection-1?view=net-7.0

BlockingCollection Class (System.Collections.Concurrent)

Provides blocking and bounding capabilities for thread-safe collections that implement IProducerConsumerCollection.

Zendist•13mo ago

I'm arguing to not create the task in the event handler, because that is slightly more expensive than just putting a file name in a queue structure.

Denis•13mo ago

Alright, thanks a lot!

Gaming

Programming

✅ FileSystemWatcher - avoiding buffer overflow