C#•2y ago

✅ FileSystemWatcher - avoiding buffer overflow

I have a file-based communication protocol - insane, I know. It is used to communicate many messages a second. The FileSystemWatcher has a default buffer size of 8KB, which allows up to 15 notifications, assuming the filenames are 260 characters long. Let's say I increase the buffer size 2x. Is there anything else I can do to avoid the notification buffer from overflowing? I have also configured the following:

      _watcher = new FileSystemWatcher
      {
        Path = _path,
        InternalBufferSize = 16_384, // 16KB
        IncludeSubdirectories = false,
        Filter = "*_*_*_*_*" + Constants.FILE_EXTENSION,
        NotifyFilter = NotifyFilters.CreationTime
      };
      _watcher.Created += OnCreated;

      _watcher = new FileSystemWatcher
      {
        Path = _path,
        InternalBufferSize = 16_384, // 16KB
        IncludeSubdirectories = false,
        Filter = "*_*_*_*_*" + Constants.FILE_EXTENSION,
        NotifyFilter = NotifyFilters.CreationTime
      };
      _watcher.Created += OnCreated;

I'm also thinking about invoking the body of the OnCreated handler in a background thread. Is that a bad idea?

private void OnCreated(object source, FileSystemEventArgs e)
{
  ThreadPool.QueueUserWorkItem(_ =>
  {
    var fullName = Path.GetFileNameWithoutExtension(e.FullPath);
    var isMatch = Regex.IsMatch(fullName, pattern);

    if (!isMatch)
      return;

    if (DisableFileQueue)
      NewMessage?.Invoke(this, e.FullPath);
    else
      FileQueue.Add(e.FullPath);
  });
}

private void OnCreated(object source, FileSystemEventArgs e)
{
  ThreadPool.QueueUserWorkItem(_ =>
  {
    var fullName = Path.GetFileNameWithoutExtension(e.FullPath);
    var isMatch = Regex.IsMatch(fullName, pattern);

    if (!isMatch)
      return;

    if (DisableFileQueue)
      NewMessage?.Invoke(this, e.FullPath);
    else
      FileQueue.Add(e.FullPath);
  });
}

12 Replies

DenisOP•2y ago

I chose ThreadPool over Task.Run to avoid running on the same thread as the OnCreated. Afaik, I must exit OnCreated as fast as I can, to prevent the buffer from overloading

Zendist•2y ago

What is your problem exactly?

DenisOP•2y ago

The buffer overflows, and I am no longer receiving notifications via the FSW

Zendist•2y ago

Gotcha. Consider reading all of https://learn.microsoft.com/en-us/dotnet/api/system.io.filesystemwatcher.internalbuffersize?view=net-7.0

FileSystemWatcher.InternalBufferSize Property (System.IO)

Gets or sets the size (in bytes) of the internal buffer.

Zendist•2y ago

There are some important details if you want to increase the buffer size.

DenisOP•2y ago

And what about running the body of the OnCreated in a worker thread? WIll that help? Assuming that the Regex operations are slow and cannot be optimized

Zendist•2y ago

Sure--you could minimize that operation by placing the work in a blocking collection and return immediately, passing along the heavier work to some always-running thread to take care of.

DenisOP•2y ago

And... can it be done the way I've written? Using the ThreadPool?

Zendist•2y ago

I wouldn't personally do that, I would use BlockingCollection, but sure 😊.

Zendist•2y ago

https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.blockingcollection-1?view=net-7.0

BlockingCollection Class (System.Collections.Concurrent)

Provides blocking and bounding capabilities for thread-safe collections that implement IProducerConsumerCollection.

Zendist•2y ago

I'm arguing to not create the task in the event handler, because that is slightly more expensive than just putting a file name in a queue structure.

DenisOP•2y ago

Alright, thanks a lot!

Gaming

Programming

✅ FileSystemWatcher - avoiding buffer overflow