FileSystemWatcher Service
Hi! I'm building a background service that monitors file paths and uploads new or modified files to blob storage. The service should auto-start on boot, process changes since the last run, and automatically restart after crashes. I want to store the state to track processed files and prevent duplicate uploads while resuming operations seamlessly.
I'm looking for guidelines and best practices for building a such application. Any advice on libraries, tools, or strategies for robust state handling, error management, and ensuring reliable auto-restart mechanisms would be greatly appreciated.
9 Replies
The sollution which you are seeking depends on what scenario you need it for.
Just an example, if you have a masive ammount of files, and want to store them in the cloud or transfer them to the cloud (in this case a situation where there are no new files added), you can suffice with a read directory and sort all the files by name and size and folder.
after every transfer update a txt file of the latest transfer.
the second option is when you get batches of files to upload but do not want to put in doubles to keep in a database the uploaded files with checksum, size and name.
Before every upload you can check the checksum and if it reoccurs, you could download and compare (checksums are not 100% secure).
another option is to rename the files, a lot of file transfers for archves do this.
Before starting they put an extension to the filename like something:
test.pdf ==> test.pdf.processing and after the transfer ==> test.pdf.transferComplete .
This can also be done with folders and then you can process all which are not excluded through the extension.
After the transfer is completed, the data most times is transferred to a folder like processed or transferred so it will be available to transfer again
Hope this helps
is there already something existing in this architecture? like for example is there already a db you can use?
before starting programming i would consider if this can be solved by the os itself, like a scheduled rsync
apart from that i would say always pay attention if you want to use FSW over network, because it's not always trustworthy
about crashes, if service crashes more than few times it has to send an emai or an alert to tell an human to give a look, but again this depends if there are already monitoring systems in the architecture
As an aside FileSystemWatcher is ass and can't be trusted.
Thanks for the inputs! I haven’t built anything yet since this is just a personal hobby project.
FileSystemWatcher needs to die in a fire. You might not receive updates from the OS if it's too busy, antivirus can cause issues, a number of other small cases. The events that FileSystemWatcher have need to be carefully understood. Including lifecycle.
Common file system operations might raise more than one event. For example, when a file is moved from one directory to another, several OnChanged and some OnCreated and OnDeleted events might be raised. Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, antivirus software) might cause additional file system events that are detected by FileSystemWatcher.
yeah, i know, i had to deal with that for years
i could still think of using it again anyway, maybe just in a limited way
Is there a more reliable alternative to fsw for file monitoring?
Polling on a timer
And the strategies @Bailey mentioned above.
FileSystemWatcher is probably fine for a hobby project
It's just production critical workloads, it sucks
I see, I think I will start with the fsw and go with the first option @Bailey suggested.