Systemd SMB Mount and Automount leading to incorrect sha256sums on some files.
Anyone else here that use systemd mounts to automount a smb share running into issues with some of their files? It seems to only be on the systemd mounts and for some reason small files like compressed photos, text files, configs, etc. don't run into this same issue. SSH and the Network locations built into nautilus will get the correct sha256sums. I have a Microsoft Surface Pro 1 that also runs Bazzite, and I have confirmed it also has the same problem. This systemd mount has been working fantasticly the past few months, but yesterday I ran into this issue and it scared me that my hard drives were about to give up. Luckily that is not the problem and have finally narrowed down the issues to only my systemd mounts. If anyone has any insight, or ideas I'd love to hear it! :peepoSalute:
sha256sum results for sample file
5 Replies
let me ask you this question, for the both systems that are running bazzite, are they both automounting the same NFS network drive? have you check the drive health on the NFS server?
I personally am using NFS server off a rocky linux as well as a synology running both SMB and NFS for other backups, never had an issue but I also only tend to use
rsync -a for all file copy operations.
I do not trust regular cp or even dolphin/nautilus to handle large file copying, rsync is fault tolerant and has built-in mechanisms to ensure data integrity is intact.
Do the hashes change when you copy the files in?
Do the hashes change randomly while the files are in storage?
Have you checked to make sure the NFS server is not performing any type of compression on the drives where data is hosted?
Is the data with the questionable hashes backed up anywhere else where you can check hashes against?I have done multiple health checks on the drives. The data on the hard drives is NOT being corrupted. The data on the hard drive itself is as expected. I get consistent sha256sums when using SSH, as well as when I mount the SMB share using the built in network locations in nautilus.
let me ask you this question, for the both systems that are running bazzite, are they both automounting the same NFS network drive?Yes, both of the Bazzite instances are mounting the same SMB shares.
Do the hashes change when you copy the files Do the hashes change randomly while the files are in storage?No, the ONLY time I get inconsistent sha256sums is when I use my systemd mounts. And this doesn't seem to effect small files like compressed images, text docs, etc.
ssh - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 ssh - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 sysd - f38b8b8222d3e47d50634295462f32e0483bcbf53f8fb467a736989973f9626d sysd - 6f9fef919a4c0e3328165da6b3e6d760c648d3bd5b1f6f451a466cf4f039c4c9 nautilus network locations - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954These are the sha256sums from one file which I've confirmed to act as expected, as long as I'm not accessing it through the sysd mount. SSH, and Nautilus Network locations mount all have the correct sha256sums. The sysd mount is the only one that is spitting out inconsistent sha256sums. It's a very weird issue, and I've yet to try using a VM to try and replicate this issue on other distros. It would take me some time, as I have yet to delve into using Virtual Machines on Linux. I'm still relatively new to Linux in general only about 11 months so far, but having a blast. :letsgo:
never had an issue but I also only tend to use rsync -a for all file copy operations.I attempted to use rsync -a to copy the above file to my home dir from both the sysd mount and the Nautilus Network Locations mount, and getting the same results.
rsync sysd mount - 3dfe5d2dc7cb7e45c6ab0989c57fb254f96fefbef36145bcd05b365996d3d5dc rsync Nautilus mount - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954
if this is the case, i am suspecting the system hosting the NFS server is performing some sort of compression then
if you rsync the file back, does the original hash get recomputed?
if this is the case, i am suspecting the system hosting the NFS server is performing some sort of compression thenThere is LZ4 compression on my datasets, though I really am unsure how this can be the problem. Compression should effect all different ways of mounting the SMB Shares, should it not? If this was the problem I should be getting inconsistent sha256sums when I access the SMB share through the Nautilus SMB mount.
if you rsync the file back, does the original hash get recomputed?Below I used rsync on the sysd mount to copy the same file as before to my home dir, then again used rsync to copy the file back using the sysd mount path. The file has a wrong but consistent sha256sum, only when I use the sysd mount to access the file does the sha256sum change again.
start sum - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 rsync sysd mount to home dir - 0d70fadfa66c089951920c7eb81ef03e306416efb97b98be3e4410b39868fb8a rsync sysd mount after file was copied back to share from home dir - 2f6294655c433e8ef9c7105941dbcbd5422a3a367ccf8f67f61a9bdf5541f48b rsync Nautilus mount to access file that was copied from home dir to SMB share - 0d70fadfa66c089951920c7eb81ef03e306416efb97b98be3e4410b39868fb8aI'll do the same as above, but this time I will use the Nautilus SMB mount to rsync -a the above file.
rsync Nautilus mount to home dir - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 cp Nautilus mount after file was copied back to share from home dir - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 (note used cp, rsync doesn't like copying to the Nautilus mount see here: https://bbs.archlinux.org/viewtopic.php?id=287443) rsync sysd mount after file was copied back to share from home dir - 478fe48616edb906ad8ffa6220422cb2e03da38d169596dbfcf2b484ef809954 (copied using rsync to the sysd mount dir, and so long I use the Nautilus mount to sha256sum the sums are consistent)Ran out of allocated words in last message.
rsync sysd mount after file was copied back to share from home dir - 24a8a09321c3a18ac56eee9c10145a41befe3ea54f170e198adefa6235168225 (This is the same exact file that was last sha256sumd, the only difference is how it was accessed. The one above with the 478 (correct) sha256sum was wrtitten to the sysd mount but read through the Nautilus mount. This one was read through the sysd mount. So this means that the sums are consistent when writing to the sysd mount, just not reading from the sysd mount.):clown: I have no idea how the discrepancy between the Systemd mount and Nautilus mount happened, but it seems to be fixed. My SMB share is hosted on a Truenas Scale system, and somehow Truenas pushed an update that messed up SMB AIO read buffers (https://ixsystems.atlassian.net/browse/NAS-132365), and it could be that my Systemd mount which used CIFS was affected. I am still confused how the Nautilus mount was NOT affected. I would expect this to also affect the Nautilus mount, but I suppose I don't know enough to explain the behavior. If anyone comes across this same issue and is also using a Truenas Scale system running SMB shares ensure that you update your system to 24.10.2 Thank you for the replies @DevilFish303! Sorry to drag you along on a bug in a different system.
nah dw about it, sometimes rubber ducking method helps 🙂