C#•4y ago

File comparison

I want to compare two files very fast, I've implemented a full comparison (Comparing size / 1024 bytes spans),
but I feel like there might some very unique info in File headers that I can use for my comparison.
What do you think?

toddlahakbar•9/6/22, 6:27 PM

If you want a full comparison, then do a full comparison

toddlahakbar•9/6/22, 6:27 PM

Also last- first is usually a really quick algorithm to search

mtreit•9/6/22, 6:33 PM

@bookuha what do you mean by unique info in file headers? What kind of files are we talking about?

Susko3•9/6/22, 10:09 PM

if you want to do it really fast, then look into vectorization with eg. class

Susko3•9/6/22, 10:10 PM

I suppose you're comparing for (in)equality?

Jester•9/7/22, 1:05 AM

the bottle neck is probably just reading the files

Mmtreit @bookuha what do you mean by unique info in file headers? What kind of files are...

bookuhaOP•9/7/22, 8:22 AM

any i guess, there must be some info that describes the file

bookuhaOP•9/7/22, 8:22 AM

to the OS

bookuhaOP•9/7/22, 8:22 AM

in some of these bytes

SSusko3 I suppose you're comparing for (in)equality?

bookuhaOP•9/7/22, 8:22 AM

yes

Kouhai•9/7/22, 9:19 AM

what do you mean ?

KKouhai what do you mean `info that describes the file`?

bookuhaOP•9/7/22, 9:27 AM

Headers that OS use to identify it.
I am just guessing

Kouhai•9/7/22, 9:28 AM

If you're doing full file comparison, why would the file headers matter

KKouhai If you're doing full file comparison, why would the file headers matter 😅 ?

bookuhaOP•9/7/22, 9:45 AM

I dont really want to do full comparison, I want to do it as fast as possible

bookuhaOP•9/7/22, 9:45 AM

I need to compare them, but going full size is more expensive

bookuhaOP•9/7/22, 9:45 AM

so I decided to find some array of bytes that can be compared

bookuhaOP•9/7/22, 9:45 AM

instead of the entire file

Susko3•9/7/22, 9:47 AM

compare sections and stop when you find a non-matching section, makes sense

Susko3•9/7/22, 9:47 AM

the important question for you may be: "where do the files usually differ?"

Susko3•9/7/22, 9:47 AM

maybe in the beginning, maybe in the end?

SSusko3 the important question for you may be: "where do the files usually differ?"

bookuhaOP•9/7/22, 9:54 AM

yes! i want to find out

Kouhai•9/7/22, 9:54 AM

The header for two files can be the same but the rest Is different, even for a simple bmp file.

KKouhai The header for two files can be the same but the rest Is different, even for a s...

bookuhaOP•9/7/22, 9:54 AM

hmm, got it

Bbookuha yes! i want to find out

Susko3•9/7/22, 9:56 AM

well.... you're the one with the files

Jester•9/7/22, 10:11 AM

its possible to listen for file chanfes in a folder. maybe make a hash for every changed file so you can compare hashes?

Jester•9/7/22, 10:12 AM

oh yea you can also compare the file name first and then the last modified date

mtreit•9/7/22, 4:07 PM

Files are completely arbitrary sequences of bytes. Unless you are restricted to a very specific file format you can't rely on headers - many file formats have no headers. (Think of a text file)

File names and things like the modified time, which are file system metadata and not part of the file itself, tell you nothing about the file contents and should never be relied upon.

If you have two files and want to compare them for equality, hashing both of them is much of the time far more expensive than doing a byte-by-byte comparison. (If you pre-hash and store the hashes for later comparison it can be useful.)

Comparing file size first will immediately tell you if the files are different without doing any file I/O.

If the file sizes are the same you can proceed with a byte-wise comparison.

If you do precompute hashes consider using a very fast hash (crc32, murmurhash, etc) on portions of the file (first 1K, last 1K, etc) and store that for fast invalidation of files that don't match.

Note that I can easily pad files with arbitrary extra bytes to make your diff check fail - this is why detection by exact match for things like malware is fragile and not generally a great approach.

Mmtreit Files are completely arbitrary sequences of bytes. Unless you are restricted to ...

bookuhaOP•9/13/22, 9:56 AM

Thank you