C#•3w ago

✅ Getting duplicates out of a list

Hey, I have a C# code where I have a list of byte arrays (which are extracted from Images of a folder and I don‘t know how many there are - ps. They are too many - the code I wrote the Program in was in Windows forms) and there are duplicates in it, but I don’t know how get the duplicates out of the list dynamically. I couldn’t find a really good solution that does what I want. Thanks for help!

36 Replies

Keswiik•3w ago

What are you considering a duplicate? If you have the exact same image visually, but at a different resolution, is that a duplicate? How about the same image in two different formats?

Daiko GamesOP•3w ago

nonono, what I meant with that is the byte array itself. The byte arrays are sometimes the same which means everything of some images is the same, isn´t it?

Keswiik•3w ago

Sure, in that specific case. But if the goal is to detect duplicate images, then that kind of comparison will miss a lot of stuff.

Daiko GamesOP•3w ago

Yes i know but the code will be larger by that, and this converter should be simple - do you know a solution?

Keswiik•3w ago

There are a ton of different methods for finding duplicate images, they're not exactly simple though

Keswiik•3w ago

https://stackoverflow.com/questions/23931/algorithm-to-compare-two-images

Stack Overflow

Algorithm to compare two images

Given two different image files (in whatever format I choose), I need to write a program to predict the chance if one being the illegal copy of another. The author of the copy may do stuff like rot...

Daiko GamesOP•3w ago

ok, if that is so, then I would have to do it complicated too I think..

ero•3w ago

your approach would only really work if the images are genuinely identical like 1:1 copy and paste and even then i'm not sure (what about time stamps?)

Daiko GamesOP•3w ago

They are, that is the point that is why I need help, is there an easy way for that.

ero•3w ago

so you have a List<byte[]>?

Daiko GamesOP•3w ago

Yes

ero•3w ago

that won't entirely be enough, you have the file names somewhere? unless you only care about the indices

Daiko GamesOP•3w ago

Yes I have them Isn´t the hash code normally enough?

MODiX•3w ago

ero

REPL Result: Success

byte[] arr = [];
arr.GetHashCode()

byte[] arr = [];
arr.GetHashCode()

Result: int

48389476

48389476

Compile: 293.479ms | Execution: 54.281ms | React with ❌ to remove this embed.

ero•3w ago

no :p or, to be more clear:

MODiX•3w ago

ero

REPL Result: Success

byte[] arr1 = [];
byte[] arr2 = [];

(arr1.GetHashCode(), arr2.GetHashCode())

byte[] arr1 = [];
byte[] arr2 = [];

(arr1.GetHashCode(), arr2.GetHashCode())

Result: ValueTuple<int, int>

{
  "item1": 27554885,
  "item2": 27554885
}

{
  "item1": 27554885,
  "item2": 27554885
}

Compile: 364.339ms | Execution: 27.222ms | React with ❌ to remove this embed.

ero•3w ago

:ReallyMad: i doubt you can compare arrays with their hashcode

Daiko GamesOP•3w ago

ok sorry, I don´t understand the topic that much, i am trying to understand it.

ero•3w ago

they're reference types, but i don't know their hashcode implementation

Daiko GamesOP•3w ago

k how would you do it if it was simple? I mean filter out the byte arrays (duplicates)

ero•3w ago

really good question honestly i mean, it's not hard or anything just hard to get right the easiest and slowest way would of course be a nested loop, comparing each item one by one with all other items you would use SequenceEquals on the arrays to make sure they're identical

Daiko GamesOP•3w ago

But that is time consuming and also power consuming if they are many images

ero•3w ago

are you doing this more than once?

Daiko GamesOP•3w ago

ero•3w ago

so?

Daiko GamesOP•3w ago

isn´t hashset the thing that does that better?

ero•3w ago

hashset still needs a comparer

Daiko GamesOP•3w ago

Ohhhhhhh, that is why it didn´t work. Now I understand, thanks I tested things bevore I wrote here ok, what else is possible instead of sequence equal, or is there a method to dynamically compare two bytes with Sequence equals from my list?

ero•3w ago

public unsafe class ByteArrayComparer : IEqualityComparer<byte[]>
{
    public bool Equals(byte[]? x, byte[]? y)
    {
        if (x == null || y == null)
        {
            return false;
        }

        return x.SequenceEqual(y);
    }

    public int GetHashCode([DisallowNull] byte[] obj)
    {
        return obj.GetHashCode();
    }
}

public unsafe class ByteArrayComparer : IEqualityComparer<byte[]>
{
    public bool Equals(byte[]? x, byte[]? y)
    {
        if (x == null || y == null)
        {
            return false;
        }

        return x.SequenceEqual(y);
    }

    public int GetHashCode([DisallowNull] byte[] obj)
    {
        return obj.GetHashCode();
    }
}

var dups = list
    .Select((bytes, i) => (bytes, i))
    .GroupBy(t => t.Item1, new ByteArrayComparer())
    .Where(g => g.Count() > 1)
    .SelectMany(g => g.Select(t => t.Item2));

var dups = list
    .Select((bytes, i) => (bytes, i))
    .GroupBy(t => t.Item1, new ByteArrayComparer())
    .Where(g => g.Count() > 1)
    .SelectMany(g => g.Select(t => t.Item2));

this should return an enumerable of duplicate indices

MODiX•3w ago

ero

sharplab.io (click here)

var b1 = GetBytes(50, 1337);
var b2 = GetBytes(50, 1337);
var b3 = GetBytes(50, 42);
Console.WriteLine(b1 == b2); // not reference equal
var list = new List<byte[]>([b1, b2, b3]);
var dups = list
    .Select((bytes, i) => (bytes, i))
    .GroupBy(t => t.Item1, new ByteArrayComparer())
    .Where(g => g.Count() > 1)
    .SelectMany(g => g.Select(t => t.Item2));
// 34 more lines. Follow the link to view.

var b1 = GetBytes(50, 1337);
var b2 = GetBytes(50, 1337);
var b3 = GetBytes(50, 42);
Console.WriteLine(b1 == b2); // not reference equal
var list = new List<byte[]>([b1, b2, b3]);
var dups = list
    .Select((bytes, i) => (bytes, i))
    .GroupBy(t => t.Item1, new ByteArrayComparer())
    .Where(g => g.Count() > 1)
    .SelectMany(g => g.Select(t => t.Item2));
// 34 more lines. Follow the link to view.

React with ❌ to remove this embed.

ero•3w ago

here's some fixed code

Daiko GamesOP•3w ago

Ok thanks for help!

ero•3w ago

the problem with hashcode not being enough to implement is that the hashcode can still contain collisions it's meant as a pre-check before doing the actual equality check

Daiko GamesOP•3w ago

👍🏻 k

ero•3w ago

if the hashcodes already don't line up, we don't need to check for equality there's certainly many ways to optimize this, but that's a topic for #allow-unsafe-blocks $close

MODiX•3w ago

If you have no further questions, please use /close to mark the forum thread as answered

Gaming

Programming

✅ Getting duplicates out of a list