C#•14mo ago

❔ How to convert a hex string to a Span<byte>?

Is there an allocation free method to convert a hex string into the corresponding bytes without allocation? I already have a Span<byte> as the destination, but Convert.FromHexString(...) insists on returning a newly allocated byte array. BitConverter also has no useful methods, it seems.

15 Replies

Angius•14mo ago

A new array has to be allocated, because the actual hexadecimal value doesn't exist anywhere in memory yet, so you can't take a peek at it with a span "0x3F" is not the same as 63

MarkusSchaberOP•14mo ago

I want to pass the destination Span<byte> as a parameter. Similar to the BitConverter.TryWriteBytes(...) or Base64.EncodeToUtf8 get a destination span. Right now, I have to copy the bytes from the byte[] returned by FromHexString back into my destination span, using useless CPU and producing garbage for the GC.

Angius•14mo ago

Well, yes, but TryWriteBytes() actually returns the byte representation of the given data So when writing an int you get a representation of an int What you're trying to do is take a string and get a representation of an int You can't peek into the string's memory space and magically pull an int out of there that will be the string converted to a decimal number

MarkusSchaberOP•14mo ago

I don't want that.

Angius•14mo ago

The string "0x3F" is stored in the memory as 30 78 33 46. You can't pull out the value 63 out of this, no matter where you peek with the span

MarkusSchaberOP•14mo ago

Convert.FromHexString(...) already converts a hex encoded string into a byte array. I want exactly the same semantics, just the bytes should be written into a Span<byte> I provide instead of a newly allocated byte[].

Angius•14mo ago

So you want to convert a hex string to an array of bytes and write them directly to memory without storing them anywhere in the meantime, basically I don't think there's any simple built-in way to do that. Maybe you could try playing with unsafe and what not Maybe people in #allow-unsafe-blocks will know more, they like thinking of weird microoptimizations like this

MarkusSchaberOP•14mo ago

Right now, I'm using the following code:

    private static void FromHex(ReadOnlySpan<char> source, Span<byte> dest)
    {
        Debug.Assert(source.Length == 2*dest.Length);
        while (source.Length > 0)
        {
            dest[0] = byte.Parse(source[..2], NumberStyles.HexNumber);
            source = source[2..];
            dest = dest[1..];
        }
    }

    private static void FromHex(ReadOnlySpan<char> source, Span<byte> dest)
    {
        Debug.Assert(source.Length == 2*dest.Length);
        while (source.Length > 0)
        {
            dest[0] = byte.Parse(source[..2], NumberStyles.HexNumber);
            source = source[2..];
            dest = dest[1..];
        }
    }

It seems to work and be allocation free (without unsafe), but byte.Parse seems to have a lot of overhead. I could also write my own hex parsing, but I thought with all that optimizations for Span and Memory etc. there should be a built-in way nowadays.

reflectronic•14mo ago

there isn't one in .NET 8, but there will be one in .NET 9

reflectronic•14mo ago

https://github.com/dotnet/runtime/issues/78472

GitHub

[API Proposal]: Convert.TryFromHexString · Issue #78472 · dotnet/ru...

Background and motivation Convert.FromHexString("not-hex"); is throwing a System.FormatException when input string is not a valid hex. Like Convert.TryFromBase64Chars() Convert.TryFromBas...

MarkusSchaberOP•14mo ago

Will still take some time till .NET 9 is ready then. 😦

Angius•14mo ago

A year and some change, yeah

nukleer bomb•14mo ago

You can avoid using byte.Parse() something like this:

if(char >= '0' && char <= '9') 
    num = char - '0';
else if(char >= 'A' && char <= 'F') 
    num = char - 'A';
else
    throw;

if(char >= '0' && char <= '9') 
    num = char - '0';
else if(char >= 'A' && char <= 'F') 
    num = char - 'A';
else
    throw;

MarkusSchaberOP•14mo ago

Yes. I just thought I could avoid writing that kind of low level code... Seems I can't get around this before .NET 9.

Accord•14mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ How to convert a hex string to a Span<byte>?