C#•2y ago

Brotli compression for JSON producing newlines - any work around?

I am running ASP.NET 7.0 and I don't want to use Newtonsoft because I would have to downgrade. Can I minify a JSON string without it?

31 Replies

Jayy•2y ago

what does minify mean here

oe•2y ago

like remove whitespaces & new lines on it @Jayy the brotli compression algorithm i'm using doesn't interpret new lines well so when decompressing my angular frontend can't read it

Jayy•2y ago

this is the default output

oe•2y ago

wdym

Jayy•2y ago

stj outputs json with no spaces and no newlines by default

MODiX•2y ago

Jayy#6249

REPL Result: Success

System.Text.Json.JsonSerializer.Serialize(new {foo=1, bar=2})

System.Text.Json.JsonSerializer.Serialize(new {foo=1, bar=2})

Result: string

{"foo":1,"bar":2}

{"foo":1,"bar":2}

Compile: 417.071ms | Execution: 65.127ms | React with ❌ to remove this embed.

oe•2y ago

maybe i should rephrase the question i'm using this https://khalidabuhakmeh.com/compress-strings-with-dotnet-and-csharp @Jayy to compress my json string the issue is that the response is this:

"[\n  {\n    \"requestID\": \"a65e2ed5-5a23-45ab-adef-09b6ac857cdb\",\n    \"url\": \"https://example.com/\",\n    \"headers\": [\n

"[\n  {\n    \"requestID\": \"a65e2ed5-5a23-45ab-adef-09b6ac857cdb\",\n    \"url\": \"https://example.com/\",\n    \"headers\": [\n

\n everywhere and newlines everywhere i thought minifying would fix it but \n is not a part of json i tried maybe converting to base64 before compression and then converting back to string after decompression but that produced a bunch of errors

mtreit•2y ago

Can you use $paste to show your actual program?

MODiX•2y ago

If your code is too long, you can post to https://paste.mod.gg/ and copy the link into chat for others to see your shared code!

mtreit•2y ago

Minimal program that reproduces the issue

Jayy•2y ago

So this has nothing to do with json and everything to do with ur compression algorithm lol

oe•2y ago

absolutely

oe•2y ago

https://github.com/OEvans117/RequestConverter/blob/master/RequestConverterAPI/Controllers/RequestController.cs

GitHub

RequestConverter/RequestController.cs at master · OEvans117/Request...

Convert request bundles from Fiddler, Wireshark etc into clean, readable code. Current support for C# & Python - RequestConverter/RequestController.cs at master · OEvans117/RequestConverter

oe•2y ago

yeah phrased the question wrong, sorry

mtreit•2y ago

        await using var input = new MemoryStream(bytes);
        await using var output = new MemoryStream();
        await using var stream = new BrotliStream(output, level);

        await using var input = new MemoryStream(bytes);
        await using var output = new MemoryStream();
        await using var stream = new BrotliStream(output, level);

Um...what the heck are these awaits? (Not that that's relevant to your question...)

oe•2y ago

😵‍💫 didn't even have a look it's from the article, i just copy pasted

mtreit•2y ago

oe•2y ago

mtreit•2y ago

This compression code shouldn't be using async at all It's purely in-memory

oe•2y ago

true ok. i figured out the issue for some reason i was serializing the conversionresult here var CompressedResult = await Compression.ToBrotliAsync(ConversionResult ,CompressionLevel.SmallestSize);

var CompressedResult = await Compression.ToGzipAsync(JsonSerializer.Serialize(ConversionResult) ,CompressionLevel.SmallestSize);

var CompressedResult = await Compression.ToGzipAsync(JsonSerializer.Serialize(ConversionResult) ,CompressionLevel.SmallestSize);

conclusion i am dumb thanks for taking a look anyway. @mtreit much appreciated

mtreit•2y ago

Hmm, I didn't quite understand your conclusion about what the issue is - it's related to that gzip code?

oe•2y ago

well when compressing the string (which was already json) i reserialized it and i don't know why i did that and as soon as i fixed that it seemed to work

mtreit•2y ago

Your original input json that you re-serialized must have had newlines in it

oe•2y ago

it did yes

mtreit•2y ago

Oh ok, now I understand

oe•2y ago

are you actually an ms employee ? @mtreit

mtreit•2y ago

So minimal repro:

using System.Text.Json;

var r = new MyAwesomeRecord("Joe", "Schmoe");

var json = JsonSerializer.Serialize(r, new JsonSerializerOptions {  WriteIndented = true});

Console.WriteLine(json);

var compressed = Compression.ToBrotliAsync(JsonSerializer.Serialize(json)).Result;
var decompressed = Compression.FromBrotliAsync(compressed.Result.Value).Result;

Console.WriteLine(compressed.Result.Value);
Console.WriteLine(decompressed);

record MyAwesomeRecord(string FirstName, string LastName);

using System.Text.Json;

var r = new MyAwesomeRecord("Joe", "Schmoe");

var json = JsonSerializer.Serialize(r, new JsonSerializerOptions {  WriteIndented = true});

Console.WriteLine(json);

var compressed = Compression.ToBrotliAsync(JsonSerializer.Serialize(json)).Result;
var decompressed = Compression.FromBrotliAsync(compressed.Result.Value).Result;

Console.WriteLine(compressed.Result.Value);
Console.WriteLine(decompressed);

record MyAwesomeRecord(string FirstName, string LastName);

[16:08:19] ✓ dotnet run
{
  "FirstName": "Joe",
  "LastName": "Schmoe"
}
i2IAAICqqqrqH09HAzvbQcGOBnc7GBjYxa5yUTAwALPDAqBmdjIwqMlhyt9azIfhCwaAEQY7RM9hyXftE4pwqlTCK0lard6uuD0m2WEVRNAt1IOLUhp7xaPs7tizxeZ4yM5QhNOvZgA=
"{\r\n  \u0022FirstName\u0022: \u0022Joe\u0022,\r\n  \u0022LastName\u0022: \u0022Schmoe\u0022\r\n}"

[16:08:19] ✓ dotnet run
{
  "FirstName": "Joe",
  "LastName": "Schmoe"
}
i2IAAICqqqrqH09HAzvbQcGOBnc7GBjYxa5yUTAwALPDAqBmdjIwqMlhyt9azIfhCwaAEQY7RM9hyXftE4pwqlTCK0lard6uuD0m2WEVRNAt1IOLUhp7xaPs7tizxeZ4yM5QhNOvZgA=
"{\r\n  \u0022FirstName\u0022: \u0022Joe\u0022,\r\n  \u0022LastName\u0022: \u0022Schmoe\u0022\r\n}"

I am, yes.

oe•2y ago

that's really cool, i can't imagine how you have time on your hands to be helpful here

mtreit•2y ago

I'm pretty good at multi-tasking 🙂 Also it gives me something to do when I'm waiting for a build to finish or that kind of thing...

mtreit•2y ago

oe•2y ago

hahah

Gaming

Programming

Brotli compression for JSON producing newlines - any work around?