C#•2mo ago

✅ How do Avro, ORC and Parquet work?

I have been learning about big data and these came across as a few of the popular "optimized file formats". I want to understand how they work.

7 Replies

Sossenbinder•2mo ago

Not very familiar with the others, but at least for Avro the unique selling point is probably the potential to differ in reader and writer schema. The consumer and producer do not always have to stick to the exact same encoding, they just need to be compatible I also dug up a small example of how the Avro variable length binary encoding works

Sossenbinder•2mo ago

Sossenbinder•2mo ago

For the schema

record Person {
 string userName;
 union { null, long } favoriteNumber = null;
 array<string> interests;
}

record Person {
 string userName;
 union { null, long } favoriteNumber = null;
 array<string> interests;
}

tommyOP•2mo ago

👍

Unknown User•2mo ago

Message Not Public

tommyOP•2mo ago

okay

Unknown User•2mo ago

Message Not Public

Gaming

Programming

✅ How do Avro, ORC and Parquet work?