C#•4d ago

✅ How do Avro, ORC and Parquet work?

I have been learning about big data and these came across as a few of the popular "optimized file formats". I want to understand how they work.

7 Replies

Sossenbinder•4d ago

Not very familiar with the others, but at least for Avro the unique selling point is probably the potential to differ in reader and writer schema. The consumer and producer do not always have to stick to the exact same encoding, they just need to be compatible I also dug up a small example of how the Avro variable length binary encoding works

Sossenbinder•4d ago

Sossenbinder•4d ago

For the schema

record Person {
 string userName;
 union { null, long } favoriteNumber = null;
 array<string> interests;
}

record Person {
 string userName;
 union { null, long } favoriteNumber = null;
 array<string> interests;
}

tommyOP•4d ago

👍

Unknown User•4d ago

Message Not Public

tommyOP•4d ago

okay

Unknown User•4d ago

Message Not Public

Gaming

Programming

✅ How do Avro, ORC and Parquet work?