Hey Cloud Gurus.
Welcome back. We've talked a little bit about data zones.
We've talked about folder structure.
Now let's talk about the files actually going
into that structure.
In this lesson, we're going to start by taking a look
at the file types available to us,
and we're going to go over specific ones,
such as Avro, Parquet, and ORC.
After that, we'll do a little comparison of those three
and wrap everything up with a review.
We've talked a little bit about data lakes
and how they're a great place to store
all kinds of information.
The good news is that all formats are welcome,
be that CSV, JSON, whatever. You can throw it all
in your data lake.
The bad news is all formats are welcome.
And I don't have all these pictures of fruit for no reason.
If you think about a basket of fruit, if I tell you,
hey, there's a bunch of fruit.
Well, it doesn't really tell you much
because an apple and a lemon are very different.
And so you're going to want to be careful
which one you're using.
Just because you can use something, doesn't mean you should.
Even though CSV and JSON are common files
and easy to work with,
they're not necessarily ideal for analytics situations.
They can lead to lots of small files,
which creates performance and cost issues for us.
And they're not designed to work with parallel systems.
Luckily, even if our data comes in these formats,
we can consolidate them into ones more useful
to our purpose.
The first of these that we'll discuss is Avro,
and it uses row-based storage, where all the record data
is stored together.
If we take a look at this visually,
let's say we have a table with names
and course topics in it.
You can see that each of the rows is stored sequentially.
So we have all of our guru1 information,
all of our guru2 information,
and then all of our guru3 information.
The schema for Avro is stored in JSON,
which means it's easy for humans to read.
And the data is stored in binary,
which is easy for computers to read.
Meaning it's very efficient.
Some of the benefits of Avro include
easily handling schema changes.
We all know the fast pace of IT.
And schema changes are a real thing these days.
You need to adapt to growing business needs
and Avro easily handles that
because of its JSON format at schema.
It's great for write-heavy workloads because it's row-based,
and being row-based also means it's great for ETL operations
that need all the columns.
Comparing apples to oranges, Parquet is another option.
And this one uses columnar storage.
And this is where the values of each column type
are stored together.
And so using the same example from earlier,
instead of having all of our row information
stored sequentially, we have each column type.
So we have all of our ID columns stored together,
all of our name column, and all of our topic column.
A standout feature of Parquet is that it has nested data
structures, allowing nested fields to be read individually.
And just like Avro, it's data is stored in binary,
also making it efficient for computers to use.
Some of its benefits include: it's great for wide column
queries, it's efficient for read-heavy workloads,
and this makes it a great choice for analytics in general,
especially in the curated zone,
where read performance really matters.
And you can easily query a subset of columns
or nested data.
Another option is ORC, or Optimized Row Columnar.
And this one was designed to improve
on the other file formats.
It too uses columnar storage, just like Parquet,
but it uses stripes, and so it has collections of rows
that contain row data in columnar format.
And like all the others, its data is stored in binary.
Some of its benefits include: it supports ACID properties,
which can be a big deal if your business requires that;
it compresses more efficiently;
and its stripes enable large efficient reads.
That was a lot of information.
So let's do a bit of a TLDR on which file you would pick
for which situation.
For our analytical queries,
we're going to use either Parquet or ORC
because of their read performance benefits.
Conversely, if we have heavy write operations,
such as ETL work, we're going to use Avro
because of its row-based storage.
For nested data, that's a standout feature of Parquet,
and having ACID properties is going to point us
toward using ORC.
Lastly, if you need a lot of flexibility
in your schema evolution,
that is going to bring you to Avro.
And so they are each created to work efficiently
and analytic solutions.
It just depends on your particular needs
as to which you'll use.
By way of review, while you can store any type of data,
you want to use formats that are designed
for big data processing.
As I said before, lots of CSV or JSON files
could lead to a large number of small files.
And that's your enemy in these analytics situations.
It gives you less performance at a higher cost.
And so you're going to want to stick with Avro,
Parquet, or ORC.
Your primary decision may come down to
whether you need row-based, Avro, or columnar storage
in Parquet and ORC.
And while parquet-K and ORC are similar,
they do have differences.
And so evaluate those based on your business needs
in order to choose which is best for you.
I hope that you have enjoyed this lesson
and that the differences in these files are clear to you.
If you have any questions,
please feel free to reach out to me
and I'll see you in the next lesson.