Let's talk about Hadoop.
So as companies started to generate more and more data are databases that we handed like my ask you
all became too inefficient or unable to hold the large amounts of data and Hadoop became a cornerstone
of data engineering.
You see Hadoop is an open source distributed processing framework.
It allows us to do a data processing and storage for big data.
And it was actually developed by Yahoo.
Back in the day.
And then later donated to the Apache Software Foundation Hadoop was all the rage when the Big Data craze
started Hadoop was a solution that allowed all these companies with petabytes of data to store this
information in Hadoop.
Now Hadoop was essentially a data lake right.
It's a data lake solution.
It allows us to just store all this data for us.
But the popularity of Hadoop was because of two big things.
One was HD F S that is the Hadoop distributed file system Hadoop was able to store so much data.
Because of this HD DFS a file system just like you have on your computer that allowed it to store files
on multiple computers.
That is we can use Hadoop on multiple computers to store as much data as we want.
It was scalable we could scale up so data is stored across different physical computers then was map
reduced because once we store data we need to perform some jobs some processing on that data right and
map produce in Hadoop allowed us to perform jobs against this data that we had in a data lake using
languages like Java or Python you can use map reduce and Hadoop to do what we call batch processing
that is every night we can run some sort of a job to clean our data out to process our data and make
it useful for the company now Map Reduce isn't used as often anymore as we'll see later on in the videos
because we have something called Apache Spark that's actually a little bit faster but we'll get to that.
The idea here is that we finally had a tool with Hadoop that allows us to store a lot of data across
multiple machines using the CFS and run batch jobs using Map Reduce on this large sets of data and a
lot of other tools started popping up around Hadoop tools such as hive love their logo by the way check
that out anyway hive makes your Hadoop cluster feel like it's a relational database even though it isn't
it allows you to use hive and right ask you well queries against your HDD DFS file system.
So in a sense you kind of had this data lake of Hadoop and also a little bit of a data warehouse type
thing with hive by allowing ask who all commands on your data that hopefully was processed using map
produce and you may not have heard of Hadoop before because it's used by data engineers.
Remember data engineers usually don't work on systems that are customer facing that is they don't build
databases that you would use with your app or with an web app.
Instead they build the systems behind the scenes that ingests collects runs EDL jobs that is extract
transform load on data so Hadoop is that behind the scenes storage layer that stores large amount of
data running batch processing against this data so that we can have some useful information and Hadoop
is still very popular and very capable when it comes to large data petabytes of data.
Let's learn a few more tools in the next video.