Let's talk about Apache Spark an Apache flank you see before Apache Spark.
Usually people used Hadoop map reduce to process everything spark came along and actually improved the
map produce and who dupe by doing something called in memory processing which essentially allowed spark
to run processing jobs much much faster than map reduce so Apache Spark became really really popular.
And it's probably the go to batch processing framework.
So if you want to process a lot of data well you can use Hadoop to store that data data and a DFS and
use Apache Spark to run EDL jobs like extract transform load to clean and transform that data now.
Up until now we had batch processing which essentially means give me a chunk of data and I'll process
this data over a bit of time.
Usually you'd run a batch processing job at the end of the night and in the morning after a couple hours
the job is done.
But as of a few years ago the idea of real time processing started to happen things like spark streaming
came out which almost made real time processing possible that is instead of batch processing where you
waited until the end of the night to run some jobs on amount of data instead every time data comes in.
You process that data in this base Apache flank really took on the charge it offers real time stream
processing.
Now all those sparks still popular tools like Apache flank are certain to become popular.
Now this idea of streaming data and stream processing is becoming more and more popular.
It's still new but there are some exciting things happening in this landscape.
So let's talk about that in the next video.