All right, Gurus, welcome back.
In this lesson, we are going to be talking about
Azure Databricks.
And specifically,
I'm going to be teaching you what Azure Databricks is.
So like the other lessons,
we're going to walk through an introduction
to Azure Databricks.
Then we're going to talk about
the when, where, and why it's used,
how it works,
and I'm going to do that by showing you a little bit
of a live look in the demo,
kind of like we did on Synapse Analytics.
So with that, let's dive in and get going.
Introducing Azure Databricks.
What is Azure Databricks?
Well, it has 3 main functions.
First, it serves as database.
First, it serves as Databricks SQL.
And what that means is Databricks SQL
is just a simple secure access to data
that allows you to create and reuse SQL queries.
Next, it's used in data engineering.
And then third, it's used in machine learning.
Now for our purposes,
we really only care about data engineering.
So the other things we're not even going to talk about,
because it doesn't relate to the DP-203.
Now specifically for this course,
when we're talking about data engineering,
we're going to be talking about transformations.
So as you can see
from the sign of over here on the right, we have an error.
Now, someone has gone along and messed up the sign.
This is a common fix for something like Azure Databricks.
It's going to process and curate,
if you remember from our Databricks lesson,
and it does that by taking and standardizing.
So remember we about the country codes,
where you could have USA or US or United States,
or even just a, you know, 02 for a country code, right?
Well, Databricks is going to be able to go in
and make all of that data look the same
so that when we do processing later on,
we're all looking at the same source of data.
And we can be reasonably certain that we've cured it
from all of the least obvious errors in the data.
So looking at where it actually lives,
I've pulled up a Microsoft diagram here,
and you can see so we're looking at this,
and hopefully you're starting to sense a pattern.
But we have our ingest, far left, we have our store.
So we ingest data into the system,
then we store the data,
and then we pull it out of our storage, our data lake,
or our BLOB, and we use Databricks to prep and train.
And so that's going to be the transformation.
Now, when we talk about training data,
that's more machine learning.
What we're talking about is prepping data for the
model-and-serve phase in Azure Synapse Analytics, or Cosmos
DB, or something else.
So Databricks is going to serve
as that transformational layer
to curate and process your data into something
that it needs to be.
Now, at the very top, you'll also notice Data Factory.
This is something you're going to see continuously
on all of the diagrams,
because Data Factory, if you remember,
serves as the movement in the pipelines,
to take that data, and move it through those stages,
including Databricks.
So now let's talk about some core concepts of Databricks.
The first is clusters.
Clusters are just a group of compute resources
that you're going to use to process
your Data Factory transformations.
Think of it as the engine, if you will.
Next up, we have our workspace.
So workspaces kind of serve as the filing cabinet
to store all of our files,
which we'll talk about here in just a second.
But the workspace is kind of like a filing cabinet.
Now in that workspace
you're going to have individual folders.
Folders are the notebooks.
The notebooks are going to store all of your cells
which are your individual pieces of code.
So we have our individual pieces code, written in cells,
stored in notebooks.
The notebooks are stored in our filing cabinet
or our workspace.
And then, lastly, we have our libraries.
Libraries are a package or a module
that conveys or gives additional functionality to Databricks
so that we can do something specific
that wouldn't normally be there in Databricks
as a standard unit.
So we can create and import our libraries as needed as well.
Lastly don't forget, are tables.
Tables are just the storage for all
of the structured data that's created in Databricks.
So you'll see that as well.
These are the 6 main concepts that go with Databricks.
All right, so now let's hop over into the portal
and take a look at Databricks in action.
So just like with Synapse, I have opened up the Azure portal
and I have clicked on Launch Workspace
to give us our Azure Databricks workspace.
Now you'll note that over here on the left-hand side,
again, you're going to see that menu bar that's
going to walk you through the main pieces that you need.
You can also find it here in the middle,
including additional documentation
if you really want to get down into the weeds
on a particular topic.
So for us, you can see first, at our very far left,
this is our workspace.
So remember that is our filing cabinet,
and it's going to break it down,
so that we can see our individual users.
And we're going to go ahead
and click on the Quickstart Notebook.
Now the notebook, remember that's the folder, right?
That's going to store our individual cells.
And you can see that each one of these
is an individual Cell.
And I've just opened up a Databricks Quickstart notebook.
So we can see some data right at the start to begin with.
And I want to scroll down and just show you
a few different things.
So the first is we can go ahead
and start creating snippets of code in any of these cells.
In addition, it's going to give us data,
as we run those cells.
And in this one you can see at the bottom,
it's already went ahead and run all of the cells for us.
So we can see a table format of the data.
And the other nice thing is with Databricks,
I can also click on
and I can see graphs of the data as well,
if I want to see the data in different formats,
which is very, very helpful.
So as a data engineer,
I can go through, write some snippets of code,
and start to see how it's working
right at the very beginning,
rather than having to complete the entire code base
before I can run and get real data.
The other piece that I want to show you is here,
see this little % sign, and then python,
Databricks allows you to use different code languages,
and we can actually switch between cells.
So I can just do %python and start writing in Python.
Or I could do %scala and start writing in Scala.
That's also really handy, if you have, for some reason,
a need to write in different languages,
you can actually change that on the fly
as you go through individual cells.
So there you go,
there is a super high-level look at Databricks
in the Azure portal.
Finally, let's jump in and talk about our review.
First, we have our introduction.
So, know what Databricks is, you need to know what it is
and where you would use it, which is in transformations.
And specifically, large-scale transformations.
That's really where you're going to want to use Databricks.
Core concepts, remember the cluster of the workspace,
cell, library, and table.
That's going to be important just for you
to kind of get in your head, how all of this works together.
And then finally, we looked at this in the portal.
So again, we're going to dive further into these concepts
for what you need to know,
but right now, what I really want you to do
is to be able to get in your head
where these services live and what they do.
If you've done that, then you should be good to go.
All right, we are almost done with this section
in the next lesson are going to jump in and have a review.
I'll see you there.