Hello everyone and welcome to the r d d transformations and actions lecture and this lecture are going
to begin to dive deeper into using Sparc and Python.
There's important terms and definitions.
You can check out the Jupiter notebook that goes along with this lecture.
I'm going to go ahead and jump that note book now and walk you through a few of them.
All right.
Here I am at the notebook that goes along with this lecture.
I just wanted to quickly go over some important terms that we reviewed before.
But here's just a refresher.
Remember that an R D D is a resilient distributed data set and we have transformations and actions.
A transformation is a spark operation that will produce an RTD.
While the action is a spark operation that produces a local object.
So you can think of a transformation as just a set of instructions.
Basically a recipe to follow.
They'll produce an RTD.
However until you call an action you're not going to get some local object as a result that you can
then either view or play around with.
And then finally a spark job is a sequence of transformations on data of some sort of final action so
Sparke job is just the Arkadi that you have a transformation and then call an action on all three of
them combined.
There's a few common ways to create d.
There's SC textfile where Essie's that spark contex we saw earlier.
So if you want to create r d d of lines from a file you can use S-E that text file.
Or if you're dealing with some sort of elements of array or maybe a list you can also use C the parallelize
.
And then we have transformation.
So you've actually seen the filter transformation earlier but you have quite a few examples that are
available to you.
And these are some of the transformations we're actually going to be working through and showing examples
through this lecture.
Some of the more common ones are things such as philtre which will discard elements that don't return
True the pending on what lambda expression or function you pasan.
Then you have a map and there's a couple of examples here.
For instance if you want to multiply.
Already the element by 2.
You could use that second example or the third example of map will split each string into words.
Then you have a flat map.
Remember the difference between map and flat map.
In this case this flat map will split each string into words and flatten out a sequence.
You also have sample which is another transformation sample just creates a sampling with replacement
and you can actually specify whether or not you want with replacement to be true or false.
And then you can specify what percentage you want a sample of the entire R D D.
So just the way to take out a sample of certain elements from the original RTD.
Then you have union which can actually append and RTD to an existing RTD you have distinct which removes
duplicates and in our.
And you actually also have sort by which will soar elements in either ascending or descending order
depending on what parameters you pass in.
So those are just transformation examples and we're going to walk through a lot of those today.
And then we have already the actions.
So once you have that recipe transformations ready to go the next thing you're going to do is execute
them by calling an action.
So some common actions are collect which is actually seen before.
It just converts the entire RTD into an in-memory list.
Then there's take with some and number passed into it which takes the first elements of the RTD top
which takes the top elements of the deity.
Maybe you want to use that after you've sorted.
There's also take sample which creates sample of three elements of replacement.
In this case because we pass and 3 for end and with replacement equals true and then there's some mean
and standard deviation which basically do what they say.
All right so the best way to show all of this is just three examples.
So we're going to do is work and create through all the examples here in the note book.
So your DD key value parodies etc. and we're actually going to work through all of this.
So let me go ahead and jump right back to a empty notebook to get started.
OK.
So let's actually create a text file to work with.
We've done this last time but you can just say write.
File example.
Let's go and call this example to 60 and I'm going to write first second line.
The third line then of fourth line.
So I'm going to run this cell to actually create that text file.
Here we have writing sample to the text.
Let's go ahead and perform some transformations and actions on this textfile to get started.
I'm going to say from Paice spark import spark context.
Run that and then I'm going to say s c is equal to spark context.
Then I'm going to go ahead and show the RTD by saying the text file and I'm just using tab autocomplete
there and I'm going to pass an example to 60.
So here I can see the actual aready where describes it as a map partition.
R d d text file at some sort of location in Java in order to actually save a reference to sorry we can
go ahead and make another object we'll call it text underscore.
R d d is equal to see text file and then pass an example again to let's run that.
OK so now we have a reference to the RTT.
But I just wanted to show you what it actually looks like if you just say that by itself.
Let's go ahead and map a function or lambda expression to each line and then collect the results.
I'm going to go ahead and say text underscored DD and do a transformation on it I'll say Lamda line
and then go ahead and line split.
So so far I just have a transformation that will split everything.
Every line there to a list of words so I'm going to call this words run there and then notice words
doesn't do anything until I actually perform an action on it.
So I can listen then words that collect this may take a wild painting on your connection but there it
is.
If I just take a look at my original text RTD and call collect on it.
Notice the difference here where here I said split every line into a list of words.
If I just call collect on the original text I actually get each string line in that list so I notice
the difference here between mapping this line split like expression versus the collection of just the
original R D D.
All right let's go and explore map versus flat map.
So if I wanted to collect everything as a single flat map I could use the flat map transformation so
I can say text underscore RTD dots and instead of saying map I'm going to say flat map and I'm going
to pass in the same land expression for every line take it in and split it and I'm going to go ahead
and call collect on that.
So notice here on this previous example I separated this into a step my transformation step and then
my action step.
But you can actually just put it all in one line if you want.
It's up to you.
You're not wasting any memory by splitting this up into multiple steps because remember nothing actually
gets evaluated until the final action call.
Let's go in and run this and see the difference.
All right.
So this is a flat map the exact same expression except instead of using map or using flat map.
And look what happens here.
We actually collect everything as a single flat map.
So now we just have one single array or one single list of all the words in that text file.
Ok so now that we've worked with just the text already does and how to aggregate values with them.
We're going to begin looking for is key value pair are Didi's and in order to do this what I'm going
to go ahead and do is create some fake data as a new text file.
Now if you go back to the notebook for this lecture here we just finished the map versus flat map section
.
There's a cell here with the actual text file information.
You should probably just copy this and then paste it back into a cell here that we don't spend any time
writing all of this is going to run this.
And this is a services that text file.
This basically looks like to say typical CSP file.
We have an event ID some sort of time stamp for that event.
The customer that did some sort of purchase that that even the state they did the purchase and what
service they purchased so that service has an ID and then the amounts in dollars of how much that service
cost.
OK so let's just pretend that there's some data that represents services sold to maybe some sort of
software as a service business.
We'll go in and say Services is equal to see the text file and all say Services t x t run that.
All right.
And usually when you're dealing with some sort of more complicated textfile like the one we have here
you probably have to clean it or manipulate it in some sort of fashion.
Let's go ahead and just grab using take the top two lines of this already.
So here we have basically the header and then the first line of data.
This right now that it's all just a single string per line.
Let's go ahead and explore just splitting this up using map.
So I'm going to say services that map Lamda taken a line and then split that line.
And here I have again that Python already.
So nothing's going to happen until I actually call an action on this.
And since map is just a transformation I need to call an action on it to actually see the results output
and some sort of local object saying that take the top three results of this so we can just say take
three.
Run this and we can see here we begin to actually transform this string as every item in this list to
an actual list of items.
So now we can actually get some separation between each variable in the columns.
Now what we want to do is maybe try removing this hash tag here.
We don't want hash tags of anti-ID we just want event ID.
So let's go ahead and explore how we can do that using this RTD transformation action methodology.
First thing I'd do is map Lamda and let's go ahead and say line and I'm going to grab everything from
index one forward if the first element of that line is equal to a hash tag
else I'm just going to return that line and then I'm going to collect the results and see how it works
.
OK and that worked perfectly we were able to remove that hash tag.
Basically all I said was going back to the original services check the first item of every string.
If it's a hash tag.
Make sure to remove it by grabbing everything beyond it.
If it's not just go ahead and return the entire line.
That's all I was doing here of this land expression.
So you can see the flexibility of a land the expression they can actually be quite precise in what you
want to do.
So now that we have that I'm going to go ahead and then do this same map call of splitting it up.
So we're going to not collect this yet.
I'll call this clean run that and then off of clean I'm going to say clean is equal to clean that map
and then I'm going to map the previous thing here in order to split everything.
So I'll say Lamda line line thoughts splits and let's go ahead and run that.
And then I will say clean dot collect.
And there it is.
Now we have our previous services mapped to a list of items and we no longer have that hash tag.
All right.
Now that we've seen the basic transformation action relationship that we can use Let's go ahead and
begin to use methods that combine lambda expressions using a bike key arguments and these by key methods
will assume that your data is in the form of a key value pair.
This is the reference back to those pair already.
I mentioned back in the spark overview lecture.
Let's go ahead and check out clean one more time here by saying clean that collect to see the data we're
working with.
All right let's go to and start by practicing how to grab fields for instance let's go ahead and explore
how we could get the total sales per state meaning I want to know how many sales in the amount column
were for New York.
So I kind of want to do this group by but it doesn't work exactly like that with SPARK.
You're gonna have to use a reduced by key statement.
First thing I'm going to do is just say clean and then going to map and I just want to grab these fields
.
State field and the amount field.
So I'm going to say lambda.
And then there's a line I remember that line is a list so if you want just to make it very clear I'm
going to say ls t and then I'm going to return a tuple and I want to grab the state column and the amount
column values.
So if we look at states that's 0 1 2 3 in those lists all say ls t of 3.
And then the amount is the last one second actually is using negative indexing to grab that last item
and then I'm going to go ahead and show you what this looks like.
By calling collect on it and there you have it.
Now you have a list of these tuple pairs and that's exactly what we need for these pair RDD commands
to work which is going to be the reduce by key statement.
So now after collect on I'm going to go ahead and do just delete that and I'm going to go ahead and
call this pares equals cleaned up map.
So go in and run that and then I'm going to say pairs and I will use the method reduce by key and reduce
by key takes in Ilium the expression and it assumes that you have your data already in the tuples form
.
So I can say amount one amounts to Colon mount one plus amount to 82.
And if we take a look I just want to show you what pairs looks like again.
So I can explain this clearly if I look at the collection of just pairs by itself.
What reduced by Key does is it assumes that the very first item in this tuple is the key and it performs
this lambda expression as a reduction on the second item.
So automatically assumes that you're going to use this first column this case state as the key.
So I'm going to call this Reekie.
So reduce by key and then Afrique key I'm going to say collects
and here you can see that the amounts are actually still strings so this didn't quite work exactly like
we expected it to.
So we have the land expression working correctly we are adding the amounts together but it's concatenating
strings because if you notice here we can still see single quotes.
So that means these are actually still strings that is concatenating together so it's actually adding
100 and 750 as concatenation of strings instead of actual numbers.
So let's go ahead and fix that and you can fix that by just converting these two amounts to float here
.
And this is the kind of thing you're going to be doing often of spark kind of filtering through your
data cleaning and then step by step through each transformation check back and make sure you fix things
to get it in the way you want.
So now if I run that run Reekie again I get exactly what I wanted which is the total sales per state
and that uses the reduced by key method.
Again in order to use the reduced bakey method you need to have your data in a format that looks like
this some sort of key value pair with these tuples and you can do that by using map on your data and
grabbing whatever tuples you want for the columns of data.
Let's go ahead and continue our analysis by sorting this output.
So I'm going to grab the state in the mount's add them then get rid of some columns and show you how
to do all this.
So I'm going to basically start from clean.
So when I say clean Let's go ahead and collect that just so we can see what it looks like.
So here's clean.
Let's go ahead and show all the steps we're going to do.
So I'll say step one is take clean and map lambda list and grab the tuples are the items we want and
set it as a tuple.
So I want the third item and the last item and I want to return that as a tuple.
So that's step 1.
Let's go in and make sure that works correctly.
So step 1 is Schuss grab the states and the münze and we'll grab it in the form of a tuple.
Now let's go to Step Two.
Step two is going to reduce by key.
So say Step two is equal to step 1 dot reduce by key.
And notice the capitalization there will grab lambda takes in amounts one empty one empty two and I'm
going to return the addition of those 81 plus floats a. To then the next thing to do is go ahead and
try to get rid of.
If we look back we had this states and the mounts.
I don't want those I just want these values here at the state and the total amount.
So it's going to get rid of those.
So let's get rid of states amounts titles.
So then step three we're going to go ahead and say this is step two dots and I'm going to filter out
lambda for let's go and just say X and we'll say Knot's x 0 equal to states and that will remove any
item where your first item in that tuple is state which essentially you're going to get rid of that
first line.
OK.
And now let's go ahead and sort the results by the amounts.
So we'll go ahead and say Step four is equal to step three and I'm going to call sort by a transformation
and call lambda.
We'll go ahead and say ST amount and we'll take ST amounts.
Grab the second item at index 1 and then say ascending is equal to false.
So this basically tells you a sort by the second item in those tuples and then say ascending is equal
to false.
And then finally performed the action.
So we'll say Step 4 collect everything.
Let's go ahead and run this and see the results.
All right perfect this is exactly what we wanted.
It's really similar to everything we just did.
But he really broke it down into step by step.
So we grabbed our data and this clean data remember that was getting rid of that hash tag we go back
appear clean.
Was are getting rid of the hash tag there and splitting it up into lines.
Then we actually grabbed the items we wanted returned that as a tuple.
Now that it's in that tuple form we can do the reduce by key.
Then we got rid of the state and amount titles sort of those results and performed that action on it
.
So you should remember to try to use unpacking or tuple and packing for readability.
Let me go ahead and show you a quick example of that.
Here we were using indexing.
But a lot of times you want to replace that indexing with tuple and packing.
So I'm going to make a list called X just to make this clear I'll say Id States and then amounts a lot
of times you may not be using lambda expressions here but you'll actually define your own function outside
of it using the F and then does the player function which helps for readability and just clears things
up.
So let's go in and pretend you are going to do that.
I'll call ADF phunk one that maybe it takes in a list and then it returns.
Ls t negative 1.
So that's one method you could use which is essentially what we're doing here right.
These were what are land the expressions were they took in an item and return some indexed item off
of that.
However this isn't really readable if you come back to it later.
So if a couple of days pass by you're trying to remember what is the index tree and what is the last
index what you will want to do is try using tuple and packing instead.
So you can say something like the f..
Let's go and make this phunk to and do something like this.
Will say Id underscore s t underscore and T and there is just one object.
But I can unpack the values
so I'll say I d s t and t tuple and packing is equal to ID underscore s t underscore and T and then
return.
And I can actually now return singular values here.
So then I can just return the amount and that function works the exact same way.
So if I say func 1 on X member X looks like this that returns amounts say func to an X that also returns
amounts.
But when you go back after a couple of days of working on this function to here is much more readable
than a function of one reading function to you I can immediately tell what I was trying to do here.
I was trying to create a function to return the amount and function one.
All I know is I was trying to return the last item.
OK that's it for this lecture.
Make sure you go and review back to the notebook here.
Everything we did there's a lot of extra information here on transformations and actions that you can
play around with that are really helpful for you to review.
Hope you enjoyed this lecture and I'll see at the next one.
Thanks everyone.