Scruffy overview.
Hi, everyone.
Today, we're going to talk about a very interesting tool and this is actually the next level that we're
going to build up after we learn the beautiful SoPE tool kit.
So today we're going to talk about the scrapie, which obviously is a python tool used for weps cropping
so you can find all the information about scrapping scrapie dot org.
And this, too, essentially allows you to extract the data from the Web pages.
And many people use it for data mining, information processing and so on.
Just two rows actually in absolutely every platform since it is an item.
So if you have Mac Windows on Linux operating system, you can use it to all of them without any issues.
So even though the main purpose of scrapie is to extract data from the Web pages using Python, you
can also do it with scrappier when using the APIs.
So let's talk about the advantages of Krutch.
So Skrappy's actually the bridge fruit for you to the partner automation, because here you simply write
the rules and then scrap it does the job instead of you.
So once you write your rules, then scrappier, goes to the Web, find the Web page and extract all
the information that you require without the need for you to do any additional work.
You can obviously easily modify your sole source code used in different platforms, regardless of whether
you use Linux, Windows, MapQuest and so on.
So this is quite a useful tool that will definitely simplify the way you're extracting information from
websites to extract data from the e-mail file.
Scrap usually uses the successful selection and expat expressions that we talked about in the previous
lectures.
You can simply write your code in Python console and then use these rules in order to extract information
from the Web.
It also support Muj performers like Jason Sias v. Ximo and so on.
So regardless what you use, you can definitely assess most of the more popular file types out there.
And the other thing actually that I would like about scrapple is that it allows you to be more flexible
with what you write and what you get.
So if you have errors in the code, there was a big chance of scrap rules to execute it and find the
best solution.
You can also use your own expressions, such as signals, extensions and pipelines.
But let's talk now specifically about the architecture of those Croppy Scrappy allow you to simply scan
the content of a website and extract the information that is useful for you.
There is a very important architecture going into the background of that tool.
So the first part of the architecture is the interpreter.
So the interpreter simply allows you to create projects and test them quickly.
The interpreter used some colder routines called spiders and are used for making a request to a list
of domains that you predefine.
So once you predefine these rules on HTP, request is made and then scrapie applies the rules that you
predefined to this list of HTP requests and so you can execute all of them.
So, as I said, scrappiest expat expressions and with them you can actually get pretty detailed information
about them, about what we extract from the website.
So, for example, if let's say you want to extract download links from a page, you can simply write
an expert expression using scrapie and you easily be able to assess all the attributes in that website.
And finally, there are the items which are basically containers of information that allow storing information
that's returned from the rules.
So, for example, if you write the rule that assists a website and you get a response from this website,
you're basically going to store all this information into these containers or the items that the return
from the rules.
And on the next few guys, you can actually see the workflow of scrapie and how actually the scrap engine
is implemented in order to cope with all of these web features.
OK, so we can see that here.
First we have the Internet and with the Internet, with the downloader, OK, that is connected to scrapie
and.
We keep doing requests and we're getting responses, so once we do a request to Internet, we're downloading
their responses and we're using the spiders to do that.
So as you can see, all of those four features are actually connected to the scrap agent and this information
that goes inside their moves dynamically, which means that constantly the Ruža exchange with the website,
so constantly you can access different locations in Internet depending on the rules that.
That's right.
So as you can see on the image, the spiders use the items in order to pass the data to diatom pipelines.
And as you can see, scrapie actually has different spiders.
Some of them pass items through the pipelines and other ones are sending request to a scheduler over
here.
So the requests from the spiders to the scheduler are actually the ones that are making requests to
the actual server, because from the requests you're getting the requests sent to the server and this
how in the information.
So once we request information from the server, the information is in the back after that of the spiders.
And so the spider is fed back with each request that we get from the server.
So, guys, this was the overview of scrapie.
I hope it was clear, because in the next video, we're actually going to start coding and we are going
to do some scrapbooking requests to the Web.
So that's it.
Thanks very much for watching that video for today.
And I will see you in the next one.