Working with spiders.
Hello, everyone.
Today we're going to talk about what spiders are and how we can actually work with them.
So spiders are classis, OK, spiders are python classes that helps us that help us to navigate to a
specific website or domain.
And it basically defines how to extract the information from there.
So you can simply write a class at some rules there.
And then once you run this file or actually describe a project, you are actually running this spider
and extracting the information depending on the rules there.
And you saw last time we actually create the spider, which extracted the information from two links
and we got all the information for the links in the HTML format.
So today we are going to extend this a little bit.
And in general, spiders are basically generating an initial request to the website.
And once you generate the request of the URL, it gets a response.
And this response is usually at all the polling that you get inside your computer.
So once you get this downloadable and basically you have all the information for the website.
So, guys, now we're in the terminal.
And let's do one thing.
Let me show you how you can actually use Scribe to extract data.
So we are going to use the show today, and I will show you a very simple way.
So let's write scrapie.
Case Croppy.
Shell, OK, and now we're actually going to run this show, so we're going to run it for the following
website, HTP then.
That's right.
Quotes.
Not to.
Scrub.
Dot com and then as Fred slash page slash one, same as the last time.
OK, let's go to the parenthesis here and now you can see that we're actually downloading here all the
information for the website.
OK, we're getting a list here over and say things we don't already to information and so on and so
forth.
And you can see different options here that you can use.
So one particular of the options, one particular option that we can use is the.
Response option, OK, response, because there is stored information for the website, so if we do
response dot success and then they do litho, we're getting the actual name of the website.
So you can see that we get Selecter, XPath and so on and so forth.
And this is the title of the website calls to Scrap.
OK, and this is directly extracted from the HTML file using the expert expressions here.
So for example, let's try another thing if we do response.
That success and then I thought, OK, let's close the bracket and let's write extract.
That's it.
So in that way we can actually extract text from the website.
So let me do it.
And here I specified that they just want to extract the text without the additional code.
So two things are very important here.
The first is that we actually added text with double column here, and the second thing is the extract.
So the text means that we won't extract specifically only the text elements from the file.
And the only text element here is cost of coal.
And also we specify that we want them from the title element.
So since there is only one text in the title element, we extracted the actual title of the website.
And there is an important thing because like if we don't write text actually, so if we do response
that says, OK, and if I simply right here I thought, OK.
Not extract, what we're getting is still the title, but also we get the bounce or we get the exact
markers of the course scope, and for that reason we use this text to get specifically only the exact
insight without the markers.
So you can see the difference is actually quite obvious.
Also, another thing we can use, actually, if you'd like to remove the brackets here, you can simply
write response.
Not cases, and in the brackets you can write title.
Text, OK?
And after that, they can write to zero dot.
Extract.
OK.
And you can see that we're actually remove the parentheses and there's only course a scope in parentheses
because the brackets are removed.
So actually this is the best way to extract the header from the from the website.
Also, our way to do that is, for example, you can write a response.
Cyesis and then here we can write I.
Next, OK.
And after DataDot, Ari.
And we can do here are.
And then quotes.
Dot star.
So I don't care what's after that and close the bracket and if I hit Enter here, you can see that we
get basically the same thing.
So get quotes of scope.
So whatever metal to use, I think actually the first one is a bit better.
You can always export the title of the website that you found with the heap.
So that said, guys, this was the brief intro of what we are going to do in with the bull crap, too,
actually.
So that said, thank you very much for watching.
In the next video, we'll continue with the exploration exploration of the scrapie tool kit.
Thank you very much for watching.