Website corruption overview.
Hello, everyone.
In the last section, we're talking how we're using API in order to extract data from the Web, but
many of the websites do not actually offer API.
So you can extract the data.
And I'm talking about the small websites, not the big ones like Amazon or Twitter, but smaller ones
that do not actually offer this option.
So what do we do in that case?
In that case, we're actually using scrapping techniques in order to recover data from the web automatically.
And some of the most powerful tools that are known and actually are implemented into item three point
seven are the beautiful soap and scrap it all.
So, for example, scrapie is specifically for a framework written for Python and it is used for extraction
from the Web in an automated way.
And even today, and actually, especially today, many people are using scrapie for data processing
and specifically for data mining.
So if you would like to use Python for your Web extraction of data and for data mining, Skrappy's a
very, very good tool that you can specifically implement for mining your data.
So in general, in this section, I would like to cover the following topics so you can be aware of
what we are going to learn today.
We're going to talk about the web scrapping tools.
As I told you, we're going to do an overview of scrapie and beautiful.
So we're going to use them to extract information from pages and for passing the HTML files, which
are the files that all the websites do to their data.
We're going to obviously learn how to use a scrappy toolkit.
We're going to learn the web crawling process and how to analyze data with scrapie.
And we're going to do some Theros cropping on the web using cloud.
So I will show how you can use croppy and scrap web data in a cloud.
So when we're extract content from Internet, there are few widely used techniques that they'd like
to discuss today.
So the first and the more simple one is the screen scrapping when you can obtain information by moving
around the screen and checking for registered users.
Then you have the website cropping, which aims to obtain the data or the information from a resource
such as an HTML resource and process information once we extract the relevant data.
The next technique is the report mining, and this is a technique that allows trees to obtain information.
If you're not familiar with trees, I advise to researchers from the Web because this is quite an interesting
structure and can be used in multiple files such as HTML, RDF, C, ASV and so on.
And in that way we can create simple and quite fast mechanism without needing to use an API.
And sometimes we wouldn't require any connection because we can simply extract the data and analyze
it afterwards.
So this is quite a good way to mine the data you need.
But let's focus now on the next one, which is called spiders.
So spiders refer to defining specific rules for moving around a website.
And while moving around following these rules, it's like a robot.
So while moving around to this website, we can extract information about the interactions with the
users, what the data has been created on the website and so on.
So the idea here is that the software developers like you and me only need to write the rules for the
data extraction.
And after we write that rules, we can just leave the code to execute and obtain the relevant information.
And finally, we have the craws that are not spiders, but there are processes in this process automatically
pass and extract content from a website.
And they ECDIS this content from the website and provide it to the search engine immediately.
And this is how the page indexes are actually built.
And before ending this video for today, let's talk about what is Web scrubbing.
So the Web scrubbing is a technique that allows you to extract information from websites, and it basically
helps you to transform the unstructured data from the website that is collected into the HTML file,
into a structured data that is easy to use for programmers and from programming languages.
So that was the intro for that section.
Guys, I hope you really enjoyed it.
And I think it will be quite useful for you, especially once you start writing the code, which is
actually going to be from the next video.
So that's it.
Thanks very much for watching this video for today and I will see you in the next one.