In this article, we are going to learn the difference between crawling, scraping, and parsing in computer science. Sometimes these expressions are used interchangeably, but they are definitely not synonyms. Usually, crawling is the process that comes first – it is about following internal and external links, then comes scraping, which is about extracting bits of data that resulted from crawling, and then at the end comes parsing, which is all about breaking that data into specific, meaningful parts. Nevertheless, let’s look into these terms in more detail.
So, what is (web) crawling?
Have you ever heard of Google crawlers or Google (ro)bots? Sometimes they are even called spiders or spiderbots. They are the ones that do the crawling. They visit websites and search for new data or new content on these websites.
In general, it is not only Google that has them, but every search engine (and some other websites) also operates these internet bots because they serve the main purpose of all search engines – they systematically browse or search the World Wide Web and index that web. Crawlers copy pages they find and update them whenever they find new data on them so that search engines can index and process these pages later on because end users (you and me) could search for content more expeditiously.
This process is called (web) crawling. In general, website owners, want their pages to be found. However, it is important to bring out that these bots do consume a substantial amount of resources. For this reason, sometimes, owners of public websites decide to limit or even prevent access to crawling bots.
Of course, to avoid overloading websites, search engines did come up with a policy that defines the behavior of web crawlers and how to coordinate web crawlers that are distributed through the web (parallelization policy). Selection policy defines which pages should be downloaded, a revisiting policy defines how often crawlers should check for changes and updates on pages, and there is even a politeness policy, which defines how to prevent overloading websites.
What is scraping?
As said in the introduction, sometimes people consider crawling and scraping as being the same and use these terms interchangeably, and yes, there is a connection. However, there is also a substantial difference. In short, as presented in the section above, crawling is about discovering and then updating links on the web. You don’t even know which links or domains you (web crawler) are searching for and this is the reason you crawl for them – first, you need to find these links. Scraping, on the other hand, it is all about data and taking that data from a specific website. You already know the website or at least the domain you’re targeting.
So, what is that connection between crawling and scraping? This connection is obvious in projects, where you have to extract data from a specific website. You know the domain, but you don’t know all the URLs that are on that domain. So, what do you do? First, you have to do the crawling – you have to discover all the URLs that are located on the domain, and this is basically what web crawlers do, too – they search the web so they can index webpages. When we say all URLs, we mean all URLs that you think are important, for example, the URLs from a specific category or only the URLs that contain a specific keyword. For this purpose, you need to create a little script called a crawler. This crawler will collect all website URLs that are important to you, so the result of crawling will be a list of URLs.
And then comes the part with scraping. When scraping you will extract or scrape the data from these URLs and for example store this data into some sort of database. When scraping, the result will be data that is listed on these URLs. If you want to perform scraping, you will need a scraper, which is a script that visits web pages. Scraper does not collect new URLs as a crawler does. Instead, you give a scraper a list of URLs that you collected with a crawler, and the scraper retrieves data and stores that data.
Maybe you’ve heard of data scraping and web scraping. There isn’t much difference, except that with web scraping you need the internet so you can perform it, while with data scraping we are dealing with data that can be located either on your computer (in that case you don’t need internet) or on some website. In both cases, you are importing the information you scrapped to some local file on your computer or to another website or server.
And then comes the parsing.
What is parsing?
Sometimes parsing and scraping are used as synonyms, but they are definitely not the same. However, scraping tools often include the functionality of a parser.
When parsing, you are organizing the unstructured data you’ve gathered or scrapped either from some data storage or directly from the web in a form of HTML into specific data bits that are meaningful to you. Following this logic, a parser is a bot that can be used offline and helps you analyze and structure data into a useful structure – it breaks down the unorganized data into useful pieces. Depending on a business, this could be a product name or price, or some text or other information such as headlines or reviews from any type of website.
In conclusion – why do we need all crawling, scraping, and parsing tools?
In theory, scraping and parsing could be done by hand, however, it would be super time-consuming and most of all not as accurate as actually needed. Obviously, in the past, this was all done by manual collection, but these days, we can no longer rely on that. Scraping (and parsing) scripts are highly accurate and they eliminate human errors from analyzing operations. This way you can be confident that the information you receive is definitely accurate. In terms of time consumption, scraping and parsing tools are cost-efficient because they can save you on employee costs and at the end of the day – they are more or less automated. The additional advantage is that they help you pinpoint the exact data you need.
Nevertheless, let’s take these pink glasses off and see what are the main challenges when you’re dealing with crawling, scraping, and parsing tools, or God forbid when trying to create one. Yeah, not all website developers are stupid – some even build into websites’ data blockades, which make it challenging to collect the data you need. “Challenging” is a good thing – it challenges you to think differently and find a way to pass that anti-crawling and anti-scraping policy and gather that data after all. And when we said crawling, scraping, and parsing is super efficient in terms of labor and time, well we were comparing these tools to manual analysis. There are whole companies involved in providing these services and they have the people who know how to do the job. High-level programming skills are often involved. However, even beginners can sometimes do the job. Here’s an example where we applied low-level PHP knowledge to parse data in a form of URLs and accompanying headlines from an HTML source: