Ethical Web Scraping and Crawling: Navigating the Digital World Responsibly

The wealth of data available on the internet and the infinite potential that it has to offer requires much diligence and technique to unlock. This is where ‘Web Crawling’ and ‘Web Scraping’ come in.

However, since its introduction, the term “Web Scraping” has been associated with a common misconception – the question of its legality. Even today, Web Crawling and Scraping are often regarded as terms that are related to hacking. But this may not be entirely true.  In this blog we will clarify the myths hovering around these terms and understand how to crawl and scrape the web ethically.

Web Crawling and Web Scraping

To put things into perspective, Web scraping, also referred to as web harvesting or web data extraction is the automated process of mining for data or collecting information from a specific web page in the world wide web.  It was  originally created for the purpose of making the World Wide Web easier to use.

Much like an endless library with its infinite supply of books that require a meticulous exploration of every bookshelf to get the information that you need, web crawling is a technique of scanning every website thoroughly, to make a comprehensive list of all the available information on the web. While this technique is focused on locating or discovering URLs or links across the web, ‘Web scraping’ involves the extraction of data from one or multiple websites.  It makes it possible to extract vast amounts of information from websites. A typical web data extraction project requires a combination of both crawling and scraping techniques.

Debunking 4 Common Myths Surrounding Web Scraping

Myth #1 – Web scraping is illegal

No, it is legal to extract information  from publicly available data, but one should take care not to cross the lines.’ Intruding or hacking into somebody’s personal data or intellectual property is illegal. While there is no worldwide law that outright bans web scraping, it doesn’t imply that one can scrape everything without consequences.

Myth #2 – Web scrapers operate in a Grey area of the Law

Definitely not! Data is the most powerful thing in the digital age of today and responsible companies understand this. They use the technique of ‘web scraping’  only from publicly available data for their respective businesses. The point is,  if you adhere to the ethical company practices then you are not operating in any grey area of the law.

Myth #3 – Web scraping is like Hacking

While ‘Hacking’ simply means doing something unauthorized or illegal,  ‘Web scraping’ is a technique of browsing the websites and capturing publicly available data like any normal user. It does not imply exploiting the website or information for malicious gains. So web scraping is not like hacking.

Myth #4 – Web Scrapers are Stealing Data

Let me explain this with an example. If a person’s public posts about buying new clothes allow the owner of a clothing store to observe his/her shopping patterns and recommend new collections from the store – does this behavior constitute data stealing? Similarly, web scrapers collect the publicly available data to find more insights from it. Also, any data which is in the public domain cannot be stolen.

Existing Regulations for Web Scraping

While there is no concrete, global law which mentions the do’s and don’ts of ‘web scraping,’ there are a few regulations under which one can be penalized for unauthorized web scraping. Here is the list:

1. Violation of the Digital Millennium Copyright Act (DMCA)
2. Violation of the Computer Fraud and Abuse Act (CFAA)
3. Breach of Contract
4. Copyright Infringement
5. Trespassing

A quick summary of the above regulations:

Do’s:

o Follow the terms and conditions of the website from which they are scraping data. (robots.txt)

o Only gather data that is required for business use (Use customized web scraper rather than generic)

o Be crystal clear about where the information is going to be used and are able to document it on a public forum.

Don’ts:

o Use it to perform exhaustive scraping as it will lead to your web scraper getting blocked

o Engage in scraping personal, critical or sensitive data

o Refrain from displaying scraped data publicly

3 Noteworthy Lawsuits related to Web Scraping:

• eBay vs Bidder’s Edge Case: During the 2000s, eBay filed a well-known lawsuit against EBidger, an online price comparison website for consumers, marking one of the earliest publicly known web scraping legal cases. The court order prevented Bidger’s Edge from scraping eBay content again. The main argument with which eBay won the case was that Bidger’s Edge was exhausting their system, and others following Bidger’s Edge’s technique were likely to cause more harm to eBay’s system.
• Facebook vs Power Ventures Case: In 2009, Facebook took legal action against Power Ventures for extracting content from its websites that had been uploaded by its users. This set the example for a case where web scraping was evaluated from an intellectual property standpoint. The court sided with Facebook and ordered a substantial fiscal penalty for Power Ventures.
• Linkedin vs hiQ Labs Case: This most recent major web scraping case started in 2019. HiQ Labs, a data analytics company, faced legal action from LinkedIn for scraping publicly available profiles to perform a professional skill analysis. The case underwent review in multiple courts, including the Supreme.

Precautions to take for Ethical Web Surfing

Now coming to the heart of our topic, let’s look closely at the precautions that are necessary while web scraping from any website. The purpose is to ensure that all the details are collected legally without getting blocked.

1. Verify and always follow ROBOTS.TXT

Robots.txt is used for robots exclusion protocol. It provides web scrapers or bots instructions on what part of the website they can access and which part they should not access. You can verify this by checking on robots.txt at http://website_name.com/robots.txt

2. Enable Proper User Agent

If the website you intend to scrape contains mostly unchanging data, you have the option to utilize its cached version. Extracting the data from Google’s cached copy of the website ensures you can evade any worries about detection or being blocked completely.

3. Reduce Scrapping Speed and Crawl during Off-Peak hours

Although it is considered that legal web scrapers act like humans accessing the website to access public data, there is a catch here. Web crawlers can move between the pages at a speed that humans cannot. This is where the defence mechanism catches the crawlers and bots, often blocking them. Also, there are greater chances of crawlers getting blocked during peak hours, when in comparison to human users, crawlers are able to affect higher server load. Once again, this results in bots getting blocked to prevent other users’ user interface from getting hampered.

4. Avoid Crawling Admin Pages

This specifically applies to Content Management System(CMS) websites who have predefined admin or login pages. Web Crawlers should take care and avoid crawling such webpages. Special checks are often added to monitor traffic on these pages. So, in case of any abnormal activity observed on these pages, it leads to quick detection and blocking of requests for that IP, preventing access to the entire website. Example: wp-config. wp-admin pages in WordPress.

5. Rotating IP Addresses

Websites that are concerned about their security always have mechanisms to block any IP who is observed to be making continuous requests. The best way to avoid getting blocked in situations like these is to keep rotating your IP address.

6. Scraping from Google Cache

If the website you want to scrape has relatively static data, you can opt to use a cached version of the site. By scraping the data from Google’s cached version of the website, you can avoid concerns about detection or being blocked entirely.Syntax: http://webcache.googleusercontent.com/search?q=cache:URL

7. Use a Referer

The Referer header is an HTTP request header used to inform a website about the site from which a user is coming. It is advisable to set this header in a way that it appears as if the user is arriving from Google. This can be achieved by including the following header: “Referer”: https://www.google.com/

Conclusion:

In conclusion, you can rest easy, knowing that Website Scraping or Crawling is not illegal. Embrace the power of gathering publicly available data without facing blocks or blacklisting. And, let’s always remember to tread responsibly, in adherence to the suggested guidelines, maintaining transparency about our business intentions, and ensuring that the data we gather is never misused. Only this way, we can harness the full potential of this tool while upholding ethical standards to unlock boundless opportunities for progress and innovation.

References:

1. https://research.aimultiple.com/web-scraping-ethics/
2. https://scrapingant.com/
3. https://www.scraperapi.com/
4. https://dev.to/digitallyrajat/

Author:

Pratik Raosaheb Kadam

Quick Heal Security Labs