How to Make Money From Web Scraping? The Top 20 Tips

What etiquette should web scrapers follow? – Web scraping code of conduct:

Scraping articles/data that is otherwise not publicly available and re-publishing it for a for-profit company is generally a no-no. There are a lot of grey areas here, and there’s usually a paragraph on scraping policy in the Terms of Use on a website.

Scraping for your own personal use: no-one cares. Just make sure to throttle the process so you don’t hammer a website to the point it becomes a DDoS attack.

Scraping is legal. https://techcrunch.com/2022/04/18/web-scraping-legal-court/

I’m not sure if there is any real law against scraping, but there are licensing issues regarding data published. If someone is paying for a data provider, and you scrape that data, that may not be legal for you to collect and redistribute.

Web Scraping with Python: from Fundamentals to Practice

Access it here

How do deal with https-domains with SSL certificates in BeautifulSoup? And please don’t say use verify = False:

BeautifulSoup is a library for pulling data out of HTML and XML. You have to make a request using another library(e.g. requests) to get HTML content of the page and pass it to BeautifulSoup for extracting useful information.

I haven’t faced with any problems during scraping HTTPs sites using requests lib.

For anyone who goes with requests as your HTTP client, I would highly recommend adding requests-cache for a nice performance boost.

Why does Python not separate data into columns when exporting web scraping results to .csv?

Make sure to set the separator to , (I think the default is ;?).

Also, you should use BeautifulSoup(page.text) instead of BeautifulSoup(page.content). If you give it bytes rather than text, BeautifulSoup has to guess the text encoding, which is slow and can produce incorrect results.

And at the end, remember to call soup.decompose() to let python free up the memory.

How do I turn web scraping into a business?

Start by identifying the problem your service can solve. Eg, e-commerce companies wanting real time data on retail trends in their space, or financial firms wanting data on hiring trends gleaned from jobs postings, etc. If you can show how your tool addresses that problem better or cheaper than the current solution, and thus creates value and $ for your audience, you’ve got a business.

Is it possible to do web scraping without using any third-party modules?

Uh, of course you can. Here I wrote this just for you. I tried to make it slightly realistic so I gave it some error handling, a stopping point, absolute URL handling, and multithreading.

I think the first barrier you’ll run into with this is Python’s native HTML parser is very strict about what valid HTML is so it won’t interpret things the same way your web browser will. For that, I suggest using lxml as a parser (but that is a third-party module).

from collections import deque
from html.parser import HTMLParser
from threading import Lock
from urllib.error import HTTPError
from urllib.parse import urljoin
from urllib.request import urlopen
from concurrent.futures import ThreadPoolExecutor

NUMBER_OF_THREADS = 10
MAX_DEPTH = 3
TARGET_URL = r"https://www.reddit.com/r/Python/comments/v89fm9/is_it_possible_to_do_web_scraping_without_using/"


class MyHTMLParser(HTMLParser):
    def __init__(self, url=None):
        super().__init__()
        self.links = []
        self.url = url

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            if "href" not in dict(attrs):
                return

            href = dict(attrs)["href"]

            # Convert relative links to absolute links
            if self.url:
                href = urljoin(self.url, href)

            self.links.append(href)


def get_html(url):
    """
    Get the content of a URL.
    """
    try:
        return urlopen(url).read().decode("utf-8")
    except HTTPError as e:
        return e.read().decode("utf-8")



def parse_html(html, url=None):
    """
    Parse the HTML of a web page.
    """
    parser = MyHTMLParser(url)
    parser.feed(html)
    return parser


def handle(url, depth, callback, lock):
    """
    Handle a web page.
    """
    html = get_html(url)
    links = parse_html(html, url).links

    # Lock when printing to the terminal to avoid two threads printing at the same time
    with lock:
        print(depth, url)

    for link in links:
        # Lock when adding to the queue to avoid two threads adding to the queue at the same time
        with lock:
            callback((depth + 1, link))


def crawl(url, max_depth):
    """
    Crawl a web page.
    """
    seen = set()
    crawling = deque([(0, url)])
    lock = Lock()
    with ThreadPoolExecutor(max_workers=NUMBER_OF_THREADS) as executor:
        tasks = []
        while crawling:
            depth, url = crawling.popleft()

            # If the depth is equal to the maximum depth, skip the URL (remember depth starts at 0)
            if depth == max_depth:
                continue

            # If the URL has already been seen, skip it
            if url in seen:
                continue
            seen.add(url)

            # Submit the task and add the task to the list of tasks
            tasks.append(executor.submit(handle, url, depth, crawling.append, lock))

            # If the queue is empty and we still have tasks, wait for them one by one until we have something to do
            while tasks and not crawling:
                tasks.pop().result()


if __name__ == "__main__":
    crawl(TARGET_URL, max_depth=MAX_DEPTH)