Web Scraping With Python: A Step-By-Step Beginner’s Guide

Web scraping with Python is the most common way developers, analysts and marketers turn public web pages into structured data they can sort, chart and feed into other tools. Python has mature, well-documented libraries for every stage of the job, from downloading a page to crawling thousands of them, and the syntax stays readable even when the project grows. This tutorial takes you from an empty folder to a working scraper, one step at a time.

Quick answer: to scrape a website with Python, confirm the site allows it and has no official API, install requests and Beautiful Soup, download the page with a clear User-Agent, a timeout and a delay, select the data with CSS selectors, follow the “next” links for pagination and write the results to CSV. Use Playwright when the content is rendered by JavaScript, and move to Scrapy when you need to crawl at scale.

Every example below runs against Books to Scrape and its sister site Quotes to Scrape, two sandboxes built specifically for practicing scrapers. That keeps this Python web scraping tutorial legitimate and repeatable: the pages never change under you, and nobody’s server is harmed while you learn. If you want the concepts before the code, our beginner’s guide to web scraping covers how scrapers work in plain language.

What You Need Before You Start

You need Python 3.10 or newer, because current releases of Requests, Playwright and Scrapy all require it. You also need a terminal, a code editor and a basic grasp of Python lists, dictionaries and loops. Some familiarity with HTML helps: if you know that a page is a tree of tags with classes and attributes, you know enough to start.

Here is the toolkit this guide uses and what each piece is for:

Library	What it does	When to use it
Requests	Sends HTTP requests and returns the raw HTML	Static pages, simple scripts
HTTPX	HTTP client with a Requests-style API plus async support	When you want async requests or HTTP/2
Beautiful Soup	Parses HTML into a searchable tree	Extracting data from downloaded HTML
lxml	Fast HTML and XML parser that Beautiful Soup can use	Speeding up parsing
Playwright	Drives a real browser (Chromium, Firefox, WebKit)	Pages that render content with JavaScript
Scrapy	Full crawling framework with scheduling, retries and exports	Crawling many pages or whole sites

How To Scrape A Website With Python, Step By Step

The nine steps below build one scraper that collects every book title, price, rating and link from the practice catalog, then extend it to JavaScript pages and larger crawls. Work through them in order: each step reuses code from the one before.

Step 1: Check The Terms, robots.txt And Whether An API Exists

Before writing any code, check three things. First, read the site’s terms of service for rules about automated access. Second, look at its robots.txt file (the site root plus /robots.txt), which tells crawlers which paths the owner wants left alone. Third, look for an official API or a data export. An API gives you clean, structured data with documented limits, and it is almost always the better choice when one exists.

Python’s standard library can read robots.txt for you, so you can build the check into your scraper instead of doing it by eye:

from urllib.robotparser import RobotFileParser

robots = RobotFileParser("https://books.toscrape.com/robots.txt")
robots.read()
print(robots.can_fetch("books-tutorial-scraper", "https://books.toscrape.com/catalogue/page-2.html"))

Books to Scrape has no robots.txt at all (the request returns a 404), and in that case the parser treats every path as allowed, so this prints True. On a real site, respect a False. Keep in mind that robots.txt is a convention, not a permission slip: it does not override a site’s terms, copyright or privacy law. Stick to public data, avoid collecting personal information, and get legal advice for anything commercial or sensitive.

Step 2: Install Requests, Beautiful Soup And lxml

Create a project folder, set up a virtual environment so these packages stay isolated from the rest of your system, and install the three libraries:

python -m venv .venv
source .venv/bin/activate        # on Windows: .venv\Scripts\activate
pip install requests beautifulsoup4 lxml

Requests handles the HTTP side. Beautiful Soup (installed as beautifulsoup4, imported as bs4) turns HTML into a tree you can search. lxml is an optional parser that Beautiful Soup can use for speed; Python’s built-in html.parser also works if you would rather install less. If you prefer HTTPX, pip install httpx gives you a nearly identical httpx.get() call, with one difference worth knowing: HTTPX does not follow redirects unless you pass follow_redirects=True.

Step 3: Fetch A Page Politely

A polite request identifies itself, gives up if the server is slow and does not hammer the site. In practice that means three habits: send a descriptive User-Agent header (ideally with a way to contact you), always set a timeout, and pause between requests.

import requests

HEADERS = {"User-Agent": "books-tutorial-scraper/1.0 (you@example.com)"}
url = "https://books.toscrape.com/"

response = requests.get(url, headers=HEADERS, timeout=10)
response.raise_for_status()          # raises an error for 4xx and 5xx responses
print(response.status_code)          # 200
print(response.content[:200])        # the first bytes of raw HTML

raise_for_status() turns error responses into exceptions so a 404 or 503 never slips through as if it were real data. The timeout=10 matters more than it looks: without a timeout, the Requests documentation notes your code can hang indefinitely if the server never answers.

Step 4: Parse The HTML With CSS Selectors

This is the heart of any BeautifulSoup tutorial. Open the page in your browser, right-click a book and choose Inspect. You will see that each book sits inside an <article class="product_pod"> element, the full title lives in the title attribute of the link inside the h3, the price is in p.price_color, and the star rating is stored as a second class on p.star-rating (for example star-rating Three).

Beautiful Soup’s select() and select_one() methods accept the same CSS selectors you would write in a stylesheet, which makes those observations easy to translate into code:

from urllib.parse import urljoin
from bs4 import BeautifulSoup

soup = BeautifulSoup(response.content, "lxml")

books = []
for card in soup.select("article.product_pod"):
    link = card.select_one("h3 a")
    books.append({
        "title": link["title"],
        "price": card.select_one("p.price_color").get_text(strip=True),
        "rating": card.select_one("p.star-rating")["class"][1],
        "url": urljoin(url, link["href"]),
    })

print(len(books), books[0])

Two details are worth noticing. The code passes response.content (raw bytes) rather than response.text, which lets Beautiful Soup work out the character encoding itself; this practice site does not declare a charset in its HTTP headers, so response.text would mangle the pound sign in prices. And the links on the page are relative, so urljoin() converts them into full URLs you can visit later. The Beautiful Soup documentation covers every search method if you need more than CSS selectors, such as find_all() with attribute filters.

Step 5: Handle Pagination

The catalog spreads 1,000 books over 50 pages, 20 per page. Each page has a “next” button inside li.next, so the cleanest approach is to keep following that link until it disappears. That works even when you do not know how many pages there are, which is the usual situation on real sites.

import time

session = requests.Session()
session.headers.update(HEADERS)

url = "https://books.toscrape.com/"
all_books = []

while url:
    response = session.get(url, timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "lxml")

    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        all_books.append({
            "title": link["title"],
            "price": card.select_one("p.price_color").get_text(strip=True),
            "rating": card.select_one("p.star-rating")["class"][1],
            "url": urljoin(url, link["href"]),
        })

    next_link = soup.select_one("li.next a")
    url = urljoin(url, next_link["href"]) if next_link else None
    time.sleep(1)                     # one request per second is plenty

print(f"Collected {len(all_books)} books")

Using a requests.Session reuses the same connection across requests, which is faster and lighter on the server. The time.sleep(1) keeps you to roughly one request per second. On a production site, slow down further if responses get slower, and stop if you start seeing errors.

Step 6: Save The Results To CSV

Python’s built-in csv module writes a list of dictionaries straight to a file that opens in Excel or Google Sheets:

import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price", "rating", "url"])
    writer.writeheader()
    writer.writerows(all_books)

Pass newline="" so Windows does not insert blank rows, and encoding="utf-8" so special characters survive. For anything bigger than a spreadsheet, the same list of dictionaries loads into pandas with one line, or into SQLite if you want to query it.

Step 7: Scrape JavaScript Pages With Playwright

Some sites send a nearly empty HTML shell and build the content in your browser with JavaScript. Requests only sees that shell. You can confirm this on quotes.toscrape.com/js/: the raw HTML contains the quotes as a JavaScript array, but none of the div.quote elements you see on screen. To scrape pages like this, you need a real browser that runs the scripts. Playwright drives Chromium, Firefox or WebKit from Python.

pip install playwright
playwright install chromium

The second command downloads the browser build that matches your Playwright version. Then:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()     # headless by default
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")

    quotes = []
    for quote in page.locator("div.quote").all():
        quotes.append({
            "text": quote.locator("span.text").inner_text(),
            "author": quote.locator("small.author").inner_text(),
        })

    browser.close()

print(quotes[0])

wait_for_selector() pauses until the quotes have rendered, and locators query elements inside the live page. The Playwright for Python documentation also offers an async API if you want to drive several pages at once. Browsers are far heavier than HTTP requests, so before reaching for one, open your browser’s developer tools, check the Network tab and see whether the page loads its data from a JSON endpoint you could request directly. For production work on heavily scripted sites, a managed option from our list of web scraping APIs for JavaScript-rendered sites can take the browser infrastructure off your hands.

Step 8: Scale Up With Scrapy

Requests and Beautiful Soup are perfect for one-off scripts. Once you are crawling thousands of pages, you start rebuilding things a framework already has: a request queue, concurrency, retries, duplicate filtering, throttling and export formats. Scrapy is that framework. Here is the same book scraper as a Scrapy spider:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "DOWNLOAD_DELAY": 1,
        "USER_AGENT": "books-tutorial-scraper/1.0 (you@example.com)",
    }

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
                "url": response.urljoin(card.css("h3 a::attr(href)").get()),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Save it as books_spider.py and run it without creating a full project:

pip install scrapy
scrapy runspider books_spider.py -O books.json

The -O flag writes (and overwrites) the output file, and Scrapy picks the format from the extension, so books.csv works too. The settings in the spider matter: Scrapy’s built-in default for ROBOTSTXT_OBEY is False, although new projects created with scrapy startproject switch it on. AutoThrottle adjusts the crawl speed based on how quickly the server responds. The official Scrapy tutorial is the best next read once this spider makes sense.

Step 9: Add Error Handling And Retries

Real-world scrapers fail in predictable ways: timeouts, temporary 5xx errors, 429 “too many requests” responses and pages where an element is missing. Retrying temporary failures with growing waits, and giving up cleanly on everything else, covers most of it. Requests can do this through urllib3’s Retry class:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=3,
    backoff_factor=1,                              # waits grow between attempts
    status_forcelist=[429, 500, 502, 503, 504],    # retry only temporary failures
)
session = requests.Session()
session.headers.update(HEADERS)
session.mount("https://", HTTPAdapter(max_retries=retry))

url = "https://books.toscrape.com/"
try:
    response = session.get(url, timeout=10)
    response.raise_for_status()
except requests.RequestException as exc:
    print(f"Giving up on {url}: {exc}")

A 429 is the server asking you to slow down, so treat it as a signal to reduce your request rate, not as an obstacle to get around. urllib3 honors the server’s Retry-After header by default when it retries. Inside the parsing code, guard against missing elements so one odd page does not crash a long run:

def text_or_none(parent, selector):
    element = parent.select_one(selector)
    return element.get_text(strip=True) if element else None

Log every failure with its URL so you can re-run just those pages later, and save progress as you go rather than holding everything in memory until the end.

Python Web Scraping Best Practices

The difference between a scraper that runs for years and one that gets blocked in a day is mostly etiquette. These habits keep your Python web scraping projects reliable and respectful:

• Prefer official APIs and data exports whenever they exist. They are more stable than HTML and come with clear terms.
• Respect robots.txt and rate limits. Crawl slowly, run large jobs off-peak and back off when the server slows down.
• Identify yourself with an honest User-Agent and contact details so site owners can reach you instead of blocking you.
• Cache what you download during development so you are not re-requesting the same pages every time you tweak a selector.
• Collect only what you need. Avoid personal data, check copyright before republishing anything, and review a site’s terms before using its content to train AI models.
• Expect change. Sites redesign without notice, so keep selectors in one place and add simple checks that alert you when a page suddenly returns no results.

At commercial scale, teams often hand the request layer to a managed service that handles rendering, retries and IP rotation within each provider’s acceptable-use rules. Our comparison of the best web scraping APIs in 2026 covers the main options, and if you would rather hire the whole project out, our list of top Python development companies is a good place to start.

Frequently Asked Questions

Is web scraping with Python legal?

Scraping publicly available data is generally lawful in many jurisdictions, but the answer depends on what you collect, how you access it and what you do with it. Logging in to scrape, ignoring terms you agreed to, collecting personal data or republishing copyrighted content all raise the risk. For commercial projects, check the site’s terms and take legal advice.

Should I use Beautiful Soup or Scrapy?

Start with Requests and Beautiful Soup for small, one-off jobs and for learning, because there is less to set up. Switch to Scrapy when you need to crawl many pages, run jobs on a schedule or want built-in retries, throttling and exports. Many developers use both: Beautiful Soup for quick scripts and Scrapy for anything that has to run reliably.

Why does my scraper return empty results?

The two usual causes are a wrong selector and JavaScript rendering. Print part of the HTML your code actually received and search it for the text you expect. If the text is missing, the page builds it with JavaScript, so look for the underlying JSON request or switch to Playwright. If the text is there, adjust your selector.

Do I need proxies for web scraping?

Not for learning, and not for small, polite jobs. Proxies come into play for legitimate commercial work such as price intelligence, ad verification or checking how search results differ by country. If you get there, choose providers that publish how they source their IPs and use them within each target site’s terms.

How long does it take to learn Python web scraping?

If you already know basic Python, you can write a working scraper like the one in this guide in an afternoon. Getting comfortable with pagination, error handling, JavaScript pages and Scrapy usually takes a few weeks of small projects.

Conclusion

Web scraping with Python comes down to a repeatable loop: check that you are allowed, fetch politely, parse with CSS selectors, follow pagination, save the results and handle errors. Requests and Beautiful Soup cover most beginner projects, Playwright handles pages built with JavaScript, and Scrapy takes over when a script turns into a crawler. Treat this as web scraping with Python: a step-by-step beginner’s guide you can come back to, practice on the sandbox sites until each step feels routine, and then apply the same habits to real projects.

Related Reading

• Scrapy vs BeautifulSoup: Which Python Scraping Tool Should You Use?
• Top Open-Source Web Scraping Tools In 2026
• Web Scraping Best Practices: How To Collect Data Without Getting Blocked
• Web Scraping & Proxies hub: every list, guide and comparison in one place