Web Scraping Best Practices: How To Collect Data Without Getting Blocked

Web scraping best practices are the habits that let you collect public data reliably for years instead of days: crawling slowly, reading robots.txt, identifying your bot, caching what you already have and using official APIs when they exist. Most scrapers that get blocked are not blocked because of the technique. They are blocked because they behave in ways no site owner would accept from any visitor.

The quick answer: to collect data without getting blocked, make your scraper look and act like a considerate, identifiable client. Check for an API or feed first, follow robots.txt, limit your request rate and back off when the server signals strain, cache pages and only re-fetch what changed, and monitor your output so you notice problems early. Proxies help with geography and reliability, but they are not a way around a site that has asked you to stop.

This guide covers each of those practices in turn, with web scraping tips for planning, JavaScript-heavy pages, monitoring and legal checks. It stays at the level of responsible web scraping: there are no instructions here for defeating a particular site’s bot protection or solving CAPTCHAs, because that is where technical risk becomes legal risk. If you are just getting started, our beginner’s guide to web scraping explains the basics.

Why Sites Block Scrapers In The First Place

Understanding the site owner’s point of view explains almost every best practice that follows. Websites block automated traffic for a few concrete reasons:

• Load and cost. Thousands of uncached requests per minute can slow a site for real customers and raise hosting bills.
• Anonymity. Traffic with no contact details and a fake browser signature looks the same as credential stuffing or fraud.
• Unpredictability. Bursty traffic at odd intervals, hitting search pages and filters, is hard to plan capacity around.
• Commercial concerns. Some owners do not want competitors copying their content or catalog, and say so in their terms.

You can address the first three entirely through how you build your crawler. The fourth is a business and legal question, and the right answer is sometimes to license the data or leave the site alone.

Web Scraping Best Practices At A Glance

1. Look For An Official API, Feed Or Dataset First

The most reliable scraper is the one you never have to write. Before building anything, check whether the data is already offered in a structured form:

• A public or partner API in the site’s developer documentation.
• XML sitemaps, RSS or Atom feeds, and product feeds built for shopping engines.
• Open data portals, published datasets and bulk downloads.
• Commercial data providers or a web scraping API that already handles collection and compliance at scale.

APIs come with documented rate limits and terms, return clean data and rarely break when a page is redesigned. Even if an API covers only part of what you need, using it for that part reduces the load and the risk of the rest of the project.

2. Plan The Crawl Before You Write Code

Good scrapers are designed, not improvised. Write down a short plan before the first request:

1. Define the fields. List exactly which data points you need. Collecting only what you need is faster and keeps you away from personal data you do not want.
2. Map the URLs. Find the smallest set of pages that contains those fields. Category pages or sitemaps often beat crawling every link.
3. Set the frequency. Prices may need daily checks, while company descriptions may only need monthly ones.
4. Estimate the load. Multiply pages by frequency. If the number would be noticeable to the site, redesign the plan or ask for a feed.
5. Record the legal check. Note the terms of service, robots.txt rules and whether any personal data is involved.

3. Read And Respect robots.txt

The robots.txt file at the root of a domain tells automated clients which paths they may crawl. It was formalized by the IETF as RFC 9309 in 2022, and a few details from the standard are worth building into your code:

• Match the most specific user agent group that applies to your bot, and fall back to the * group.
• Do not cache robots.txt for more than 24 hours. Re-check it at least daily on long-running crawls.
• If robots.txt returns a 4xx status such as 404, the standard allows crawling. If it returns a 5xx server error, assume everything is disallowed until it recovers.
• Crawl-delay is not part of RFC 9309. Many sites still use it, and honoring it is a simple way to show good faith.

robots.txt is not an access control system, and it does not settle every legal question, but it is the clearest statement of what a site owner wants. Ignoring it is the fastest way to turn a tolerated crawler into a blocked one.

4. Rate Limiting, Backoff And Retries

Request rate is the practice that matters most for staying unblocked. A scraper that sends a few requests per second to a large site will usually go unnoticed. The same scraper running 200 concurrent connections against a small store can take it offline.

1. Cap concurrency per domain. Start with one or two simultaneous connections per site and only raise it if the site is clearly built for heavy traffic.
2. Add delays. Wait between requests, with a little random variation so you do not create a perfectly regular spike. Respect any Crawl-delay value.
3. Watch response times. If pages start taking noticeably longer to load, slow down automatically. Rising latency is the server telling you it is busy.
4. Treat 429 and 503 as stop signs. A 429 Too Many Requests or 503 Service Unavailable response means you are going too fast. Honor the Retry-After header when it is present.
5. Use exponential backoff. Double the wait after each failed attempt, cap the number of retries and log persistent failures for a human to review instead of hammering the same URL.
6. Schedule for off-peak hours. Run large crawls when the site’s audience is asleep, based on its main time zone.

If a site responds with a block page or 403 Forbidden after you have done all of this, take it as a decision, not a puzzle. Contact the owner, look for a licensed source, or remove the site from your crawl.

5. Identify Your Scraper Honestly

A transparent user agent string is one of the simplest ethical web scraping practices and one of the most effective. Include your organization or bot name and a URL or email address where a site owner can reach you, for example a short page explaining what your crawler collects and how to opt out.

Site administrators who can see who you are and why you are crawling often choose to rate-limit or contact you rather than block you outright. Disguising a bot as a consumer browser has the opposite effect: once it is detected, it looks deceptive, and deception is what turns a technical dispute into a legal one.

6. Cache Pages And Only Fetch What Changed

The cheapest request is the one you do not send. Store raw responses with a timestamp so you can re-parse them when your extraction logic changes, instead of crawling the site again. For pages you revisit, use HTTP conditional requests: send the ETag value in an If-None-Match header, or the last fetch time in If-Modified-Since, and a well-configured server will reply with a short 304 Not Modified when nothing has changed.

XML sitemaps often include a lastmod date for each URL. Reading the sitemap first and fetching only URLs that changed since your last run can reduce request volume dramatically on large catalogs.

7. Handle JavaScript-Heavy Pages Efficiently

Many modern sites render content in the browser with JavaScript, so a plain HTTP request returns an almost empty page. Headless browsers such as Playwright or Puppeteer load and run the page like a real browser, which solves the problem but costs far more CPU, memory and bandwidth per page.

Before reaching for a headless browser, open your browser’s developer tools and check the network tab. Many pages load their data from a JSON endpoint you can request directly, which is lighter for both you and the site. When you do need a browser, block images, fonts and ads you do not need, reuse browser contexts and keep the same rate limits you would use for plain requests. Our list of web scraping APIs for JavaScript-rendered sites covers managed options if you would rather not run browsers yourself.

8. Proxies: What They Are For And What They Are Not

A proxy routes your requests through another IP address. In data collection, proxies have legitimate jobs: seeing the prices, search results or ads that users in a specific country or city see, spreading a large but polite crawl across infrastructure so no single connection carries all the traffic, and keeping collection running when one network path fails. Rotating sessions, where requests are distributed across a pool of addresses, is a common general pattern for those goals.

What proxies should not be used for is ignoring a site that has told you to stop. If you are blocked after respecting robots.txt and rate limits, switching IP addresses to keep going turns a technical signal into a deliberate workaround, and recent US litigation over getting around access controls shows that courts take that seriously.

Where you get proxies also matters. Residential proxy networks route traffic through real consumer devices, so ask any provider how those devices joined the network and whether their owners gave informed consent. Reputable vendors publish their IP sourcing policies; avoid free proxy lists and any provider linked to malware or botnet-sourced addresses. Our roundup of proxy-backed data extraction services is a starting point for comparing managed options.

9. Monitoring, Data Quality And Maintenance

Scrapers fail quietly. A redesigned page can turn every price into an empty string without a single error in your logs. Treat your scraper as a data pipeline and monitor it like one:

• Validate every record against a schema: required fields present, numbers parse as numbers, dates fall in sensible ranges.
• Track volumes. Alert when a run returns far fewer records than usual, or when error and block rates rise.
• Review samples by hand on a schedule. Compare a few scraped records with the live page.
• Deduplicate and normalize currencies, units and product identifiers before data reaches analysts.
• Keep the raw pages for a short period so you can fix extraction bugs without re-crawling.
• Log decisions, including which sites were excluded and why, so the project is auditable.

For larger programs, it is often cheaper to buy this layer than to build it. Our guides to web scraping tools for data extraction and cloud-based scraping tools for ecommerce monitoring compare services that include scheduling, monitoring and data delivery.

10. Legal And Ethical Checks

Technical politeness does not make every project lawful. Before you scale up, run through a short legal review, and get proper advice for anything sensitive:

• Terms of service: have you created an account or otherwise agreed to terms that restrict automated access?
• Logins and barriers: collect only what is public and logged out, and do not get around CAPTCHAs, paywalls or bot detection.
• Personal data: in the EU and UK, data that identifies people needs a lawful basis under the GDPR, even when it is public. Leave it out if you can.
• Copyright: facts such as prices are generally free to use, but articles, images and reviews are protected. Reuse, republication and AI training each need their own review.
• Opt-outs: respect site and rightsholder opt-outs, including machine-readable text and data mining reservations in Europe.

This section is general information, not legal advice.

How To Avoid Getting Blocked Web Scraping: The Short Version

If you searched for how to avoid getting blocked web scraping, the honest answer is less exciting than most tutorials suggest. Sites block scrapers that are fast, anonymous and careless. Slow down, identify yourself, cache aggressively, fetch only what changed, use APIs where they exist and stop when a site says no. Those habits solve the large majority of blocking problems without any tricks, and they keep you on the right side of both the site owner and the law.

Frequently Asked Questions

How many requests per second is safe when scraping?

There is no universal number. A large retailer’s infrastructure can absorb far more than a small business site. Start low, around one request every few seconds per domain, honor any Crawl-delay, and adjust based on response times and error rates. If latency rises or you see 429 or 503 responses, slow down.

Do I have to follow robots.txt?

In most countries robots.txt is not legally binding on its own, but it is the clearest expression of what a site owner wants, and courts and regulators increasingly treat it as relevant. Following it is a core part of responsible web scraping and one of the best ways to avoid being blocked.

Should I rotate user agents to avoid detection?

No. Rotating fake browser user agents to disguise a bot works against you. A single, honest user agent that names your crawler and gives contact details is the better long-term approach, because it gives site owners a reason to talk to you instead of blocking you.

When do I need a headless browser?

Only when the data you need is rendered by JavaScript and is not available from an underlying JSON endpoint, feed or API. Headless browsers are heavier for you and the site, so check the network requests first and keep the same rate limits when you do use one.

Is it better to build a scraper or use a scraping API?

Building gives you full control and suits small, stable projects. A managed scraping API or data provider makes sense when you need many sites, JavaScript rendering, geographic coverage or monitoring at scale. In both cases, the same best practices on rate, respect for robots.txt and data scope still apply.

Conclusion

Web Scraping Best Practices: How To Collect Data Without Getting Blocked comes down to a simple idea: behave like a visitor the site owner would be happy to have. Use official APIs and feeds where they exist, read robots.txt, limit your rate and back off when asked, identify your bot, cache and fetch only what changed, render JavaScript only when you must, and monitor your data as carefully as you collect it. Add a legal check before you scale, and treat a block as a message rather than a challenge. Follow these web scraping best practices and your data pipeline will be more reliable, cheaper to run and far easier to defend.

Related Reading

• Is Web Scraping Legal In 2026? What The Courts Have Actually Said
• Top Rotating Proxy Providers For Web Scraping In 2026
• Web Scraping With Python: A Step-By-Step Beginner’s Guide
• Web Scraping & Proxies hub: every list, guide and comparison in one place