You've written the scraper. It pulls a few thousand product pages, or every dentist in Leeds from a directory, or a pile of docs for your RAG bot. Then the question hits: could this get me sued?
Usually, no. Scraping public pages without logging in is legal in the US and most of Europe. But "is scraping legal" is really four separate questions, and each one has its own trap. One of them turns on a single click you probably made months ago and forgot about. We'll get to it.
Court opinions, statutes and regulator pages checked on 10 November 2026. This is general information, not legal advice. For a real project with real money behind it, talk to a lawyer in your country.
The short answer: scraping is legal, but what you take and how you take it can make it illegal
No US or EU law bans automated collection of web pages as such. Search engines do it billions of times a day. What courts and regulators look at is narrower:
- Access. Did you get past a password, a paywall or a block meant for you?
- Contract. Did you agree to terms that forbid scraping, usually by creating an account?
- Personal data. Are you collecting information about identifiable people?
- Copyright and databases. Are you copying creative work, or a protected database, and republishing it?
Public product prices scraped while logged out clear all four easily. Scraping people's profiles from inside an account can fail three of them at once. Most real projects sit somewhere between, so let's take the questions one at a time.
Can scraping a public website be a crime in the US?
The law people worry about is the Computer Fraud and Abuse Act (CFAA). It punishes anyone who accesses a computer "without authorization" or "exceeds authorized access", with penalties that reach 10 years in prison. For years, companies used it to threaten scrapers with criminal liability.
Two decisions shrank that threat. The first was Van Buren v. United States in June 2021. The Supreme Court read the statute as a "gates-up-or-down inquiry". Either you can enter a system or part of it, or you can't.
Using access you already have for a purpose someone dislikes isn't the crime. The Court also said civil claims under the CFAA need technological harm, such as corrupted files, not just unwanted copying.
Then in April 2022 the Ninth Circuit applied that to scraping in hiQ Labs v. LinkedIn. hiQ scraped public LinkedIn profiles. LinkedIn sent a cease-and-desist letter and said further scraping was "without authorization". The court disagreed: a public page has "erected no gates to lift or lower in the first place."
So is scraping public pages CFAA-proof? Not quite. The same opinion says areas that need a login are still protected, and dodging an IP block or using someone else's password puts you back on the wrong side of the gate. The practical line is simple: if a browser with no account can see it, the CFAA is a weak weapon against you.
But hiQ didn't win in the end. That's the second question.
Does breaking a site's terms make scraping illegal?
Breaking terms of service isn't a crime. It can be a breach of contract, and contract claims are where scrapers actually lose money.
Here's the click we promised. When you create an account, you accept the site's terms, and most big platforms forbid scraping in them. That turns "I copied public data" into "I broke an agreement I signed."
hiQ learned this the hard way. After the appeal, the district court found in November 2022 that hiQ had breached LinkedIn's User Agreement, partly through contractors who made fake accounts. In December 2022 hiQ accepted a $500,000 judgment and a permanent injunction that bars it from scraping LinkedIn, logged in or not, and requires it to delete the data.
Compare that with Meta v. Bright Data. In January 2024 a federal judge in California ruled that Facebook's and Instagram's terms "do not bar logged-off scraping of public data." Bright Data scraped without logging in, so the terms didn't reach it, and Meta's contract claims failed.
Put those two cases side by side and the pattern is clear:
| How you collect | Contract risk | Why |
|---|---|---|
| Logged out, public pages | Low | Terms you never accepted are hard to enforce (Meta v. Bright Data) |
| Logged in with your own account | High | You clicked "I agree" to terms that usually ban scraping (hiQ, 2022) |
| Fake or bought accounts | Highest | Breach of contract plus deception; hiQ lost on exactly this point |
A footer link to the terms that nobody has to click (a "browsewrap") is weaker than a checkbox at sign-up (a "clickwrap"). Still, don't plan a business around that distinction. If the data you need sits behind a login, assume the terms apply to you.
Does robots.txt have legal force?
Not on its own. The robots.txt file is a request, and the standard that defines it says so in plain words.
That doesn't make it safe to ignore. Courts look at conduct. Picture a scraper that ignored robots.txt, rotated IPs to dodge blocks and kept going after a cease-and-desist letter. Now picture one that behaved like a polite search crawler. A judge sees two very different defendants. The Ninth Circuit noted that site owners can still bring claims such as trespass to chattels when scraping causes real harm to their servers.
In the EU, machine-readable opt-outs matter more.
The text-and-data-mining exception in the Digital Single Market Directive applies only where rightholders haven't reserved their rights, and Recital 18 says that for online content the reservation should use "machine-readable means, including metadata and terms and conditions." The directive doesn't name robots.txt, but it's the obvious place a site will put that signal. Respecting it costs you a few pages. Ignoring it can cost you the exception.
Personal data is where scrapers actually get fined
This is the part most "scraping is legal" articles skip, and it's the one with the biggest numbers attached.
In Europe, the GDPR (General Data Protection Regulation) applies to personal data whether it's public or not. A name and job title on a public profile is still personal data. To scrape it you need a legal basis, usually "legitimate interests", and you have to tell people you hold their data under Article 14, even though you never spoke to them. The UK applies the same rules through the UK GDPR.
Clearview AI is the extreme case. It scraped more than 30 billion public photos and turned faces into biometric codes.
The Dutch Data Protection Authority fined it €30.5 million. Its reasons: no legal basis, illegal processing of biometric data, failing to inform people under Article 14 and ignoring access requests. Clearview says it isn't subject to the GDPR at all. The regulator also warned that Dutch companies using Clearview's service could be fined too.
You're probably thinking your lead list is nothing like a face database. True, and the risk scales with what you collect. Company names, prices and product specs carry almost none. Work emails and names of employees carry some. Health, religion, politics, biometrics and anything about children are "special category" data in the GDPR, and scraping them is a different order of risk.
In the US there's no single federal privacy law, but state laws are filling the gap. California's CCPA (California Consumer Privacy Act) gives residents rights over data businesses hold about them, and it explicitly covers data brokers.
If you build a list of Californians from scraped profiles and sell or share it, those rights to know, delete and opt out apply to you. If you're building outreach lists, our guide to finding business email addresses covers the verification side; the legal side starts here.
Copyright and databases: facts are free, the way they're written isn't
Copyright protects expression, not facts. A price, a phone number or an opening time isn't anyone's creative work. The DSM Directive says it directly in Recital 9: mining "mere facts or data that are not protected by copyright" needs no authorisation.
Article text, photos, reviews and product descriptions are different. Scraping them to analyse is one thing; republishing them is copying. The Ninth Circuit's hiQ opinion cites a case where a news monitoring service that scraped and republished articles was not protected by fair use.
Europe adds a second layer: the sui generis database right, which protects a database where someone made a substantial investment in collecting the contents. Extracting a substantial part of one, even if every single fact in it is unprotected, can infringe. Recital 8 of the DSM Directive names both copyright and this database right as things text and data mining can trip over.
So what about scraping for AI training? In the EU, Article 4 of the DSM Directive allows text and data mining by anyone, commercial or not, unless the rightholder has opted out in a machine-readable way. Article 3 gives research organisations a wider exception they can't be opted out of.
In the US, whether training on scraped work counts as fair use is still being fought in court. If you're building a web scraping pipeline for AI, honour opt-outs and keep the raw copies internal. We covered the output side of this in whether you can use AI images commercially.
A quick check before you run a scraper
None of this needs a law degree to apply. Run through these before you schedule anything:
- Can you see the data in a private browser window with no account? If not, stop and rethink.
- Did you, or anyone on your team, accept the site's terms with an account you'll scrape from? Read the scraping clause.
- Does robots.txt disallow the paths you want? Respect it, especially for EU content.
- Is any of it about people? Collect only the fields you need, write down your legal basis, and plan how you'll tell people and delete on request.
- Will you republish text or images, or only extract facts? Facts are safe; copies aren't.
- Are you rate-limiting? A scraper that slows a site down hands the owner a harm-based claim.
Tools don't change the answer. Apify and Firecrawl handle rendering and blocking, and proxies hide your IP, but the legal responsibility for what you collect stays with you. Our proxy guide is about getting pages to load, not about permission.
So, should you scrape it?
If it's public, logged out, factual and not about people (prices, specs, listings, docs), go ahead and be polite about rate limits and robots.txt.
If it's public but about people, as in a Google Maps lead list, you can usually do it, but treat GDPR and CCPA duties as part of the build: minimal fields, a written legal basis and a way to delete.
If it's behind a login, or you'd need fake accounts or a way around a block, don't. That's the exact setup that cost hiQ $500,000 and its business.
And if a cease-and-desist letter arrives, stop and call a lawyer before you reply.
