A Comprehensive Legal Guide to Web Scraping in the US

//

Date of Last Update: December 8, 2022

TL;DR

This is a long article about a complicated issue involving more than 20 different laws. But you can read the bolded sections in just a few minutes, and you’ll get the gist of it.

When I first published this guide, in the summer of 2020, I wrote that the recent trend had been toward greater permissiveness with web scraping. 

That sentence isn’t as true today as the day when I first wrote it. Recent policy trends have wavered. The few jurisdictions where courts began making broad policy statements in favor of permissiveness toward web scraping in 2017 and 2019 have backtracked. Savvy plaintiffs’ lawyers have started to eschew the more sensitive and uncertain legal battleground of the CFAA and progress toward predictable state-law claims such as breach of contract, where their track record of success—when plaintiffs’ attorneys effectively lay the foundation for litigation—is essentially unbroken.

Some like to describe web scraping as a “gray area of the law.” I don’t think that’s true. The laws around web scraping are as black and white as with any other legal domain. It’s just that few people know how to apply these laws, and that there’s a total disconnect between the law related to web scraping and social norms for how it is enforced.

The goal with this article will be to provide a lay introduction to web scraping legal issues in a way that is accessible to lawyers and non-lawyers alike.

  1. Introduction

There are a few websites online that purport to answer the question of “whether web scraping is legal.” And way too many of those websites, with unwavering confence and a complete absence of caution, provide clear and concise answers to that question that are laughably and dangerously false.

One such website claims to have a “three-part test” to determine whether web scraping is legal.

But a web scraper could follow that test and still violate a dozen state and federal laws. The blog post would be certifiable legal malpractice—except it wasn’t written by a lawyer in the first place.

With the increasing importance of data collection from privately owned websites, web scraping has grown from a niche enterprise to a bona fide industry in the last decade.

As a lawyer (and a python programmer) who has worked with dozens of clients in the web-scraping industry, from Y Combinator startups to Fortune 50 companies, I figured the internet was long overdue for a practitioner’s guide to web-scraping law that was grounded in the law. With that, I took the time to read every web-scraping case and scholarly journal on the subject published in the last ten years. I constantly monitor new and developing case law on web-scraping legal issues, and I review every new case published that touches on web scraping. I also have litigated this issue on behalf of both companies that engage in web scraping, and on behalf of companies looking to stop web scraping.

What you see here is the end-product of that research and experience.

To get a complete copy of this article, please contact us through the contact form below.

MGL Website Contact Form

Form on new MGL site
  • Privacy Policy
  • This field is for validation purposes and should be left unchanged.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.