Tech article

Scraped 300 pages successfully. Site updated robots.txt at page 187 and blocked me.

Site admin updated robots.txt while my scraper was running. 187 pages succeeded, then everything got blocked.

Dev.to | Apr 3, 2026 | Nico Reyes

Read the original article