Topic

AI web scraping

4 mentions in 2 episodes

Also known as AI website scraping

The hosts discuss AI crawlers accessing websites despite robots.txt rules and tools intended to block them.

Mentions

Jul 6, 2024

0:58
scraping websites without consent and ignoring the robots.txt file
11:57
a new free tool that prevents AI companies' bots from scraping its clients' websites

Jun 29, 2024

20:33
AI companies are reportedly still scraping websites despite protocols meant to block them.
21:29
AI agents from multiple sources ... are opting to bypass the robots.txt protocol