Grumpy Old Geeks Topics Web crawler restrictions Topic Web crawler restrictions 2 mentions in 1 episode The hosts discuss allegations that a crawler bypassed robots.txt rules and web application firewalls. Mentions 708: Spicy Mode Aug 8, 2025 9:44 disguising their identities to get around robots.txt files 10:27 rotate through IP addresses not listed in Perplexity's official IP range