If robots.txt said to jump off a bridge, would your spider do it? | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

elwell on May 8, 2014 | parent | context | favorite | on: The Scraping Problem and Ethics

If robots.txt said to jump off a bridge, would your spider do it?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact