I guess it depends too much on the nature of the crawler. Does it actually extract links from robots.txt or is it merely ignoring them? If the crawlers are distributed, do page hits come from the same IP that the robots.txt was hit from?

It gets harder and harder to get away from CDNs and captchas, which are not exactly good things from an open source POV for the most part.