minus-squaremidribbon_action@lemmy.blahaj.zonetoTechnology@lemmy.ml•Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countrieslinkfedilinkarrow-up8·6 days agoYeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it. linkfedilink
Yeah, I’ve been curious if you could explicitly block a page in robots.txt, hide an invisible link to the same page in your footer, then kinda have it act like an immediate IP block when someone requests it.