Hubi@feddit.de to Reddit@lemmy.worldEnglish · 9 months agoReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comexternal-linkmessage-square186fedilinkarrow-up1856arrow-down114
arrow-up1842arrow-down1external-linkReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comHubi@feddit.de to Reddit@lemmy.worldEnglish · 9 months agomessage-square186fedilink
minus-squareLaterRedditor@lemmy.worldlinkfedilinkarrow-up3arrow-down2·9 months agoWasn’t there robots.txt at least?
minus-squareSacrificedBeans@lemmy.worldlinkfedilinkarrow-up12·9 months agoRobots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Wasn’t there robots.txt at least?
There was, and still is
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.