Luu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 10 hours agoTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comexternal-linkmessage-square60fedilinkarrow-up1299arrow-down13cross-posted to: hackernews@lemmy.bestiver.se
arrow-up1296arrow-down1external-linkTikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAIfortune.comLuu Tuyen@lemmy.world to Technology@lemmy.worldEnglish · 10 hours agomessage-square60fedilinkcross-posted to: hackernews@lemmy.bestiver.se
minus-squarezod000@lemmy.mllinkfedilinkEnglisharrow-up14·4 hours agoBullshit. This bot doesn’t identify itself as a bot and doesn’t rate limit itself to anything that would be an appropriate amount. We were seeing more traffic from this thing that all other crawlers combined.
minus-squarejagged_circlelinkfedilinkEnglisharrow-up2·56 minutes agoNot rate limiting is bad. Hate them because of that, not because they’re a not. Some bots are nice
Bullshit. This bot doesn’t identify itself as a bot and doesn’t rate limit itself to anything that would be an appropriate amount. We were seeing more traffic from this thing that all other crawlers combined.
Not rate limiting is bad. Hate them because of that, not because they’re a not.
Some bots are nice