Bandwidth usage almost doubled

[Arantor]

If you create a file called robots.txt with that content, and put it in the same folder as your top level stuff, it will block any well behaved robots.

I say 'top level stuff' because I don't know where your .htaccess file is (since you can have one in every folder if you want!).

If your site is example.com/index.php, example.com/robots.txt is where the file goes (same folder as SMF in that situation), but if your site is example.com/forum/index.php, it's *still* example.com/robots.txt (but this time it's not the same folder as SMF, if that makes sense)

[Steve]

Makes perfect sense and thank you.

My site is actually forums.example.com but it still goes in example.com ... prior to the 'public html' folder, yes?

[Arantor]

Um... example.com/robots.txt will not apply to forum.example.com and vice versa a forum.example.com/robots.txt file will not apply to example.com.

[Steve]

Ah, gotcha now. Thanks.

[Gwenwyfar]

These bots are more annoying than I thought they'd be, getting spammed by yet another unnamed chinese bot now

From some of the info I found (is is changing the IP always as usual), seems to be from something called ShenZhen Sunrise Technology Co.,Ltd., which might be just baidu under another name. So yeah, they're still annoying, I guess, or at least the chinese are

It doesn't get identified by smf as a bot, though.