How do I keep bots from crawling board?

[ML]

I have had over 15 bots crawl my board in one day.

[NEMINI]

bots = good

why would you want to stop them?

[TheDisturbedOne]

If you want to stop them put:
      Disallow: /index.php
In your robots.txt file.

[ML]

bots = good

why would you want to stop them?

I don't want a lot of members other than people I invite. It's just a board for me to blow off steam at and post stupid stuff. :-)

[ML]

If you want to stop them put:
      Disallow: /index.php
In your robots.txt file.

I'll try that. Thanks.

[ML]

Hmmm. I wonder how long it should take before it starts working?

bots = not good

They are still at it. I just do not want the inevitable porn postings that I got with my old board. Bots crawl, weirdos post porn sites. Go figure.

I now have it set up so that everyone has to be approved, but one with a real sounding name made it by me. His first post? Porn sites.

You want to post a dirty joke on my board that's fine. But advertising porn sites will not happen.

Unless one signs up, there are only three boards that can be seen at all. Should I make these boards off limits too? You recon that would stop the little crawlers?

[kat]

Why not disallow guest viewing, if your board's private?

That'll stop 'em.

[Toadmund]

This will stop the spammers 'right quick'!
http://custom.simplemachines.org/mods/index.php?mod=1516
I neglected my board for awhile and got spammed hard, I installed it, they can't answer the simple questions.

[kat]

Have a read of this:

http://www.simplemachines.org/community/index.php?topic=273816.0

God knows why that thread's not been stickied...

[evolution88]

bots = good

why would you want to stop them?

either that response was sarcastic or I really don't know what bots are? I thought bots are to spam forums ?

[NEMINI]

bots = good

why would you want to stop them?

either that response was sarcastic or I really don't know what bots are? I thought bots are to spam forums ?

googlebot, msnbot, yahoobot ... etc.  without them no ones site would everr appear in search engines.  without search engines you'd never be able to grow a site.
either that response was sarcastic or I really don't know what bots are? I thought bots are to spam forums ?

[evolution88]

ooh okay ... yeah that makes more sense now , thanks


but how do you get those searchmachine bots? you have to sign up yourself?

[SaltwaterRedneck]

ooh okay ... yeah that makes more sense now , thanks


but how do you get those searchmachine bots? you have to sign up yourself?

They automatically crawl the web on their own. I've seen them on my sites a time or two and never signed up for them.

[evolution88]

oh okay, thanks. learned something new

[ML]

Why not disallow guest viewing, if your board's private?

That'll stop 'em.

I did but they still tried to crawl the site. It created errors. I guess that's what is creating th errors. It is the bots IP address that is noted in the error anyway.

Since I added the index.php to the text file, I haven't seen any today.

As for growing a site: I would rather not have many more members than I already have.

Its a place for me to vent. Whether anyone else reads it or not doesn't matter at all to me.

[Rumbaar]

If it's legit bots, they'll respect your robots.txt.  But if it is in fact spam bots, they'll not even look at the robots.txt and still try trawl your forum.

User approval is the solution, and/or various anti-bot mods for then they try to register.  If you don't want more members, just turn off registration.

[ML]

If it's legit bots, they'll respect your robots.txt.  But if it is in fact spam bots, they'll not even look at the robots.txt and still try trawl your forum.

User approval is the solution, and/or various anti-bot mods for then they try to register.  If you don't want more members, just turn off registration.

That explains why they are still crawling. I don't suppose it's a big deal. It's just aggravating sometimes.

I have it set so that only a few boards can even be "viewed only" without joining. I have some restaurant and business reviews for places that I have gone to and some announcements and rules and things like that.

To get to the Ramblings boards though, you have to register and that would require my approval.

I don't want to turn off registration completely as every once in a while I get someone I know, or am acquainted with, to join up.

The bots just had me wondering as I had a robots txt file and they were still doing their bot thing. It is not as bad as it was before I added the index to the txt file though.

I appreciate the comments and advice from all.

[MrMike]

May I suggest using this: http://BotScout.com

BotScout helps prevent automated web scripts, known as "bots", from registering on forums, polluting databases, spreading spam, and abusing forms on web sites. We do this by tracking the names, IPs, and email addresses that bots use and logging them for future comparison. We also provide a simple API that you can use to test forms when they're submitted on your site.

We provide some sample code that you can use in your forums, contact forms, mailing lists, and other parts of your site.

You can manually search the bot database to find bots in your forums, but you can also use a little bit of code in your forum, contact form, or other web application to test for bots and automatically ban or reject them on the spot. To make use of our database in your application, all you need to do is query it correctly.

There are also anti-bot plugins for popular forums, including SMF,  VBulletin, phpBB, Phorum, PunBB, Invision Power Board, FUDForum, Ikonboard, and W-Agora.         

[kat]

On a similar note, I use this:

http://english-72682862726.spampoison.com

Works like a charm.

[Toadmund]

I've been using something similar:
http://www.mytrashmail.com/anti_spam.aspx
Feels nice to foil, them