SMF Support > SMF 2.0.x Support

Bots are not entering in my site!

<< < (4/4)

SwapsRulez:

--- Quote from: Nigel on April 26, 2008, 04:15:13 PM ---
--- Quote ---Then what might be the reason to not to even enter into any section.
--- End quote ---

Dunno... I'm not a SEO expert. One thing to bear in mind is that there's a lot of links on a forum page. Bots tends to spend a certain amount of time or follow a certain number of links per visit and it's gonna take time. Plus the newness of your site is gonna make a difference. My forum isn’t live yet but with my website it took about a year before Google started to fully index the pages, despite visiting regularly.


--- Quote from: starlion on April 26, 2008, 02:45:51 PM ---If you create robots.txt bots are can entrance to your web site.
--- End quote ---

--- Quote from: SwapsRulez on April 26, 2008, 03:06:57 PM ---Yep, i created robots.txt here.
--- End quote ---

Firstly the robots.txt file tells the bots where NOT to go. Normally they go everywhere and you can exclude them from certain areas you don’t want crawled. Note however that only the good bots abide by those rules. The bad ones will go there anyway.

Second, your robots.txt file isn’t doing anything.


--- Code: ---User-agent: *
Disallow:

--- End code ---

The first line is targeting every bot, then in the disallow line you specify the files/folders you don’t want crawled. But as I said, that will keep them out of certain areas, NOT coax them in.

I'd suggest forgetting about all this stuff for now and instead focus on building up a good site first. Then they will come...

Nigel


--- End quote ---

Yep... i do know robots.txt as well, & the creator of the standard knows me :P
Coz while reading his site, i told him some mistakes in the site & he corrected it. :D

Also i'm going to wait now... Lets see what happens!!!! :)

karlbenson:
Do you still require assistance with this?

CraigPal:
SwapsRules,

You have 403 pages listed in Google, in case you didn't know.
http://www.google.com/search?q=site%3Awww.project-bb.org&rls=com.microsoft:*:IE-SearchBox&ie=UTF-8&oe=UTF-8&sourceid=ie7&rlz=1I7GWYE

Navigation

[0] Message Index

[*] Previous page