Does GOOGLE spider SMF without mod?

[andrejc]

I don't understand  While I was having PHPbb, google indexed every single post and board, and now nothing!
Please tell me what to do, because I don't want to return to PHPbb again

[Thunderace]

It's not SMF, my guess is that google tried to find all the old PHPbb boards and got constant errors so dumped you for a bit.

I would have added a load of permanent redirects (for the main boards at least) to help google find the new stuff.

Also google is currently going through jagger3 so lots of weird things are happening in googleworld.

FYI changeing back to PHPbb would just make it worse as there'd be a whole new range of missing boards.

I'm afraid that we have to accept that atm if we change our sites google may dump us, happened to me when I changed my main site from .com to .co.uk

[coolparth]

Hey can anyone post the sitemap.php by unknown here...and how i can use it>>?

Parth

[crappyday]

something nobody has mentioned in this topic is whether or not we need a robots.txt file.  I looked through the files on my server and don't see one, should I add one?  My site isn't live yet so I'm not too worried, I'm just planning ahead.  Thanks in advance for any help.

[Trekkie101]

robots.txt is only use to stop robots, not having it allows them to spider everything they can see.

[redone]

In addition to that many bot's don't even pay attention to a robots.txt file anyway and will crawl your site regardless.

Common sense really applies regarding search engines and them crawling your site. If your site is worth crawling on a regular basis and you have lots of natural inbound and outbound links your site will be crawled by the search engines.

[beemer]

To find out what google knows about your site try the following in a google search

info:yourdomainname

[redone]

I don't think this site needs any help being crawled by Google. Just look what is at the bottom of every SMF install!

[Thunderace]

In addition to that many bot's don't even pay attention to a robots.txt file anyway and will crawl your site regardless.

If we're talking majors

yahoo now follows robots.txt

[Sanhux]

Hi

The script URL now return a not found page

Can anibody post the script or provide instruction on how to make a google sitemap in SMF?

tks

[NOLA]

The question originally asked was... Does GOOGLE spider SMF without mod?

Now, I see a ton of post on site maps and how well smf gets indexed but the question remains somewhat unanswered. Rather than tell you that the spiders or in my site every day and I have "some" PR. I have decided to just ask a more direct question because telling you the above would just cause us more  confuseion with the answers we get....lol


Unknown posted that we should search for "mod_rewrite"  I did this and still don't get it... sorry

Can anyone tell me where do I get the mod_rewrite or how do I do it? I don't have an Apache server just to know.

edit:Thank You Harro !!!

[Harro]

Google will spider your forum without a mod!

And mod_rewrite is something that is installed on your (apache webserver).
It'll allow you to rewrite your url from eg
http://harro.sin.khk.be/forums/index.php?topic=704 to http://harro.sin.khk.be/forums/topic-704.html

I've done it on my forum in the past (lost it with upgrading, so now waiting for 1.1 final to do it again), but it's not really needed.

Maybe those sitemaps can help the spiders to find the topics a bit faster, but it's not needed.

Here's a (very good) mod_rewrite tutorial: http://www.doriat.com/mod_rewrite6.html

[iettafune]

sorry but i have smf rc 1.3 integrate in mkportal 1.1...but my forum doesn't indexing at google spiders why?

[Harro]

How long has your forum been online?
can take quit some time before your site gets indexed.

[iettafune]

before i used phpbb but from 2 months use smf integrate in mkportal....but i don't have installed any mod for indexed