Google spiders...

Started by BrookanHub, January 03, 2008, 08:16:52 PM

Previous topic - Next topic

BrookanHub

What is the trick to getting Google spiders to crawl your forum? The Forum is included in the .xml site map, but I dont think its crawling it. Any tips would be sooooo helpfull.

Thanx
Brookanhub

karlbenson

It can takes several weeks to get a spider to crawl a new website.
Even then it can take months to get them to regularly crawl your forum.

If you forum has 0 pagerank the spiders don't necessarily think your worth fully crawling.

BrookanHub

The site itself handles about 250000 page requests  a day, and is crawled a fair bit.. what Im asking, is there anything in the .xml file that has to be added. Is there a trick I dont know about.. and so on.

Thanx
Brookanhub

karlbenson

Ah right.

Don't think so.
Have you submitted the xml file link directly to the google?
Have you tested it works?

BrookanHub

I did just that.. test and ok! My stats show Im being crawled about every two weeks.

Cheers!
brookanhub

ApplianceJunk

Appliancejunk.com and appliancejunk.com/forums was getting hit about every 4-5 day's.

Appliancejunk.com was last hit on Dec. 21st
our .com/forums has been getting hit everday for the last week. :)

Other then using the Google Webmaster tools we don't do anything special.

karlbenson

I get Spidered everyday by google.

kizer

Ok, Ive had my forum up for a while I'd guess 4years on my old software. Google and every other engine spidered the heck out of it.

Now my question is I was reading this and trying to figure out what sitemap file are we talking about. I did a search on the forum and found this mod.
http://www.simplemachines.org/community/index.php?topic=170813.0

I installed it and now Im on the google tools. Its asking for a .xml file. I used the URL of
http://www.my_domain_here.com/forum/index.php?action=sitemap;xml

or should I of used

http://www.my_domain_here/forum/index.php?action=sitemap

I wouldn't of asked about it if it wasn't for the fact that "karlbenson" is using it and figured maybe its a worth wild mod to use. If it isn't the correct one then what is everybody else using? Google has been to my front page of my forum several times and its listed in google, but it doesn't seem to go any farther. My old forum I want to shut down, but I'd rather give my new forum some spidering so Im not leaving people that are looking for my content nothing to look at.
Own a Jeep? Links4Jeeps.com

kizer

#8
Welp I 'might' of answered my own question. I used
http://www.my_domain_here.com/forum/index.php?action=sitemap;xml

Checked the status in the tools section and it said the following:

Sitemap type Web
Submitted Jan 15, 2008
Last downloaded by Google Jan 15, 2008
Total URLs in Sitemap 3746

After that I guess it does what ever it does and builds it into the search engine.
Own a Jeep? Links4Jeeps.com

humbleworld

Quote from: karlbenson on January 11, 2008, 08:33:30 AM
I get Spidered everyday by google.

Me, too, I am spired by google everyday.

I tested one time. I wrote one topic and the next minute I searched the term on google.com, and there it was, the topic was already indexed immediately.
And it happened not just once, but many times already.

arya6000

When I try to put http://mydomain.com/index.php?action=sitemap;xml

I get an error saying:

QuoteThe Sitemap must be located at http://mydomain.com/. To add a Sitemap at http://mydomain.com/http://mydomain.com/, first add that site to your account and then click the Add a Sitemap link beside it.

Any idea why?

CmptrWz

Tell it just index.php?action=sitemap;xml perhaps?

arya6000

Quote from: CmptrWz on January 30, 2008, 07:34:34 PM
Tell it just index.php?action=sitemap;xml perhaps?

yep, Thank Youuuu

Sun11

My stasts are

Sitemap statistics:
Total URLs: 651
Indexed URLs: 1

Well it takes a LONG time to index but iv got others links on it

For a better way to get double the links

On Google Webmaster add in

www.w-aw.com
&
w-aw.com
but take "w-aw" and put your domain name in

H

As already stated it can take awhile to get into Google. Having flash for the main site content may slow things down a bit. You also get a few more results if you disable safesearch.
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Advertisement: