I was with phpbb and change to SMF. I'm with SMF for 15 days now and google stil does not index posts. (Under phpbb all posts where indexed without problem every day).
The forum has now about 20 000 messages and this indexing is a problem. Is there a mod a anything to help google spidering ?
thank you
mine is not running any mods and it is indexing fine... search for tkd fighters needed (http://www.google.com/search?hl=en&lr=&ie=UTF-8&safe=off&q=tkd+fighters+needed) in google, which is one of my topic subjects and you'll see it's first in the list... ok granted it still shows my old Yabb.cgi link (which you can still click on and will bring you to SMF, but at least the second link is index.php !!
for me I don't understand why messge are the profile ones, with message by author...
But normal pages are not indexed... :-[
what explaination for that?
my SMF:
http://www.celebrites-selection.com/phorum/
Hmm.. I don't know... the tdk fighters link doesn't make google go to the thread directly either...
Personally, I set the option to display links like this: index.php/topic,344.new
The only publically available posts on my forum are being found by google (http://www.google.com/search?sourceid=mozclient&ie=utf-8&oe=utf-8&q=site:forum.vuurwerklook.com+forum+updates) ... so you might want to try enabling that option in your settings, if you really need to have it fixed quickly... it's apache servers only though...
Also, just tested this board on google (http://www.google.com/search?hl=en&lr=&ie=UTF-8&safe=off&q=site%3Asimplemachines.org+GOOGLE+spider+SMF+without+mod&btnG=Search)...
http://www.google.com/search?q=allinurl:++site:simplemachines.org&hl=en&lr=&ie=UTF-8&safe=off&as_qdr=all&start=0&sa=N&filter=0
Quote from: Burpee on July 18, 2004, 08:55:03 AM
Hmm.. I don't know... the tdk fighters link doesn't make google go to the thread directly either...
I think the reason it still shows the Yabb one is cos that post was started when I had my board as Yabb gold. Now my yabb.cgi file (which is still uploaded) is a redirect to /forum... which is why the link goes to the board index.
this google search (http://www.google.com/search?hl=en&lr=&ie=UTF-8&safe=off&q=site%3Aairport-tkd.com+kicks) shows a few links to my main site, and also a link to a topic in the forum...
Every three days I see about 15 Google robots crawling my forum. And I'm not running any mods. It's the meta tags that also helps alot. If you have the same meta tags in your forum as you have on your main website, plus a few links directly to the forum from the main site and otherwise, Google will crawl your site every once in a while to keep itself updated with your hits. Plus, you are very likely, if you have your meta tags set properly, to become one of the first ten hits on almost every search word that contains items you know is on your forum.
Unless things have changed, Google places very little value on meta tags.
True Ben... I get most of my hits from strategically used keywords on my main page...
Quote from: zom on July 18, 2004, 02:40:09 AM
I was with phpbb and change to SMF. I'm with SMF for 15 days now and google stil does not index posts. (Under phpbb all posts where indexed without problem every day).
The forum has now about 20 000 messages and this indexing is a problem. Is there a mod a anything to help google spidering ?
thank you
Unless you are a high PR site google wont dig very deep.
Make the forums 2 levels down max, my advice would be to build a site map with a "one link" access to each of your forums, google will start indexing from there I recon.
I've found the best way to get spidered is exposure, exposure, exposure. I belong to a bunch of community forums, post quite a bit (always put my site's url in my sig/profile), and always use the same username. The more popular the sites you post on, the more traffic you'll generate.
Seems like Google has indexed my site pretty well with no mods. For example, if you type DezinaControl in Google, both results link to posts on my site.
Quote from: kegobeer on July 18, 2004, 10:31:30 AM
I've found the best way to get spidered is exposure, exposure, exposure. I belong to a bunch of community forums, post quite a bit (always put my site's url in my sig/profile), and always use the same username. The more popular the sites you post on, the more traffic you'll generate.
Seems like Google has indexed my site pretty well with no mods. For example, if you type DezinaControl in Google, both results link to posts on my site.
That's called link popularity ;) always get as many links incomming as possible, as long as they're high PR and relevant, and make sure your links have tags that match your keywords. (link reputation)
Quote from: Burpee on July 18, 2004, 08:55:03 AM
Hmm.. I don't know... the tdk fighters link doesn't make google go to the thread directly either...
Personally, I set the option to display links like this: index.php/topic,344.new
how do you do that?
SMF works fine like all php scripts a site map speeds things up.
If you want more use mod_rewrite rule (find on the web)
could it be possible to have a sit_map page to help indexing like with phpbb. This php page contains all the leads to all the posts directely. If you know scripting this could help: here is the code for phpbb:
<?php
/***************************************************************************
* Save this file as: site_map.php (or anything you like)
* Version: Friday, Oct 4, 2002
* Email:
[email protected] * Purpose of hack: Basically generates a list of topics and
* displays them with link to the topic. Goal
* is to provide search engines like Google
* with a static page of links to dynamic pages
* You should link to this page from your sites
* home page somewhere.
* Demo: http://www.aussiecelebs.com/forums/site_map.php
* Tested on: phpBB 2.01, 2.02
*
***************************************************************************/
/***************************************************************************
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
***************************************************************************/
echo "
<!-- Start header here. Make sure you dont use any \" s -->
<html>
<head>
</head>
<h1>TITLE</h1>
<!-- End Header -->
";
define('IN_PHPBB', true);
$phpbb_root_path = './';
include($phpbb_root_path . 'extension.inc');
include($phpbb_root_path . 'common.'.$phpEx);
echo "<h2>Catégories</h2>";
$result = mysql_query("SELECT cat_title, cat_id FROM phpbb_categories ORDER BY cat_title");
while($row= mysql_fetch_assoc($result))
{
echo "<a href='index.php?c=".$row["cat_id"]."'>".$row["cat_title"]."</a><br>";
}
echo "<h2>All forums</h2>";
$result = mysql_query("SELECT forum_id, forum_name, forum_desc FROM phpbb_forums ORDER BY forum_name");
while($row= mysql_fetch_assoc($result))
{
echo "<a href='viewforum.php?f=".$row["forum_id"]."'>".$row["forum_name"]."</a>"." - ".$row["forum_desc"]."<br>";
}
echo "<h2>All posts</h2>";
$result = mysql_query("SELECT topic_title, topic_id FROM phpbb_topics ORDER BY topic_title");
while($row= mysql_fetch_assoc($result))
{
echo "<a href='viewtopic.php?t=".$row["topic_id"]."'>".$row["topic_title"]."</a><br>";
}
echo "
<!-- Start footer here. -->
</body>
</html>
<!-- End footer -->
";
?>
I have to highly recommend the mod_rewrite function. You'll have to rewrite your urls in the source files, but I did this on one of my sites which has almost 60,000 posts and every single post is on google. So are all my user's profiles. This gets my site very high rankings for a wide variety of search terms.
does anyone have an .htaccess for those mod rewrite ?
errmm, how do you do this mod rewrite function?
thanks
Quote from: Shadow's Pawn on July 19, 2004, 03:19:05 PM
I have to highly recommend the mod_rewrite function. You'll have to rewrite your urls in the source files, but I did this on one of my sites which has almost 60,000 posts and every single post is on google. So are all my user's profiles. This gets my site very high rankings for a wide variety of search terms.
I would like to know about mod rewrite as well! I have quite a few topics show on google, but no posts showing.
Sea Of Sin
Search for mod_rewrite.
-[Unknown]
Guys;
I've got google-bots crawling my site every single day without any special mods done. I've got the meta tags, adsense installed and that's about it.
http://www.latinagorditas.com is the site. I need to do the mod rewrite thing myself although finding time is the problem.
And by the way, yesterday. Google released a new tool/service that may take care of this problem.
Check out https://www.google.com/webmasters/sitemaps/. It's a hot topic right now in the blogosphere. I've done it to both of my blogs and are waiting for the results.
http://www.truckingmonkey.com & http://www.retromex.com
http://www.simplemachines.org/community/sitemap.php ;).
Did that yesterday.
-[Unknown]
The search engine friendly url's in smf are great.
But I noticed they don't work anymore hwen you use mkportal.
So I don't have them anymore... :(
Google won't crawl very deem into your forum, cuz it doesn't like those dynamic url's.
I had over 4000 topics/posts in google's index when I had phpbb with static url's, went down to 675 since I switched to smf and can't use them anymore, cuz I have mkportal.
http://www.google.com.au/search?hl=en&lr=&q=+site:web.aanet.com.au+ftgforum
I'm only getting 7, although I'm sure i used to have more. possibly those are ones that google was linked to from elsewhere.
I got a question what template file of the smf templates do i change the meta information cause the meta that it is using now is the smf one, below is the meta its using
<meta name="keywords" content="PHP, MySQL, bulletin, board, free, open, source, smf, simple, machines, forum" />
index.template.php
Quote from: Harro on June 05, 2005, 06:21:17 AM
The search engine friendly url's in smf are great.
But I noticed they don't work anymore hwen you use mkportal.
So I don't have them anymore... :(
Google won't crawl very deem into your forum, cuz it doesn't like those dynamic url's.
I had over 4000 topics/posts in google's index when I had phpbb with static url's, went down to 675 since I switched to smf and can't use them anymore, cuz I have mkportal.
I have MKportal too and had it wrapped in the forum, but Idecided to try something this month and unwrapped it! and it really pushed my site high up in the search engines! I was lagging on the 5th page with one of my keywords and now I'm 3rd! :)
And no-one on my forum yet has noticed a difference between when it was wrapped and unwrapped!! YAY! Can you tell?
http://www.revolutionforum.net
I wrote a little php script and put it in my index.template to make it look (as close as I can get it!) to look like the MKPortal top banner and buttons (which I use) :) I am even suprised by what I can do sometimes! from someone who says they cannot programme! :)
But it does make a big difference! and would say it really helps not to have them connected like that!
Sea Of Sin
Quote from: eddie43302 on June 05, 2005, 02:24:22 PM
I got a question what template file of the smf templates do i change the meta information cause the meta that it is using now is the smf one, below is the meta its using
<meta name="keywords" content="PHP, MySQL, bulletin, board, free, open, source, smf, simple, machines, forum" />
You only have to edit if for the default template btw, cuz that's the one that spiders will see.
@SeaOfSin: could be a good idea to try to get the left menu (and rights if you wish), also in the index.template.... Or just a little menu bar to the mkportal's "Main Menu" or something like that.
Think I'll give that a try after I finished my exams :)
Google certainly indexes the SMF forums here:
http://www.google.co.uk/search?q=site:www.simplemachines.org&num=100&hl=en&lr=&safe=off&filter=0
Shows 52,000 results.
Using the mod_rewrite function in the admin section could certainly help - but it is definitely worth making the distinction between Google actually *indexing* pages, and Google *ranking* pages.
In other words, just because a page is indexed, doesn't mean to say it will actually rank for anything on its own merits.
Google has always been a links-based search engine, so links into your sites and pages for relevant and descriptive keywords will certainly be helpful.
Quote from: zom on July 19, 2004, 02:25:21 AM
could it be possible to have a sit_map page to help indexing like with phpbb. This php page contains all the leads to all the posts directely. If you know scripting this could help: here is the code for phpbb:
Is that file for PHPBB .. or did you modify to use with smf?
That would be PERFECT... if in XML format to be submitted to the Google Sitemap section.
Would this be possible to do?
I just got a file that will do this for my store:
www.egameaddiction.com/store/gsmap.php (http://www.egameaddiction.com/store/gsmap.php)
Search for "sitemaps".
-[Unknown]
update, update... how about it.. you people have now some experiences with that sitemap...
i made me also some for my sites, put them in my rootdirectory, en submit them with google submit.. does it funktion?
i saw this topic was from june.. perhaps some people have it running allready for that period..
greetz
Hello everyone,
I have the same problem with google indexing my forum.Only main page is showed in results,but not forums or posts.I submited a XML site map,but nothing changed.When I used PHPbb,all forums and posts were indexed :(
Can you tell me what should I do to help google index all pages on my forum? www.andforum.net
Thank you
Google indexes SMF very easily and with no mods.
However your change of forum software will mean that the links google was used to finding were all gone (it is finding bad links) .. it therefore may "sandbox" you for a while.
Page rank n/a. In other words google doesn't even know it exists let alone think it's important.
Quote
However your change of forum software will mean that the links google was used to finding were all gone (it is finding bad links) .. it therefore may "sandbox" you for a while.
And how long it can last?
Who knows?
Google is a law unto itself.
My advice is get some incomming links so that google can find you again.
But google finds main page of my forum,how is then possible that he didn't find my site?
Because google doesn't care to look if you have no page rank.
Google will take a cursery glance at your front page but it will only spider down through layers if it decides that you have a worthwhile site.
I don't understand :( While I was having PHPbb, google indexed every single post and board, and now nothing! :(
Please tell me what to do, because I don't want to return to PHPbb again :(
It's not SMF, my guess is that google tried to find all the old PHPbb boards and got constant errors so dumped you for a bit.
I would have added a load of permanent redirects (for the main boards at least) to help google find the new stuff.
Also google is currently going through jagger3 so lots of weird things are happening in googleworld.
FYI changeing back to PHPbb would just make it worse as there'd be a whole new range of missing boards.
I'm afraid that we have to accept that atm if we change our sites google may dump us, happened to me when I changed my main site from .com to .co.uk
Hey can anyone post the sitemap.php by unknown here...and how i can use it>>?
Parth
something nobody has mentioned in this topic is whether or not we need a robots.txt file. I looked through the files on my server and don't see one, should I add one? My site isn't live yet so I'm not too worried, I'm just planning ahead. Thanks in advance for any help.
robots.txt is only use to stop robots, not having it allows them to spider everything they can see.
In addition to that many bot's don't even pay attention to a robots.txt file anyway and will crawl your site regardless.
Common sense really applies regarding search engines and them crawling your site. If your site is worth crawling on a regular basis and you have lots of natural inbound and outbound links your site will be crawled by the search engines.
To find out what google knows about your site try the following in a google search
info:yourdomainname
I don't think this site needs any help being crawled by Google. Just look what is at the bottom of every SMF install!
:)
Quote from: redone on December 05, 2005, 08:52:09 AM
In addition to that many bot's don't even pay attention to a robots.txt file anyway and will crawl your site regardless.
If we're talking majors
yahoo now follows robots.txt
Hi
The script URL now return a not found page
Can anibody post the script or provide instruction on how to make a google sitemap in SMF?
tks
The question originally asked was... Does GOOGLE spider SMF without mod?
Now, I see a ton of post on site maps and how well smf gets indexed but the question remains somewhat unanswered. Rather than tell you that the spiders or in my site every day and I have "some" PR. I have decided to just ask a more direct question because telling you the above would just cause us more confuseion with the answers we get....lol
Unknown posted that we should search for "mod_rewrite" I did this and still don't get it... sorry
Can anyone tell me where do I get the mod_rewrite or how do I do it? I don't have an Apache server just to know.
edit:Thank You Harro !!!
Google will spider your forum without a mod!
And mod_rewrite is something that is installed on your (apache webserver).
It'll allow you to rewrite your url from eg
http://harro.sin.khk.be/forums/index.php?topic=704 to http://harro.sin.khk.be/forums/topic-704.html
I've done it on my forum in the past (lost it with upgrading, so now waiting for 1.1 final to do it again), but it's not really needed.
Maybe those sitemaps can help the spiders to find the topics a bit faster, but it's not needed.
Here's a (very good) mod_rewrite tutorial: http://www.doriat.com/mod_rewrite6.html
sorry but i have smf rc 1.3 integrate in mkportal 1.1...but my forum doesn't indexing at google spiders why?
How long has your forum been online?
can take quit some time before your site gets indexed.
before i used phpbb but from 2 months use smf integrate in mkportal....but i don't have installed any mod for indexed