I want to de-index everything except topics & categories

[MohammedFK]

Hi all

Like title: I want to de-index everything except topics & categories in my forums.

what's the best code for robots.txt ?

[shadav]

not sure why you'd want to do that and not really sure there's a way to block everything else and just leave the posts

but you could add into the header of the pages that you don't want indexed

Code Select Expand

<meta name="robots" content="noindex">
make sure that the page doesn't already have a meta name="robots" on it

make sure that the pages you do not want indexed are not in your sitemap.xml

[Aleksi "Lex" Kilpinen]

There really isn't a magic trick to do what you want - what has been indexed, can not easily be "de-indexed" from everywhere. At the very least, going through the robots.txt route will take a long time to have any meaningful effect.

[MohammedFK]

There really isn't a magic trick to do what you want - what has been indexed, can not easily be "de-indexed" from everywhere. At the very least, going through the robots.txt route will take a long time to have any meaningful effect.

My forum is NEW; I want to do this right from the start.

[MohammedFK]

make sure that the pages you do not want indexed are not in your sitemap.xml

I haven't  sitemap.xml .. how can I?

[Aleksi "Lex" Kilpinen]

There really isn't a magic trick to do what you want - what has been indexed, can not easily be "de-indexed" from everywhere. At the very least, going through the robots.txt route will take a long time to have any meaningful effect.

My forum is NEW; I want to do this right from the start.

Any particular reason you feel the need to manually limit things like that then?
SMF already does a pretty good job with search engines as it is, and I haven't really noticed any need to tamper with it.
Just trying to understand what it is you are trying to accomplish.

[drewactual]

using the structure of SMF out of the box, de-indexing categories isn't possible.  it isn't mentioned in the URL as it simply looks for the topic, which is numbered. 

using something like pretty url's will remedy this solution, but it will also render the listings you currently have indexed on whatever engine useless- making your site take a nose dive initially in the rankings (unless you do one key thing, which is a touch complicated and outside of the scope of this discussion). 

if you don't care about the already indexed pages and wish to block them, as your title suggests, with the exception of category and topic, then PrettyURL's is your answer- and your solution will be simply to allow indexing via robots.txt with those category names and the trailing slash indicating anything following the index is allowed.  you'd disallow anything else. 

basically, PrettyURL's is the beginning of the solution for you- and makes manipulation of the robots.txt simpler...

caveat:  those URL's as presented by SMF out of the box are still available as a means to reach the page in question... it's just that your site translates that URL to the structure of PrettyURL's.  The question this presents is one i don't know the answer to- and that is "if robots.txt blocks indexing to a particular URL via one path, will it break the PrettyURL path too?"... and I don't know...

[Kindred]

drewactual -- your suggestion is based on the rough translation. he's not actually asking to "remove an existing index"


Based on his previous request, he thinks that there are SEO things that he "needs to do" -- and he's essentially asking "how do I prevent every page other than a topic page from being indexed in the first place"  (possibly because he thinks that other pages getting indexed will harm his search engine ranking (which they will not))

In short, everything this user has asked so far is based on a basically incomplete understanding of how search engines work and what can be "easily" done in a forum system.

[shadav]

wouldn't having a bunch of noindex and nofollows also hurt your pr/search rank?
not that i've ever cared about pagerank and crap (is pr even still a thing)....just work on your sites content, I mean yes add title tags, alt tags, ect to help but mainly if you don't have good content and active members then it isn't going to matter one way or another

one of my sites is the 2nd search result on google, didn't mean a thing without the content on it....and I didn't do anything out of my way to get it there, other than it's not a much used key phrase (it is a niche site)..as for the main keywords, lmfao that's never gonna happen as I'm competing against youtube on that.

[MohammedFK]

Any particular reason you feel the need to manually limit things like that then?
SMF already does a pretty good job with search engines as it is, and I haven't really noticed any need to tamper with it.
Just trying to understand what it is you are trying to accomplish.

Too many links to personal files and unimportant links, I think the important thing is the articles; But it is not yet topics; I know it takes time, but I thought this was useful.


Support Team edit: fixed missing open quote

[MohammedFK]

drewactual -- your suggestion is based on the rough translation. he's not actually asking to "remove an existing index"


Based on his previous request, he thinks that there are SEO things that he "needs to do" -- and he's essentially asking "how do I prevent every page other than a topic page from being indexed in the first place"  (possibly because he thinks that other pages getting indexed will harm his search engine ranking (which they will not))

In short, everything this user has asked so far is based on a basically incomplete understanding of how search engines work and what can be "easily" done in a forum system.

OMG! Smart mind ..