How to Configure my robot.txt?

[selinamelez]

Hi there  ı m tryıng to figure it out how to configure my robot.txt file


ı have created robot.txt on the root directory and added my sitemap link

but also do ı need to add this code?

User-agent: *
Disallow: /forum/*action
Disallow: /forum/*PHPSESSID
Disallow: /forum/*;
Allow: /forum/$
Allow: /forum/*board,*.0.html$
Allow: /forum/*topic,*.0.html$
Allow: /forum/*.xml
Allow: /*.css$
Allow: /*.js$
Allow: /*.png$
Allow: /*.jpg$
Allow: /*.gif$


Also, my forum has more than 2000 topics not indexing  even ı have SEO  plugging https://custom.simplemachines.org/index.php?mod=2659





You cannot view this attachment.

[Kindred]

the robots.txt is fine to have, or not have -- nothing in that would stop your topics from being crawled.

the Optimus mod is "SEO" only in a broad sense...  it adds opengraph and metatags...   but the search engines only barely use those these days.

Did you confirm that the "missing" topics are actually IN the sitemap?

On the other hand...  it takes time for spiders to crawl sites, and the good ones only do a little bit at a time.

[Bs1418]

You need to add it to a Google search console

[Aleksi "Lex" Kilpinen]

No, no you do not. Search engines either request for it, or ignore it.
Also, note that I have deleted a few of your posts recently as either digging up ancient topics, or simply adding nothing to the discussion.
Please stop that.

[Steve]

Flagging @Bs1418 to be sure he sees our Project Manager's post.