Spiders trying to index things that are not there anymore

[deansmar]

Hi Folks,

i hope i have placed this request in the right place, if not then could the Moderators move it to the right place

i am running a SMF 2 RC3 with SimplePortal 2.3.2

i have installed a MOD called More Spiders 1.2 and Spiders Don't Increase Topic Views 1.1.1 

i also have Aeva Media 1.3a  installed.

the situation;
Spiders are indexing the site, this is quite normal i have tried to give them limited access, under Members Settings "Apply restrictive permissions from group"
but if a Spider is classed as a Guest then it will still index the site ...

i have deleted some of the pages and images from the site

the problem;
my error log is now overflowing with things like Page can not be found and Album can not be found

is there a way to stop this ??

thanks for reading this request.

[deansmar]

no body up for this one..?? 

[Kindred]

no, there is no way to stop that....  once the spiders have cataloged pages on your site, they will try to re-find those same pages 2-3 times before they remove that page from their catalog.   In the mean time, that page will show up for anyone who searched the correct term, leading to more pages not found reports....    it is the nature of deleting a page on a site.

[deansmar]

thanks for the answer... this is very much as i thought..

but with a bit of editing in the robots.txt at least it can be minimised...