That's not a normal load time, but it can happen when many people are searching at the same time.
We need the max possible search index and that almost doubles the database size, and when many people use it, it can slow it down a bit sometimes.
If you have a forum without a lot of searching that is a lot easier to configure and way less hardware is needed.
You should seriously consider using sphinx if you're not already. We run a dedicated Sphinx server for this site and searches happen very fast with no measurable load on the web or database servers.
Derek, I have found the reason.
It was Microsofts poor MSNbot 2.0b, it misbehaves and ignores robots.txt so I completly banned that bot via .htcaccess on their IP's.
That bot is hitting our server extremely hard, probably other sites like this one too.
Simply entered this into it:
#Block Inktomi and MSNbot 2.0b
order allow,deny
deny from 74.6
deny from 72.30
deny from 65.55.207
deny from 65.55.106
deny from 65.55.107
allow from all
And hoppa, problem solved

One does not want misbehaving bots...and Inktomi and MSNbot 2.0b are the worst on the planet.
Yesterday MSNbot 2.0b hit our website with about 50000 pageviews! That 1 every 2 seconds.
Even after I set crawldelay to 120 seconds, all others obey my setting, MSNbot 2.0b does not, and I suspect it to do even near 90k a day, as the peaks are very high.
Not even Google does anywhere close to that.