So what is wrong on the first page load, before you do a hard refresh?
What settings have you for caching in the admin panel?
Who are you hosted with, and do you know if the host has some caching at play as well?
I just recently saw something very similar on GoDaddy, when GoDaddy's firewall and cache was at fault.