I've been using SMF for a car forum since 2014 and it's been pretty stable and trouble free, so well done to all involved. I've not needed to ask for help until now....
On Jan 19, users reported a problems with features that have never been an issue:
Tapatalk didn't work
"Recents" post listing failed (no error and nothing happened when clicked on)
We were running v2.0.15 and an outdated Tapatalk MOD, so decided the issues may be due to this. On Jan 20 I updated through v2.0.16/17/18 to 19.
Tapatalk MOD v4.5.8 installed
Enhancements to recent posts MOD v2.0.10 installed
This solved the issues reported and the forum appeared to be working fine BUT, I noted the error log was showing zero entries. It has remained with zero entries over the past 4 days, yet I have spotted a couple of minor issues for which I expected an error log. (Error logging is ticked the "Security and....").
More worryingly, although the forum is 99.99% functioning, I have looked back at the host performance logs and noted a very worrying set of 100% CPU spikes, which started around the time of users reporting the issues, PRIOR to my upgrades. The spiking has continued, see attached image below
I've raised it with the host who replied:
I have had a look into the resource usage for the 19th of January and it appears that the same script is causing the CPU to spike throughout the day, this would be the following PHP file:
......../public_html/forum/index.php
From the snapshot which you can also view in the cPanel under Resource Usage and then Snapshot, if you go to the 19th of January from around 5 pm onwards this specific script appears to cause the CPU to spike, in this case, I do believe there is an issue with the form and I would advise that you contact the form provider and they may be able to explain the reason the form index is causing the CPU to spike......
No-one made any changes to the software on 19 Jan, so this is a bit baffling.
Full disclosure - I'm not very adept at coding, but could someone advise as to how I can go about fixing these issues?
yes, there is definitely something wrong with your installation -- but it's not specifically 2.0.19 causing it -- because I manage 4 sites running 2.0.19 and have NOT seen this sort of spike.
SO... are those the only two or do you have any other mods installed?
I'd suggest removing tapatalk as a first step/trial.
Also, 2.0.15+ included security updates... so I'd also start checking for hacks or other unknown scripts in your directories/files.
maybe worth looking into https://wiki.simplemachines.org/smf/How_to_upload_a_fresh_set_of_files
Thanks for those suggestions, which seem a good way forward. I'll think I'll start by uninstalling all the MODs, one at a time, to see what impact that has. There are about a dozen MODs in all, but none gave issues before 19 Jan, so still baffled as to how one of them might have started to cause issues so suddenly.
If that doesn't solve the spiking, I'll look to the clean install, but hope not to have to take that step if at all possible. Even by backing everything up first, I'd be concerned at losing what, for the moment (from a user perspective) is an operational forum.
Forum content (members, posts, pms, etc.) are in the database, not the files. If you have a good database backup and backup attachments, you won't lose anything by uploading clean files. Mods and customizations would have to be reinstalled.
I think uninstalling mods one at a time, starting with Tapatalk, is the way to go first. Some mods need to be updated for 2.0.19 ... there aren't many but there are some.
If that fails, then go the fresh set of files route.
If it were me.
One other thing to look at is the web access log. See if any unusual activity corresponds to the spikes.
Or an unusual volume of activity.
I find that access logs can be read almost directly into Excel (as csv, but using space as a delimiter). Pluck the day+hour out of the time, pivot by ip over day+hour, sort by count. You can only do 1M rows at a time. It's surprising what you can find...
Quote from: Sir Osis of Liver on January 24, 2022, 08:53:51 PMForum content (members, posts, pms, etc.) are in the database, not the files. If you have a good database backup and backup attachments, you won't lose anything by uploading clean files. Mods and customizations would have to be reinstalled.
Thanks for that reassurance!
So, if I have a database backup (which I take regularly) and I screw everything up, if I then install a fresh copy of SMF v2.0.19 and restore my database, I'll have all the forum board structure, members, posts, pm's etc, as now, but without any customisation?
I will then just need to re-install my custom theme and MODs?
Quote from: casey_mrc on January 26, 2022, 01:05:29 PMI'll have all the forum board structure, members, posts, pm's etc, as now, but without any customisation?
Yes, all of that is in the database. Everything except attachments and avatars is in the database.
Thanks again for these valuable suggestions to a newbie on here :-[
Quote from: shawnb61 on January 25, 2022, 08:46:00 AMOne other thing to look at is the web access log. See if any unusual activity corresponds to the spikes.
Good call! I haven't ever needed to look before, but just took a peak at some stats and the raw access log.
It's possible the spikes could be down to web crawlers. The robots/txt file has no disallows listed, so I've copied one over from another car forum I
manage** with a loooong list of disallows (that someone gave me!).
I'll monitor the spikes and access logs to see what difference that makes.
**manage sounds grand. In reality, all the admins gradually drifted away for two car forums I frequent. The other forum crashed and the community thought it was gone forever. I volunteered to take a look and was thrown the keys. By some miracle/fluke I managed to get it back and it's still running 8 years later with me as the sole admin :(
That led me to volunteer to look after the SMF site that is the subject of this topic, as there was no-one else with any web knowledge (although they were very good with spanners and welders ;D )
My only real coding experience is BBC Basic :-[
I have in fact had to shut down overaggressive crawlers multiple times. Poorly done, it's basically a ddos attack.
One was Tapatalk itself... Yes, they will crawl you. Frankly I would just get rid of it entirely.
Note that robots.txt provides guidance for those that follow that guidance. It is not mandatory, and bad actors just ignore it.
If you do find a bad actor, you may need to block it outright.
Have your issues been resolved? If so, please click the green 'Mark Topic Solved' button.
It looks like someone has ticked the box for me - which is fine, as the issue appears to be resolved :)
The problem appears to be caused by numerous crawlers deciding to target the forum. The access logs were stuffed with bot entries >:(
I did a check this morning and only ONE unwanted bot was active, and I've added that to the disallow list.
You can see the impact of the improved robots.txt file in this graph (cf graph in my first post). This graph cover the last 5 days.
Performance log 27 Jan -1 Feb 2022.JPG
I will continue to monitor, but all looking good.
Odd that the bots should suddenly seem to swarm and cause issues like this, but many thanks to the swift and great advice on here. Much appreciated.