Dear all,
Forbes has a very annoying page: when you get the url of an article of Forbes, you do not read the article web-page, but land in a page, where you have to click on a link, AND THEN are directed automatically to the actual article page.
We never thought that we would consider installing such a mechanism. But in the last weeks we were flooded by Chinese and other spammers, who spider the forum and abuse its contents on their web-farms. 2 Chinese internet access providers are responsible for over 500000 forum reads in just 2 weeks. This is a severe abuse, and we are facing problems with our web-hoster.
The abuse is much worse, because the attackers show up with up to 1000 diffferent IPs simultaneously. So we can not block their IPs. The Chinese were easy to block, but a cloud of IPs we can not handle.
So we need some mechanism to separate spiders from real humans. But we were not able to find such a mechanism in the forum. Did we miss something?
If someone knows a way to handle the problem, we would be very happy.
Regards,
Walrus
Having the landing page doesn't really protect you significantly in terms of bandwidth etc.
Something like Cloudflare is probably a better bet, and you can just block all the traffic from China and be done with it.
As far as I understand, Cloudflare is a commercial company. But we are guests on a webspace given to us by friends.
As I said: We can not block all the various IPs, because they do not come from distinguishable internet access providers. There are hundreds of simultaneous read requests. We think they are from hijacked PCs. There is no other explanation because of the sequence of requests.
If the spiders were hindered to access the forum pages, after some time they would give up, because they can no get what the want to harvest.
We do know that the additional layer is annoying, but we think that human guests of the forum will understand the matter.
That's why you use something like CloudFlare, by way of them having a list of IP addresses linked to a given country.
And no, I guarantee you that for the bots to give up in the fashion you're talking about, it will take months for them to actually notice.
Cloudflare does have a free plan which I believe will give some of the blocking features. Give it a try.
To me Cloudflare just looks line one more parasite. Just like the anti-virus companies.
There must be better things than those gangs which just drain money.
We have a small and tough forum, and we are guests. We can not put the burden of attacks onto our generous hosts. we have to do it our way.
Then I wish you the very best possible luck.
Quote from: vbgamer45 on September 04, 2019, 04:23:34 PM
Cloudflare does have a free plan which I believe will give some of the blocking features. Give it a try.
That looks so at the first sight. But we have 30 to 50 GB traffic per month. This is the region where Cloudflare DOES want money. We encountered other "free" offers which are much too small, like for 10 web pages, or something like that. But we already outgrew that size.
You can do geoip solution with apache or other firewall.
Quote from: vbgamer45 on September 04, 2019, 04:35:29 PM
You can do geoip solution with apache or other firewall.
This will not work, because they come from all over the world via hijacked PCs. That is the problem. They behave like ordinary guests, which - in fact - they are not.
So we need a means to make guests undergo a manual procedure, just like Forbes. As I said: We do not like it that way, but we see no other chance.
The entry page, even if hit by hundreds of thousands of requests, is so small in size that it will save more than 95 percent of the traffic. The forum pages easily exceed 300 to 500 kB.
I use recaptcha mod only. No smf security questions no nothing. Yet I have no spammers. Tho, i see them sign up but they dont post.
Quote from: Mick. on September 04, 2019, 04:48:43 PM
I use recaptcha mod only. No smf security questions no nothing. Yet I have no spammers. Tho, i see them sign up but they dont post.
Those spammers who want to register in the forum, do not succeed.
Our problem is GUESTS. They are not registered. They simply drain traffic. This is why we need something to force guests to undergo a manual procedure. But we do not see any way to do that.
Quote from: Mick. on September 04, 2019, 04:48:43 PM
I use recaptcha mod only. No smf security questions no nothing. Yet I have no spammers. Tho, i see them sign up but they dont post.
I have to add something about the term "spammers":
The spammers that we encounter are NOT those which place crappy forum posts. The spammers we encounter read forum pages, then shred them to particles of some sentences, and then mount those particles with particles taken from other web-sites in new web-pages the fill their web-sites with: hundreds of thousands of pages.
This way they abuse the original writers of the original web-pages, attract search engines, and pull the major traffic of real people to their sites.
This has a dramatic effect:
1. The huge number of targets for search engines leads to only a fraction of the surfers go and read the original pages. The original pages simply drown in an ocean of crap. This is a deadly means to suppress unwanted web-sites.
2. The search engines might consider the original sites as paying partners of spammers with the INTENT of flooding the search engines - and so the search engines will drop the original sites in ranking. This, too, is a deadly means to suppress unwanted web-sites.
The waves of attack show up suddenly, exceed HUNDREDS of read requests per 15 minutes, and disappear as fast as they began. Which definitely is not the usual traffic caused by humans living in other time zones. Such traffic goes up slowly, and fades away, on a daily basis.
Also, very peculiar: the requests are not for the latest forum entries, but for stuff several years old. Humans would rather read new topics.
You can try mod_evasive on apache might help just adjust the settings.
If "mod-evasive" is something to be installed on the server: we have no means to do that. We can only modify the forum software. Access to the server itself we do not have.
Quote from: walrus2019 on September 04, 2019, 05:37:40 PM
If "mod-evasive" is something to be installed on the server: we have no means to do that. We can only modify the forum software. Access to the server itself we do not have.
it is. could be installed though you might be able to control though .htaccess or contact your host.
Quote from: vbgamer45 on September 04, 2019, 05:48:18 PM
it is. could be installed though you might be able to control though .htaccess or contact your host.
I have no idea what you mean. Where is that modification? How can it be controlled by .htaccess?
https://www.linode.com/docs/web-servers/apache-tips-and-tricks/modevasive-on-apache/
Quote from: vbgamer45 on September 04, 2019, 06:06:32 PM
https://www.linode.com/docs/web-servers/apache-tips-and-tricks/modevasive-on-apache/
Thank you, but we have no access to that. Our hosts only can work on a cpanel. so they have no access to the server itself.
You can use the SMF GeoIP (https://github.com/Spuds/SMF-GeoIP) modification written by a former SMF Member...
It needs a bit of updating, but there are here people around who can help you with that...
Quote from: Herman's Mixen on September 04, 2019, 06:36:17 PM
You can use the SMF GeoIP (https://github.com/Spuds/SMF-GeoIP) modification written by a former SMF Member...
It needs a bit of updating, but there are here people around who can help you with that...
NO! WE CAN NOT! The spammers come from all over the globe. So it is NOT possible to use ANYTHING related with "geo" or alike.
As I wrote: The spammers use hijacked PCs an com from thousands of IPs all over the globe.
so you can not use of instal anything, well you be doomed by those hijackers ;)
if you cannot install stuff or your host cant cause its a reseller, or some weird provider.. you should switch hosts !
Do not shout in posts like this, this will not help you anyway.
so your research yourself what you will doing about this one then :P
Quote from: Herman's Mixen on September 04, 2019, 06:51:09 PM
so you can not use of instal anything, well you be doomed by those hijackers ;)
if you cannot install stuff or your host cant cause its a reseller, or some weird provider.. you should switch hosts !
Do not shout in posts like this, this will not help you anyway.
so your research yourself what you will doing about this one then :P
There is no reason for us to do so. We want an additional intermediate page which forces the surfers to some manual action. This is enough to reduce the load and it will frighten the spammers.
Simply adding a new pageload will not work, it will add to your current bandwidth usage, and do nothing much to fight the issue. You could try this though https://custom.simplemachines.org/mods/index.php?mod=2155
Quote from: Aleksi "Lex" Kilpinen on September 04, 2019, 10:48:15 PM
Simply adding a new pageload will not work, it will add to your current bandwidth usage, and do nothing much to fight the issue. You could try this though https://custom.simplemachines.org/mods/index.php?mod=2155
1. The total traffic is NOT the sum of ALL incoming requests and of the pages the surfers WANT. If instead of giving them a page sized 300 to 500 kB they only get some few kB, the traffic load is reduced by over 90 percent.
Only those who manually go on will receive the full size of the web page. The traffic saved will be BY FAR more than 95 percent.
2. The honeypot stuff and the blacklists are a) useless and are b) abused by frauds who enter persons IDs in faked entries. We strongly advice NOT to use such lists, because in effect they support criminals.
Again: We need an additional page that must be passed by the surfers with a manual entry. This will block ALL automatic requests to the forum and lets through only real persons.
Maybe an analogy may help?
Let's assume you live in a house, which is on a narrow street. The street is a public road.
Now there are thousand of people that for some reason want to look at your house. Mind you: they do not get onto your lawn and they don't get in.
Due to the masses of people in front of your house your guests cannot get through, or with difficulty.
What you are now asking is to somehow block visitors from accessing your street so they look at the house, but at the same time you state that you are in no position to prevent them from entering the street in front of your house... there really is no solution for that.
Putting a page (large fence) in front of your site (house) does not block the street: people will still block street, but instead of looking at the house they will looking at the fence, but the street is still blocked...
If you do not want guests to pull a lot of traffic: close your forum for guests, that way the traffic will go down... that's as close to a fence that you will get.
at least, that's my opinion...
Your information on how things work overall seems to be pretty much incorrect in almost every detail...
You've been given really good suggestions, but you refuse every one with excuses based on incorrect information.... however, I'm sure that you know best (although, if so, why even bother coming here to ask the question?)
1. Please do note that we are online for over 2 decades now.
2. You have absolutely no clues either about traffic nor about how spammers work now about internals of maintaining a forum.
We need help by people who REALLY do know the materia.
without access to your server control panel, there is very little that can be done to even attempt to fix the issues you are having.
you need to obtain that access then you can look into getting someone to fix the issues.
Well, you've been told by people who've been running forums longer than you, and you've been told by people who have literally been on the SMF dev team at this point.
Since you clearly know better than us, why do you still want our opinion or help?
Quote from: Arantor on September 05, 2019, 09:43:13 AM
Well, you've been told by people who've been running forums longer than you, and you've been told by people who have literally been on the SMF dev team at this point.
Since you clearly know better than us, why do you still want our opinion or help?
1. You do not know who we are.
2. I explained the situation as precise as possible. If you do not understand it, I am sorry, this is not our fault.
As I said: We need an entry page, because that is the only means to counter the wild hordes attacks. Anything else does not work - for a number of reasons.
Quote from: walrus2019 on September 05, 2019, 09:49:22 AM
As I said: We need an entry page, because that is the only means to counter the wild hordes attacks. Anything else does not work - for a number of reasons.
So you have found YOUR solution yes ?
Oh, I understand what you think you want and why you think you want it. I just don't think it will actually fix your problem.
But again, I'm only literally a former developer of this software, and you have deliberately misconstrued my point: you are coming here and telling us we need to build you a solution to your problem, as though you are the only site to ever have had this problem in the history of the world. You are not a unique and beautiful snowflake, this is a problem that happens every day and the suggestions made were entirely the correct ones.
But since you don't trust the views of people who have been on the dev team, why do you trust the software they helped make?
You know best what you can do we have come up with alot of suggestions and idea's but you do deny on them... well build your custom page and you will see that, that doesn't help you
Quote from: Arantor on September 05, 2019, 09:55:13 AM
Oh, I understand what you think you want and why you think you want it. I just don't think it will actually fix your problem.
But again, I'm only literally a former developer of this software, and you have deliberately misconstrued my point: you are coming here and telling us we need to build you a solution to your problem, as though you are the only site to ever have had this problem in the history of the world. You are not a unique and beautiful snowflake, this is a problem that happens every day and the suggestions made were entirely the correct ones.
1. That you wrote a forum script does not automatically mean that you understand the situation we are faced with.
2. We are unique. That you can be sure of.
Quote from: Arantor on September 05, 2019, 09:55:13 AM
But since you don't trust the views of people who have been on the dev team, why do you trust the software they helped make?
That is totally irrelevant. There is a big difference in writing a forum script and in maintaining a forum, especially a forum like ours. 99,999 percent of people are not able to maintain a forum. The "trends" in the media show this day for day. We are not in the media business, but we very well could be...
Quote from: Herman's Mixen on September 05, 2019, 10:02:00 AM
You know best what you can do we have come up with alot of suggestions and idea's but you do deny on them... well build your custom page and you will see that, that doesn't help you
Well, since Forbes is not the only company which has an intermediate page we of course assume that OTHER FORUMS might, too, have installed means like intermediate pages. This is all we are looking for. It is a simple construction. But we do OUR part of the job, and that is maintaining forums in hurricanes.
Maybe it's time you stopped freeloading off your friend and got a real server because hundreds of requests in 15 minutes should not be a problem even on a modest VPS (I reckon with proper tuning you could do it for $20 a month on a self managed VPS)
Mind you I'm only looking after sites for small players in their field, getting only up to 10k requests a minute, or the one governmental agency that requires 15 web servers to serve the load, I'm sure I'll never hit the level of busy you seem to have.
As for the protection of content, the quickest route is to simply make everyone log in. Achieves the same as your front page with the same level of inconvenience to your actual users.
Forbes do it for other reasons, not just to deal with the issue you seem to have (like making sure you see an ad to cover their costs)
Quote from: Arantor on September 05, 2019, 10:12:08 AM
Maybe it's time you stopped freeloading off your friend and got a real server because hundreds of requests in 15 minutes should not be a problem even on a modest VPS (I reckon with proper tuning you could do it for $20 a month on a self managed VPS)
Mind you I'm only looking after sites for small players in their field, getting only up to 10k requests a minute, or the one governmental agency that requires 15 web servers to serve the load, I'm sure I'll never hit the level of busy you seem to have.
As for the protection of content, the quickest route is to simply make everyone log in. Achieves the same as your front page with the same level of inconvenience to your actual users.
Forbes do it for other reasons, not just to deal with the issue you seem to have (like making sure you see an ad to cover their costs)
As I wrote: We are only guests, and we want to stay that way. The forum is a small one. The wave of requests is artificial and in in no way related with the normal life of the forum.
We do not make any SEO stuff. We are not commercial. So, Forbes may have some financial ideas in their heads on WHY to make that entry page. But we have different reasons.
And your reasons as they are don't make sense.
A simple added step. Does literally next to nothing to slow down any modern bot. But - worst case scenario, can be used to bring your forum down.
Quote from: Aleksi "Lex" Kilpinen on September 05, 2019, 10:33:12 AM
And your reasons as they are don't make sense.
A simple added step. Does literally next to nothing to slow down any modern bot. But - worst case scenario, can be used to bring your forum down.
It is very easy to understand: The entry pages must contain a simple question. Bots will not be able to pass that stage.
Believe me, you are wrong. Sure, it'll work for a day, perhaps a week, but you have added a function that adds to the load when they try, and they do learn. I too run a forum, one that started in 2005, with some 3+ million posts, and have shared your frustration with crawlers - until I decided to simply not give a fudge. It's not like they affect my VPS much.
Quote from: Aleksi "Lex" Kilpinen on September 05, 2019, 10:52:41 AM
Believe me, you are wrong. Sure, it'll work for a day, perhaps a week, but you have added a function that adds to the load when they try, and they do learn. I too run a forum, one that started in 2005, with some 3+ million posts, and have shared your frustration with crawlers - until I decided to simply not give a fudge. It's not like they affect my VPS much.
We have some "special" enemies, and some "special" spammers. Of course they will learn. But it will not help them. We have some very agile mods.
Incidentally, before you denigrate the team further and claim that we don't know what we are talking about, be aware that I have been running a forum since dial-up days -- since 1988 to be exact.
Quote from: Kindred on September 05, 2019, 12:07:31 PM
Incidentally, before you denigrate the team further and claim that we don't know what we are talking about, be aware that I have been running a forum since dial-up days -- since 1988 to be exact.
1. But still NO ONE understood how the mechanisms of a forum work. Writing a script is not the same as steering a forum through a hurricane. That is worlds apart.
2. We do not run a gossip forum. As I said: We are unique.
The discussion is boring. I will not respond anymore unless some useful hint show up.
see.... I really doubt that you are as unique as you believe...
and we do fully understand the mechanisms of designing, implementing AND running a forum.
I don't run a "gossip" forum either...
You came with a preconceived notion.
Your notion, as it turns out, is pretty much wrong... and this has been pointed out to you by people with more experience in every area of forum-ness.
Believe me or not -- it is the plain truth and your insistence on "only my thought is the correct way" is nothing more than pure mule-headedness.
You asked for a new feature.
Sorry, but what you asked for
1- will not accomplish what you think it will
2- even if it did, would not be useful to 99.9999% of the universe of forums (actually, I would argue that it would be actually damaging to most, if not every, forum site.)
3- because of 1 and 2 -- Will not be implemented as a feature.
If you are willing to pay someone to design it specially for you -- you might find someone who is willing to do it. Post in the Help Wanted (not for support) board as a PAID request.
Honestly I'm tempted to do it to just be proven right at the end that it won't solve the problem, but that's more effort than I can really be bothered.
As it seems what you want is Cloudflare without Cloudflare, I don't think we can help you further. I have moved this topic to Applied or Declined Requests (https://www.simplemachines.org/community/index.php?board=38.0) as a declined request.
If you wish to seek this further, I suggest you follow Kindred's advice and seek paid help with this.