Discuss mailing list

[Jeff Taylor <shdwdrgn AT sourpuss.net>]

Yeah there's plenty of options, and I actually use fail2ban on some of my other VMs, but I generally haven't had any problems with the apache servers.  It's not enough of a problem to require drastic measures yet, and I certainly don't want to go crazy with it and block legitimate lookups by opennic members, but I'm sort of stumped as to the source of this flood.  As I mentioned, they all have the same signature so it must be some sort of script or bot, and it has some minimal intelligence to it because the flood stopped as soon as I started returning unexpected answers... I wonder what sort of results I might see if I compared the IPs making these queries with a list of IPs sending email spam to my servers?

Anyway the only real problem here is the number of queries.  I set up the VM with very low resources expecting only an occasional request for an API or the servers page.  The actual bandwidth used didn't even put a dent in my connection and I don't have metered traffic.  I'll probably restart the VM tonight with more memory though just to handle the extra traffic and see how it does.  Fortunately this VM runs on my biggest machine so I can throw a lot more resources at it as needed.

On 10/04/2017 03:48 PM, Rouben wrote:

May I suggest using either

https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?

you'd need to get Apache 2.4, though, looks like you're still on 2.2.

I'd also disable HTTP KeepAlive, since API calls by their nature are atomic, and clients generally have no business asking the server to keep the connection alive for a single question-answer transaction typical of APIs.

I'd add also a second layer using IPTables, similar to how the DoS is mitigated for OpenNIC DNS servers:

-p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode srcip --hashlimit-upto 30/m --hashlimit-burst 10 --hashlimit-name HTTPSTHROTTLE --dport 443 -j ACCEPT
-p udp -m udp --dport 53 -j DROP

Above rule adapted from https://wiki.opennic.org/opennic/tier2security

Alternatively, perhaps fail2ban can automate the iptables banning/unbanning based on a more sophisticated detection rule:

https://www.maketecheasier.com/fail2ban-protect-apache-ddos/

I like layered security solutions... :) Apache can handle the low-frequency "reasonable" DoS, and iptables can handle the high-frequency heavy abuse that would be too much for Apache (or even Varnish) to tackle.

Rouben

On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund <deep.alexander AT gmail.com> wrote:
Have you considered putting Varnish in front of it?

Best regards
Alex

> On 4 Oct 2017, at 20:12, Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:
>
> You may have noticed some issues reaching either the API or servers page recently.  I've tracked down the problem to some extremely excessive calls to the geoip page (https://api.opennicproject.org/geoip/).
>
> If you are the owner of 208.82.39.26... your script is doing lookups four times per second.  Just how often do you think the list of servers changes?  I blocked this IP completely for now, please fix your script and let me know if you want access again.
>
> Of course this one user wasn't enough to bring the server to its knees, this problem was because of yet another script that seems to be getting shared around the globe.  There are two aspects of the query that lead me to believe there is a common script running here:
> "GET /geoip/?bare&pct=95 HTTP/1.1"
> "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
>
> I'm seeing well over 2000 unique IP addresses making the same query up to once ever five seconds.  That translated to about 200 queries per second.  Now the geoip page is rather expensive in terms of resources, because it has to look up the user's IP and try to match it geographically to the list of Tier-2 servers.  I wrote up some code this morning to cache the queries by IP address for 5 minutes before re-checking.  Now this made a huge difference but still wasn't enough.  I may have another bottleneck in my network that was causing problems even with the cached content so I'll be looking into that.
>
> In the meantime I've added a level of blocking for any server making queries faster than every 15 seconds.  This will return a message warning the requester that server information doesn't change that fast, and doesn't give the expected reply.  I'm hoping whoever set up this script will see broken results and get it fixed.  At the moment this 15-second warning message is accounting for about 25% of all the queries.  I'll keep working on it, but just wanted to let folks know WHY in case anyone happens to see the warning message.
>
>
> --------
> You are a member of the OpenNIC Discuss list.
> You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org



--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list. 
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org