Discuss mailing list

[Jonah Aragon <jonah AT triplebit.net>]

Nobody here likes _javascript_ or analytics for some reason (paranoia?) so there’s no good way for us to check that out. I sort of doubt just normal web traffic to this one page is causing that level of load but I’m not sure how many visitors we actually get.

Jonah 

Sent from my iPhone

On Oct 5, 2017, at 7:57 AM, Rouben <rouben AT rouben.net> wrote:

Hmmm... any correlation with the website? As mentioned on this thread, it invokes the API via client-side _javascript_.

On Thu, Oct 5, 2017 at 08:53 Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:

Possibly, but no single IP is making requests quite that fast.  What they're doing is flooding me with requests from all over the place, and "they" seem to be a whole lot more active after I go to bed.  Yesterday I was seeing about 200 IPs per hour and around 50 queries per second.  As I wake up this morning I find over 750 IPs per hour and over 300 queries per second.  And I'm pretty sure opennic didn't just happen to pick up that many new users overnight.

On 10/04/2017 08:27 PM, Theo B. wrote:

Would it be possible to have a stacking rate limit per IP? For example, if an IP requests the list 10 times in a second, they get a 20 second rate limit, and if they keep requesting it gets higher?

-Theo

On Oct 4, 2017, 10:21 PM -0400, Jeff Taylor <shdwdrgn AT sourpuss.net>, wrote:

Oh yeah, forgot about that part.

On 10/04/2017 06:50 PM, Jonah Aragon wrote:

It’s client side (_javascript_) for obvious reasons, so the API key would have to be embedded in the code which would kind of defeat the point. The browser is making the request, not the opennic.org server. 

Jonah

Sent from my iPhone

On Oct 4, 2017, at 7:06 PM, Rouben <rouben AT rouben.net> wrote:

Not necessarily.... each individual or application could be issued an API key to use; www.opennic.org included. This has to be done for some API calls already anyway, and is generally a good idea...

On Wed, Oct 4, 2017 at 19:47 Jonah Aragon <jonah AT triplebit.net> wrote:

That would be unfortunate, it’d break the nearest servers list on www.opennic.org. 

Jonah

Sent from my iPhone

On Oct 4, 2017, at 5:39 PM, Rouben <rouben AT rouben.net> wrote:

On second thought, a more practical option would be to change the geoip API to require authentication, similar to the BIND ACL api. That way at least you can determine the identity of the abuser and contact them, asking to correct the problem.

Rouben

On Wed, Oct 4, 2017 at 6:02 PM, Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:

Yeah there's plenty of options, and I actually use fail2ban on some of my other VMs, but I generally haven't had any problems with the apache servers.  It's not enough of a problem to require drastic measures yet, and I certainly don't want to go crazy with it and block legitimate lookups by opennic members, but I'm sort of stumped as to the source of this flood.  As I mentioned, they all have the same signature so it must be some sort of script or bot, and it has some minimal intelligence to it because the flood stopped as soon as I started returning unexpected answers... I wonder what sort of results I might see if I compared the IPs making these queries with a list of IPs sending email spam to my servers?

Anyway the only real problem here is the number of queries.  I set up the VM with very low resources expecting only an occasional request for an API or the servers page.  The actual bandwidth used didn't even put a dent in my connection and I don't have metered traffic.  I'll probably restart the VM tonight with more memory though just to handle the extra traffic and see how it does.  Fortunately this VM runs on my biggest machine so I can throw a lot more resources at it as needed.

On 10/04/2017 03:48 PM, Rouben wrote:

May I suggest using either

https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?

you'd need to get Apache 2.4, though, looks like you're still on 2.2.

I'd also disable HTTP KeepAlive, since API calls by their nature are atomic, and clients generally have no business asking the server to keep the connection alive for a single question-answer transaction typical of APIs.

I'd add also a second layer using IPTables, similar to how the DoS is mitigated for OpenNIC DNS servers:

-p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode srcip --hashlimit-upto 30/m --hashlimit-burst 10 --hashlimit-name HTTPSTHROTTLE --dport 443 -j ACCEPT
-p udp -m udp --dport 53 -j DROP

Above rule adapted from https://wiki.opennic.org/opennic/tier2security

Alternatively, perhaps fail2ban can automate the iptables banning/unbanning based on a more sophisticated detection rule:

https://www.maketecheasier.com/fail2ban-protect-apache-ddos/

I like layered security solutions... :) Apache can handle the low-frequency "reasonable" DoS, and iptables can handle the high-frequency heavy abuse that would be too much for Apache (or even Varnish) to tackle.

Rouben

On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund <deep.alexander AT gmail.com> wrote:

Have you considered putting Varnish in front of it?

Best regards
Alex

> On 4 Oct 2017, at 20:12, Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:
>
> You may have noticed some issues reaching either the API or servers page recently.  I've tracked down the problem to some extremely excessive calls to the geoip page (https://api.opennicproject.org/geoip/).
>
> If you are the owner of 208.82.39.26... your script is doing lookups four times per second.  Just how often do you think the list of servers changes?  I blocked this IP completely for now, please fix your script and let me know if you want access again.
>
> Of course this one user wasn't enough to bring the server to its knees, this problem was because of yet another script that seems to be getting shared around the globe.  There are two aspects of the query that lead me to believe there is a common script running here:
> "GET /geoip/?bare&pct=95 HTTP/1.1"
> "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
>
> I'm seeing well over 2000 unique IP addresses making the same query up to once ever five seconds.  That translated to about 200 queries per second.  Now the geoip page is rather expensive in terms of resources, because it has to look up the user's IP and try to match it geographically to the list of Tier-2 servers.  I wrote up some code this morning to cache the queries by IP address for 5 minutes before re-checking.  Now this made a huge difference but still wasn't enough.  I may have another bottleneck in my network that was causing problems even with the cached content so I'll be looking into that.
>
> In the meantime I've added a level of blocking for any server making queries faster than every 15 seconds.  This will return a message warning the requester that server information doesn't change that fast, and doesn't give the expected reply.  I'm hoping whoever set up this script will see broken results and get it fixed.  At the moment this 15-second warning message is accounting for about 25% of all the queries.  I'll keep working on it, but just wanted to let folks know WHY in case anyone happens to see the warning message.
>
>
> --------
> You are a member of the OpenNIC Discuss list.
> You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org



--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.  
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--

Rouben

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.  
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list. 
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org

--

Rouben

--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org