Discuss mailing list

[Al Beano <albino AT autistici.org>]

What are the user agents on these requests?

On 5 October 2017 13:52:44 BST, Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:
>Possibly, but no single IP is making requests quite that fast.  What 
>they're doing is flooding me with requests from all over the place, and
>
>"they" seem to be a whole lot more active after I go to bed. Yesterday
>I 
>was seeing about 200 IPs per hour and around 50 queries per second.  As
>
>I wake up this morning I find over 750 IPs per hour and over 300
>queries 
>per second.  And I'm pretty sure opennic didn't just happen to pick up 
>that many new users overnight.
>
>On 10/04/2017 08:27 PM, Theo B. wrote:
>> Would it be possible to have a stacking rate limit per IP? For 
>> example, if an IP requests the list 10 times in a second, they get a 
>> 20 second rate limit, and if they keep requesting it gets higher?
>>
>> -Theo
>>
>>
>> On Oct 4, 2017, 10:21 PM -0400, Jeff Taylor <shdwdrgn AT sourpuss.net>, 
>> wrote:
>>> Oh yeah, forgot about that part.
>>>
>>> On 10/04/2017 06:50 PM, Jonah Aragon wrote:
>>>> It’s client side (javascript) for obvious reasons, so the API key 
>>>> would have to be embedded in the code which would kind of defeat
>the 
>>>> point. The browser is making the request, not the opennic.org 
>>>> <http://opennic.org> server.
>>>>
>>>> Jonah
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Oct 4, 2017, at 7:06 PM, Rouben <rouben AT rouben.net 
>>>> <mailto:rouben AT rouben.net>> wrote:
>>>>
>>>>> Not necessarily.... each individual or application could be issued
>
>>>>> an API key to use; www.opennic.org <http://www.opennic.org> 
>>>>> included. This has to be done for some API calls already anyway, 
>>>>> and is generally a good idea...
>>>>>
>>>>> On Wed, Oct 4, 2017 at 19:47 Jonah Aragon <jonah AT triplebit.net 
>>>>> <mailto:jonah AT triplebit.net>> wrote:
>>>>>
>>>>>     That would be unfortunate, it’d break the nearest servers list
>>>>>     on www.opennic.org <http://www.opennic.org>.
>>>>>
>>>>>     Jonah
>>>>>
>>>>>     Sent from my iPhone
>>>>>
>>>>>     On Oct 4, 2017, at 5:39 PM, Rouben <rouben AT rouben.net
>>>>>     <mailto:rouben AT rouben.net>> wrote:
>>>>>
>>>>>>     On second thought, a more practical option would be to change
>>>>>>     the geoip API to require authentication, similar to the BIND
>>>>>>     ACL api. That way at least you can determine the identity of
>>>>>>     the abuser and contact them, asking to correct the problem.
>>>>>>
>>>>>>
>>>>>>     Rouben
>>>>>>
>>>>>>     On Wed, Oct 4, 2017 at 6:02 PM, Jeff Taylor
>>>>>>     <shdwdrgn AT sourpuss.net <mailto:shdwdrgn AT sourpuss.net>> wrote:
>>>>>>
>>>>>>         Yeah there's plenty of options, and I actually use
>>>>>>         fail2ban on some of my other VMs, but I generally haven't
>>>>>>         had any problems with the apache servers.  It's not
>enough
>>>>>>         of a problem to require drastic measures yet, and I
>>>>>>         certainly don't want to go crazy with it and block
>>>>>>         legitimate lookups by opennic members, but I'm sort of
>>>>>>         stumped as to the source of this flood.  As I mentioned,
>>>>>>         they all have the same signature so it must be some sort
>>>>>>         of script or bot, and it has some minimal intelligence to
>>>>>>         it because the flood stopped as soon as I started
>>>>>>         returning unexpected answers... I wonder what sort of
>>>>>>         results I might see if I compared the IPs making these
>>>>>>         queries with a list of IPs sending email spam to my
>servers?
>>>>>>
>>>>>>         Anyway the only real problem here is the number of
>>>>>>         queries. I set up the VM with very low resources
>expecting
>>>>>>         only an occasional request for an API or the servers
>>>>>>         page.  The actual bandwidth used didn't even put a dent
>in
>>>>>>         my connection and I don't have metered traffic. I'll
>>>>>>         probably restart the VM tonight with more memory though
>>>>>>         just to handle the extra traffic and see how it does.
>>>>>>         Fortunately this VM runs on my biggest machine so I can
>>>>>>         throw a lot more resources at it as needed.
>>>>>>
>>>>>>
>>>>>>         On 10/04/2017 03:48 PM, Rouben wrote:
>>>>>>>         May I suggest using either
>>>>>>>        
>https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?
>>>>>>>         you'd need to get Apache 2.4, though, looks like you're
>>>>>>>         still on 2.2.
>>>>>>>
>>>>>>>         I'd also disable HTTP KeepAlive, since API calls by
>their
>>>>>>>         nature are atomic, and clients generally have no
>business
>>>>>>>         asking the server to keep the connection alive for a
>>>>>>>         single question-answer transaction typical of APIs.
>>>>>>>
>>>>>>>         I'd add also a second layer using IPTables, similar to
>>>>>>>         how the DoS is mitigated for OpenNIC DNS servers:
>>>>>>>
>>>>>>>         -p udp -m hashlimit --hashlimit-srcmask 24
>--hashlimit-mode srcip --hashlimit-upto 30/m --hashlimit-burst 10
>--hashlimit-name HTTPSTHROTTLE --dport 443 -j ACCEPT
>>>>>>>         -p udp -m udp --dport 53 -j DROP
>>>>>>>         Above rule adapted from
>>>>>>>         https://wiki.opennic.org/opennic/tier2security
>>>>>>>
>>>>>>>         Alternatively, perhaps fail2ban can automate the
>iptables
>>>>>>>         banning/unbanning based on a more sophisticated
>detection
>>>>>>>         rule:
>>>>>>>        
>https://www.maketecheasier.com/fail2ban-protect-apache-ddos/
>>>>>>>
>>>>>>>         I like layered security solutions... :) Apache can
>handle
>>>>>>>         the low-frequency "reasonable" DoS, and iptables can
>>>>>>>         handle the high-frequency heavy abuse that would be too
>>>>>>>         much for Apache (or even Varnish) to tackle.
>>>>>>>
>>>>>>>         Rouben
>>>>>>>
>>>>>>>         On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund
>>>>>>>         <deep.alexander AT gmail.com
>>>>>>>         <mailto:deep.alexander AT gmail.com>> wrote:
>>>>>>>
>>>>>>>             Have you considered putting Varnish in front of it?
>>>>>>>
>>>>>>>             Best regards
>>>>>>>             Alex
>>>>>>>
>>>>>>>             > On 4 Oct 2017, at 20:12, Jeff Taylor
>>>>>>>             <shdwdrgn AT sourpuss.net
>>>>>>>             <mailto:shdwdrgn AT sourpuss.net>> wrote:
>>>>>>>             >
>>>>>>>             > You may have noticed some issues reaching either
>>>>>>>             the API or servers page recently.  I've tracked down
>>>>>>>             the problem to some extremely excessive calls to the
>>>>>>>             geoip page (https://api.opennicproject.org/geoip/).
>>>>>>>             >
>>>>>>>             > If you are the owner of 208.82.39.26... your
>script
>>>>>>>             is doing lookups four times per second.  Just how
>>>>>>>             often do you think the list of servers changes?  I
>>>>>>>             blocked this IP completely for now, please fix your
>>>>>>>             script and let me know if you want access again.
>>>>>>>             >
>>>>>>>             > Of course this one user wasn't enough to bring the
>>>>>>>             server to its knees, this problem was because of yet
>>>>>>>             another script that seems to be getting shared
>around
>>>>>>>             the globe.  There are two aspects of the query that
>>>>>>>             lead me to believe there is a common script running
>here:
>>>>>>>             > "GET /geoip/?bare&pct=95 HTTP/1.1"
>>>>>>>             > "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0;
>>>>>>>             Touch; rv:11.0) like Gecko"
>>>>>>>             >
>>>>>>>             > I'm seeing well over 2000 unique IP addresses
>>>>>>>             making the same query up to once ever five seconds.
>>>>>>>             That translated to about 200 queries per second. 
>Now
>>>>>>>             the geoip page is rather expensive in terms of
>>>>>>>             resources, because it has to look up the user's IP
>>>>>>>             and try to match it geographically to the list of
>>>>>>>             Tier-2 servers.  I wrote up some code this morning
>to
>>>>>>>             cache the queries by IP address for 5 minutes before
>>>>>>>             re-checking.  Now this made a huge difference but
>>>>>>>             still wasn't enough.  I may have another bottleneck
>>>>>>>             in my network that was causing problems even with
>the
>>>>>>>             cached content so I'll be looking into that.
>>>>>>>             >
>>>>>>>             > In the meantime I've added a level of blocking for
>>>>>>>             any server making queries faster than every 15
>>>>>>>             seconds.  This will return a message warning the
>>>>>>>             requester that server information doesn't change
>that
>>>>>>>             fast, and doesn't give the expected reply.  I'm
>>>>>>>             hoping whoever set up this script will see broken
>>>>>>>             results and get it fixed.  At the moment this
>>>>>>>             15-second warning message is accounting for about
>25%
>>>>>>>             of all the queries.  I'll keep working on it, but
>>>>>>>             just wanted to let folks know WHY in case anyone
>>>>>>>             happens to see the warning message.
>>>>>>>             >
>>>>>>>             >
>>>>>>>             > --------
>>>>>>>             > You are a member of the OpenNIC Discuss list.
>>>>>>>             > You may unsubscribe by emailing
>>>>>>>             discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>            
><mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>             --------
>>>>>>>             You are a member of the OpenNIC Discuss list.
>>>>>>>             You may unsubscribe by emailing
>>>>>>>             discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>            
><mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         --------
>>>>>>>         You are a member of the OpenNIC Discuss list.
>>>>>>>         You may unsubscribe by
>emailingdiscuss-unsubscribe AT lists.opennicproject.org
>>>>>>>         <mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>         --------
>>>>>>         You are a member of the OpenNIC Discuss list.
>>>>>>         You may unsubscribe by emailing
>>>>>>         discuss-unsubscribe AT lists.opennicproject.org
>>>>>>         <mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>     --------
>>>>>>     You are a member of the OpenNIC Discuss list.
>>>>>>     You may unsubscribe by emailing
>>>>>>     discuss-unsubscribe AT lists.opennicproject.org
>>>>>>     <mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>
>>>>>
>>>>>     --------
>>>>>     You are a member of the OpenNIC Discuss list.
>>>>>     You may unsubscribe by emailing
>>>>>     discuss-unsubscribe AT lists.opennicproject.org
>>>>>     <mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>>
>>>>> --
>>>>>
>>>>> Rouben
>>>>>
>>>>>
>>>>> --------
>>>>> You are a member of the OpenNIC Discuss list.
>>>>> You may unsubscribe by emailing 
>>>>> discuss-unsubscribe AT lists.opennicproject.org 
>>>>> <mailto:discuss-unsubscribe AT lists.opennicproject.org>
>>>>
>>>>
>>>> --------
>>>> You are a member of the OpenNIC Discuss list.
>>>> You may unsubscribe by
>emailingdiscuss-unsubscribe AT lists.opennicproject.org
>>>
>>>
>>>
>>> --------
>>> You are a member of the OpenNIC Discuss list.
>>> You may unsubscribe by emailing 
>>> discuss-unsubscribe AT lists.opennicproject.org
>>
>>
>>
>> --------
>> You are a member of the OpenNIC Discuss list.
>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>
>
>
>------------------------------------------------------------------------
>
>
>
>--------
>You are a member of the OpenNIC Discuss list. 
>You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org