Discuss mailing list

[Al Beano <albino AT autistici.org>]

Don't need to use JS analytics, just feed the web server logs into some kind 
of script. there are plenty online. 

Doesn't the request on the website set a specific user-agent or other header 
anyway?

albino

On 5 October 2017 14:26:04 BST, Jonah Aragon <jonah AT triplebit.net> wrote:
>Nobody here likes JavaScript or analytics for some reason (paranoia?)
>so there’s no good way for us to check that out. I sort of doubt just
>normal web traffic to this one page is causing that level of load but
>I’m not sure how many visitors we actually get. 
>
>Jonah 
>
>Sent from my iPhone
>
>> On Oct 5, 2017, at 7:57 AM, Rouben <rouben AT rouben.net> wrote:
>> 
>> Hmmm... any correlation with the website? As mentioned on this
>thread, it invokes the API via client-side JavaScript.
>> 
>>> On Thu, Oct 5, 2017 at 08:53 Jeff Taylor <shdwdrgn AT sourpuss.net>
>wrote:
>>> Possibly, but no single IP is making requests quite that fast.  What
>they're doing is flooding me with requests from all over the place, and
>"they" seem to be a whole lot more active after I go to bed.  Yesterday
>I was seeing about 200 IPs per hour and around 50 queries per second. 
>As I wake up this morning I find over 750 IPs per hour and over 300
>queries per second.  And I'm pretty sure opennic didn't just happen to
>pick up that many new users overnight.
>>> 
>>> 
>>>> On 10/04/2017 08:27 PM, Theo B. wrote:
>>>> Would it be possible to have a stacking rate limit per IP? For
>example, if an IP requests the list 10 times in a second, they get a 20
>second rate limit, and if they keep requesting it gets higher?
>>>> 
>>>> -Theo
>>>> 
>>>> 
>>>>> On Oct 4, 2017, 10:21 PM -0400, Jeff Taylor
><shdwdrgn AT sourpuss.net>, wrote:
>>>>> Oh yeah, forgot about that part.
>>>>> 
>>>>>> On 10/04/2017 06:50 PM, Jonah Aragon wrote:
>>>>>> It’s client side (javascript) for obvious reasons, so the API key
>would have to be embedded in the code which would kind of defeat the
>point. The browser is making the request, not the opennic.org server. 
>>>>>> 
>>>>>> Jonah
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>> On Oct 4, 2017, at 7:06 PM, Rouben <rouben AT rouben.net> wrote:
>>>>>> 
>>>>>>> Not necessarily.... each individual or application could be
>issued an API key to use; www.opennic.org included. This has to be done
>for some API calls already anyway, and is generally a good idea...
>>>>>>> 
>>>>>>>> On Wed, Oct 4, 2017 at 19:47 Jonah Aragon <jonah AT triplebit.net>
>wrote:
>>>>>>>> That would be unfortunate, it’d break the nearest servers list
>on www.opennic.org. 
>>>>>>>> 
>>>>>>>> Jonah
>>>>>>>> 
>>>>>>>> Sent from my iPhone
>>>>>>>> 
>>>>>>>> On Oct 4, 2017, at 5:39 PM, Rouben <rouben AT rouben.net> wrote:
>>>>>>>> 
>>>>>>>>> On second thought, a more practical option would be to change
>the geoip API to require authentication, similar to the BIND ACL api.
>That way at least you can determine the identity of the abuser and
>contact them, asking to correct the problem.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Rouben
>>>>>>>>> 
>>>>>>>>> On Wed, Oct 4, 2017 at 6:02 PM, Jeff Taylor
><shdwdrgn AT sourpuss.net> wrote:
>>>>>>>>>> Yeah there's plenty of options, and I actually use fail2ban
>on some of my other VMs, but I generally haven't had any problems with
>the apache servers.  It's not enough of a problem to require drastic
>measures yet, and I certainly don't want to go crazy with it and block
>legitimate lookups by opennic members, but I'm sort of stumped as to
>the source of this flood.  As I mentioned, they all have the same
>signature so it must be some sort of script or bot, and it has some
>minimal intelligence to it because the flood stopped as soon as I
>started returning unexpected answers... I wonder what sort of results I
>might see if I compared the IPs making these queries with a list of IPs
>sending email spam to my servers?
>>>>>>>>>> 
>>>>>>>>>> Anyway the only real problem here is the number of queries. 
>I set up the VM with very low resources expecting only an occasional
>request for an API or the servers page.  The actual bandwidth used
>didn't even put a dent in my connection and I don't have metered
>traffic.  I'll probably restart the VM tonight with more memory though
>just to handle the extra traffic and see how it does.  Fortunately this
>VM runs on my biggest machine so I can throw a lot more resources at it
>as                                         needed.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On 10/04/2017 03:48 PM, Rouben wrote:
>>>>>>>>>>> May I suggest using either
>>>>>>>>>>> https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?
>>>>>>>>>>> you'd need to get Apache 2.4, though, looks like you're
>still on 2.2.
>>>>>>>>>>> 
>>>>>>>>>>> I'd also disable HTTP KeepAlive, since API calls by their
>nature are atomic, and clients generally have no business asking the
>server to keep the connection alive for a single question-answer
>transaction typical of APIs.
>>>>>>>>>>> 
>>>>>>>>>>> I'd add also a second layer using IPTables, similar to how
>the DoS is mitigated for OpenNIC DNS servers:
>>>>>>>>>>> 
>>>>>>>>>>> -p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode
>srcip --hashlimit-upto 30/m --hashlimit-burst 10 --hashlimit-name
>HTTPSTHROTTLE --dport 443 -j ACCEPT
>>>>>>>>>>> -p udp -m udp --dport 53 -j DROP
>>>>>>>>>>> Above rule adapted from
>https://wiki.opennic.org/opennic/tier2security
>>>>>>>>>>> 
>>>>>>>>>>> Alternatively, perhaps fail2ban can automate the iptables
>banning/unbanning based on a more sophisticated detection rule:
>>>>>>>>>>> https://www.maketecheasier.com/fail2ban-protect-apache-ddos/
>>>>>>>>>>> 
>>>>>>>>>>> I like layered security solutions... :) Apache can handle
>the low-frequency "reasonable" DoS, and iptables can handle the
>high-frequency heavy abuse that would be too much for Apache (or even
>Varnish) to tackle.
>>>>>>>>>>> 
>>>>>>>>>>> Rouben
>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund
><deep.alexander AT gmail.com> wrote:
>>>>>>>>>>>> Have you considered putting Varnish in front of it?
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards
>>>>>>>>>>>> Alex
>>>>>>>>>>>> 
>>>>>>>>>>>> > On 4 Oct 2017, at 20:12, Jeff Taylor
><shdwdrgn AT sourpuss.net> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > You may have noticed some issues reaching either the API
>or servers page recently.  I've tracked down the problem to some
>extremely excessive calls to the geoip page
>(https://api.opennicproject.org/geoip/).
>>>>>>>>>>>> >
>>>>>>>>>>>> > If you are the owner of 208.82.39.26... your script is
>doing lookups four times per second.  Just how often do you think the
>list of servers changes?  I blocked this IP completely for now, please
>fix your script and let me know if you want access again.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Of course this one user wasn't enough to bring the server
>to its knees, this problem was because of yet another script that seems
>to be getting shared around the globe.  There are two aspects of the
>query that lead me to believe there is a common script running here:
>>>>>>>>>>>> > "GET /geoip/?bare&pct=95 HTTP/1.1"
>>>>>>>>>>>> > "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch;
>rv:11.0) like Gecko"
>>>>>>>>>>>> >
>>>>>>>>>>>> > I'm seeing well over 2000 unique IP addresses making the
>same query up to once ever five seconds.                               
>That translated to about 200 queries per second.  Now the geoip page is
>rather expensive in terms of resources, because it has to look up the
>user's IP and try to match it geographically to the list of Tier-2
>servers.  I wrote up some code this morning to cache the queries by IP
>address for 5 minutes before re-checking.  Now this made a huge
>difference but still wasn't enough.  I may have another bottleneck in
>my network that was causing problems even with the cached content so
>I'll be looking into that.
>>>>>>>>>>>> >
>>>>>>>>>>>> > In the meantime I've added a level of blocking for any
>server making queries faster than every 15 seconds.  This will return a
>message                                                   warning the
>requester that server information doesn't change that fast, and doesn't
>give the expected reply.  I'm hoping whoever set up this script will
>see broken results and get it fixed.  At the moment this 15-second     
>warning message is accounting for about 25% of all the queries.  I'll
>keep working on it, but just wanted to let folks know WHY in case
>anyone happens to see the warning message.
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > --------
>>>>>>>>>>>> > You are a member of the OpenNIC Discuss list.
>>>>>>>>>>>> > You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --------
>>>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --------
>>>>>>>>>>> You are a member of the OpenNIC Discuss list.  
>>>>>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --------
>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --------
>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --------
>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>>> 
>>>>>>> --
>>>>>>> 
>>>>>>> Rouben
>>>>>>> 
>>>>>>> 
>>>>>>> --------
>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>>> 
>>>>>> 
>>>>>> --------
>>>>>> You are a member of the OpenNIC Discuss list.  
>>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>>> 
>>>>> 
>>>>> 
>>>>> --------
>>>>> You are a member of the OpenNIC Discuss list.
>>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>>> 
>>>> 
>>>> --------
>>>> You are a member of the OpenNIC Discuss list. 
>>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>>> 
>>> 
>>> 
>>> --------
>>> You are a member of the OpenNIC Discuss list.
>>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>> 
>> -- 
>> 
>> Rouben
>> 
>> 
>> --------
>> You are a member of the OpenNIC Discuss list. 
>> You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org
>
>
>------------------------------------------------------------------------
>
>
>
>--------
>You are a member of the OpenNIC Discuss list. 
>You may unsubscribe by emailing
>discuss-unsubscribe AT lists.opennicproject.org