Discuss mailing list

[Jonah Aragon <jonah AT triplebit.net>]

Maybe I don’t know enough about JavaScript but I have no idea why it would 
change the user-agent. I’d imagine it’d just send the browser’s default 
user-agent. 

The homepage is hosted at Netlify which has worked great for us but doesn’t 
exactly give us server logs. I’d like to use Piwik Analytics for the 
homepage, wiki, and servers page, but based on reactions from some people 
here I suspect even Do Not Track respecting, IP obfuscating, first-party 
JavaScript hosted by us would still be looked down upon :(

To be fair though we clearly have JavaScript on the page anyways, so maybe 
we’ll go for that route regardless. 

Jonah

Sent from my iPhone

> On Oct 5, 2017, at 8:28 AM, Al Beano <albino AT autistici.org> wrote:
> 
> Don't need to use JS analytics, just feed the web server logs into some 
> kind of script. there are plenty online. 
> 
> Doesn't the request on the website set a specific user-agent or other 
> header anyway?
> 
> albino
> 
>> On 5 October 2017 14:26:04 BST, Jonah Aragon <jonah AT triplebit.net> wrote:
>> Nobody here likes JavaScript or analytics for some reason (paranoia?)
>> so there’s no good way for us to check that out. I sort of doubt just
>> normal web traffic to this one page is causing that level of load but
>> I’m not sure how many visitors we actually get. 
>> 
>> Jonah 
>> 
>> Sent from my iPhone
>> 
>>> On Oct 5, 2017, at 7:57 AM, Rouben <rouben AT rouben.net> wrote:
>>> 
>>> Hmmm... any correlation with the website? As mentioned on this
>> thread, it invokes the API via client-side JavaScript.
>>> 
>>>> On Thu, Oct 5, 2017 at 08:53 Jeff Taylor <shdwdrgn AT sourpuss.net>
>> wrote:
>>>> Possibly, but no single IP is making requests quite that fast.  What
>> they're doing is flooding me with requests from all over the place, and
>> "they" seem to be a whole lot more active after I go to bed.  Yesterday
>> I was seeing about 200 IPs per hour and around 50 queries per second. 
>> As I wake up this morning I find over 750 IPs per hour and over 300
>> queries per second.  And I'm pretty sure opennic didn't just happen to
>> pick up that many new users overnight.
>>>> 
>>>> 
>>>>> On 10/04/2017 08:27 PM, Theo B. wrote:
>>>>> Would it be possible to have a stacking rate limit per IP? For
>> example, if an IP requests the list 10 times in a second, they get a 20
>> second rate limit, and if they keep requesting it gets higher?
>>>>> 
>>>>> -Theo
>>>>> 
>>>>> 
>>>>>> On Oct 4, 2017, 10:21 PM -0400, Jeff Taylor
>> <shdwdrgn AT sourpuss.net>, wrote:
>>>>>> Oh yeah, forgot about that part.
>>>>>> 
>>>>>>> On 10/04/2017 06:50 PM, Jonah Aragon wrote:
>>>>>>> It’s client side (javascript) for obvious reasons, so the API key
>> would have to be embedded in the code which would kind of defeat the
>> point. The browser is making the request, not the opennic.org server. 
>>>>>>> 
>>>>>>> Jonah
>>>>>>> 
>>>>>>> Sent from my iPhone
>>>>>>> 
>>>>>>>> On Oct 4, 2017, at 7:06 PM, Rouben <rouben AT rouben.net> wrote:
>>>>>>>> 
>>>>>>>> Not necessarily.... each individual or application could be
>> issued an API key to use; www.opennic.org included. This has to be done
>> for some API calls already anyway, and is generally a good idea...
>>>>>>>> 
>>>>>>>>> On Wed, Oct 4, 2017 at 19:47 Jonah Aragon <jonah AT triplebit.net>
>> wrote:
>>>>>>>>> That would be unfortunate, it’d break the nearest servers list
>> on www.opennic.org. 
>>>>>>>>> 
>>>>>>>>> Jonah
>>>>>>>>> 
>>>>>>>>> Sent from my iPhone
>>>>>>>>> 
>>>>>>>>>> On Oct 4, 2017, at 5:39 PM, Rouben <rouben AT rouben.net> wrote:
>>>>>>>>>> 
>>>>>>>>>> On second thought, a more practical option would be to change
>> the geoip API to require authentication, similar to the BIND ACL api.
>> That way at least you can determine the identity of the abuser and
>> contact them, asking to correct the problem.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Rouben
>>>>>>>>>> 
>>>>>>>>>> On Wed, Oct 4, 2017 at 6:02 PM, Jeff Taylor
>> <shdwdrgn AT sourpuss.net> wrote:
>>>>>>>>>>> Yeah there's plenty of options, and I actually use fail2ban
>> on some of my other VMs, but I generally haven't had any problems with
>> the apache servers.  It's not enough of a problem to require drastic
>> measures yet, and I certainly don't want to go crazy with it and block
>> legitimate lookups by opennic members, but I'm sort of stumped as to
>> the source of this flood.  As I mentioned, they all have the same
>> signature so it must be some sort of script or bot, and it has some
>> minimal intelligence to it because the flood stopped as soon as I
>> started returning unexpected answers... I wonder what sort of results I
>> might see if I compared the IPs making these queries with a list of IPs
>> sending email spam to my servers?
>>>>>>>>>>> 
>>>>>>>>>>> Anyway the only real problem here is the number of queries. 
>> I set up the VM with very low resources expecting only an occasional
>> request for an API or the servers page.  The actual bandwidth used
>> didn't even put a dent in my connection and I don't have metered
>> traffic.  I'll probably restart the VM tonight with more memory though
>> just to handle the extra traffic and see how it does.  Fortunately this
>> VM runs on my biggest machine so I can throw a lot more resources at it
>> as                                         needed.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On 10/04/2017 03:48 PM, Rouben wrote:
>>>>>>>>>>>> May I suggest using either
>>>>>>>>>>>> https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?
>>>>>>>>>>>> you'd need to get Apache 2.4, though, looks like you're
>> still on 2.2.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'd also disable HTTP KeepAlive, since API calls by their
>> nature are atomic, and clients generally have no business asking the
>> server to keep the connection alive for a single question-answer
>> transaction typical of APIs.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'd add also a second layer using IPTables, similar to how
>> the DoS is mitigated for OpenNIC DNS servers:
>>>>>>>>>>>> 
>>>>>>>>>>>> -p udp -m hashlimit --hashlimit-srcmask 24 --hashlimit-mode
>> srcip --hashlimit-upto 30/m --hashlimit-burst 10 --hashlimit-name
>> HTTPSTHROTTLE --dport 443 -j ACCEPT
>>>>>>>>>>>> -p udp -m udp --dport 53 -j DROP
>>>>>>>>>>>> Above rule adapted from
>> https://wiki.opennic.org/opennic/tier2security
>>>>>>>>>>>> 
>>>>>>>>>>>> Alternatively, perhaps fail2ban can automate the iptables
>> banning/unbanning based on a more sophisticated detection rule:
>>>>>>>>>>>> https://www.maketecheasier.com/fail2ban-protect-apache-ddos/
>>>>>>>>>>>> 
>>>>>>>>>>>> I like layered security solutions... :) Apache can handle
>> the low-frequency "reasonable" DoS, and iptables can handle the
>> high-frequency heavy abuse that would be too much for Apache (or even
>> Varnish) to tackle.
>>>>>>>>>>>> 
>>>>>>>>>>>> Rouben
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund
>> <deep.alexander AT gmail.com> wrote:
>>>>>>>>>>>>> Have you considered putting Varnish in front of it?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>> Alex
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 4 Oct 2017, at 20:12, Jeff Taylor
>> <shdwdrgn AT sourpuss.net> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> You may have noticed some issues reaching either the API
>> or servers page recently.  I've tracked down the problem to some
>> extremely excessive calls to the geoip page
>> (https://api.opennicproject.org/geoip/).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If you are the owner of 208.82.39.26... your script is
>> doing lookups four times per second.  Just how often do you think the
>> list of servers changes?  I blocked this IP completely for now, please
>> fix your script and let me know if you want access again.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Of course this one user wasn't enough to bring the server
>> to its knees, this problem was because of yet another script that seems
>> to be getting shared around the globe.  There are two aspects of the
>> query that lead me to believe there is a common script running here:
>>>>>>>>>>>>>> "GET /geoip/?bare&pct=95 HTTP/1.1"
>>>>>>>>>>>>>> "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch;
>> rv:11.0) like Gecko"
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm seeing well over 2000 unique IP addresses making the
>> same query up to once ever five seconds.                               
>> That translated to about 200 queries per second.  Now the geoip page is
>> rather expensive in terms of resources, because it has to look up the
>> user's IP and try to match it geographically to the list of Tier-2
>> servers.  I wrote up some code this morning to cache the queries by IP
>> address for 5 minutes before re-checking.  Now this made a huge
>> difference but still wasn't enough.  I may have another bottleneck in
>> my network that was causing problems even with the cached content so
>> I'll be looking into that.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In the meantime I've added a level of blocking for any
>> server making queries faster than every 15 seconds.  This will return a
>> message                                                   warning the
>> requester that server information doesn't change that fast, and doesn't
>> give the expected reply.  I'm hoping whoever set up this script will
>> see broken results and get it fixed.  At the moment this 15-second     
>> warning message is accounting for about 25% of all the queries.  I'll
>> keep working on it, but just wanted to let folks know WHY in case
>> anyone happens to see the warning message.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --------
>>>>>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --------
>>>>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --------
>>>>>>>>>>>> You are a member of the OpenNIC Discuss list.  
>>>>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --------
>>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --------
>>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --------
>>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>>> 
>>>>>>>> --
>>>>>>>> 
>>>>>>>> Rouben
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --------
>>>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>>> 
>>>>>>> 
>>>>>>> --------
>>>>>>> You are a member of the OpenNIC Discuss list.  
>>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --------
>>>>>> You are a member of the OpenNIC Discuss list.
>>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>>> 
>>>>> 
>>>>> --------
>>>>> You are a member of the OpenNIC Discuss list. 
>>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>>> 
>>>> 
>>>> 
>>>> --------
>>>> You are a member of the OpenNIC Discuss list.
>>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>>> 
>>> -- 
>>> 
>>> Rouben
>>> 
>>> 
>>> --------
>>> You are a member of the OpenNIC Discuss list. 
>>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
>> 
>> 
>> ------------------------------------------------------------------------
>> 
>> 
>> 
>> --------
>> You are a member of the OpenNIC Discuss list. 
>> You may unsubscribe by emailing
>> discuss-unsubscribe AT lists.opennicproject.org
> 
> 
> 
> --------
> You are a member of the OpenNIC Discuss list. 
> You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org