The agent string for all of these requests is "Mozilla/5.0 (Windows
NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko"
They also always make the same request "?bare&pct=95"
I would have to agree that I don't think the _javascript_ code is
changing the agent string, there would simply be no point of it.
I added a progressive rate limiter to the code now. I may have been
too generous with it, but I'll wait and see. With the traffic rate
this morning I just went ahead and blocked that one query completely
for a couple hours, then it slowed down to the more normal rate of
around 20 queries/second. I've removed the block so I'm waiting for
the traffic to pick up again, then I'll fine-tune the progression
rate. Basically it doubles the expected wait period for every query
made (up to a point), so after you make a query you need to wait 1
second before the next, the 2 seconds, then 4 seconds. It stores
your activity over the past 2 minutes. There's probably a sweet
spot in there for the fastest continuous rate you can query, but I'm
too lazy to work it out.
On 10/05/2017 07:25 AM, Al Beano wrote:
What are the user agents on these requests?
On 5 October 2017 13:52:44 BST, Jeff Taylor <shdwdrgn AT sourpuss.net> wrote:
Possibly, but no single IP is making requests quite that fast. What
they're doing is flooding me with requests from all over the place, and
"they" seem to be a whole lot more active after I go to bed. Yesterday
I
was seeing about 200 IPs per hour and around 50 queries per second. As
I wake up this morning I find over 750 IPs per hour and over 300
queries
per second. And I'm pretty sure opennic didn't just happen to pick up
that many new users overnight.
On 10/04/2017 08:27 PM, Theo B. wrote:
Would it be possible to have a stacking rate limit per IP? For
example, if an IP requests the list 10 times in a second, they get a
20 second rate limit, and if they keep requesting it gets higher?
-Theo
On Oct 4, 2017, 10:21 PM -0400, Jeff Taylor <shdwdrgn AT sourpuss.net>,
wrote:
Oh yeah, forgot about that part.
On 10/04/2017 06:50 PM, Jonah Aragon wrote:
It’s client side (_javascript_) for obvious reasons, so the API key
would have to be embedded in the code which would kind of defeat
the
point. The browser is making the request, not the opennic.org
<http://opennic.org> server.
Jonah
Sent from my iPhone
On Oct 4, 2017, at 7:06 PM, Rouben <rouben AT rouben.net
<mailto:rouben AT rouben.net>> wrote:
Not necessarily.... each individual or application could be issued
an API key to use; www.opennic.org <http://www.opennic.org>
included. This has to be done for some API calls already anyway,
and is generally a good idea...
On Wed, Oct 4, 2017 at 19:47 Jonah Aragon <jonah AT triplebit.net
<mailto:jonah AT triplebit.net>> wrote:
That would be unfortunate, it’d break the nearest servers list
on www.opennic.org <http://www.opennic.org>.
Jonah
Sent from my iPhone
On Oct 4, 2017, at 5:39 PM, Rouben <rouben AT rouben.net
<mailto:rouben AT rouben.net>> wrote:
On second thought, a more practical option would be to change
the geoip API to require authentication, similar to the BIND
ACL api. That way at least you can determine the identity of
the abuser and contact them, asking to correct the problem.
Rouben
On Wed, Oct 4, 2017 at 6:02 PM, Jeff Taylor
<shdwdrgn AT sourpuss.net <mailto:shdwdrgn AT sourpuss.net>> wrote:
Yeah there's plenty of options, and I actually use
fail2ban on some of my other VMs, but I generally haven't
had any problems with the apache servers. It's not
enough
of a problem to require drastic measures yet, and I
certainly don't want to go crazy with it and block
legitimate lookups by opennic members, but I'm sort of
stumped as to the source of this flood. As I mentioned,
they all have the same signature so it must be some sort
of script or bot, and it has some minimal intelligence to
it because the flood stopped as soon as I started
returning unexpected answers... I wonder what sort of
results I might see if I compared the IPs making these
queries with a list of IPs sending email spam to my
servers?
Anyway the only real problem here is the number of
queries. I set up the VM with very low resources
expecting
only an occasional request for an API or the servers
page. The actual bandwidth used didn't even put a dent
in
my connection and I don't have metered traffic. I'll
probably restart the VM tonight with more memory though
just to handle the extra traffic and see how it does.
Fortunately this VM runs on my biggest machine so I can
throw a lot more resources at it as needed.
On 10/04/2017 03:48 PM, Rouben wrote:
May I suggest using either
https://httpd.apache.org/docs/trunk/mod/mod_ratelimit.html ?
you'd need to get Apache 2.4, though, looks like you're
still on 2.2.
I'd also disable HTTP KeepAlive, since API calls by
their
nature are atomic, and clients generally have no
business
asking the server to keep the connection alive for a
single question-answer transaction typical of APIs.
I'd add also a second layer using IPTables, similar to
how the DoS is mitigated for OpenNIC DNS servers:
-p udp -m hashlimit --hashlimit-srcmask 24
--hashlimit-mode srcip --hashlimit-upto 30/m --hashlimit-burst 10
--hashlimit-name HTTPSTHROTTLE --dport 443 -j ACCEPT
-p udp -m udp --dport 53 -j DROP
Above rule adapted from
https://wiki.opennic.org/opennic/tier2security
Alternatively, perhaps fail2ban can automate the
iptables
banning/unbanning based on a more sophisticated
detection
rule:
https://www.maketecheasier.com/fail2ban-protect-apache-ddos/
I like layered security solutions... :) Apache can
handle
the low-frequency "reasonable" DoS, and iptables can
handle the high-frequency heavy abuse that would be too
much for Apache (or even Varnish) to tackle.
Rouben
On Wed, Oct 4, 2017 at 4:09 PM, Alex Nordlund
<deep.alexander AT gmail.com
<mailto:deep.alexander AT gmail.com>> wrote:
Have you considered putting Varnish in front of it?
Best regards
Alex
> On 4 Oct 2017, at 20:12, Jeff Taylor
<shdwdrgn AT sourpuss.net
<mailto:shdwdrgn AT sourpuss.net>> wrote:
>
> You may have noticed some issues reaching either
the API or servers page recently. I've tracked down
the problem to some extremely excessive calls to the
geoip page (https://api.opennicproject.org/geoip/).
>
> If you are the owner of 208.82.39.26... your
script
is doing lookups four times per second. Just how
often do you think the list of servers changes? I
blocked this IP completely for now, please fix your
script and let me know if you want access again.
>
> Of course this one user wasn't enough to bring the
server to its knees, this problem was because of yet
another script that seems to be getting shared
around
the globe. There are two aspects of the query that
lead me to believe there is a common script running
here:
> "GET /geoip/?bare&pct=95 HTTP/1.1"
> "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0;
Touch; rv:11.0) like Gecko"
>
> I'm seeing well over 2000 unique IP addresses
making the same query up to once ever five seconds.
That translated to about 200 queries per second.
Now
the geoip page is rather expensive in terms of
resources, because it has to look up the user's IP
and try to match it geographically to the list of
Tier-2 servers. I wrote up some code this morning
to
cache the queries by IP address for 5 minutes before
re-checking. Now this made a huge difference but
still wasn't enough. I may have another bottleneck
in my network that was causing problems even with
the
cached content so I'll be looking into that.
>
> In the meantime I've added a level of blocking for
any server making queries faster than every 15
seconds. This will return a message warning the
requester that server information doesn't change
that
fast, and doesn't give the expected reply. I'm
hoping whoever set up this script will see broken
results and get it fixed. At the moment this
15-second warning message is accounting for about
25%
of all the queries. I'll keep working on it, but
just wanted to let folks know WHY in case anyone
happens to see the warning message.
>
>
> --------
> You are a member of the OpenNIC Discuss list.
> You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by
emailingdiscuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--
Rouben
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
<mailto:discuss-unsubscribe AT lists.opennicproject.org>
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by
emailingdiscuss-unsubscribe AT lists.opennicproject.org
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
------------------------------------------------------------------------
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing
discuss-unsubscribe AT lists.opennicproject.org
--------
You are a member of the OpenNIC Discuss list.
You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org
|
