Discuss mailing list

[Verax <verax AT 8chan.co>]

Out of curiosity, what are you using to do the geoip lookups?  A
properly indexed version of the maxmind DB should be able to to process
hundreds of queries per second.

--Verax

Jon Hebb wrote:
> I think this is more than reasonable. Continued abuse even when getting
> the error message on a regular basis just means someone is intentionally
> trying to flood the server, or probably set some script and forgot about
> it. Either way, a ban is an easy enough way to fix that issue.
> 
> On Wed, Nov 1, 2017 at 3:28 PM, Jeff Taylor <shdwdrgn AT sourpuss.net
> <mailto:shdwdrgn AT sourpuss.net>> wrote:
> 
>     The API and the servers pages are run from a single VM machine that
>     doesn't expect a lot of traffic.  The servers page sees what I
>     consider 'typical' traffic, where people come in and check out the
>     information, browse around a bit, or perform updates to their own
>     server entries.  The API pages provide information that may
>     periodically change.  For instance, the ACL lists may change at
>     1-minute intervals but will generally see hours between any
>     updates.  Something like the geoip page will probably not see
>     changes for *days* at a time.
> 
>     Recently someone released what I imagine is a very poorly programmed
>     script which was absolutely flooding the PHP code used to compile
>     the information on the API pages, with queries as frequent as five
>     times per second.  I mean really, what does this idiot hope to
>     achieve with such frequent queries for the same information?  As a
>     result, access to the servers page has been very slow or dropped out
>     completely at times, so in October I added new code to cache the
>     replies and rate-limit the queries and send a warning message if you
>     exceeded a certain threshold.  Of course the caching greatly reduced
>     the load on the script, and once they started receiving invalid
>     information in the form of the warning message most of the bots
>     completely dropped out.  Unfortunately there always has to be that
>     one guy...
> 
>     Since at least October 4th, someone at 208.82.39.26 has been running
>     a script with the following query:
>     /geoip/?ip=174.49.73.80&pct=95&list&res=4&nearest&noscript
>     What this means is they are doing a lookup for servers nearest to
>     the location of the IP at 174.49.73.80, and they apparently have no
>     idea what they are requesting because 'noscript' and 'nearest'
>     aren't even recognized parameters.
> 
>     This query has been coming consistently at a rate of more than once
>     per second, which means they haven't received a single valid reply
>     since I implemented the rate-limiting warning messages.  Because
>     this person obviously isn't paying attention to what is happening,
>     I've completely blocked this IP at the firewall, so they will have
>     no access to either the API or servers pages.  Any such activity I
>     see which is impacting access by opennic members will be considered
>     abusive and will be shut down without notice.  For reference, the
>     'normal' traffic I see for geoip lookups amounts to around five
>     queries per *minute*, globally.  Any time a single IP exceeds the
>     amount of queries that the rest of the world combined is performing,
>     I will look at it with suspicion.
> 
>     For anyone wishing to actually implement a script to query the geoip
>     information, please have some realistic expectations.  You could ask
>     for updates once per hour and at reboots, and you would still have a
>     reasonably accurate and usable list of nearby DNS servers to query
>     from.  Of course as mentioned above some of the API pages will have
>     more frequent updates.  The ACL list provides a timestamp to signal
>     when you should grab a new copy, but checking that list once a
>     minute is expected.
> 
>     Note that bandwidth is NOT an issue here!  Despite the abusive
>     traffic, the combination of traffic for all of the services I run on
>     my network is using less than 1/50th of my available bandwidth.  The
>     problem lies in features like the geoip page which requires a lot of
>     calculations to generate the information, and each IP address has to
>     be resolved individually.  I could further optimize the caching and
>     such, but I think the real problem lies in people who are using
>     opennic services for nefarious purposes.  We already know someone
>     has been using opennic and .bit domains to spread malware, and I've
>     seen many suggestions over the years that opennic DNS servers are
>     being used to perform lookups by spam bots.  I believe the recent
>     large number of queries to the geoip page were being performed for a
>     similar purpose (the agent string and specific query were identical
>     for each, but coming from IPs all around the globe).  I'll continue
>     to keep an eye on the traffic though and will only block access when
>     there is a clear case to do so.
> 
> 
> 
>     --------
>     You are a member of the OpenNIC Discuss list.
>     You may unsubscribe by emailing
>     discuss-unsubscribe AT lists.opennicproject.org
>     <mailto:discuss-unsubscribe AT lists.opennicproject.org>
> 
> 
> 
> 
> -- 
> Best Regards,
> Jon Hebb
> 
> 
> 
> 
> --------
> You are a member of the OpenNIC Discuss list. 
> You may unsubscribe by emailing discuss-unsubscribe AT lists.opennicproject.org
>