{ "info": { "author": "Dmitry Fillo", "author_email": "fillo@fillo.me", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Topic :: Internet :: Name Service (DNS)", "Topic :: Utilities" ], "description": "==================\nBerserker Resolver\n==================\n\n.. image:: https://travis-ci.org/DmitryFillo/berserker_resolver.svg?branch=master\n :target: https://travis-ci.org/DmitryFillo/berserker_resolver\n.. image:: https://coveralls.io/repos/DmitryFillo/berserker_resolver/badge.svg?branch=master&service=github\n :target: https://coveralls.io/github/DmitryFillo/berserker_resolver?branch=master\n\nFast mass dns resolver which can bypass loadbalancers.\n\n.. contents::\n\nMotivation\n==========\n\nDNS servers can provide load balancing of many types. It can be simple round-robin or another algorithm that\ndepends on the implementation of a particular DNS server. See `RFC 1794 `_ \nto understand the capabilities and flexibility of the DNS protocol. As a result, it is possible that when the ordinary\nresolver is not able to get *all* IP addresses, e.g. DNS server can return small sets of IP addresses (or even only one IP)\nwith low TTL per request. Number and presence of addresses may vary depending on the load of servers. Important to note\nthat this information can be cached by other DNS servers and can be distorted. Getting all IP addresses problem arises when\nyou want to resolve many domains with sufficient accuracy which useful for blocking network purposes, e.g. websites filtering,\nparental controls, etc.\n\nLet's try to make a couple probes for www.twitter.com.\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @8.8.8.8 www.twitter.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.twitter.com.\t593\tIN\tCNAME\ttwitter.com.\n twitter.com.\t\t23\tIN\tA\t199.16.156.70\n twitter.com.\t\t23\tIN\tA\t199.16.156.198\n twitter.com.\t\t23\tIN\tA\t199.16.156.102\n twitter.com.\t\t23\tIN\tA\t199.16.156.6\n ;; Query time: 5 msec\n ;; SERVER: 8.8.8.8#53(8.8.8.8)\n ;; WHEN: Sat Aug 08 04:01:43 MSK 2015\n ;; MSG SIZE rcvd: 122\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @8.8.8.8 www.twitter.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.twitter.com.\t596\tIN\tCNAME\ttwitter.com.\n twitter.com.\t\t26\tIN\tA\t199.16.156.230\n twitter.com.\t\t26\tIN\tA\t199.16.156.70\n twitter.com.\t\t26\tIN\tA\t199.16.156.198\n twitter.com.\t\t26\tIN\tA\t199.16.156.102\n ;; Query time: 4 msec\n ;; SERVER: 8.8.8.8#53(8.8.8.8)\n ;; WHEN: Sat Aug 08 04:01:44 MSK 2015\n ;; MSG SIZE rcvd: 122\n\nYou see different RR sets with small TTL. What about another public DNS?\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @77.88.8.8 www.twitter.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.twitter.com.\t30\tIN\tCNAME\ttwitter.com.\n twitter.com.\t\t30\tIN\tA\t199.16.156.102\n twitter.com.\t\t30\tIN\tA\t199.16.156.6\n twitter.com.\t\t30\tIN\tA\t199.16.156.38\n twitter.com.\t\t30\tIN\tA\t199.16.156.230\n ;; Query time: 2 msec\n ;; SERVER: 77.88.8.8#53(77.88.8.8)\n ;; WHEN: Sat Aug 08 04:04:05 MSK 2015\n ;; MSG SIZE rcvd: 111\n\nLet's try www.youtube.com.\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @8.8.8.8 www.youtube.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.youtube.com.\t21313\tIN\tCNAME\tyoutube-ui.l.google.com.\n youtube-ui.l.google.com. 13\tIN\tCNAME\twide-youtube.l.google.com.\n wide-youtube.l.google.com. 13\tIN\tA\t74.125.143.198\n ;; Query time: 5 msec\n ;; SERVER: 8.8.8.8#53(8.8.8.8)\n ;; WHEN: Sat Aug 08 04:06:08 MSK 2015\n ;; MSG SIZE rcvd: 121\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @8.8.8.8 www.youtube.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.youtube.com.\t21599\tIN\tCNAME\tyoutube-ui.l.google.com.\n youtube-ui.l.google.com. 299\tIN\tCNAME\twide-youtube.l.google.com.\n wide-youtube.l.google.com. 299\tIN\tA\t173.194.71.198\n ;; Query time: 6 msec\n ;; SERVER: 8.8.8.8#53(8.8.8.8)\n ;; WHEN: Sat Aug 08 04:06:11 MSK 2015\n ;; MSG SIZE rcvd: 121\n\n.. code:: bash\n\n ; <<>> DiG 9.10.2-P1 <<>> @84.200.70.40 www.youtube.com +nocomments +noquestion\n ; (1 server found)\n ;; global options: +cmd\n www.youtube.com.\t55513\tIN\tCNAME\tyoutube-ui.l.google.com.\n youtube-ui.l.google.com. 271\tIN\tA\t216.58.209.46\n ;; Query time: 41 msec\n ;; SERVER: 84.200.70.40#53(84.200.70.40)\n ;; WHEN: Sat Aug 08 04:07:29 MSK 2015\n ;; MSG SIZE rcvd: 94\n\nThis outputs may be outdated soon, but it is only necessary to show the behavior of DNS. Any website can use\nload balancer and so you are not able to do full resolve these sites.\n\nThe solution is query many nameservers many times for each domain. Yes, it's a bit clumsy, but works well enough\nin many cases. The resolving should be performed in multiple threads, because resolving in one thread is slow,\nespecially in this case.\n\nAnd so Berserker Resolver is emerged.\n\n*It's worth noting that full resolving may be impossible because GEO load balancing or resolving can be occurred \n\"at the wrong time in the wrong place\" when some servers are down and their IP addresses are excluded from DNS pool by fault\ntolerance algorithm. If you need actual information you should schedule resolving attempts, maintain your DNS database,\nmaybe perform resolving from different networks/servers. There is no universal solution for that cases, but you can use Berserker\nResolver as the backend in your application.*\n\nQuery backend\n=============\n\nBerserker Resolver is using `dnspython `_ as query backend and so operates with its built-in types.\n\nSupported versions\n==================\n\n* Python 2.6\n* Python 2.7\n* Python 3.2\n* Python 3.3\n* Python 3.4\n\nInstallation\n============\n\nThe best way::\n\n pip install berserker_resolver\n\nResolver class\n==============\n\nCore of the Berserker Resolver.\n\nMethods:\n\n+ resolve\n+ query\n\nProperties:\n\n+ nameservers\n+ tries\n+ timeout\n+ qname\n+ threads\n+ www\n+ www_combine\n+ verbose\n\nProperties can be assign via constructor or directly to the object.\n\nResolver.resolve\n----------------\n\nResolve method. It takes list of domains and returns dictionary with results (dictionary of sets).\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['kernel.org', 'toster.ru']\n\n resolver = Resolver()\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'toster.ru': {\n \n },\n 'kernel.org': {\n ,\n ,\n \n }\n }\n '''\n\nResolver.query\n--------------\n\nQuery method, wrapper around ``dns.resolver.Resolver.query`` from dnspython. It takes domain and nameserver,\nand returns result of the query. Nameserver is optional, if not given, random from ``Resolver.nameservers``\nwill be used.\n\nCan throw exception, see details `here `_.\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n resolver = Resolver()\n\n result = resolver.query('facebook.com')\n print(list(result)) # []\n\n # Query to the local dns.\n result = resolver.query('facebook.com', '127.0.0.1')\n print(list(result)) # []\n\nResolver.nameservers\n--------------------\n\nList of nameservers for resolving, each of them will be queried for particular domain.\n\nThe larger the list, the more chances to get all IP addresses, but it increases\ntime needed for resolving.\n\nDefault is ``['8.8.8.8', '8.8.4.4', '77.88.8.8', '77.88.8.1', '84.200.69.80', '84.200.70.40']``. There are\n`Google Public DNS `_, `Yandex.DNS `_\nand `DNS.WATCH `_.\n\nResolver.tries\n--------------\n\nNumber of queries for each nameserver.\n\nIf the number of times increases, the resolving accuracy increases too, but it also\nincreases time to resolving.\n\nDefault is ``48``.\n\nResolver.timeout\n----------------\n\nThe total number of seconds to spend trying to get an answer to the query.\n\nNote that low timeout combined with high values of ``Resolver.tries`` and ``Resolver.threads`` can lead to\nnumerous timeout errors when nameserver does not have enough time to return a response.\n\nDefault is ``3``.\n\nResolver.threads\n----------------\n\nNumber of threads.\n\nMore threads lead to increase speed of resolving, but too many threads lead to threads switching overhead.\nYou should test different numbers and choose one suitable for your systems. Also be careful with large number of threads, you can\nflood the DNS server. If you want to use crazy large amount of threads, check\n`stackoverflow thread `_ and\nincrease ``Resolver.timeout``.\n\nDefault is ``512``.\n\nResolver.qname\n--------------\n\nDNS query type name.\n\nDefault is ``A``.\n\nResolver.www\n------------\n\nEnables automatic addition/removal of *www* prefix depending on the domain.\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['wikipedia.org', 'www.toster.ru']\n\n resolver = Resolver(www=True)\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'toster.ru': {\n \n },\n 'www.wikipedia.org': {\n \n },\n 'www.toster.ru': {\n \n },\n 'wikipedia.org': {\n \n }\n }\n '''\n\nDefault is ``False``.\n\nResolver.www_combine\n--------------------\n\nEnables automatic combining *www* prefixed domains with theirs non-*www* versions.\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['facebook.com', 'www.facebook.com']\n\n resolver = Resolver()\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'facebook.com': {\n \n },\n 'www.facebook.com': {\n ,\n ,\n ,\n \n }\n }\n '''\n\n resolver.www_combine = True\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'www.facebook.com': {\n \n ,\n ,\n ,\n \n }\n }\n '''\n\nPowerful use case is combine this property together with ``Resolver.www``.\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['facebook.com']\n\n resolver = Resolver(www=True, www_combine=True)\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'www.facebook.com': {\n \n ,\n ,\n ,\n \n }\n }\n '''\n\nDefault is ``False``.\n\nResolver.verbose\n----------------\n\nThis property enables error reporting, e.g. nxdomain, noanswer, etc. ``Resolver.resolve`` normally returns\ndictionary of sets with resolved domains, but with this option it returns dictionary with two keys:\n\n+ success\n+ error\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['nonexistent.domain', 'facebook.com']\n\n resolver = Resolver(verbose=True)\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'success': {\n 'facebook.com': {\n \n }\n },\n 'error': {\n 'nonexistent.domain': {\n '77.88.8.1': NXDOMAIN(),\n '8.8.4.4': NXDOMAIN(),\n '8.8.8.8': NXDOMAIN(),\n '77.88.8.8': NXDOMAIN()\n }\n }\n }\n '''\n\n``result['success']`` is dictionary with successfully resolved domains, as if without ``Resolver.verbose``.\n``result['error']`` is dictionary with unsuccessfully resolved domains where each key contains another dictionary\nwith per nameserver exception. Exceptions comes from dnspython backend as ``dns.exception.DNSException`` subclasses.\nCheck out `dnspython docs `_ for more\ninformation about built-in exceptions.\n\n*Note that particular domain can be placed in both dictionaries, because some nameservers can return answer and some not.*\n\n.. code:: python\n\n from berserker_resolver import Resolver\n\n domains = ['facebook.com']\n\n # 216.239.32.10 is ns1.google.com\n resolver = Resolver(nameservers=['216.239.32.10', '8.8.8.8'], verbose=True)\n result = resolver.resolve(domains)\n\n print(result)\n '''\n {\n 'success': {\n 'facebook.com': {\n \n }\n },\n 'error': {\n 'facebook.com': {\n '216.239.32.10': NoNameservers()\n }\n }\n }\n '''\n\nDefault is ``False``.", "description_content_type": null, "docs_url": null, "download_url": null, "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/DmitryFillo/berserker_resolver", "keywords": "dns resolver berserker loadbalancer", "license": "BSD", "maintainer": null, "maintainer_email": null, "name": "berserker_resolver", "package_url": "https://pypi.org/project/berserker_resolver/", "platform": "any", "project_url": "https://pypi.org/project/berserker_resolver/", "project_urls": { "Homepage": "https://github.com/DmitryFillo/berserker_resolver" }, "release_url": "https://pypi.org/project/berserker_resolver/2.0.1/", "requires_dist": null, "requires_python": null, "summary": "Fast mass dns resolver which can bypass loadbalancers", "version": "2.0.1" }, "last_serial": 1669190, "releases": { "0.0.3": [ { "comment_text": "", "digests": { "md5": "c44f8519ff96fcb2d610cd0188065c67", "sha256": "f9d6664d8ccc50afb1e70806866c99240c28da3554125d22864cf931519bbb7c" }, "downloads": -1, "filename": "berserker_resolver-0.0.3.tar.gz", "has_sig": false, "md5_digest": "c44f8519ff96fcb2d610cd0188065c67", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4838, "upload_time": "2014-11-20T04:34:29", "url": "https://files.pythonhosted.org/packages/09/30/16366e869813831a5c5aebafe4a9b1a14a00a095ae3000d42d892bd900ae/berserker_resolver-0.0.3.tar.gz" } ], "1.0.3": [ { "comment_text": "", "digests": { "md5": "64024dfde0dba527bde930da202f050f", "sha256": "8fa4215ac377b7752657373e5b613c03bfc15bb14347a0f1d1a031c5c5bc9333" }, "downloads": -1, "filename": "berserker_resolver-1.0.3.tar.gz", "has_sig": false, "md5_digest": "64024dfde0dba527bde930da202f050f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5865, "upload_time": "2014-11-25T17:01:10", "url": "https://files.pythonhosted.org/packages/0d/07/11590d84df9f52e75ac8a660d888e3d9ef5c88c64af81bd2d5ca305f15ab/berserker_resolver-1.0.3.tar.gz" } ], "2.0.0": [ { "comment_text": "", "digests": { "md5": "98f228523283c5e39bd498994ac8e117", "sha256": "07a1492d45d7e41e07cb1ad33fdee86089031bac437357e1cec976582ceb1428" }, "downloads": -1, "filename": "berserker_resolver-2.0.0.tar.gz", "has_sig": false, "md5_digest": "98f228523283c5e39bd498994ac8e117", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8394, "upload_time": "2015-08-08T01:51:04", "url": "https://files.pythonhosted.org/packages/65/8e/8feb05d3e9e00930cdd686c61fe206df151fb4bec1bde4c30cf27c66b75e/berserker_resolver-2.0.0.tar.gz" } ], "2.0.1": [ { "comment_text": "", "digests": { "md5": "1cb3f0c74c19d4a8872c1eab616acceb", "sha256": "a7ed9766162c2e2467719f3af8038b119c9d73bba0974ea553304a40ffd7eb16" }, "downloads": -1, "filename": "berserker_resolver-2.0.1.tar.gz", "has_sig": false, "md5_digest": "1cb3f0c74c19d4a8872c1eab616acceb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8392, "upload_time": "2015-08-08T02:23:33", "url": "https://files.pythonhosted.org/packages/4a/4a/2868e2d2f440650da668e58725d1d8f8cf430dc8f01fd9aced517b09b2f6/berserker_resolver-2.0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "1cb3f0c74c19d4a8872c1eab616acceb", "sha256": "a7ed9766162c2e2467719f3af8038b119c9d73bba0974ea553304a40ffd7eb16" }, "downloads": -1, "filename": "berserker_resolver-2.0.1.tar.gz", "has_sig": false, "md5_digest": "1cb3f0c74c19d4a8872c1eab616acceb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8392, "upload_time": "2015-08-08T02:23:33", "url": "https://files.pythonhosted.org/packages/4a/4a/2868e2d2f440650da668e58725d1d8f8cf430dc8f01fd9aced517b09b2f6/berserker_resolver-2.0.1.tar.gz" } ] }