{ "info": { "author": "Christophe", "author_email": "chris.perso@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "A basic but fast, persistent and threadsafe caching system\n================================================================\n\nThis package lets you efficiently retrieve pages from the Internet by caching request's results.\n\nBasic commands\n-----------------\n\nImporting required modules first:\n\n`from webscrapetools import urlcaching`\n\nInitializing the cache:\n\n`urlcaching.set_cache_path('.wst_cache')`\n\nThe option _expiry_days_ sets the cache expiry period, default is 10 days.\n\nThis is a required step: otherwise responses to url calls will simply not be cached.\nCache data are stored in the specified folder, so that re-using the same string makes the cache persistent. This creates\nthe folder on the fly if it does not exist.\nThe following command cleans up the cache, making sure we start with no prior data:\n\n`urlcaching.empty_cache()`\n\nOpening an url with the following command stores the repsonse content behind the scene, so that subsequent calls will\nnot hit the network.\n\n`urlcaching.open_url('http://www.google.com')`\n\n\nFull example\n-----------------\n\n.. code-block:: python\n\n from webscrapetools import urlcaching\n import time\n\n # Initializing the cache\n urlcaching.set_cache_path('.wst_cache')\n\n # Making sure we start from scratch\n urlcaching.empty_cache()\n\n # Demo with 5 identical calls... only the first one is delayed, all others are hitting the cache\n count_calls = 1\n while count_calls <= 5:\n start_time = time.time()\n urlcaching.open_url('http://deelay.me/5000/http://www.google.com')\n duration = time.time() - start_time\n print('duration for call {}: {:0.2f}'.format(count_calls, duration))\n count_calls += 1\n\n # Cleaning up\n urlcaching.empty_cache()\n\nThe code above outputs the following:\n\n duration for call 1: 6.74\n duration for call 2: 0.00\n duration for call 3: 0.00\n duration for call 4: 0.00\n duration for call 5: 0.00\n\nExample plugging in a custom client\n--------------------------------------\n\nThe framework lets you customize the way you access the web. It is therefore possible to drive a browser \nvia Selenium for example.\n\n.. code-block:: python\n\n from webscrapetools import urlcaching\n urlcaching.set_cache_path('./output/tests', max_node_files=10, rebalancing_limit=100)\n\n def dummy_client():\n return None\n\n def dummy_call(_, key):\n return '{:d}'.format(int(key)) * int(key), key\n\n # simulating calls using the dummy client\n keys = ('{:05d}'.format(count) for count in range(500))\n for key in keys:\n urlcaching.open_url(key, init_client_func=dummy_client, call_client_func=dummy_call)\n\n urlcaching.empty_cache()\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "https://github.com/chris-ch/webscrapetools/webscrapetools/archive/0.5.5.tar.gz", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/chris-ch/webscrapetools", "keywords": "", "license": "Apache", "maintainer": "", "maintainer_email": "", "name": "webscrapetools", "package_url": "https://pypi.org/project/webscrapetools/", "platform": "", "project_url": "https://pypi.org/project/webscrapetools/", "project_urls": { "Download": "https://github.com/chris-ch/webscrapetools/webscrapetools/archive/0.5.5.tar.gz", "Homepage": "https://github.com/chris-ch/webscrapetools" }, "release_url": "https://pypi.org/project/webscrapetools/0.5.5/", "requires_dist": [ "requests (>=2.20.0)" ], "requires_python": "", "summary": "A basic but fast, persistent and threadsafe caching system", "version": "0.5.5", "yanked": false, "yanked_reason": null }, "last_serial": 6028441, "releases": { "0.1": [], "0.2": [ { "comment_text": "", "digests": { "md5": "9873633c6a2eda886fb027f095609faf", "sha256": "7895eab01b8ddf8dcab5142bd78b00a9cf978db1f59efe8d000e8fc50849448d" }, "downloads": -1, "filename": "webscrapetools-0.2.tar.gz", "has_sig": false, "md5_digest": "9873633c6a2eda886fb027f095609faf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4654, "upload_time": "2016-10-05T14:06:59", "upload_time_iso_8601": "2016-10-05T14:06:59.380546Z", "url": "https://files.pythonhosted.org/packages/69/8c/9b66fe793931566a5ada34ecb4a91b2cb22b52ea67e545f70b0445eda31a/webscrapetools-0.2.tar.gz", "yanked": false, "yanked_reason": null } ], "0.3": [ { "comment_text": "", "digests": { "md5": "6dd942df320149ce8e4fe206c602893d", "sha256": "a68488c2ccc64f0478659ec6bda54e9fa72a5cd57244ed455844ecd2d067e09e" }, "downloads": -1, "filename": "webscrapetools-0.3.tar.gz", "has_sig": false, "md5_digest": "6dd942df320149ce8e4fe206c602893d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4600, "upload_time": "2016-10-18T16:49:06", "upload_time_iso_8601": "2016-10-18T16:49:06.471230Z", "url": "https://files.pythonhosted.org/packages/dd/43/9eeb6046a5971d8aa8305619cbbb9c12c62033d07b5e3dc9f2f7b47fb310/webscrapetools-0.3.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.0": [ { "comment_text": "", "digests": { "md5": "828750b9c9222cb3648f221830103cea", "sha256": "c296dfcbbc61aca8abd280113f4fa60418c559d99e80fccfd4df5678d59e4026" }, "downloads": -1, "filename": "webscrapetools-0.4.0.tar.gz", "has_sig": false, "md5_digest": "828750b9c9222cb3648f221830103cea", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9789, "upload_time": "2017-07-05T15:41:44", "upload_time_iso_8601": "2017-07-05T15:41:44.513473Z", "url": "https://files.pythonhosted.org/packages/dc/c5/430a8491696f4027f20bec455ffd30cb7ee7f3ec65a87e4eb9d25bd0e8c7/webscrapetools-0.4.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "70c8eb9a2780f545ff384791f0e70018", "sha256": "ee2fe3a4bd77cbd509023cc41dcdb50b61196f2c7053a19f8b309aa28f2771a7" }, "downloads": -1, "filename": "webscrapetools-0.4.1.tar.gz", "has_sig": false, "md5_digest": "70c8eb9a2780f545ff384791f0e70018", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9868, "upload_time": "2019-03-17T14:38:57", "upload_time_iso_8601": "2019-03-17T14:38:57.617931Z", "url": "https://files.pythonhosted.org/packages/0b/e4/362845e97e0006d3fec9994ac5c54befc9f0119e37c444ddf7941b76e060/webscrapetools-0.4.1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "f4842bfe992e5b4034e0d7c5f77e5b19", "sha256": "e6b3af5be0f95fd2e5100c50f9a3abf2166a00f7de47fd9bad725c900eac1475" }, "downloads": -1, "filename": "webscrapetools-0.4.2.tar.gz", "has_sig": false, "md5_digest": "f4842bfe992e5b4034e0d7c5f77e5b19", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9994, "upload_time": "2019-03-21T20:13:24", "upload_time_iso_8601": "2019-03-21T20:13:24.371340Z", "url": "https://files.pythonhosted.org/packages/5d/2c/9804344763195b304e52795e46ee91974d12143a1bcbc2c9939574d4f195/webscrapetools-0.4.2.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.3": [ { "comment_text": "", "digests": { "md5": "7591e63690925b1dd2599c90ff0d3daf", "sha256": "edd5fedd4a78dc2539df74351eba40a64673867e24633e58f3441d6de024cfc5" }, "downloads": -1, "filename": "webscrapetools-0.4.3.tar.gz", "has_sig": false, "md5_digest": "7591e63690925b1dd2599c90ff0d3daf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9997, "upload_time": "2019-03-21T20:24:33", "upload_time_iso_8601": "2019-03-21T20:24:33.599806Z", "url": "https://files.pythonhosted.org/packages/76/cf/1a214abe856efda758e85d705dba1cf42e1147ab49de2483ab076f382cda/webscrapetools-0.4.3.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.4": [ { "comment_text": "", "digests": { "md5": "abdc42512e7fe53a6b465ef1b86fb69e", "sha256": "b2b6fd6b2f358c22cfc32a03c2afd1ccd9373f3dad5c9ac5fee43ac4dd5830de" }, "downloads": -1, "filename": "webscrapetools-0.4.4-py3-none-any.whl", "has_sig": false, "md5_digest": "abdc42512e7fe53a6b465ef1b86fb69e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13070, "upload_time": "2019-09-04T20:37:58", "upload_time_iso_8601": "2019-09-04T20:37:58.923420Z", "url": "https://files.pythonhosted.org/packages/77/e9/e993e914421080d2ed5f2a78617f8b55e21fe6e451e10480d3e8bd15e714/webscrapetools-0.4.4-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "201d7af6f178c499c58ed924b3faa415", "sha256": "17e11193536a0777132157d5a779af5ce4b24747fd9a1f5ff67283043f3f154e" }, "downloads": -1, "filename": "webscrapetools-0.4.4.tar.gz", "has_sig": false, "md5_digest": "201d7af6f178c499c58ed924b3faa415", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11400, "upload_time": "2019-09-04T20:38:00", "upload_time_iso_8601": "2019-09-04T20:38:00.958310Z", "url": "https://files.pythonhosted.org/packages/e7/dd/fa43a79f1960716ad36284af23b7db2a8e3f2d70811bc60e9331ca1662ab/webscrapetools-0.4.4.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.5": [ { "comment_text": "", "digests": { "md5": "37efef1b6eb5ce9d30cdda6c2736c440", "sha256": "0215adc9677924a10510b13ef0556b10169d41b68d2bfd1856b9493e059b5d14" }, "downloads": -1, "filename": "webscrapetools-0.4.5-py3-none-any.whl", "has_sig": false, "md5_digest": "37efef1b6eb5ce9d30cdda6c2736c440", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13120, "upload_time": "2019-09-06T11:14:39", "upload_time_iso_8601": "2019-09-06T11:14:39.378665Z", "url": "https://files.pythonhosted.org/packages/0b/4b/37c3d9dc63890ee84609ad89e6ceff910310d92126bc96c62fe4273bd060/webscrapetools-0.4.5-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "e9a66fa997d65e6c9899d179f7290b2e", "sha256": "7b87711e11ed4fbcdb3724deb89b304ab768ed2894164382c93e828c1199ba9a" }, "downloads": -1, "filename": "webscrapetools-0.4.5.tar.gz", "has_sig": false, "md5_digest": "e9a66fa997d65e6c9899d179f7290b2e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11442, "upload_time": "2019-09-06T11:14:41", "upload_time_iso_8601": "2019-09-06T11:14:41.781640Z", "url": "https://files.pythonhosted.org/packages/8c/cd/be2687f8efa60020a9e6a6efcc6ac8f0f1bd47824e5fde04cf8ce2cb04dd/webscrapetools-0.4.5.tar.gz", "yanked": false, "yanked_reason": null } ], "0.4.6": [ { "comment_text": "", "digests": { "md5": "62d06c38fd416634e5b1b8d78ff904f8", "sha256": "78c5dc33dc25c3d610fa7e98b7e15cc2de41a3f4d43a12179729aa4c024025af" }, "downloads": -1, "filename": "webscrapetools-0.4.6-py3-none-any.whl", "has_sig": false, "md5_digest": "62d06c38fd416634e5b1b8d78ff904f8", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13153, "upload_time": "2019-09-07T06:06:47", "upload_time_iso_8601": "2019-09-07T06:06:47.702683Z", "url": "https://files.pythonhosted.org/packages/d5/41/56cd74a6b21434eb6f732e86b83d24d2133eced134cf3e9547b53ea2142a/webscrapetools-0.4.6-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "7ffb3410c3973d221bfb6e5dcfee811f", "sha256": "497c54177467f963b6507957de292cf6f25bfd4774d76f77844f6ee6ea05a029" }, "downloads": -1, "filename": "webscrapetools-0.4.6.tar.gz", "has_sig": false, "md5_digest": "7ffb3410c3973d221bfb6e5dcfee811f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11472, "upload_time": "2019-09-07T06:06:51", "upload_time_iso_8601": "2019-09-07T06:06:51.694729Z", "url": "https://files.pythonhosted.org/packages/8c/b7/e33d7f46b6a7208af1e59585367c4f38c8ecf16b412c1389e1faef8c3c3d/webscrapetools-0.4.6.tar.gz", "yanked": false, "yanked_reason": null } ], "0.5.1": [ { "comment_text": "", "digests": { "md5": "d50b0d9831aa33ee5b04831ee989a707", "sha256": "228b124a5fa8681de78d023bd2b239743e1648b847fbefbaf393e09ce025dd2b" }, "downloads": -1, "filename": "webscrapetools-0.5.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d50b0d9831aa33ee5b04831ee989a707", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13244, "upload_time": "2019-10-21T20:00:40", "upload_time_iso_8601": "2019-10-21T20:00:40.194779Z", "url": "https://files.pythonhosted.org/packages/5c/59/5cbb55a3cb74bb5241b460d024e3d11e6c3b8d43278b4a935f7e5c6f2b48/webscrapetools-0.5.1-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "fabcd05de9fdf4db0fadb9874aaa10a6", "sha256": "765554e97b645368cbd52035c2287298e935cc22bcfd21a34c15d6b7f22e0227" }, "downloads": -1, "filename": "webscrapetools-0.5.1.tar.gz", "has_sig": false, "md5_digest": "fabcd05de9fdf4db0fadb9874aaa10a6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11534, "upload_time": "2019-10-21T20:00:43", "upload_time_iso_8601": "2019-10-21T20:00:43.474776Z", "url": "https://files.pythonhosted.org/packages/24/7c/d86ac74b9cf45f0859454a202a6cb13fc2deb719b4726cc7a7a896eb2522/webscrapetools-0.5.1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.5.3": [ { "comment_text": "", "digests": { "md5": "a93b3e9f3bc17dba7c146d1051d77311", "sha256": "c2ec3b51d6bc8a63be690c7cc8275cc465182fb75a5acb17266daa9d65785186" }, "downloads": -1, "filename": "webscrapetools-0.5.3-py3.6.egg", "has_sig": false, "md5_digest": "a93b3e9f3bc17dba7c146d1051d77311", "packagetype": "bdist_egg", "python_version": "3.6", "requires_python": null, "size": 20324, "upload_time": "2019-10-22T06:19:46", "upload_time_iso_8601": "2019-10-22T06:19:46.618778Z", "url": "https://files.pythonhosted.org/packages/0c/01/026b153c0e8b3b5654cc1fb2dd6cf03475e62883cddd017d47531e47a25e/webscrapetools-0.5.3-py3.6.egg", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "04c73489abc118ca8257f33ca49e8dbe", "sha256": "355e1b85733d4ca774718f579fa8a57800b52cb94a7f840b2b35ebf63e3c2e74" }, "downloads": -1, "filename": "webscrapetools-0.5.3.tar.gz", "has_sig": false, "md5_digest": "04c73489abc118ca8257f33ca49e8dbe", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11791, "upload_time": "2019-10-22T06:19:48", "upload_time_iso_8601": "2019-10-22T06:19:48.974786Z", "url": "https://files.pythonhosted.org/packages/2b/82/160284360cd813afd84648dfc80555b7338b35a9942774eab425f384b2ed/webscrapetools-0.5.3.tar.gz", "yanked": false, "yanked_reason": null } ], "0.5.4": [ { "comment_text": "", "digests": { "md5": "12f02cffee2b121b997a8b2ca0172124", "sha256": "8867e9bdca01dd27c53bdefe1d4bc347f3ae965642d26e1cba9898386e9a6ab6" }, "downloads": -1, "filename": "webscrapetools-0.5.4-py3-none-any.whl", "has_sig": false, "md5_digest": "12f02cffee2b121b997a8b2ca0172124", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13519, "upload_time": "2019-10-22T06:24:17", "upload_time_iso_8601": "2019-10-22T06:24:17.158780Z", "url": "https://files.pythonhosted.org/packages/c9/16/3bc2049d16ebacccabafd28d54391aac8c4534ca8afb0af2f3533dc1713b/webscrapetools-0.5.4-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "6d9adb5f0e3b790eb29dcf2a9de959b0", "sha256": "664a29edfc0c0bc39da761ad4b5168a84ce4a8f06ca8f8723c5a0490a2f86e51" }, "downloads": -1, "filename": "webscrapetools-0.5.4.tar.gz", "has_sig": false, "md5_digest": "6d9adb5f0e3b790eb29dcf2a9de959b0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11789, "upload_time": "2019-10-22T06:24:25", "upload_time_iso_8601": "2019-10-22T06:24:25.811056Z", "url": "https://files.pythonhosted.org/packages/98/ac/16f7956370094c479cfd8e1a7cd906a1aaba7c7331e5736616ac61ab468c/webscrapetools-0.5.4.tar.gz", "yanked": false, "yanked_reason": null } ], "0.5.5": [ { "comment_text": "", "digests": { "md5": "088015edb671a4a44f4456458cc86391", "sha256": "ae3daa8fb7073a4a7a50567d41991e502069e87330cb4e3c23a2f7ebcb845de9" }, "downloads": -1, "filename": "webscrapetools-0.5.5-py3-none-any.whl", "has_sig": false, "md5_digest": "088015edb671a4a44f4456458cc86391", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13583, "upload_time": "2019-10-25T09:26:36", "upload_time_iso_8601": "2019-10-25T09:26:36.116704Z", "url": "https://files.pythonhosted.org/packages/c7/dd/ce5df3605f05e8396bf2dc183fffaf0d40ae6235c8393be31936fadffc19/webscrapetools-0.5.5-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "801217f626418ea6e3bdc7d0246543c8", "sha256": "b8ba08b4a351796c7283e4c773a571f1984af9bf066bd3f738e489370e0f73c3" }, "downloads": -1, "filename": "webscrapetools-0.5.5.tar.gz", "has_sig": false, "md5_digest": "801217f626418ea6e3bdc7d0246543c8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11835, "upload_time": "2019-10-25T09:26:40", "upload_time_iso_8601": "2019-10-25T09:26:40.749189Z", "url": "https://files.pythonhosted.org/packages/a5/49/f8f2070ac51fbfc3b0627ff31d19b199cb56c21b40c988997d4caa971761/webscrapetools-0.5.5.tar.gz", "yanked": false, "yanked_reason": null } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "088015edb671a4a44f4456458cc86391", "sha256": "ae3daa8fb7073a4a7a50567d41991e502069e87330cb4e3c23a2f7ebcb845de9" }, "downloads": -1, "filename": "webscrapetools-0.5.5-py3-none-any.whl", "has_sig": false, "md5_digest": "088015edb671a4a44f4456458cc86391", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13583, "upload_time": "2019-10-25T09:26:36", "upload_time_iso_8601": "2019-10-25T09:26:36.116704Z", "url": "https://files.pythonhosted.org/packages/c7/dd/ce5df3605f05e8396bf2dc183fffaf0d40ae6235c8393be31936fadffc19/webscrapetools-0.5.5-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "801217f626418ea6e3bdc7d0246543c8", "sha256": "b8ba08b4a351796c7283e4c773a571f1984af9bf066bd3f738e489370e0f73c3" }, "downloads": -1, "filename": "webscrapetools-0.5.5.tar.gz", "has_sig": false, "md5_digest": "801217f626418ea6e3bdc7d0246543c8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11835, "upload_time": "2019-10-25T09:26:40", "upload_time_iso_8601": "2019-10-25T09:26:40.749189Z", "url": "https://files.pythonhosted.org/packages/a5/49/f8f2070ac51fbfc3b0627ff31d19b199cb56c21b40c988997d4caa971761/webscrapetools-0.5.5.tar.gz", "yanked": false, "yanked_reason": null } ], "vulnerabilities": [] }