{ "info": { "author": "Ozzy", "author_email": "cfhamlet@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy" ], "description": "# os-spage\n\n[![Build Status](https://www.travis-ci.org/cfhamlet/os-spage.svg?branch=master)](https://www.travis-ci.org/cfhamlet/os-spage)\n[![codecov](https://codecov.io/gh/cfhamlet/os-spage/branch/master/graph/badge.svg)](https://codecov.io/gh/cfhamlet/os-spage)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/os-spage.svg)](https://pypi.python.org/pypi/os-spage)\n[![PyPI](https://img.shields.io/pypi/v/os-spage.svg)](https://pypi.python.org/pypi/os-spage)\n\n\nRead and write Spage.\n\nSpage is an incompact data structure to specify fetched record. Generally speaking, it contains four sub-blocks: *url*, *inner_header*, *http_header*, and *data*.\n\nSpage:\n- __url__: the URL.\n- __inner_header__: key-values, can be used to record fetch/process info, such as fetch-time, data-digest, record-type, ect.\n- __http_header__: key-values, server's response HTTP Header as you know.\n- __data__: fetched data, can be flat or compressed html.\n\nWe use dict type to implements Spage. A predefined [schema](https://github.com/cfhamlet/os-spage/blob/master/src/os_spage/default_schema.py) can be used for validating.\n\nIt is common to write Spage to size-rotate-file, we choice [os-rotatefile](https://github.com/cfhamlet/os-rotatefile.git) as default back-end.\n\n__Notice__: \n1. os-spage should not be used for strict serialization/deserialization purpose, it will lose type info when written, all data will be read as string(unicode python2) after all.\n2. Usually, the data stored in compressed format. You can use ``zlib.decompress`` method to decompress.\n \n-------------------------\nOffpage:\n\nFrom v0.4, this libaray support reading from offpage. Offpage is another data storage format, include url, headers and series data. You can use ``read/open_file`` methods with ``page_type=\"offpage\"`` to read from offpage.\n\n\n\nFrom v0.5, support transform spage into offpage. You can use ``read/open_file`` methods with ``page_type=\"s2o\"`` to read from spage and transform the record into offpage format. (Not fully tested yet)\n\n\n\nExample:\n\n```\nfrom os_spage import read\n\nf = open('your_spage', 'rb')\nfor offpage in read(f, page_type='s2o'):\n print(offpage )\n```\n\n\n\n\n\n# Install\n\n`pip install os-spage`\n\n# Usage\n\n * Write to size-rotate-file\n\n ```\n from os_spage import open_file\n\n url = 'http://www.google.com/'\n inner_header = {'User-Agent': 'Mozilla/5.0', 'batchID': 'test'}\n http_header = {'Content-Type': 'text/html'}\n data = b\"Hello world!\"\n\n f = open_file('file', 'w', roll_size='1G', compress=True)\n f.write(url, inner_header=inner_header, http_header=http_header, data=data, flush=True)\n f.close()\n ```\n\n * Read from size-rotate-file\n\n ```\n from os_spage import open_file\n\n f = open_file('file', 'r')\n\n for record in f.read():\n print(record)\n f.close()\n ```\n\n * R/W with other file-like object\n\n ```\n from io import BytesIO\n from os_spage import read, write\n\n s = BytesIO()\n write(s, \"http://www.google.com/\")\n\n s.seek(0)\n for record in read(s):\n print(record)\n ```\n\n\n# Unit Tests\n\n`$ tox`\n\n# License\n\nMIT licensed.", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/cfhamlet/os-spage", "keywords": "", "license": "MIT License", "maintainer": "", "maintainer_email": "", "name": "os-spage", "package_url": "https://pypi.org/project/os-spage/", "platform": "", "project_url": "https://pypi.org/project/os-spage/", "project_urls": { "Homepage": "https://github.com/cfhamlet/os-spage" }, "release_url": "https://pypi.org/project/os-spage/0.5/", "requires_dist": null, "requires_python": "", "summary": "Read and write spage.", "version": "0.5" }, "last_serial": 5960321, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "154210990a640a6858ed331de620221e", "sha256": "71c251dba410e803854503a0a60e074074144d957ffeeb5496f04097ef322761" }, "downloads": -1, "filename": "os-spage-0.1.0.tar.gz", "has_sig": false, "md5_digest": "154210990a640a6858ed331de620221e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4931, "upload_time": "2017-07-18T11:47:37", "url": "https://files.pythonhosted.org/packages/3a/38/ccf0fbb3b1213136dda380e9cca86db85e3106dbb2c9410c9cfa3d70e21f/os-spage-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "52ac29f004066a227ac0f55d026b51f5", "sha256": "200e1c00ddc08e271481a5e24ad0e13ceffe8e4c993cd466a1fb95bb9eb912fd" }, "downloads": -1, "filename": "os-spage-0.1.1.tar.gz", "has_sig": false, "md5_digest": "52ac29f004066a227ac0f55d026b51f5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4912, "upload_time": "2017-12-15T03:04:58", "url": "https://files.pythonhosted.org/packages/13/2b/afc0a6ad58b14b9f5727db4fda9bc4c831e709d466949440dbe77a68447f/os-spage-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "91e15e3a6fda3cefed25184536a2a9ba", "sha256": "cc31a459d9e4b49f732983bf7f24b53802d24093de690185e29ff644778878fd" }, "downloads": -1, "filename": "os-spage-0.1.2.tar.gz", "has_sig": false, "md5_digest": "91e15e3a6fda3cefed25184536a2a9ba", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4920, "upload_time": "2018-02-26T07:48:23", "url": "https://files.pythonhosted.org/packages/92/52/95340b51552de8d166fa7ac40a5f11fdc32d2907a805eb4584cefc7ba06a/os-spage-0.1.2.tar.gz" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "633a043e487382a73b5c20b5367869e1", "sha256": "ccaf7d85631b7f7643624425158285f58d9acc7646e6a690743c4f4bb3216011" }, "downloads": -1, "filename": "os-spage-0.1.3.tar.gz", "has_sig": false, "md5_digest": "633a043e487382a73b5c20b5367869e1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4982, "upload_time": "2018-02-28T09:22:04", "url": "https://files.pythonhosted.org/packages/7b/22/d712bac1a3a77969d536dcda807e2fe4b61319216cbcdc8787b9e6844733/os-spage-0.1.3.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "c163cfe9d3b8e64207f26326c0f503a8", "sha256": "64de2745b08ff116b3c2e24c51cbcfc03dda4d8aedbfdffbd3b05bb4bfe0aefc" }, "downloads": -1, "filename": "os-spage-0.1.4.tar.gz", "has_sig": false, "md5_digest": "c163cfe9d3b8e64207f26326c0f503a8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4980, "upload_time": "2018-02-28T09:38:26", "url": "https://files.pythonhosted.org/packages/31/bc/b1ede0d60d96f50a3c563dce2c2708856eaadcd32d0f8475ed0d7c2c1a2e/os-spage-0.1.4.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "813ad5e1d67bf1a689411984cb80b427", "sha256": "7acad1a4f11bfe55b617a7af133c8d3ed52deb27ec739f241ed448ee04d24953" }, "downloads": -1, "filename": "os-spage-0.2.2.tar.gz", "has_sig": false, "md5_digest": "813ad5e1d67bf1a689411984cb80b427", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7085, "upload_time": "2018-03-02T10:39:22", "url": "https://files.pythonhosted.org/packages/73/68/ae8a17cd6fe863a95deb58445d01feb5791280edc432a5967c32e9447370/os-spage-0.2.2.tar.gz" } ], "0.2.3": [ { "comment_text": "", "digests": { "md5": "258f01d1bea5fb3837d8a60accdfe4f7", "sha256": "f177de0d74017981b7bc2229e5ba91f6587ce1f861410c5b982cf0c608ba08a3" }, "downloads": -1, "filename": "os-spage-0.2.3.tar.gz", "has_sig": false, "md5_digest": "258f01d1bea5fb3837d8a60accdfe4f7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7172, "upload_time": "2018-03-05T06:56:43", "url": "https://files.pythonhosted.org/packages/7a/c0/28e669abcf33ac29cfd4c9175c9f8e0b1bde1647b24851f47b7836f868ef/os-spage-0.2.3.tar.gz" } ], "0.2.4": [ { "comment_text": "", "digests": { "md5": "9de5f1a55d1aed116ab200a5f954f169", "sha256": "b0e5024fbb4e88498fede953a042326c984e2f17fc01a33f086e8d9462538b0f" }, "downloads": -1, "filename": "os-spage-0.2.4.tar.gz", "has_sig": false, "md5_digest": "9de5f1a55d1aed116ab200a5f954f169", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7177, "upload_time": "2018-03-05T07:39:47", "url": "https://files.pythonhosted.org/packages/66/93/9ec4aa7d7deb4aa80488e0f19b9230a65f7416589c8bf0ee88472ab0f4f2/os-spage-0.2.4.tar.gz" } ], "0.2.5": [ { "comment_text": "", "digests": { "md5": "b6e9e93bcde5508364f897dfb6ed274e", "sha256": "b01c0e975eff1f102db6ff9ddb75cd55aaa07665210d6d6a694495c1850777f0" }, "downloads": -1, "filename": "os-spage-0.2.5.tar.gz", "has_sig": false, "md5_digest": "b6e9e93bcde5508364f897dfb6ed274e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7241, "upload_time": "2018-03-07T11:46:12", "url": "https://files.pythonhosted.org/packages/17/c1/033ac11456d36289149d60246c84b661423e31369ca9001d8a9100ef77b2/os-spage-0.2.5.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "d4d793001ba3551aaf70ccee32aa368e", "sha256": "ae68bcab17bd17604c90991965c41734a3f515609e9e15ca3eac84c8543f1739" }, "downloads": -1, "filename": "os-spage-0.3.0.tar.gz", "has_sig": false, "md5_digest": "d4d793001ba3551aaf70ccee32aa368e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7539, "upload_time": "2018-03-20T07:47:29", "url": "https://files.pythonhosted.org/packages/51/52/011e21a71b2618978078f6f7995afc0b23adf348d532ee1fe3d656bc5129/os-spage-0.3.0.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "4bbf834a7424ba95df3ab1273c5ed0ac", "sha256": "e38f52a6fe349b619e0e7fd30ed5ecef327dedc53bbf85a29a5340f4b361e83c" }, "downloads": -1, "filename": "os-spage-0.3.1.tar.gz", "has_sig": false, "md5_digest": "4bbf834a7424ba95df3ab1273c5ed0ac", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7565, "upload_time": "2018-03-21T08:42:34", "url": "https://files.pythonhosted.org/packages/95/3d/25589ae449902c713aa0a965e918318433f67f51a5932a6ab033f983d2cb/os-spage-0.3.1.tar.gz" } ], "0.3.2": [ { "comment_text": "", "digests": { "md5": "76a293a42ba0b295fa16114f38f484de", "sha256": "a08f92067f9f56d2ae3b89563bfe35a3195a8c6cec3a421b2b351e8155cda52e" }, "downloads": -1, "filename": "os-spage-0.3.2.tar.gz", "has_sig": false, "md5_digest": "76a293a42ba0b295fa16114f38f484de", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7596, "upload_time": "2018-03-22T03:16:38", "url": "https://files.pythonhosted.org/packages/82/55/4aa42482644001a1a452267e8c4c58ceafe09d429e13e66ab3728848ef9f/os-spage-0.3.2.tar.gz" } ], "0.3.4": [ { "comment_text": "", "digests": { "md5": "9942d359f42c20fb49b9d6d83e267630", "sha256": "753723947ce1819287da2599eaf2ceadc8a6814158fc3e8d33cd98be0d28a144" }, "downloads": -1, "filename": "os-spage-0.3.4.tar.gz", "has_sig": false, "md5_digest": "9942d359f42c20fb49b9d6d83e267630", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7776, "upload_time": "2018-04-08T04:20:57", "url": "https://files.pythonhosted.org/packages/19/6e/12a09e402474a1d023eaa21ccf6d47945b8951e9f14ee25e8ead2a822887/os-spage-0.3.4.tar.gz" } ], "0.3.5": [ { "comment_text": "", "digests": { "md5": "6353bb037c61fc61f4ecccbbd03710c6", "sha256": "fc1a28a912b7b1f3224c9ab73d6e728870076c36328345289560c6e948ee6a50" }, "downloads": -1, "filename": "os-spage-0.3.5.tar.gz", "has_sig": false, "md5_digest": "6353bb037c61fc61f4ecccbbd03710c6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8123, "upload_time": "2018-04-10T12:15:58", "url": "https://files.pythonhosted.org/packages/b9/a5/d345b39bceb89248dd77cc00334ff269db7196be87baa8bee8a2797ab7a1/os-spage-0.3.5.tar.gz" } ], "0.3.6": [ { "comment_text": "", "digests": { "md5": "5a6e4e0794f78fef69cedfc9e720742b", "sha256": "99cddfd544089421238fa97b03a45e08a148aaabb8d446bcca0a893dcd60f21a" }, "downloads": -1, "filename": "os-spage-0.3.6.tar.gz", "has_sig": false, "md5_digest": "5a6e4e0794f78fef69cedfc9e720742b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8397, "upload_time": "2018-04-11T03:17:34", "url": "https://files.pythonhosted.org/packages/b8/a2/2af31b5caded5a11437aa75ae16fd1b83163c68f97639e2c460558a77067/os-spage-0.3.6.tar.gz" } ], "0.3.7": [ { "comment_text": "", "digests": { "md5": "31baa374c436df26d508519da0175e66", "sha256": "f212dafd41be03ae8b0e438feae84c30aec92d51713da06184394a305ce6d5c7" }, "downloads": -1, "filename": "os-spage-0.3.7.tar.gz", "has_sig": false, "md5_digest": "31baa374c436df26d508519da0175e66", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8429, "upload_time": "2018-04-12T07:56:26", "url": "https://files.pythonhosted.org/packages/69/95/6d62307a88accedc2e9ddfb563bfe5559844518337c7a7de0a69aba36d81/os-spage-0.3.7.tar.gz" } ], "0.4": [ { "comment_text": "", "digests": { "md5": "8c92aabf36e354874d3a5c0bc5767303", "sha256": "5b4941e7d6dd9b760d4d3f6521464a95200519f1fb2f0755e406c14c553eeb71" }, "downloads": -1, "filename": "os-spage-0.4.tar.gz", "has_sig": false, "md5_digest": "8c92aabf36e354874d3a5c0bc5767303", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9165, "upload_time": "2018-12-17T15:12:53", "url": "https://files.pythonhosted.org/packages/e8/cf/3a386736722d24b39b81be5e539a21694a1eb03fa63a6fff59f4c16bd7dc/os-spage-0.4.tar.gz" } ], "0.4.1": [ { "comment_text": "", "digests": { "md5": "77ed53cea6615ece59999a65248cd72f", "sha256": "3343b96412b15d64a660bda4c480a8ac040601b2cdf0d376648aefee428d11c7" }, "downloads": -1, "filename": "os-spage-0.4.1.tar.gz", "has_sig": false, "md5_digest": "77ed53cea6615ece59999a65248cd72f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9406, "upload_time": "2018-12-18T10:30:20", "url": "https://files.pythonhosted.org/packages/c7/6a/b8f6a7a067af951687dd63069a661d67425b3d9ea580e96dab155357e884/os-spage-0.4.1.tar.gz" } ], "0.4.2": [ { "comment_text": "", "digests": { "md5": "10bad3a4d3f1e2ab49abdacb79e335d2", "sha256": "3dc9bd4d0caee075c864fe2c1f110f17777fa363f34cf8d7d545257ab3f9df2f" }, "downloads": -1, "filename": "os-spage-0.4.2.tar.gz", "has_sig": false, "md5_digest": "10bad3a4d3f1e2ab49abdacb79e335d2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9947, "upload_time": "2019-09-24T07:09:40", "url": "https://files.pythonhosted.org/packages/8c/88/371c341dd1c0468c859f12abe15de61b1dee24c29315f3aec4fd5567a341/os-spage-0.4.2.tar.gz" } ], "0.4.3": [ { "comment_text": "", "digests": { "md5": "197c4fdb2404b6044f57b4a812016b6a", "sha256": "a57b7fc9d62ae126d3bb04312a5883ff3fd43441a44728e4bc146b2f6efa54e1" }, "downloads": -1, "filename": "os-spage-0.4.3.tar.gz", "has_sig": false, "md5_digest": "197c4fdb2404b6044f57b4a812016b6a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10184, "upload_time": "2019-09-29T09:48:32", "url": "https://files.pythonhosted.org/packages/7c/d6/ba0bb45846bc17e1d1e1ab023b3fb8ff771e3554449c01e31879629349a3/os-spage-0.4.3.tar.gz" } ], "0.4.4": [ { "comment_text": "", "digests": { "md5": "e339924795278ecb2dd637d5b3f1ebee", "sha256": "f5cbabe40b849d008bdb3d167cf43b7c920e83667471d8dac3730888efb527ae" }, "downloads": -1, "filename": "os-spage-0.4.4.tar.gz", "has_sig": false, "md5_digest": "e339924795278ecb2dd637d5b3f1ebee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10213, "upload_time": "2019-09-30T08:33:30", "url": "https://files.pythonhosted.org/packages/93/6e/a03d4bc85f2f76e0ba2668b166c7bf55be338b90e06d6b3e7a38813209f7/os-spage-0.4.4.tar.gz" } ], "0.4.5": [ { "comment_text": "", "digests": { "md5": "6f92161fe9b7f64533e39e0e88d8cb62", "sha256": "0d25d2612032e4ff8c84859585e9f867d4d010b4f21d1811cc4ddc16fd7cd4d0" }, "downloads": -1, "filename": "os-spage-0.4.5.tar.gz", "has_sig": false, "md5_digest": "6f92161fe9b7f64533e39e0e88d8cb62", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10219, "upload_time": "2019-09-30T08:55:29", "url": "https://files.pythonhosted.org/packages/cc/3a/5a38e98984e89c8623573b5aafa340372a7fdc31ce092b4bc46027feba5f/os-spage-0.4.5.tar.gz" } ], "0.5": [ { "comment_text": "", "digests": { "md5": "72cb16efc681abf43759e3e8bac93198", "sha256": "bb39c2978e94a5d6bf7dd41f2507d2e132fae03c124160199e84cc05a9b49ddc" }, "downloads": -1, "filename": "os-spage-0.5.tar.gz", "has_sig": false, "md5_digest": "72cb16efc681abf43759e3e8bac93198", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11789, "upload_time": "2019-10-11T13:27:08", "url": "https://files.pythonhosted.org/packages/1d/d9/a7c063b4c870d56e20abaa36ddf968bd1a541eba9768e3791b45e2dcd361/os-spage-0.5.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "72cb16efc681abf43759e3e8bac93198", "sha256": "bb39c2978e94a5d6bf7dd41f2507d2e132fae03c124160199e84cc05a9b49ddc" }, "downloads": -1, "filename": "os-spage-0.5.tar.gz", "has_sig": false, "md5_digest": "72cb16efc681abf43759e3e8bac93198", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11789, "upload_time": "2019-10-11T13:27:08", "url": "https://files.pythonhosted.org/packages/1d/d9/a7c063b4c870d56e20abaa36ddf968bd1a541eba9768e3791b45e2dcd361/os-spage-0.5.tar.gz" } ] }