{
    "info": {
        "author": "Fujiao Liu",
        "author_email": "fujiaoliu@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Environment :: Console",
            "Intended Audience :: Developers",
            "Intended Audience :: System Administrators",
            "License :: OSI Approved :: BSD License",
            "Operating System :: MacOS :: MacOS X",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.5",
            "Topic :: Software Development :: Object Brokering",
            "Topic :: System :: Distributed Computing"
        ],
        "description": "Rezina\n=======\n\nrezina is a scalable, distributed and easy to use system for executing python code in parallel across multiple processors or many machines.  It provides a simple way to make parallel programming in python more easier, flexible and scalable and shipped with  features like periodically schedule, load balance, fail tolerate, dynamically add tasks and save result to backend.\n\nwhy use rezina?\n================\n\nConsider this task, I want to gather weather data of ten cities from yahoo weather api. I wrote a scirpt `cityweather.py` to do this.\n::\n   liufujiaodeMacBook-Pro:workspace liufujiao$ time python cityweather.py\n   {'city': 'Beijing', 'weather': u'Mostly Sunny'}\n   {'city': 'Berlin', 'weather': u'Partly Cloudy'}\n   {'city': 'New York', 'weather': u'Mostly Clear'}\n   {'city': 'London', 'weather': u'Mostly Cloudy'}\n   {'city': 'Tokyo', 'weather': u'Mostly Sunny'}\n   {'city': 'Paris', 'weather': u'Mostly Cloudy'}\n   {'city': 'Chicago', 'weather': u'Partly Cloudy'}\n   {'city': 'washington', 'weather': u'Clear'}\n   {'city': 'Venice', 'weather': u'Breezy'}\n   {'city': 'Houston', 'weather': u'Breezy'}\n\n   real\t0m18.726s\n   user\t0m0.080s\n   sys\t0m0.034s\n\nThe whole process of fetching data take 20s and we could think one city takes 2s on average.\n\nNow here is the problem, what if I want to get the ten weather data every 10 seconds so I could always know the least weather?  this requires all data should be fetched in 10s.\n\nOne simple way to do this is we could start more threads (or processes) to fetch data in parallel and implement logic to run periodically.\n\nProblem again, now I want to get weather data of 50,000 cities so I can build a weather website to provide weather conditions for my user, how to do it?\n\nOne server could not launch 10,000 threads. maybe we could split all cites into several parts and run every part with threads on different servers, but this kind of solution need consider the following problems.\n\n* what if one or more servers down?\n* what if we want to add or remove some cites?\n* what if we want to launch more processes for time consuming logic?\n* what if we want to change the fetch interval from 10s to 5s.\n* what if we want to save data in another backend?\n* \u2026\n\nWe need a easier and flexible way to execute tasks in parallel without pay much attentions on these problems and make us focus only on our business logic. this is the motivation of rezina project.\n\nSee Quick Guide for how rezina do this.\n\nSupport\n==========\n**Python**\n\n* python2.7\n* python3.5\n\n**OS**\n\n* OS X\n* Linux\n\nDependencies\n=============\n\nrezina requires `pyzmq <https://github.com/zeromq/pyzmq>`_ for message transports.\n\npyzmq is python bindings for `\u00d8MQ <http://zeromq.org/>`_. \u00d8MQ is a lightweight and fast messaging implementation.\n\nInstallaion\n=============\n\n\nYou can install reizna either via the Python Package Index (PyPI) or from source.\n\nTo install using pip:\n\n``pip install rezina``\n\nDownloading and installing from source\n---------------------------------------\n\nBefore install rezina, `building-and-installation <https://github.com/zeromq/pyzmq#building-and-installation>`_\npyzmq first.\n\nAfter pyzmq installed, download the latest version of rezina from PyPI:\n\nhttp://pypi.python.org/pypi/rezina\n\nand install it by doing the following:\n\n``pip install /path/to/rezina-0.x.y.tar.gz``\n\nor\n\n``tar xvfz rezina-0.x.y.tar.gz``\n\n``cd rezina-0.x.y``\n\n``python setup.py install``\n\nQuick Guide\n==============\n\nStart rezina\n---------------\n\nOnce rezina is installed, you can run\n\n``rezina-cli runmaster -H master_ip``\n\nto start rezina master, and run\n\n``rezina-cli runworker -H master_ip -WIP worker_ip``\n\nto start rezina worker\n\n.. note:: we could use `-D` to run master as daemon and `-W` to specify a new workspace. please see documentation for starting rezina correctly.\n\nExample cityweather\n--------------------\n\ncityweather code\n^^^^^^^^^^^^^^^^^\n\nscript name: ``cityweather.py``\n\nput this script into rezina workspace (``~/rezina/workspace`` by default, use -W /path/to/your/workspace when starting master if you want to change it)\n\n::\n\n    #!/usr/bin/evn python\n\n    import urllib2\n    import urllib\n    import json\n\n\n    def get_cities():\n        cities = ['Beijing', 'Berlin', 'New York', 'London', 'Tokyo', 'Paris',\n                  'Chicago', 'washington', 'Venice', 'Houston']\n        return cities\n\n\n    # get city weather data from yahoo weather api\n    def get_city_weather(city):\n        baseurl = \"https://query.yahooapis.com/v1/public/yql?\"\n        yql_query = \"select item.condition.text from weather.forecast \\\n                     where woeid in (select woeid from geo.places(1) \\\n                     where text='%s')\" % (city)\n        yql_url = baseurl + urllib.urlencode({'q': yql_query}) + \"&format=json\"\n        result = urllib2.urlopen(yql_url).read()\n        data = json.loads(result)\n        # because resule from yahoo api does not include the city name, we add it.\n        data['city'] = city\n        return data\n\n\n    # process diffrent output and convert data to a simple format\n    def one_word_conditions_for_city(city_weather_result):\n        simple_format_data = {}\n        simple_format_data['city'] = city_weather_result['city']\n        if city_weather_result['query']['results'] is not None:\n            weather = city_weather_result['query']['results']['channel']['item']['condition']['text']\n        else:\n            weather = \"Unkonw\"  # simplely set unkonw when result is not avaliable\n        simple_format_data['weather'] = weather\n        return simple_format_data\n\n    if __name__ == \"__main__\":\n        for city in get_cities():\n            print one_word_conditions_for_city(get_city_weather(city))\n\n\n\nBuild a typology to run cityweather with rezina\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nscript name: ``weathertypo.py``\n\nput it info rezina workspace\n\n.. hint:: you could regard hydrant, notch, bocca as input, filter, output respectively for now. A typology looks like `input | filter1 | filter2 | output` in shell. check the Documentation for more info.\n\n\nrun ``get_city_weather`` function with 2 processes and every process run 5 threads and each thread fetch one city.\n\nrun ``one_word_conditions_for_city`` function with 1 process with 1 thread because it is not time consuming one.\n\n::\n\n    #!/usr/bin/env python\n\n    from rezina.utils.network import get_ip\n    from rezina import TypologyBuilder\n    from rezina.backends import Stdout\n\n    from cityweather import get_cities, get_city_weather, one_word_conditions_for_city\n\n    ip = get_ip()  # get master_ip\n    tb = TypologyBuilder(ip, 12345, 'weather_typo2')\n    tb.add_hydrant(get_cities)\n    tb.add_notch(get_city_weather, 2, 5)\n    tb.add_notch(one_word_conditions_for_city, 1, 1)\n    tb.add_bocca(Stdout, persistent_mode='stream')\n\n    if __name__ == \"__main__\":\n        tb.restart(interval=10)\n\n\nrun typology\n^^^^^^^^^^^^\n\nrezina typology file is just a python script, run it with\n\n``python weathertypo.py`` or ``./weathertypo.py``` and you get the results.\n\nyou could also save the results of your typology to another backend rather than print them.\n\nSee documentation for more details.\n\nrezina console\n----------------\n\nrezina provides command line tool and web console to manage master, workers, typologies.\n\nyou could run\n\n```rezina-cli runconsole -H master_ip``\n\n to start cml or access ``master_ip:31218`` to see web console.\n\nDocumentation\n================\n\nSee http://rezina.readthedocs.io/en/latest/ for more info.\n\n\n",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://rezina.readthedocs.org",
        "keywords": "",
        "license": "BSD",
        "maintainer": "",
        "maintainer_email": "",
        "name": "rezina",
        "package_url": "https://pypi.org/project/rezina/",
        "platform": "",
        "project_url": "https://pypi.org/project/rezina/",
        "project_urls": {
            "Homepage": "https://rezina.readthedocs.org"
        },
        "release_url": "https://pypi.org/project/rezina/0.1.1/",
        "requires_dist": [
            "pyzmq (>=16.0.0)"
        ],
        "requires_python": "",
        "summary": "a scalable, distributed task execution system",
        "version": "0.1.1"
    },
    "last_serial": 2510617,
    "releases": {
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "438976e8d5e4624fcc567eaaff8245ef",
                    "sha256": "b6746e1253a81383829d264d4b03574088d1531242a315363e596c45b74aca10"
                },
                "downloads": -1,
                "filename": "rezina-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "438976e8d5e4624fcc567eaaff8245ef",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 2450561,
                "upload_time": "2016-12-03T19:57:27",
                "url": "https://files.pythonhosted.org/packages/29/76/01a3546ed4c3cb4a9154fbcf6557b4cf685f0a2b90661d8b2055e50de5e0/rezina-0.1.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "78c394e50fce27572887235a0c527c89",
                    "sha256": "2af92185b5ceeb73b72d59d9c3e81d22d03a450964e59f5ea6f0015fbedd6088"
                },
                "downloads": -1,
                "filename": "rezina-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "78c394e50fce27572887235a0c527c89",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2633806,
                "upload_time": "2016-12-03T20:01:48",
                "url": "https://files.pythonhosted.org/packages/95/7b/8abc20968d4a72c91374e9068055d4dca2528728778b241474e06a555ebb/rezina-0.1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "438976e8d5e4624fcc567eaaff8245ef",
                "sha256": "b6746e1253a81383829d264d4b03574088d1531242a315363e596c45b74aca10"
            },
            "downloads": -1,
            "filename": "rezina-0.1.1-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "438976e8d5e4624fcc567eaaff8245ef",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 2450561,
            "upload_time": "2016-12-03T19:57:27",
            "url": "https://files.pythonhosted.org/packages/29/76/01a3546ed4c3cb4a9154fbcf6557b4cf685f0a2b90661d8b2055e50de5e0/rezina-0.1.1-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "78c394e50fce27572887235a0c527c89",
                "sha256": "2af92185b5ceeb73b72d59d9c3e81d22d03a450964e59f5ea6f0015fbedd6088"
            },
            "downloads": -1,
            "filename": "rezina-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "78c394e50fce27572887235a0c527c89",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 2633806,
            "upload_time": "2016-12-03T20:01:48",
            "url": "https://files.pythonhosted.org/packages/95/7b/8abc20968d4a72c91374e9068055d4dca2528728778b241474e06a555ebb/rezina-0.1.1.tar.gz"
        }
    ]
}