{
    "info": {
        "author": "Kriszti\u00e1n Sz\u0171cs",
        "author_email": "krisztian.szucs@lensa.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "[![Build Status](http://drone.lensa.com:8000/api/badges/lensacom/satyr/status.svg)](http://drone.lensa.com:8000/lensacom/satyr)\n[![Join the chat at https://gitter.im/lensacom/satyr](https://badges.gitter.im/lensacom/satyr.svg)](https://gitter.im/lensacom/satyr?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n\n![satyr](https://s3.amazonaws.com/lensa-rnd-misc/satyr2.png)\n\n# An extensible Mesos library for Python\n###### aka. the distributed snake-charmer\n\n\nSatyr's intention is to simplify the process of writing python frameworks\nfor Mesos. Satyr provides multiple components and interfaces to cover various\nlevels of complexity needs.\n\n## Notable Features\n\n- Comfortable Pythonic interface instead of the C++ syntax\n- Magical Protobuf wrapper to easily extend messages with custom functionality\n- Multiple weighted Bin-Packing heuristics for optimized scheduling\n- Easily extensibe QueueScheduler implementation\n- Python multiprocessing.Pool interface\n\n## Install\n\n`pip install satyr` or use [lensa/satyr](https://hub.docker.com/r/lensa/satyr/) Docker image\n\nRequirements:\n- mesos.interface (installable via pip)\n- mesos.native (binary .egg downloadable from mesosphere.io)\n\nConfiguration:\n- `MESOS_MASTER=zk://127.0.0.1:2181/mesos`\n\n\n## Examples\n\n### Futures Interface\n\nIt's almost identical to python's\n[futures interface](https://docs.python.org/3/library/concurrent.futures.html)\nbut runs processes on a Mesos cluster (concurrently).\n\n```python\nfrom satyr.apis.futures import MesosPoolExecutor\nfrom satyr.proxies.messages import Cpus, Mem\n\nwith MesosPoolExecutor(name='futures-pool') as executor:\n    def mul(a, b):\n        return a * b\n\n    future = executor.submit(mul, args=[3, 5])\n    assert future.result(timeout=5) == 3\n\n    it = executor.map(mul, range(10), range(10), timeout=5,\n                      resources=[Cpus(0.1), Mem(128)])\n    assert list(it) == [i**2 for i in range(10)]\n```\n\n### Multiprocessing\n\nIt's similar to python's\n[multiprocessing interface](https://docs.python.org/2/library/multiprocessing.html)\nbut runs processes on a Mesos cluster (concurrently).\n\n```python\nfrom __future__ import print_function\nfrom satyr.apis.multiprocessing import Pool\n\n\nwith Pool(name='satyr-pool') as pool:\n    def mul(a, b):\n        return a * b\n\n    res1 = pool.apply_async(lambda a, b: a + b, [1, 2])\n    res2 = pool.apply_async(mul, [2, 3])\n    pool.wait()\n\n    print(res1.get())\n    print(res2.get())\n```\n\n### Work Queue Scheduler\n\nBasic scheduler to submit various kind of workloads, eg.:\n - bash commands\n - docker executable containers\n - python callables\n - customized tasks (e.g. function executed via pypy)\n\n```python\nfrom __future__ import print_function\nfrom satyr.scheduler import QueueScheduler, Running\nfrom satyr.messages import PythonTask\nfrom satyr.proxies.messages import Disk, Mem, Cpus\n\n\nscheduler = QueueScheduler()\ntask = PythonTask(fn=sum, args=[range(10)], name='satyr-task',\n                  resources=[Cpus(0.1), Mem(128), Disk(512)])\n\nwith Running(scheduler, name='satyr-scheduler'):\n    res = scheduler.submit(task)  # return AsyncResult\n    print(res.get(timeout=30))\n```\n\n### Custom Scheduler\n\nYou can make your own scheduler built on QueueScheduler or for more complex\nneeds there's a [Scheduler](satyr/interface.py) interface which you can use\nto create one from scratch. (However in this case you'll have to implement\nsome of the functionalities already in [QueueScheduler](satyr/scheduler.py))\n\n```python\nfrom __future__ import print_function\nfrom satyr.scheduler import QueueScheduler, Running\nfrom satyr.messages import PythonTask\nfrom satyr.proxies.messages import Disk, Mem, Cpus\n\n\nclass CustomScheduler(QueueScheduler):\n\n    def on_update(self, driver, status):\n        \"\"\"You can hook on the events defined in the Scheduler interface.\n\n        They're just more conveniantly named methods for the basic\n        mesos.interface functions but this is how you can add some\n        custom logic to your framework in an easy manner.\n        \"\"\"\n        logging.info(\n            \"Status update received for task {}\".format(status.task_id))\n        super(CustomScheduler, self).on_update(driver, status)\n\n\nscheduler = CustomScheduler()\ntask = PythonTask(fn=sum, args=[range(9)], name='satyr-task',\n                  resources=[Cpus(0.1), Mem(128), Disk(512)])\n\nwith Running(scheduler, name='satyr-custom-scheduler'):\n    res = scheduler.submit(task)\n    print(res.get(timeout=60))\n```\n\nAlso this way you can easily implement your own resource offer handling logic by\noverriding the `on_offers(self, driver, offers)` method in which we give you a\nhelping hand with comparable Offers and TaskInfos (basic arithmetic operators\nare also overloaded).\n\n```python\nfrom satyr.interface import Scheduler\nfrom satyr.proxies.messages import Offer, TaskInfo\n\n\nclass CustomScheduler(Scheduler):\n    ...\n    def on_offers(self, driver, offers):\n        ...\n        task = self.get_next_task()\n        for offer in offers\n            if task < offer:\n                task.slave_id = offer.slave_id\n                driver.launch(offer, [task])\n        # decline unused offers or launch with empty task list\n        ...\n```\n\n## Optimized Task Placement\n\nSatyr implements multiple weighted heuristics to solve the\n[Bin-Packing Problem](https://en.wikipedia.org/wiki/Bin_packing_problem):\n\n- First-Fit\n- First-Fit-Decreasing\n- Max-Rest\n- Best-Fit\n- Best-Fit-Decreasing\n\nsee [binpack.py](satyr/binpack.py).\n\nThe benefits of using bin-packing has been proven by\n[Netflix/Fenzo](https://github.com/Netflix/Fenzo) in\n[Heterogeneous Resource Scheduling Using Apache Mesos](http://events.linuxfoundation.org/sites/events/files/slides/Prezo-at-MesosCon2015-Final.pdf)\n\n## Built in Task Types\n\n### Command\n\nThe most basic task executes a simple command, Mesos will run CommandInfo's\nvalue with `/bin/sh -c`. Also, if you want to run your task in a Docker\ncontainer you can provide some additional information for the task.\n\n```python\nfrom satyr.proxies.messages import TaskInfo, CommandInfo\n\n\ntask = TaskInfo(name='command-task', command=CommandInfo(value='echo 100'))\ntask.container.type = 'DOCKER'\ntask.container.docker.image = 'lensacom/satyr:latest'\n```\n\n### Python\n\n[PythonTask](/satyr/messages.py) is capable of running arbitrary python code on\nyour cluster. It sends [cloudpickled](https://github.com/cloudpipe/cloudpickle)\nmethods and arguments to the matched mesos-slave for execution.\nNote that python tasks run in [lensa/satyr](https://hub.docker.com/r/lensa/satyr/)\nDocker container by default.\n\n```python\nfrom satyr.messages import PythonTask\n\n\n# You can pass a function or a lambda in place of sum for fn.\ntask = PythonTask(name='python-task', fn=sum, args=[range(5)])\n```\n\n## Custom Task\n\nCustoms tasks can be written by extending [TaskInfo](/satyr/proxies/messages.py)\nor any existing descendants.\nIf you're walking down the former path you'll most likely have to deal with\nprotobuf in your code; worry not, we have some magic wrappers for you to provide\ncustomizable messages.\n\n```python\nfrom __future__ import print_function\nfrom satyr.proxies.messages import TaskInfo\nfrom mesos.interface import mesos_pb2\n\n\nclass CustomTask(TaskInfo):\n    # descriptive protobuf template the wrapper matched against\n    proto = mesos_pb2.TaskInfo(\n        labels=mesos_pb2.Labels(\n            labels=[mesos_pb2.Label(key='custom')]))\n\n    @property\n    def uppercase_task_name():\n        return self.name.upper()\n\n    def on_update(self, status):\n         logging.info('Custom task has received a status update')\n\n    def custom_method(self):\n         print(\"Arbitrary stuff\")\n```\n\n## One-Off Executor\n\nThis Executor implementation simply runs the received python function with the\nprovided arguments, then sends back the result in a reliable fashion.\n\n```py\nclass OneOffExecutor(Executor):\n\n    def on_launch(self, driver, task):\n        def run_task():\n            driver.update(task.status('TASK_RUNNING'))\n            logging.info('Sent TASK_RUNNING status update')\n\n            try:\n                logging.info('Executing task...')\n                result = task()\n            except Exception as e:\n                logging.exception('Task errored')\n                driver.update(task.status('TASK_FAILED', message=e.message))\n                logging.info('Sent TASK_RUNNING status update')\n            else:\n                driver.update(task.status('TASK_FINISHED', data=result))\n                logging.info('Sent TASK_FINISHED status update')\n\n        thread = threading.Thread(target=run_task)\n        thread.start()\n```\n\n## Warning (at the end)\n\nThis is a pre-release!\n\n- proper documentation\n- python futures api\n- more detailed examples\n- and CONTRIBUTION guide\n- dask.mesos backend\n\nare coming!",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://github.com/lensacom/satyr",
        "keywords": "mesos framework multiprocessing",
        "license": "Apache License, Version 2.0",
        "maintainer": null,
        "maintainer_email": null,
        "name": "satyr",
        "package_url": "https://pypi.org/project/satyr/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/satyr/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "http://github.com/lensacom/satyr"
        },
        "release_url": "https://pypi.org/project/satyr/0.2.1/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "Extensible Python Framework for Apache Mesos",
        "version": "0.2.1"
    },
    "last_serial": 2297766,
    "releases": {
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "33f0d90553297f0c796ccd20b3e6b87e",
                    "sha256": "52d7c88b8f9c1c40b09e61873c6ced4e768939bc1f284c29715c6bcc08ba84be"
                },
                "downloads": -1,
                "filename": "satyr-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "33f0d90553297f0c796ccd20b3e6b87e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 30878,
                "upload_time": "2016-05-09T13:44:14",
                "url": "https://files.pythonhosted.org/packages/02/22/7093d09ac6f80c584cb7a232be3aceec46d34c2732a142c70e0f679d25e6/satyr-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5fec48c8c10055e1d16b2a01a9c1b949",
                    "sha256": "ecf3eb871945cb8f56a575670d250ddc56ccc5afffd1dfc34aa4c1d01b94ee21"
                },
                "downloads": -1,
                "filename": "satyr-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "5fec48c8c10055e1d16b2a01a9c1b949",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 31174,
                "upload_time": "2016-05-13T10:59:08",
                "url": "https://files.pythonhosted.org/packages/6c/4b/a4c16cf4de373187ec0ecbe4f21db5d9ae6f56314b052442e51d382fc429/satyr-0.1.4.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a2c9f471329b9e817106605832718b60",
                    "sha256": "1215b48427beb77ba25cf7e54e2848915524b44e562502ce7a5151a42abf836b"
                },
                "downloads": -1,
                "filename": "satyr-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "a2c9f471329b9e817106605832718b60",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32802,
                "upload_time": "2016-06-09T08:04:19",
                "url": "https://files.pythonhosted.org/packages/ef/42/61cc8f6e37f0c9e3de8897c8e099114e70c42998d45577047a0070a8fd08/satyr-0.2.0.tar.gz"
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "30e19dcf3ed76df27c88010a38ef8e31",
                    "sha256": "f25a109de4a54a99640dd5c7192b320eda710c9ecec5722df04e8be743b5a088"
                },
                "downloads": -1,
                "filename": "satyr-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "30e19dcf3ed76df27c88010a38ef8e31",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 34195,
                "upload_time": "2016-08-23T12:34:12",
                "url": "https://files.pythonhosted.org/packages/07/7a/646b3b31dc57f35010bcdabd0ff51d320578f614443aa071c24d89629ea0/satyr-0.2.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "30e19dcf3ed76df27c88010a38ef8e31",
                "sha256": "f25a109de4a54a99640dd5c7192b320eda710c9ecec5722df04e8be743b5a088"
            },
            "downloads": -1,
            "filename": "satyr-0.2.1.tar.gz",
            "has_sig": false,
            "md5_digest": "30e19dcf3ed76df27c88010a38ef8e31",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 34195,
            "upload_time": "2016-08-23T12:34:12",
            "url": "https://files.pythonhosted.org/packages/07/7a/646b3b31dc57f35010bcdabd0ff51d320578f614443aa071c24d89629ea0/satyr-0.2.1.tar.gz"
        }
    ]
}