{ "info": { "author": "Jan Kalkan", "author_email": "jan.kalkan@mailbox.org", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "\n

\n lazycluster\n
\n

\n\n

\n Distributed machine learning made simple.\n

\n\n

\n \n \n \n

\n\n

\n Getting Started \u2022\n Highlights \u2022\n Features \u2022\n API Docs \u2022\n Support \u2022\n Report a Bug \u2022\n Contribution\n

\n\n**lazycluster** is a Python library intended to liberate data scientists and machine learning engineers by abstracting \naway cluster management and configuration so that they are be able to focus on its actual tasks. Especially, the easy \nand convenient cluster setup with Python for various distributed machine learning frameworks is emphasized.\n\n## Highlights\n\n- **High-Level API for starting clusters:** \n - DASK\n - PyTorch *(WIP)* \n - *Further supported *lazyclusters* to come ...*\n- **Lower-level API for:**\n - Managing Runtimes or RuntimeGroups to:\n - a-/synchronously execute RuntimeTasks remotely by leveraging the power of ssh\n - expose services (e.g. a DB) from or to a Runtimes or in a whole RuntimeGroup\n\n## Getting Started\n\n### Installation\n\n`pip install lazycluster` \n\n### Usage Example\n\nPrerequisite: Passwordless ssh needs to be setup for the used hosts.\n\n```python\nfrom lazycluster import RuntimeTask, Runtime\n\n# Define a Python function which will be executed remotely\ndef hello(name:str):\n return 'Hello ' + name + '!'\n\n# Compose a `RuntimeTask`\ntask = RuntimeTask('my-first_task').run_command('echo Hello World!') \\\n .run_function(hello, name='World')\n\n# Actually execute it remotely in a `Runtime` \ntask = Runtime('host-1').execute_task(task, execute_async=False)\n\n# The stdout from from the executing `Runtime` can be accessed via the execution log of teh `RuntimeTask`\ntask.print_log()\n\n# Print the return of the `hello()` call\ngenerator = task.function_returns\nprint(next(generator))\n```\n## Support\n\nThe **lazycluster** project is maintained by [Jan Kalkan](https://www.linkedin.com/in/jan-kalkan-b5390284/). Please \nunderstand that we won't be able to provide individual support via email. We also believe that help is much more\nvaluable if it's shared publicly so that more people can benefit from it.\n\n| Type | Channel |\n| ------------------------ | ------------------------------------------------------ |\n| \ud83d\udea8 **Bug Reports** | |\n| \ud83c\udf81 **Feature Requests** | |\n| \ud83d\udc69\u200d\ud83d\udcbb **Usage Questions** | |\n| \ud83d\uddef **General Discussion** | \n\n## Features\n\n### Create *Runtimes* & *RuntimeGroups*\n```python\nfrom lazycluster import Runtime, RuntimeGroup\n\nrt_1 = Runtime('host-1')\nrt_2 = Runtime('host-1', root_dir='/workspace')\n\nruntime_group = RuntimeGroup([rt_1, rt_2])\nruntime_group = RuntimeGroup(hosts=['host-1', 'host-2'])\n```\n\n### Use *RuntimeManager* to create a *RuntimeGroup* based on the local ssh config\n```python\nfrom lazycluster import RuntimeManager, RuntimeGroup\n\nruntime_group = RuntimeManager().create_group()\n```\n\n### Easily launch a DASK cluster \n```python\nfrom lazycluster import RuntimeManager\nfrom lazycluster.cluster.dask_cluster import DaskCluster\n\ncluster = DaskCluster(RuntimeManager().create_group())\ncluster.start()\n```\n\n### Expose a service from or to a *Runtime*\n```python\nfrom lazycluster import Runtime\n\n# Create a Runtime\nruntime = Runtime('host-1')\n\n# Make the port 50000 from the Runtime accessible on localhost\nruntime.expose_port_from_runtime(50000)\n\n# Make the local port 40000 accessible on the Runtime\nruntime.expose_port_to_runtime(40000)\n\n```\n\n### Expose a service to a whole *RuntimeGroup* or from one contained Runtime in the RuntimeGroup\n```python\nfrom lazycluster import RuntimeGroup\n\n# Create a RuntimeGroup\nruntime_group = RuntimeGroup('host1', 'host-2', 'host-3')\n\n# Make the local port 50000 accessible on all Runtimes contained in the RuntimeGroup\nruntime_group.expose_port_to_runtimes(50000)\n\n\n# Make the port 40000 which is running on host-1 accessible on all other Runtimes in the RuntimeGroup\nruntime_group.expose_port_from_runtime_to_group('host-1', 40000)\n```\n\n## Contribution\n\n- Pull requests are encouraged and always welcome. Read [`CONTRIBUTING.md`](https://github.com/ml-tooling/lazycluster/tree/master/CONTRIBUTING.md) and check out [help-wanted](https://github.com/ml-tooling/lazycluster/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+label%3A\"help+wanted\"+sort%3Areactions-%2B1-desc+) issues.\n- Submit github issues for any [feature enhancements](https://github.com/ml-tooling/lazycluster/issues/new?assignees=&labels=feature-request&template=02_feature-request.md&title=), [bugs](https://github.com/ml-tooling/lazycluster/issues/new?assignees=&labels=bug&template=01_bug-report.md&title=), or [documentation](https://github.com/ml-tooling/lazycluster/issues/new?assignees=&labels=enhancement%2C+docs&template=03_documentation.md&title=) problems. \n- By participating in this project you agree to abide by its [Code of Conduct](https://github.com/ml-tooling/lazycluster/tree/master/CODE_OF_CONDUCT.md).\n\n---\n\nLicensed **Apache 2.0**. Created and maintained with \u2764\ufe0f by developers from SAP in Berlin.\n\nRequirements:\n['fabric >= 2.2', 'stormssh', 'cloudpickle', 'distributed', 'psutil']\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/ml-tooling/lazycluster.git", "keywords": "", "license": "Apache License 2.0", "maintainer": "", "maintainer_email": "", "name": "lazycluster", "package_url": "https://pypi.org/project/lazycluster/", "platform": "", "project_url": "https://pypi.org/project/lazycluster/", "project_urls": { "Homepage": "https://github.com/ml-tooling/lazycluster.git" }, "release_url": "https://pypi.org/project/lazycluster/0.1.1/", "requires_dist": [ "fabric (>=2.2)", "stormssh", "cloudpickle", "distributed", "psutil" ], "requires_python": ">=3.6", "summary": "Distributed machine learning made simple.", "version": "0.1.1" }, "last_serial": 5698022, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "f3089d5d9d735bfd49733d953358fd07", "sha256": "7891266935f853b78d1e3831b00ada097850fb39bce7ea2d8eec7187d9c4c869" }, "downloads": -1, "filename": "lazycluster-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "f3089d5d9d735bfd49733d953358fd07", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 33384, "upload_time": "2019-08-09T15:19:02", "url": "https://files.pythonhosted.org/packages/39/93/bdf27d439d43d404bb1e29edd517096aa0a37a55fea7e1522fb90e712af7/lazycluster-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ab7268dd16a25efa5fabe40d5fa08aea", "sha256": "f04202d4f87e0c843d3570f1429282dcd9a2282b544a0d9ab70e70ef2ef0cc22" }, "downloads": -1, "filename": "lazycluster-0.1.0.tar.gz", "has_sig": false, "md5_digest": "ab7268dd16a25efa5fabe40d5fa08aea", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 27389, "upload_time": "2019-08-09T15:19:05", "url": "https://files.pythonhosted.org/packages/96/b4/02156a7e60103ffc12baabfedd9eadde1e738cd7309d66dbfcbbeb745de0/lazycluster-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "d5a456a82d8a30e011177e04456a8524", "sha256": "9bdc22e4bee71e16666f6d2acbc6c7b536216264db450a74303d161ad3df947e" }, "downloads": -1, "filename": "lazycluster-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d5a456a82d8a30e011177e04456a8524", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 36456, "upload_time": "2019-08-19T12:12:55", "url": "https://files.pythonhosted.org/packages/de/bc/a3884ecd233a2fdab89936b9b6fc8e786eecb8ff77691cb6368998e09f7f/lazycluster-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "23b8ac8ba1a17ec231f6788c3a4a6289", "sha256": "3803d97e25147494079d631303252874a0f4e9df12fba9f6594484707a223d1a" }, "downloads": -1, "filename": "lazycluster-0.1.1.tar.gz", "has_sig": false, "md5_digest": "23b8ac8ba1a17ec231f6788c3a4a6289", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 30612, "upload_time": "2019-08-19T12:12:57", "url": "https://files.pythonhosted.org/packages/16/3b/97798d74f4d80b9a1aa39cfc322a4cbf8e7a1b40207aa73b4ab535e48164/lazycluster-0.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "d5a456a82d8a30e011177e04456a8524", "sha256": "9bdc22e4bee71e16666f6d2acbc6c7b536216264db450a74303d161ad3df947e" }, "downloads": -1, "filename": "lazycluster-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d5a456a82d8a30e011177e04456a8524", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6", "size": 36456, "upload_time": "2019-08-19T12:12:55", "url": "https://files.pythonhosted.org/packages/de/bc/a3884ecd233a2fdab89936b9b6fc8e786eecb8ff77691cb6368998e09f7f/lazycluster-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "23b8ac8ba1a17ec231f6788c3a4a6289", "sha256": "3803d97e25147494079d631303252874a0f4e9df12fba9f6594484707a223d1a" }, "downloads": -1, "filename": "lazycluster-0.1.1.tar.gz", "has_sig": false, "md5_digest": "23b8ac8ba1a17ec231f6788c3a4a6289", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 30612, "upload_time": "2019-08-19T12:12:57", "url": "https://files.pythonhosted.org/packages/16/3b/97798d74f4d80b9a1aa39cfc322a4cbf8e7a1b40207aa73b4ab535e48164/lazycluster-0.1.1.tar.gz" } ] }