{
    "info": {
        "author": "Sebastien Campion",
        "author_email": "sebastien.campion@inria.fr",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 1 - Planning",
            "License :: OSI Approved",
            "Natural Language :: French",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3.6",
            "Topic :: Communications"
        ],
        "description": "![](orqal/static/images/orqal.svg) \n\nOrqal for [ORQ]chestration of [AL]gorithms is a simple batch scheduler for docker cluster which can be used remotly and without overhead in scientific experiment.\n\n## \ud83d\udcd0 Design\n\n### Overview\n\tOrqal claim to be and stay as simple as possible. \n![](orqal/static/images/orqal_overview.svg)\n\n- We use the docker api to manage docker nodes, which means that there is **no configuration** on node except open the api port. \n- **HTTP/REST API** enable to schedule jobs and retrieve data.\n- A **dashboard** is provided to monitor load average per nodes, jobs scheduling and redoc api.\n\n![](orqal/static/images/dashboard.png)\n\n### Wrapper\n\nThe glue between docker image and orqal need to be implemented in python with the ArbstractWorker class in order to :\n- get command to execute\n- set the result if necessary\n\nHere is a simple example using radare2 \n\n\t\n```python\nclass Rabin2(AbstractWorker):\n    docker_url = \"radare/radare2\"\n    volumes = {'/database': {'bind': '/database', 'mode': 'ro'}}\n    threads = 1\n    memory_in_gb = 1\n\t\n    def get_cmd(self, params):\n        return \"rabin2 -I %s\" % self.job.input\n\t\n    def set_result(self, job):\n        r = {l.split()[0].replace('.', '_'): l.split()[1] for l in job.stdout if len(l.split()) == 2}\n        job.set_result(r)\n```\n\n## Install \n\n    pip3 install orqal \n\n## Run \n\n\nStart the web worker : \n\n    orqal-worker\n\n\nStart the web interface : \n\n    orqal-web\n    \nNB: In a production mode, use gunicorn, in orqal dir : \n\n    gunicorn3 web:app --bind 0.0.0.0:5001 --worker-class aiohttp.GunicornWebWorker --workers 8 --timeout 120\nuser\n\n\n    \n\n## Configuration \n\nIn order to establish a DB connection, process will search the following environment variable : \n    \n    ORQAL_MONGO_URI\n\nBy default :  'mongodb://localhost/'\n\nOther settings will be loaded from the conf collection, using the request : active=True\n\nHere is a example : \n\n`NB : The first run will initiate it if necessary`\n\n```json\n    {\n        \"_id\" : ObjectId(\"5d1a78eca307b20dd5a660ce\"),\n        \"active\" : true,\n        \"mongourl\" : \"mongodb://localhost/\",\n        \"docker_hosts\" : [ \n            \"192.168.100.51:2376\", \n            .....\n            \"192.168.100.64:2376\"\n        ],\n        \"docker_api_version\" : \"1.37\",\n        \"mongo_replicaset\" : \"madlabReplSet\",\n        \"registry_auth_config\" : {\n            \"password\" : \"65sX2-9sSXSp-hs-XeZ8\",\n            \"username\" : \"test\"\n        },\n        \"services\" : \"/home/madlab/services.py\",\n        \"nb_disp_jobs\" : 30,\n        \"contact\" : \"orqal@example.com\",\n        \"jobs_dir\" : \"/scratch/jobs\"\n    }\n```\n\n\n\n\n## Services\n\nThe python module is defined in the configuration (see services above)\n\nHere is a example : \n\n```python\nimport json\nimport logging\nimport conf\nimport os\nfrom orqal.abstract import AbstractWrapper\n\nclass TestProd(AbstractWrapper):\n    docker_url = \"madlab:5000/test_module\"\n\n    def get_cmd(self, params):\n        return \"python3 simple_job.py %s %s %s\" % (params['echo'], params['time'], params['exit_code'])\n\n    def set_result(self, job):\n        job.set_result(\"My results\")\n\n\nclass AngrExtraction(AbstractWrapper):\n    docker_url = \"madlab:5000/scdg/madlab-v2\"\n    threads = 1\n    memory_in_gb = 10\n    create_dir = True\n\n    def get_cmd(self, params):\n        return \"python /code/src/interfaces/cli.py %s params.json -o calls.json\" % self.job.input\n\n    def set_result(self, job):\n        return os.path.join(self.job.wd, \"calls.json\")\n```\n\n\n\n### DB tuning : log collection in Mongo \n\nProcess use a mongo logging handler, in order to provide it in the web interface, you can customize parameters like that : \n\n    use orqal\n    db.createCollection('log', {capped:true, size:10000000})\u00a0\n    db.log.createIndex( { \"time\": 1 }, { expireAfterSeconds: 86400 } )\n\n\nCache : \n\n    use orqal\n    db.jobs.createIndex( { input: 1 } )\n    db.jobs.createIndex( { params: 1 } )\n    db.jobs.createIndex( { app: 1 } )\n\n\n\n## Distribution\n\n```\t\n   python setup.py bdist\n   python3 setup.py sdist\n   twine upload dist/*\n```\n\n### Licence\n\nGNU AFFERO GENERAL PUBLIC LICENSE\nhttps://www.gnu.org/licenses/agpl-3.0.txt\n\n### Credits \n\n- Dashboard template : https://github.com/puikinsh/sufee-admin-dashboard\n- Font : https://fonts.google.com/specimen/Righteous\n\n### FAQ \n#### How to open api port on docker ?\n\nAdd in file `/etc/systemd/system/docker.service.d/override.conf`\n\n    [Service]\n    ExecStart=\n    ExecStart=/usr/bin/docker daemon -H fd://  -H tcp://0.0.0.0:2376 -s overlay --insecure-registry orqal:5000\n    \n\nThen flush changes by executing :\n    \n    systemctl daemon-reload\n\nverify that the configuration has been loaded:\n    \n    systemctl show --property=ExecStart docker\n\nrestart docker:\n    \n    systemctl restart docker\n\n\n\n\n### Clean old jobs\n\nAdd a index (7 days here):\n\n\tdb.jobs.createIndex( { \"ctime\": 1 }, { expireAfterSeconds: 604800 } )\n\nAnd call http://<server>/api/clean/old periodically in order to delete old job directories.",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "AGPL",
        "maintainer": "",
        "maintainer_email": "",
        "name": "orqal",
        "package_url": "https://pypi.org/project/orqal/",
        "platform": "",
        "project_url": "https://pypi.org/project/orqal/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/orqal/0.0.12/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Orchestration of Algorithm on docker cluster",
        "version": "0.0.12"
    },
    "last_serial": 5539075,
    "releases": {
        "0.0.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e187f6005b43861ad1672e84c13e4b9e",
                    "sha256": "2e6ff290355346cd923595aeea94b813e738d71a2d8aadfcff07aa3c2382e71f"
                },
                "downloads": -1,
                "filename": "orqal-0.0.10.tar.gz",
                "has_sig": false,
                "md5_digest": "e187f6005b43861ad1672e84c13e4b9e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13729716,
                "upload_time": "2019-07-04T20:49:58",
                "url": "https://files.pythonhosted.org/packages/6a/e4/c9bb5e407948d11c297eaa56bccbafc7505393024ca29976a00a288b614e/orqal-0.0.10.tar.gz"
            }
        ],
        "0.0.11": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ced0773019784597d0adbec022378599",
                    "sha256": "545eecadf76704e6e3f3ec5a92ef5e00dfe1f1b5e437b587c43db7a6931a85d5"
                },
                "downloads": -1,
                "filename": "orqal-0.0.11.tar.gz",
                "has_sig": false,
                "md5_digest": "ced0773019784597d0adbec022378599",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13744039,
                "upload_time": "2019-07-12T10:23:39",
                "url": "https://files.pythonhosted.org/packages/2d/c2/e5cd1cebbe0e28f49810890b7d9364dabe3f2f77b331e39e501e6ee4d687/orqal-0.0.11.tar.gz"
            }
        ],
        "0.0.12": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a9ec4af6148305e04b50b4fcf8f7c78",
                    "sha256": "a227d05604a86e075986d1f1463675af44f6e87fe90dfc5a240adb34f80daea9"
                },
                "downloads": -1,
                "filename": "orqal-0.0.12.tar.gz",
                "has_sig": false,
                "md5_digest": "5a9ec4af6148305e04b50b4fcf8f7c78",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13744683,
                "upload_time": "2019-07-16T08:02:31",
                "url": "https://files.pythonhosted.org/packages/be/0b/c670bee5fa5c9f99b8982417044cdedd9cda0a4fedfddefbb9092b3c1a03/orqal-0.0.12.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "30af31cbd5c43e60ed2f2d1c5cee78b1",
                    "sha256": "ff0497dfab55cc7b11c8f98768adbe26da07999bdb22fb8c3ff4d97cbbf46cee"
                },
                "downloads": -1,
                "filename": "orqal-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "30af31cbd5c43e60ed2f2d1c5cee78b1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13783412,
                "upload_time": "2019-07-02T08:50:50",
                "url": "https://files.pythonhosted.org/packages/f3/f5/94faac41dd09823c48736270238376a431d9841a909c22430a7b01efd635/orqal-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "227956ae9380141a7700e1cebfcc3111",
                    "sha256": "73d890ba14b7907b95a47f37d07526500c684336c1ca98ef659135ed274e4e9f"
                },
                "downloads": -1,
                "filename": "orqal-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "227956ae9380141a7700e1cebfcc3111",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13783603,
                "upload_time": "2019-07-02T15:51:38",
                "url": "https://files.pythonhosted.org/packages/96/94/06bd0d02c976a723a4c77ea7018724142929e40ba7e6d2180ecaf5f7650b/orqal-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7d9a718ec4030eda111d43fac7b147d5",
                    "sha256": "fda1d298712ed11fe35cc3e8dd523e9634aab79e0a225265684184735c871131"
                },
                "downloads": -1,
                "filename": "orqal-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "7d9a718ec4030eda111d43fac7b147d5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13783512,
                "upload_time": "2019-07-04T15:36:01",
                "url": "https://files.pythonhosted.org/packages/25/ec/49dbe6a1a06b12bc8d9ba3c83e7ad0c8e46a67a43a2228d5214862b9dbc7/orqal-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5bd8fc8115b047d551697421c99116ef",
                    "sha256": "dc16595a02c2e458a27a9c40080853a9af4b1cef860a50e7725e6a76c2d6228f"
                },
                "downloads": -1,
                "filename": "orqal-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "5bd8fc8115b047d551697421c99116ef",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13728803,
                "upload_time": "2019-07-04T17:54:33",
                "url": "https://files.pythonhosted.org/packages/92/df/5c11b9c87f6b1d5443cccf607d874d075a229e859383e07ac5a0fd93660f/orqal-0.0.7.tar.gz"
            }
        ],
        "0.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c485938c0b57bc32cde9715c5b1286be",
                    "sha256": "ded67d0127b1cb44f80ba5fda308b2d857ca1be9ecdecd3cb9400b676c96bd01"
                },
                "downloads": -1,
                "filename": "orqal-0.0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "c485938c0b57bc32cde9715c5b1286be",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13728649,
                "upload_time": "2019-07-04T18:17:47",
                "url": "https://files.pythonhosted.org/packages/ca/0b/68ec98342d9d38534692c0e7159034b36ead2638b186315ccd90220c43d3/orqal-0.0.8.tar.gz"
            }
        ],
        "0.0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d98ca7de2bce10f65606a1b18cfd589",
                    "sha256": "ff84d94667c2a2cb6822c06679f515b756d52d0d59d03e7212f721423bd41d33"
                },
                "downloads": -1,
                "filename": "orqal-0.0.9.tar.gz",
                "has_sig": false,
                "md5_digest": "8d98ca7de2bce10f65606a1b18cfd589",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13728731,
                "upload_time": "2019-07-04T20:43:01",
                "url": "https://files.pythonhosted.org/packages/80/b6/f59bd82d5bf9b958fd3b67c4619fed8046c136e779b002919bb8cfea6c90/orqal-0.0.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5a9ec4af6148305e04b50b4fcf8f7c78",
                "sha256": "a227d05604a86e075986d1f1463675af44f6e87fe90dfc5a240adb34f80daea9"
            },
            "downloads": -1,
            "filename": "orqal-0.0.12.tar.gz",
            "has_sig": false,
            "md5_digest": "5a9ec4af6148305e04b50b4fcf8f7c78",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 13744683,
            "upload_time": "2019-07-16T08:02:31",
            "url": "https://files.pythonhosted.org/packages/be/0b/c670bee5fa5c9f99b8982417044cdedd9cda0a4fedfddefbb9092b3c1a03/orqal-0.0.12.tar.gz"
        }
    ]
}