{
    "info": {
        "author": "Itamar Turner-Trauring",
        "author_email": "itamar@itamarst.org",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3"
        ],
        "description": "# dask-memusage\n\nIf you're using Dask with tasks that use a lot of memory, RAM is your bottleneck for parallelism.\nThat means you want to know how much memory each task uses:\n\n1. So you can set the highest parallelism level (process or threads) for each machine, given available to RAM.\n2. In order to know where to focus memory optimization efforts.\n\n`dask-memusage` is an MIT-licensed statistical memory profiler for Dask's Distributed scheduler that can help you with both these problems.\n\n`dask-memusage` polls your processes for memory usage and records the minimum and maximum usage in a CSV:\n\n```csv\ntask_key,min_memory_mb,max_memory_mb\n\"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)\",44.84765625,96.98046875\n\"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)\",47.015625,97.015625\n\"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)\",0,0\n\"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)\",0,0\nsum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0\napply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625\ntask_key,min_memory_mb,max_memory_mb\n\"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 0)\",44.84765625,96.98046875\n\"('from_sequence-map-sum-part-e15703211a549e75b11c63e0054b53e5', 1)\",47.015625,97.015625\n\"('sum-part-e15703211a549e75b11c63e0054b53e5', 0)\",0,0\n\"('sum-part-e15703211a549e75b11c63e0054b53e5', 1)\",0,0\nsum-aggregate-apply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,0,0\napply-no_allocate-4c30eb545d4c778f0320d973d9fc8ea6,47.265625,47.265625\n```\n\n## Usage\n\n*Important:* Make sure your workers only have a single thread! Otherwise the results will be wrong.\n\n### Installation\n\nOn the machine where you are running the Distributed scheduler, run:\n\n```console\n$ pip install dask_memusage\n```\n\n### API usage\n\n```python\n# Add to your Scheduler object, which is e.g. your LocalCluster's scheduler\n# attribute:\nfrom dask_memoryusage import install\ninstall(scheduler, \"/tmp/memusage.csv\")\n```\n\n### CLI usage\n\n```console\n$ dask-scheduler --preload dask_memusage --memusage.csv /tmp/memusage.csv\n```\n\n## Limitations\n\n* Again, make sure you only have one thread per worker process.\n* This is statistical profiling, running every 10ms.\n  Tasks that take less than that won't have accurate information.\n\n## Help\n\nNeed help? File a ticket at https://github.com/itamarst/dask-memusage/issues/new\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/itamarst/dask-memusage",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "dask_memusage",
        "package_url": "https://pypi.org/project/dask_memusage/",
        "platform": "",
        "project_url": "https://pypi.org/project/dask_memusage/",
        "project_urls": {
            "Homepage": "https://github.com/itamarst/dask-memusage"
        },
        "release_url": "https://pypi.org/project/dask_memusage/1.0/",
        "requires_dist": [
            "distributed",
            "click",
            "pytest; extra == \"test\"",
            "flake8; extra == \"test\"",
            "numpy; extra == \"test\""
        ],
        "requires_python": ">=3.5",
        "summary": "Low-impact, task-level memory profiling for Dask.",
        "version": "1.0"
    },
    "last_serial": 4320098,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7fd515960b056169cc4bb71aa3661ca1",
                    "sha256": "32c82e9a97d1133da9519bf3fc25f31f6e4bf4a45d2a941540c3b0716cc6b67f"
                },
                "downloads": -1,
                "filename": "dask_memusage-1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7fd515960b056169cc4bb71aa3661ca1",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5",
                "size": 7249,
                "upload_time": "2018-09-28T13:58:41",
                "url": "https://files.pythonhosted.org/packages/a8/45/884279ea392565bded3582f835bbeac8681a900c454338eef22b1848c626/dask_memusage-1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c53999ddb8af64147a63a7ab401ada5e",
                    "sha256": "44f05a009a256f5fe1af323970b212f024bad9150a649e595776185c3611bf47"
                },
                "downloads": -1,
                "filename": "dask_memusage-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "c53999ddb8af64147a63a7ab401ada5e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5",
                "size": 6940,
                "upload_time": "2018-09-28T13:59:21",
                "url": "https://files.pythonhosted.org/packages/72/97/30ba80d5b782fdd4ebf497612f8475d587e8cec3043f6d6a157cb3dc06cc/dask_memusage-1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "7fd515960b056169cc4bb71aa3661ca1",
                "sha256": "32c82e9a97d1133da9519bf3fc25f31f6e4bf4a45d2a941540c3b0716cc6b67f"
            },
            "downloads": -1,
            "filename": "dask_memusage-1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "7fd515960b056169cc4bb71aa3661ca1",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.5",
            "size": 7249,
            "upload_time": "2018-09-28T13:58:41",
            "url": "https://files.pythonhosted.org/packages/a8/45/884279ea392565bded3582f835bbeac8681a900c454338eef22b1848c626/dask_memusage-1.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "c53999ddb8af64147a63a7ab401ada5e",
                "sha256": "44f05a009a256f5fe1af323970b212f024bad9150a649e595776185c3611bf47"
            },
            "downloads": -1,
            "filename": "dask_memusage-1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "c53999ddb8af64147a63a7ab401ada5e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5",
            "size": 6940,
            "upload_time": "2018-09-28T13:59:21",
            "url": "https://files.pythonhosted.org/packages/72/97/30ba80d5b782fdd4ebf497612f8475d587e8cec3043f6d6a157cb3dc06cc/dask_memusage-1.0.tar.gz"
        }
    ]
}