{ "info": { "author": "Federico Poli", "author_email": "federpoli@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Console", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5" ], "description": "View Spark Timeline\n===================\n\nCommand line application to visualize the timeline of Spark executions, reading Spark's log files.\n\nA fundamental assumption is that all the executors are added before the Spark application submit jobs.\nThat is, this tool does not support dynamic scaling.\n\n\nInstallation\n------------\n\n.. code-block:: bash\n\n pip install view-spark-timeline\n\n\nExample\n-------\n\n.. code-block:: bash\n\n view-spark-timeline -i spark-logs/app-20171115015405-0219 -o docs/example-timeline.svg -u 100\n\nOutput:\n\n.. code-block:: text\n\n Read events from 'spark-logs/app-20171115015405-0219'...\n Total cores: 144\n Total duration: 187.3s\n Number of tasks: 10169\n Min task duration: 0.0s\n Max task duration: 92.2s\n Cluster utilization: 34.84%\n Drawing events...\n Read events from 'spark-logs/app-20171115015405-0219'...\n SVG size: 1500 720\n Saving SVG...\n\nProduced image: `docs/example-timeline.svg `_\n\n.. image:: docs/example-timeline.svg\n\n\nImage explanation\n-----------------\n\nOn the vertical axis we have the executor cores (grouped by executor).\nOn the horizontal axis we have the time, going from left to right.\nEach task is a horizontal bar that starts at a certain time on a core of an executor and ends after some time.\nThe color normally ranges from green, used for shorter tasks, to red, used for longer tasks. Failed tasks are black.\nAll the white space corresponds to some unused core.\n\nUsually, the greener the image is, the better. If there is a bottleneck in the execution it is easy to spot the guilty task(s).\nBy opening the SVG in a browser and by moving the mouse over a task there should appear a tooltip with the task ID.\nIt is then useful to inspect the task using the standard Spark UI.\n\n\nUsage\n-----\n\n.. code-block:: bash\n\n view-spark-timeline --help\n\nOutput:\n\n.. code-block:: text\n\n usage: view-spark-timeline [-h] -i INPUT_LOG -o OUTPUT_IMAGE\n [-t TIME_UNCERTAINTY] [-v]\n\n Visualize the timeline of a Spark execution from its log file. (v0.2.0)\n\n optional arguments:\n -h, --help show this help message and exit\n -i INPUT_LOG, --input-log INPUT_LOG\n path to the spark's application log\n -o OUTPUT_IMAGE, --output-image OUTPUT_IMAGE\n path of the output image\n -u TIME_UNCERTAINTY, --time-uncertainty TIME_UNCERTAINTY\n maximum allowed time uncertainty (in ms) of the\n timestamps in the log file. An high uncertainty\n determines a slower, but more robust, execution.\n (Default: 0)\n -v, --version print version and exit\n\n\nLicense\n-------\n\nCopyright (c) 2017, Federico Poli \n\nThis project, except for files in the `lib` folder, is released under the MIT license.\n\n\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/fpoli/view-spark-timeline", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "view-spark-timeline", "package_url": "https://pypi.org/project/view-spark-timeline/", "platform": "", "project_url": "https://pypi.org/project/view-spark-timeline/", "project_urls": { "Homepage": "https://github.com/fpoli/view-spark-timeline" }, "release_url": "https://pypi.org/project/view-spark-timeline/0.2.8/", "requires_dist": [ "svgwrite (==1.1.11)", "ujson (==1.35)", "nose (==1.3.7); extra == 'dev'", "pycodestyle (==2.3.1); extra == 'dev'", "twine (==1.9.1); extra == 'dev'" ], "requires_python": "", "summary": "Command line application to visualize the timeline of Spark executions.", "version": "0.2.8" }, "last_serial": 3341913, "releases": { "0.2.7": [ { "comment_text": "", "digests": { "md5": "d585c9cd8a402737f41e587cbdb4762f", "sha256": "2a5b5291584ec2c66e1e4f19f81e0c5efa644d9e0db93a84d47b3b03e639968a" }, "downloads": -1, "filename": "view_spark_timeline-0.2.7-py2-none-any.whl", "has_sig": false, "md5_digest": "d585c9cd8a402737f41e587cbdb4762f", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 10239, "upload_time": "2017-11-17T16:44:52", "url": "https://files.pythonhosted.org/packages/bf/24/0dd62d111383eb9f371325cea20c6ac3c5b96ebc4b0834acb7db9d4e3c2b/view_spark_timeline-0.2.7-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "2c71170c7fac1ebfa6624c151b8aa59f", "sha256": "973cb6d4281153f89eb6e4003db6e2bc28282acb6ecc54fc36721a12122c60e1" }, "downloads": -1, "filename": "view-spark-timeline-0.2.7.tar.gz", "has_sig": false, "md5_digest": "2c71170c7fac1ebfa6624c151b8aa59f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6856, "upload_time": "2017-11-17T16:44:54", "url": "https://files.pythonhosted.org/packages/7a/ae/f34befae5ef54a5a3f03b77de38dcdbf63350352698f98d9c2c32a763bf0/view-spark-timeline-0.2.7.tar.gz" } ], "0.2.8": [ { "comment_text": "", "digests": { "md5": "4eb618d01db4e38be1f3fd60c10124be", "sha256": "9c6210e144c9e1159449b6d93fa5473e23691f7f23568958d9b16ed85e50012b" }, "downloads": -1, "filename": "view_spark_timeline-0.2.8-py2-none-any.whl", "has_sig": false, "md5_digest": "4eb618d01db4e38be1f3fd60c10124be", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 10236, "upload_time": "2017-11-17T16:46:40", "url": "https://files.pythonhosted.org/packages/ff/36/6aa0e16f1c08f8ea3c10492f0aaa2ddb7927d621b5de56f39213f08a3f94/view_spark_timeline-0.2.8-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "586ce8a482d57adb1a6487d97cb67a0a", "sha256": "cc139e3a4cc73bff10e81f65fe4167bf39b7207d61218ccd4c16e6e6ab36d743" }, "downloads": -1, "filename": "view-spark-timeline-0.2.8.tar.gz", "has_sig": false, "md5_digest": "586ce8a482d57adb1a6487d97cb67a0a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6855, "upload_time": "2017-11-17T16:46:41", "url": "https://files.pythonhosted.org/packages/7d/c1/81df046a0640bb07d152832e47e0dfe3b2155c7250926cd10a9546f80b64/view-spark-timeline-0.2.8.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "4eb618d01db4e38be1f3fd60c10124be", "sha256": "9c6210e144c9e1159449b6d93fa5473e23691f7f23568958d9b16ed85e50012b" }, "downloads": -1, "filename": "view_spark_timeline-0.2.8-py2-none-any.whl", "has_sig": false, "md5_digest": "4eb618d01db4e38be1f3fd60c10124be", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 10236, "upload_time": "2017-11-17T16:46:40", "url": "https://files.pythonhosted.org/packages/ff/36/6aa0e16f1c08f8ea3c10492f0aaa2ddb7927d621b5de56f39213f08a3f94/view_spark_timeline-0.2.8-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "586ce8a482d57adb1a6487d97cb67a0a", "sha256": "cc139e3a4cc73bff10e81f65fe4167bf39b7207d61218ccd4c16e6e6ab36d743" }, "downloads": -1, "filename": "view-spark-timeline-0.2.8.tar.gz", "has_sig": false, "md5_digest": "586ce8a482d57adb1a6487d97cb67a0a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6855, "upload_time": "2017-11-17T16:46:41", "url": "https://files.pythonhosted.org/packages/7d/c1/81df046a0640bb07d152832e47e0dfe3b2155c7250926cd10a9546f80b64/view-spark-timeline-0.2.8.tar.gz" } ] }