{
    "info": {
        "author": "Chris Jenn",
        "author_email": "jnpr-community-netdev@juniper.net",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Environment :: Console",
            "Intended Audience :: Developers",
            "Intended Audience :: Information Technology",
            "Intended Audience :: System Administrators",
            "Intended Audience :: Telecommunications Industry",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.4",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3 :: Only",
            "Topic :: System :: Networking"
        ],
        "description": "# Splitcopy\n\nImproves file transfer rates when copying files to/from JUNOS/EVO/\\*nix hosts.  \nIt achieves this by splitting a file into chunks, transferring the chunks to the remote host and recombining them.  \n\nAt a minimum, sshd must be running on the remote host.  \nOn JUNOS/EVO this requires 'system services ssh' configuration.  \n\nIf using ftp to copy files (default) then an ftp daemon must be running on the remote host.   \nOn JUNOS this requires 'system services ftp' configuration.  \n\nScript overheads include authentication, sha1 generation/comparison, disk space check, file split and join.  \nIt can be slower than normal ftp/scp for small files as a result.\n\nBecause it opens a number of simultaneous connections,\nif the JUNOS/EVO host has connection/rate limits configured like this:\n\n```\nsystem {\n    services {\n        ssh { # or ftp\n            connection-limit 10;\n            rate-limit 10;\n        }\n    }\n}\n```\n\nThe script will deactivate these limits so it can proceed, then activate them again.  \n\n## Arguments\n\n`filepath` Mandatory, path to the src file you want to copy  \n`userhost` Mandatory, username and host to connect to, in format user@host  \n`--pwd`    Optional, password.  \n`--dst`    Optional, directory to put file. The default is /var/tmp/  \n`--scp`    Optional, use scp instead of ftp to transfer files  \n`--get`    Optional, copy from remote to local host  \n`--log`    Optional, enables additional logging, specify a logging level as argument  \n\n# INSTALLATION\n\nInstallation requires Python >= 3.4 and associated `pip` tool, Python >= 3.6 is recommended due to improvements in asyncio  \n\n    pip install splitcopy\n\nInstalling from Git is also supported (OS must have git installed).\n\n    To install the latest MASTER code\n    pip install git+https://github.com/Juniper/splitcopy.git\n    -or-\n    To install a specific version, branch, tag, etc.\n    pip install git+https://github.com/Juniper/splitcopy.git@<branch,tag,commit>\n\nUpgrading has the same requirements as installation and has the same format with the addition of --upgrade\n\n    pip install splitcopy --upgrade\n\n\n# Usage Examples \n## FTP transfer (default method)\n\n```\n$ ./splitcopy.py /var/tmp/jselective-update-ppc-J1.1-14.2R5-S3-J1.1.tgz lab@192.168.1.1\nPassword:\nchecking remote port(s) are open...\nusing FTP for file transfer\nchecking remote storage...\nsha1 not found, generating sha1...\nsplitting file...\nstarting transfer...\n10% done\n20% done\n30% done\n40% done\n50% done\n60% done\n70% done\n80% done\n90% done\n100% done\ntransfer complete\njoining files...\ndeleting remote tmp directory...\ngenerating remote sha1...\nlocal and remote sha1 match\nfile has been successfully copied to 192.168.1.1:/var/tmp/jselective-update-ppc-J1.1-14.2R5-S3-J1.1.tgz\ndata transfer = 0:00:16.831192\ntotal runtime = 0:00:31.520914\n```\n\n## SCP transfer with 'get'\n\n```\n$ ./splitcopy.py /var/log/messages lab@192.168.1.1 --scp --get\nPassword:\nchecking remote port(s) are open...\nusing SCP for file transfer\nchecking remote storage...\ngenerating remote sha1...\nstarting transfer...\n10% done\n20% done\n30% done\n40% done\n50% done\n60% done\n70% done\n80% done\n90% done\n100% done\ntransfer complete\njoining files...\ndeleting remote tmp directory...\ngenerating local sha1...\nlocal and remote sha1 match\nfile has been successfully copied to /var/tmp/messages\ndata transfer = 0:00:18.768987\ntotal runtime = 0:00:44.891370\n```\n\n## Notes on using FTP\n\nFTP is the default transfer method.  \nFTP progress on --get operations is supported from py-junos-eznc v2.2.2  \n\nThe version of Python used has a big impact.  \nIf using < 3.6 the maximum number of simultaneous transfers is 5.  \nIf using 3.6+ it will allow 5 simultaneous transfers per cpu   \n\nUsing FTP method will generate the following processes on the remote host:\n- for mgmt session: 1x sshd, 1x cli, 1x mgd, 1x csh\n- for transfers: up to 40x ftpd processes (depends on Python version and number of cpus as described above)\n\nIn theory, this could result in the per-user maxproc limit of 64 being exceeded:\n```\nMay  2 04:46:59   /kernel: maxproc limit exceeded by uid 2001, please see tuning(7) and login.conf(5).\n```\nThe script modulates the number of chunks to match the maximum number of simultaneous transfers possible (based on Python version and number of cpus).   \nThe maximum number of user owned processes that could be created is <= 44\n\n## Notes on using SCP\n\nThe version of Python used has a big impact.  \nIf using < 3.6 the maximum number of simultaneous transfers is 5.  \nIf using 3.6+ it will allow 5 simultaneous transfers per cpu \n\nUsing SCP method will generate the following processes on the remote host:\n- for mgmt session: 1x sshd, 1x cli, 1x mgd, 1x csh\n- for transfers:  depends on Python version, number of cpus (see above), OpenSSH and Junos FreeBSD version (see below)\n\nIn FreeBSD 11 based Junos each scp transfer creates 2 user owned processes: \n```\nlab 30366   0.0  0.0  475056   7688  -  Ss   10:39      0:00.03 cli -c scp -t /var/tmp/\nlab 30367   0.0  0.0   61324   4860  -  S    10:39      0:00.01 scp -t /var/tmp/\n```\nIn FreeBSD 10 based Junos each scp transfer creates 2 user owned processes\n```\nlab  28639   0.0  0.0  734108   4004  -  Is   12:00PM     0:00.01 cli -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/\nlab  28640   0.0  0.0   24768   3516  -  S    12:00PM     0:00.01 scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/\n```\nIn FreeBSD 6 based Junos each scp transfer creates 3 user owned processes: \n```\nlab  78625  0.0  0.1  2984  2144  ??  Ss    5:29AM   0:00.01 cli -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \nlab  78626  0.0  0.0  2252  1556  ??  S     5:29AM   0:00.00 sh -c scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \nlab  78627  0.0  0.1  3500  1908  ??  S     5:29AM   0:00.01 scp -t /var/tmp/splitcopy_jinstall-11.4R5.5-domestic-signed.tgz/  \n```\nIn addition, if OpenSSH version is >= 7.4, an additional user owned process is created:\n```\nlab  2287  2.4  0.1 11912  2348  ??  S     3:49AM   0:00.15 sshd: lab@notty (sshd)\n```\nThis could result in the per-user maxproc limit of 64 being exceeded:\n```\nMay  2 04:46:59   /kernel: maxproc limit exceeded by uid 2001, please see tuning(7) and login.conf(5).\n```\nTo mitigate this, the script modulates the number of chunks to match the maximum number of simultaneous transfers possible (based on Python, OpenSSH, Junos FreeBSD versions and the number of cpu's).  \nThe maximum number of user owned processes that could be created is <= 45\n\n\n\n## LICENSE\n\nApache 2.0\n\n## CONTRIBUTORS\n\nJuniper Networks is actively contributing to and maintaining this repo. Please contact jnpr-community-netdev@juniper.net for any queries.\n\n*Contributors:*\n\n[Chris Jenn](https://github.com/ipmonk)\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Juniper/splitcopy",
        "keywords": "ftp,ssh,scp,transfer",
        "license": "Apache 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "splitcopy",
        "package_url": "https://pypi.org/project/splitcopy/",
        "platform": "",
        "project_url": "https://pypi.org/project/splitcopy/",
        "project_urls": {
            "Homepage": "https://github.com/Juniper/splitcopy"
        },
        "release_url": "https://pypi.org/project/splitcopy/1.0.8/",
        "requires_dist": [
            "junos-eznc (>=2.3.0)"
        ],
        "requires_python": ">=3.4",
        "summary": "Improves file transfer rates when copying files to/from JUNOS/EVO/*nix hosts",
        "version": "1.0.8"
    },
    "last_serial": 5906634,
    "releases": {
        "1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b40bda414d37a5526b686f790768c922",
                    "sha256": "5b7be60757983f18a958d100a7a277929362457223c5961bfe283cbc206ea8b8"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b40bda414d37a5526b686f790768c922",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 16321,
                "upload_time": "2019-05-09T08:29:39",
                "url": "https://files.pythonhosted.org/packages/28/a8/ba8c00474ba0c0aa34607ad2133fbcddd25aa63129a8a5e493ba4da6eacb/splitcopy-1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "da0d9c71c81e56763b55807631e8591d",
                    "sha256": "6d2d85818d93d748bfc7f9236e920fd27eaa0f48018e18a9f7964137392c799d"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "da0d9c71c81e56763b55807631e8591d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.4",
                "size": 11938,
                "upload_time": "2019-05-09T08:29:41",
                "url": "https://files.pythonhosted.org/packages/82/d7/5e0cc73e5f735f04a6a72db086c40ce5c8be6915e5921535b04bf25114f5/splitcopy-1.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "71b09f8ee2ab79dddefa58ae0ac9b62e",
                    "sha256": "c8465d91e7605d3d7bb4554e3efe9a70f8631abcc205fb11e508e43140798913"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "71b09f8ee2ab79dddefa58ae0ac9b62e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 16744,
                "upload_time": "2019-05-10T12:08:50",
                "url": "https://files.pythonhosted.org/packages/56/e1/318374a6e22a087410586192660a4d7cc49151a7706293df8b4102376e51/splitcopy-1.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d97411958ccfde528d1c47aad8787981",
                    "sha256": "99a0a8c6e51a4188f617365ad4825b87b6fb172f303f5cc6b9f38bcd089993c9"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d97411958ccfde528d1c47aad8787981",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.4",
                "size": 12477,
                "upload_time": "2019-05-10T12:08:53",
                "url": "https://files.pythonhosted.org/packages/a7/bb/0718ac45b453a26ed49930127b583bfe3bddd5f44079123f64c0786bf712/splitcopy-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "30ac1dae4cd98e45984ba2b78b922f38",
                    "sha256": "2c89341ddeb671267ee8506d64719b30ceae7bebfcb71bb8f851ea7aa32cb3e1"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "30ac1dae4cd98e45984ba2b78b922f38",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 16820,
                "upload_time": "2019-05-29T07:29:19",
                "url": "https://files.pythonhosted.org/packages/d2/de/49b9b578ac90caf82b5d482e47054431d70ff8d9bc2c5f8424bc0ad80ca3/splitcopy-1.0.2-py3-none-any.whl"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b8c9f40c47d1c1c349eba98215f3cadc",
                    "sha256": "bbbfebe0f8ae0b154c54645c83c6774834519c56ca389b52f3c16d128e3138fc"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b8c9f40c47d1c1c349eba98215f3cadc",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 16968,
                "upload_time": "2019-06-14T17:32:49",
                "url": "https://files.pythonhosted.org/packages/51/1e/0b2a450ff47858815463c1cf97ef7c1b4d4fc64c969fb00de89abcc026f2/splitcopy-1.0.3-py3-none-any.whl"
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "06a0aee18f5e6b7e5cd4401fe4001c42",
                    "sha256": "603e9fc2fd92b5e51a3d9a3ed99cefbab86a596308d44c2751cd2f8054a7ba44"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "06a0aee18f5e6b7e5cd4401fe4001c42",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 17462,
                "upload_time": "2019-07-05T15:11:27",
                "url": "https://files.pythonhosted.org/packages/7a/38/d4c3cc1ef070c8bdcacf0cd39b0b17f048a3461b2fafb039a2ec85f7c7d0/splitcopy-1.0.4-py3-none-any.whl"
            }
        ],
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "41ed6bf007da4d5129c024975a9bdcfb",
                    "sha256": "3375cc6956269ef92e7c29ef2314b1562ad42bbdfca541a9692e7f9596535a80"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "41ed6bf007da4d5129c024975a9bdcfb",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 17962,
                "upload_time": "2019-07-25T18:01:47",
                "url": "https://files.pythonhosted.org/packages/1c/dc/bb335fce8a7cdde813d13676884d053ea6660d9b5633acc15363a163a1d0/splitcopy-1.0.5-py3-none-any.whl"
            }
        ],
        "1.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "2de74c02a781431cced19eb8e6eef9a0",
                    "sha256": "94b0e039746d8b7032dc39931c5a3acdde265f26220c264edaf67fea64372493"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "2de74c02a781431cced19eb8e6eef9a0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 17965,
                "upload_time": "2019-08-01T23:51:25",
                "url": "https://files.pythonhosted.org/packages/ab/18/fb296007e2b661a09ca74c6a5537c53587ee58aa5cd0506d39dfa8083d17/splitcopy-1.0.6-py3-none-any.whl"
            }
        ],
        "1.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3c6dd5c64d022265bb88b5c958f22788",
                    "sha256": "bdecbfc40491fee5cc22b652b174bc912fe66f3800ee6009d8c125eb55020060"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3c6dd5c64d022265bb88b5c958f22788",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 17966,
                "upload_time": "2019-09-30T11:54:30",
                "url": "https://files.pythonhosted.org/packages/ec/37/26f0dbad738a24cba0377a6673142e8c0b76eaf1d9d8c1b20c114f230eff/splitcopy-1.0.7-py3-none-any.whl"
            }
        ],
        "1.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0b085c8e8d8506355b8350b106249339",
                    "sha256": "39afe5cab34d27ac150b0f9da85ee5bfc9c0603edbc22f0cc66a7dd7768fb9d8"
                },
                "downloads": -1,
                "filename": "splitcopy-1.0.8-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "0b085c8e8d8506355b8350b106249339",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.4",
                "size": 17965,
                "upload_time": "2019-09-30T12:03:54",
                "url": "https://files.pythonhosted.org/packages/fe/a1/070fe8e28b6905801a3e20fc332e2f4db100d173571905f620f79e7ccebc/splitcopy-1.0.8-py3-none-any.whl"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "0b085c8e8d8506355b8350b106249339",
                "sha256": "39afe5cab34d27ac150b0f9da85ee5bfc9c0603edbc22f0cc66a7dd7768fb9d8"
            },
            "downloads": -1,
            "filename": "splitcopy-1.0.8-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "0b085c8e8d8506355b8350b106249339",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.4",
            "size": 17965,
            "upload_time": "2019-09-30T12:03:54",
            "url": "https://files.pythonhosted.org/packages/fe/a1/070fe8e28b6905801a3e20fc332e2f4db100d173571905f620f79e7ccebc/splitcopy-1.0.8-py3-none-any.whl"
        }
    ]
}