{ "info": { "author": "Dirk Petersen, Jeff Katcher", "author_email": "dp@nowhere.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Environment :: OpenStack", "Intended Audience :: Customer Service", "Intended Audience :: Developers", "Intended Audience :: Education", "Intended Audience :: End Users/Desktop", "Intended Audience :: Healthcare Industry", "Intended Audience :: Information Technology", "Intended Audience :: Science/Research", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Operating System :: MacOS :: MacOS X", "Operating System :: POSIX", "Operating System :: POSIX :: Linux", "Operating System :: POSIX :: Other", "Operating System :: Unix", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Programming Language :: Unix Shell", "Topic :: Desktop Environment :: File Managers", "Topic :: Internet", "Topic :: Scientific/Engineering :: Bio-Informatics", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Archiving", "Topic :: System :: Archiving :: Backup", "Topic :: System :: Filesystems", "Topic :: System :: Systems Administration", "Topic :: Utilities" ], "description": "swift-commander\n===============\n\nswift commander (swc) is a wrapper to various command line client tools for openstack swift cloud\nstorage systems. The purpose of swc is 3 fold:\n\n- provide a very simple user interface to Linux users\n- provide a unified user interface to swiftclient, curl, etc with reasonable defaults\n- model commands after classic shell tools such as cd, ls, etc.\n\nBasic Operations\n================\n\nif swc is invoked without any options it shows a basic help page:\n\n::\n\n Swift Commander (swc) allows you to easily work with a swift object store.\n swc supports sub commands that attempt to mimic standard unix file system tools.\n These sub commands are currently implemented: (Arguments in square brackets are \n optional).\n\n swc upload - copy file / dirs from a file system to swift\n swc download - copy files and dirs from swift to a file system\n swc cd - change current folder to in swift\n swc ls [folder] - list contents of a folder - or the current one\n swc mkdir - create a folder (works only at the root)\n swc rm - delete all file paths that start with \n swc pwd - display the current swift folder name\n swc cat|more|less - download a file to TMPDIR and view with cat, more or less\n swc vi|emacs|nano - download a file to TMPDIR and edit it with vi|emacs or nano\n swc chgrp - grant/remove rw access to current swift account or container\n swc rw - add rw access to current swift account or container\n swc ro - add ro access to current swift account or container\n swc publish|hide - make root folder public (web server mode) or hide it\n swc list [filt] - list folder content (incl. subfolders) and filter\n swc search - search for a string in text files under /folder\n swc openwith - download a file to TMPDIR and open it with \n swc header - display the header of a file in swift\n swc meta - display custom meta data of a file in swift\n swc mtime - show the original mtime of a file before uploaded\n swc size - show the size of a swift or a local folder\n swc compare - compare size of a local folder with a swift folder\n swc hash - compare the md5sum of a local file with a swift file\n swc arch - create one tar archive for each folder level\n swc unarch - restore folders that have been archived\n swc auth - show current storage url and auth token\n swc env - show authentication env vars (ST_ and OS_)\n swc clean - remove current authtoken credential cache\n\n Examples:\n swc upload /local/folder /swift/folder\n swc upload --symlinks /local/folder /swift/folder (save symlinks)\n swc compare /local/folder /swift/folder\n swc download /swift/folder /scratch/folder\n swc download /swift/folder $TMPDIR\n swc rm /archive/some_prefix\n swc more /folder/some_file.txt\n swc openwith emacs /folder/some_file.txt\n\nImportant: What you need to know about the Swift architecture\n-------------------------------------------------------------\n\n- swift does not know sub directories such as a file system. It knows containers and in containers\n it carries objects (which are actually files).\n- if you upload a path with many directory levels such as\n /folder1/folder2/folder3/folder4/myfile.pdf to swift it will cheat a little and put an object\n called ``folder2/folder3/folder4/myfile.pdf`` into a container called ``folder1``.\n- the object is just like a filename that contains a number of forward slashes. Forward slashes are\n allowed because swift does not know any directories and can have the / character as part of a\n filename. These fake folders are also called ``Pseudo-Hierarchical Directories`` (\n http://www.17od.com/2012/12/19/ten-useful-openstack-swift-features/ )\n- the architecture has advantages and disadvantages. An advantage is that you can retrieve hundreds\n of thousands of object names in a few seconds. The disadvantage is that a single container\n eventually reaches a scalability limit. Currently this limit is at about 2 million objects per\n container. You should not put more than 2 million files into a single container or /root\\_folder.\n- swift commander (swc) allows you to ignore the fact that there are containers and pseudo folders.\n For the most part you can just treat them both as standard directories\n\nAuthentication\n--------------\n\n- ``swc`` does not implement any authentication but uses a swift authentication environment, for\n example as setup by ``https://github.com/FredHutch/swift-switch-account`` including Active\n Directory integration.\n- if a swift authentication environment is found ``swc`` creates swift auth\\_tokens on the fly and\n uses them with RESTful tools such as curl.\n\nswc upload\n~~~~~~~~~~\n\nuse ``swc upload /local_dir/subdir /my_swift_container/subfolder`` to copy data from a local or\nnetworked posix file system to a swift object store. ``swc upload`` wraps ``swift upload`` of the\nstandard python swift client:\n\n::\n\n joe@box:~/sc$ swc upload ./testing /test\n *** uploading ./test ***\n *** to Swift_Account:/test/ ***\n executing:swift upload --changed --segment-size=1073741824 --use-slo --segment-container=\".segments_test\" --header=\"X-Object-Meta-Uploaded-by:joe\" --object-name=\"\" \"test\" \"./test\"\n *** please wait... ***\n /fld11/file12\n /fld11/file11\n /fld11/fld2/fld3/fld4/file43\n /fld11/fld2/fld3/fld4/file42\n .\n\nthe swc wrapper adds the following features to ``upload``:\n\n- --segment-size ensures that uploads for files > 5GB do not fail. 1073741824 = 1GB\n- Uploaded-by metadata keeps track of the operating system user (often Active Directory user) that\n upload the data\n- setting --segment-container ensures that containers that carry the segments for multisegment\n files are hidden if users access these containers with 3rd. party GUI tools (ExpanDrive,\n Cyberduck, FileZilla) to avoid end user confusion\n- --slo stands for Static Large Object and SLO's the recommended object type for large objects /\n files.\n\nas an addional feature you can add multiple metadata tags to each uploaded object, which is great\nfor retrieving archived files later:\n\n::\n\n joe@box:~/sc$ swc upload ./test /test/example/meta project:grant-xyz collaborators:jill,joe,jim cancer:breast\n *** uploading ./test ***\n *** to Swift_Account:/test/example/meta ***\n executing:swift upload --changed --segment-size=1073741824 --use-slo --segment-container=\".segments_test\" --header=\"X-Object-Meta-Uploaded-by:petersen\" --header=X-Object-Meta-project:grant-xyz --header=X-Object-Meta-collaborators:jill,joe,jim --header=X-Object-Meta-cancer:breast --object-name=\"example/meta\" \"test\" \"./test\"\n *** please wait... ***\n example/meta/fld11/fld2/file21\n example/meta/fld11/file11\n .\n .\n /test/example/meta\n\nThese metadata tags stay in the swift object store with the data. They are stored just like other\nimportant metadata such as change data and name of the object.\n\n::\n\n joe@box:~/sc$ swc meta example/meta/fld11/file13\n Meta Cancer: breast\n Meta Collaborators: jill,joe,jim\n Meta Uploaded-By: petersen\n Meta Project: grant-xyz\n Meta Mtime: 1420047068.977197\n\nif you store metadata tags you can later use an external search engine such as ElasticSearch to\nquickly search for metadata you populated while uploading data\n\nalias: you can use ``swc up`` instead of ``swc upload``\n\nswc download\n~~~~~~~~~~~~\n\nuse ``swc download /my_swift_container/subfolder /local/subfolder`` to copy data from a swift object\nstore to local or network storage. swc download\\ ``wraps``\\ swift download\\` of the standard python\nswift client:\n\n::\n\n joe@box:~/sc$ swc download /test/example/ $TMPDIR/ \n example/meta/fld11/fld2/file21\n example/meta/fld11/file11\n\nalias: you can use ``swc down`` instead of ``swc download``\n\nswc arch\n~~~~~~~~\n\n``swc arch`` is a variation of ``swc upload``. Instead of uploading the files as is, it creates a\ntar.gz archive for each directory and uploads the tar.gz archives. swc arch is different from\ndefault tar behavior because it does not create a large tar.gz file of an entire directory structure\nas large tar.gz files are hard to manage (as one cannot easily navigate the directory structure\nwithin or get quick access to a spcific file). Instead swc arch creates tar.gz files that do not\ninclude sub directories and it creates a separate tar.gz file for each directory and directory\nlevel. The benefit of this approach is that the entire directory structure remains intact and you\ncan easily navigate it by using ``swc cd`` and ``swc ls``\n\nswc cd, swc, ls, swc mkdir\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nthese commands are simplified versions of the equivalent standard GNU tools and should work very\nsimilar to these tools.\n\nswc mtime\n~~~~~~~~~\n\nuse ``swc mtime /my_swift_container/subfolder/file`` to see the modification time data from a swift\nobject store to local or network storage. ``swc download`` wraps ``swift download`` of the standard\npython swift client:", "description_content_type": null, "docs_url": null, "download_url": "https://github.com/FredHutch/swift-commander/tarball/1.4.6", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/FredHutch/swift-commander", "keywords": "openstack,swift,cloud storage", "license": "UNKNOWN", "maintainer": null, "maintainer_email": null, "name": "swift-commander", "package_url": "https://pypi.org/project/swift-commander/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/swift-commander/", "project_urls": { "Download": "https://github.com/FredHutch/swift-commander/tarball/1.4.6", "Homepage": "https://github.com/FredHutch/swift-commander" }, "release_url": "https://pypi.org/project/swift-commander/1.4.6/", "requires_dist": null, "requires_python": null, "summary": "swift commander (swc) is a wrapper to various command line\nclient tools for openstack swift cloud storage systems.", "version": "1.4.6" }, "last_serial": 2521874, "releases": { "1.3": [], "1.3.7": [ { "comment_text": "", "digests": { "md5": "f092d70c925e931781881f467c7a50c7", "sha256": "34c3d15a99775636a1a5e890daee4d9e06074d31c01c585708eefc0860efde7d" }, "downloads": -1, "filename": "swift-commander-1.3.7.tar.gz", "has_sig": false, "md5_digest": "f092d70c925e931781881f467c7a50c7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36218, "upload_time": "2016-03-12T18:33:44", "url": "https://files.pythonhosted.org/packages/3f/29/f166dad405121aebd7206389fcfc0e4a98e3ab59d14c802262b94d1413aa/swift-commander-1.3.7.tar.gz" } ], "1.3.8": [ { "comment_text": "", "digests": { "md5": "a9673043396d64f1bc5b4e6b636dd52d", "sha256": "346cd0c31e1b337d63674349ccf02d642326d98b7b9ee456ad1ae6b43560741b" }, "downloads": -1, "filename": "swift-commander-1.3.8.tar.gz", "has_sig": false, "md5_digest": "a9673043396d64f1bc5b4e6b636dd52d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36340, "upload_time": "2016-03-12T20:07:26", "url": "https://files.pythonhosted.org/packages/4d/ff/d3174bb9323faaf6c146683f179dc10d0a5c8442a0cf63b018200a2936b5/swift-commander-1.3.8.tar.gz" } ], "1.4.0": [ { "comment_text": "", "digests": { "md5": "6976d418d35593a275bcae3b25221429", "sha256": "379b475ff0bff6ad0a178edfff8008d7c03f15935660e39d2666357dfe9ec7d0" }, "downloads": -1, "filename": "swift-commander-1.4.0.tar.gz", "has_sig": false, "md5_digest": "6976d418d35593a275bcae3b25221429", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36590, "upload_time": "2016-03-25T06:15:21", "url": "https://files.pythonhosted.org/packages/6c/4d/1407f75079d24e848b572da538d499dde0afcddb7edd9650d34c3d29ceb8/swift-commander-1.4.0.tar.gz" } ], "1.4.1": [ { "comment_text": "", "digests": { "md5": "d787a0a86a96c85c5cd3e279b22b6d51", "sha256": "a2355a7e1d7a24839c278662a11082f820e35b6aac26e20177484a0509d6d9bc" }, "downloads": -1, "filename": "swift-commander-1.4.1.tar.gz", "has_sig": false, "md5_digest": "d787a0a86a96c85c5cd3e279b22b6d51", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36619, "upload_time": "2016-03-25T06:51:20", "url": "https://files.pythonhosted.org/packages/39/0b/42f420a7065a90d5e3dfa8db00791e93c7493ab2f9f17121a06631ee7507/swift-commander-1.4.1.tar.gz" } ], "1.4.2": [ { "comment_text": "", "digests": { "md5": "886d2b1741c731c6f5e101c00d101638", "sha256": "e7e3cc6b63efa3ab8aedc3213fb9abd5b5588c9b444a8c0da6cc948931dd8658" }, "downloads": -1, "filename": "swift-commander-1.4.2.tar.gz", "has_sig": false, "md5_digest": "886d2b1741c731c6f5e101c00d101638", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37202, "upload_time": "2016-10-04T04:50:57", "url": "https://files.pythonhosted.org/packages/fc/7e/cf90cea936e9187e197f26b88ce98b5c111cdb2291374b2c3108192275af/swift-commander-1.4.2.tar.gz" } ], "1.4.3": [ { "comment_text": "", "digests": { "md5": "ef48e1a0c669cca941054d4b0b70a409", "sha256": "063ccda61a63d3ea14bcc094c05c18ec7e6adc704cbf49052ce94374572db74a" }, "downloads": -1, "filename": "swift-commander-1.4.3.tar.gz", "has_sig": false, "md5_digest": "ef48e1a0c669cca941054d4b0b70a409", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37413, "upload_time": "2016-10-04T06:59:00", "url": "https://files.pythonhosted.org/packages/ac/c1/c616b2decb8013797babc6e75c150023cc2e0ba240bc35adfc133446b53d/swift-commander-1.4.3.tar.gz" } ], "1.4.4": [ { "comment_text": "", "digests": { "md5": "a172f5bf4b9ca3468d4f7b863e6c5b48", "sha256": "0f44ad1525a3129d2fd1b2a79863e6c4cef99fff119bc82513ebe372fc116ca8" }, "downloads": -1, "filename": "swift-commander-1.4.4.tar.gz", "has_sig": false, "md5_digest": "a172f5bf4b9ca3468d4f7b863e6c5b48", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35536, "upload_time": "2016-10-29T14:13:26", "url": "https://files.pythonhosted.org/packages/cf/84/464ebe47907fe6945a6a18136c02fe814809c24ca1244d30e35b8d74db29/swift-commander-1.4.4.tar.gz" } ], "1.4.5": [ { "comment_text": "", "digests": { "md5": "5c1658bb3e2394c98df0f5fe918b7da7", "sha256": "995cd6ed695f92a6ff9ec91a097d0dd57ae71a90c469d3a52899366849aa53c8" }, "downloads": -1, "filename": "swift-commander-1.4.5.tar.gz", "has_sig": false, "md5_digest": "5c1658bb3e2394c98df0f5fe918b7da7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 38237, "upload_time": "2016-10-29T16:53:51", "url": "https://files.pythonhosted.org/packages/80/b8/d76505302365af055b2f0959e1d7a3b6504c456ecfeab7785df2b6657a90/swift-commander-1.4.5.tar.gz" } ], "1.4.6": [ { "comment_text": "", "digests": { "md5": "abbf7ee32145f14e8e81fa90069b1291", "sha256": "d5bcbf76626988de4e5a43ad334c4c57a779a0b8b3f751a8183827bb0c992974" }, "downloads": -1, "filename": "swift-commander-1.4.6.tar.gz", "has_sig": false, "md5_digest": "abbf7ee32145f14e8e81fa90069b1291", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 38560, "upload_time": "2016-12-15T19:04:38", "url": "https://files.pythonhosted.org/packages/20/69/1498ba6085da784fecd0cbd00469a70e5b8ae00f316274a50567658c0ee9/swift-commander-1.4.6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "abbf7ee32145f14e8e81fa90069b1291", "sha256": "d5bcbf76626988de4e5a43ad334c4c57a779a0b8b3f751a8183827bb0c992974" }, "downloads": -1, "filename": "swift-commander-1.4.6.tar.gz", "has_sig": false, "md5_digest": "abbf7ee32145f14e8e81fa90069b1291", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 38560, "upload_time": "2016-12-15T19:04:38", "url": "https://files.pythonhosted.org/packages/20/69/1498ba6085da784fecd0cbd00469a70e5b8ae00f316274a50567658c0ee9/swift-commander-1.4.6.tar.gz" } ] }