{ "info": { "author": "Aker Systems", "author_email": "development@akersystems.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: BSD License", "Topic :: Utilities" ], "description": "Data Transfer Microservice\n==========================\n\nThis microservice is a multi-mode file movement wizard. It can transfer files,\nat a scheduled interval, between two different storage devices, using different\ntransfer protocols and storage types.\n\nThe application is built for Python 3, but also tested against Python 2.7. It\nis **not** compatible with Python 2.6.\n\n**Note**\nThe application will not work if you have files with spaces in the names and will fail.\n\n**Standard FTP is no longer supported in this app**\n\nInstalling and getting started\n------------------------------\n\nThe application should be installed using ``pip3`` (or ``pip`` for Python 2.7).\n\nTo install from a private PyPI server we suggest using ``~/.pypirc`` to configure\nyour private PyPI connection details::\n\n pip3 install data-transfer --extra-index-url \n\nAfter installing and setting the configuration settings, the application can be\nstarted with the following command::\n\n data-transfer\n\n\nDeveloping\n----------\n\nStart by cloning the project::\n\n git clone git@github.com:UKHomeOffice/data-transfer.git\n\nEnsure that ``python3`` is installed and on your ``path``.\n\nInstalling for local development\n\nThese steps will install the application as a local pip installed package,\nusing symlinks so that any updates you make to the files are automatically\npicked up next time you run the application or tests.\n\nUsing venv\n\"\"\"\"\"\"\"\"\"\"\n\nTo install the app using the standard ``python3 venv`` run the following\ncommands from the project root folder::\n\n python3 -m venv ~/.virtualenvs/data-transfer\n source ~/.virtualenvs/data-transfer/bin/activate\n pip3 install -e . -r requirements.txt\n export PYTHONPATH=.\n\n\nUsing virtualenvwrapper\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nAlternatively, if you are using ``virtualenvwrapper`` then run the following::\n\n mkvirtualenv data-transfer -p python3\n pip3 install -e . -r requirements.txt\n export PYTHONPATH=.\n\nDependancies for local testing\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThe project's tests require the following dependencies:\n\n* An AWS S3 bucket or a mock\n* An FTP server\n* An SFTP server\n\nFor local development and testing, we suggest running Docker images. The following\nwill meet the test dependencies and match the default env vars::\n\n docker run -d --name s3server -p 8000:8000 scality/s3server\n docker run -d --name ftp_server -p 21:21 -p 30000-30009:30000-30009 onekilo79/ftpd_test\n docker run -p 2222:22 -d atmoz/sftp foo:pass:::upload\n\nTest\n\"\"\"\"\n\nOnce the application is installed and the dependencies are in place, run the\ntests::\n\n pytest tests\n\n\nBuilding & publishing\n=====================\n\nThis project uses ``setuptools`` to build the distributable package.\n\nRemember to update the ``version`` in ``setup.py`` before building the package::\n\n python setup.py sdist\n\nThis will create a ``.tar.gz`` distributable package in ``dist/``. This should be\nuploaded to an appropriate PyPI registry.\n\nDeploying\n---------\n\nThe application should be installed using ``pip3`` (or ``pip`` for Python 2.7).\n\nIf installing from a private PyPI server then we suggest using ``~/.pypirc`` to\nconfigure your private PyPI connection details::\n\n pip3 install data-transfer --extra-index-url \n\n\nConfiguration\n-------------\n\nThe application requires the following environment variables to be set before\nrunning.\n\nAll configuration settings automatically default to suitable values for the\ntests, based on the local test dependencies running in the Docker images\nsuggested in this guide.\n\nApplication settings\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nThese control various application behaviours, where a variable is not required\nthe default value is used:\n\n+---------------------+----------------------+-----------+-----------------------------------+\n|Environment Variable | Example (Default) | Required | Description. |\n+=====================+======================+===========+===================================+\n|INGEST_SOURCE_PATH | /upload/files | Yes | Source path |\n+---------------------+----------------------+-----------+-----------------------------------+\n|INGEST_DEST_PATH | /upload/files/done | Yes | Destination path |\n+---------------------+----------------------+-----------+-----------------------------------+\n|MAX_FILES_BATCH | 5 | No | Number to process each run |\n+---------------------+----------------------+-----------+-----------------------------------+\n|PROCESS_INTERVAL | 5 | No | Runs the task every (x) seconds. |\n+---------------------+----------------------+-----------+-----------------------------------+\n|FOLDER_DATE_OUTPUT | False | No | Moves files to YYYY / MM / DD |\n+---------------------+----------------------+-----------+-----------------------------------+\n|LOG_LEVEL | INFO | No | Log level |\n+---------------------+----------------------+-----------+-----------------------------------+\n|LOG_FILE_NAME | data-transfer.log | Yes | Filename for log output |\n+---------------------+----------------------+-----------+-----------------------------------+\n|USE_IAM_CREDS | False | Yes | Indicates to app to use IAM |\n+---------------------+----------------------+-----------+-----------------------------------+\n|READ_STORAGE_TYPE | See footnote | Yes | The type of read storage |\n+---------------------+----------------------+-----------+-----------------------------------+\n|WRITE_STORAGE_TYPE | See footnote | Yes | The type of write storage |\n+---------------------+----------------------+-----------+-----------------------------------+\n\nNote: the read and write storage types need to be prefixed and options are:\n\n* datatransfer.storage.FolderStorage\n* datatransfer.storage.SftpStorage\n* datatransfer.storage.S3Storage\n\n* Also ensure that the source and destination paths have the correct leading and\ntrailing slashes, this will depend on the storage type and the OS. See the\necosystem.config file for examples.\n\n\nSource / read settings\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nProvide the connection settings for either sFTP or S3. You only need to\nconfigure the settings associated with the source storage type.\n\n+----------------------------+------------------------+--------------------------+\n|Environment Variable | Example | Description |\n+============================+========================+==========================+\n|READ_FTP_HOST | localhost | Hostname or IP of server |\n+----------------------------+------------------------+--------------------------+\n|READ_FTP_PASSWORD | pass | Password |\n+----------------------------+------------------------+--------------------------+\n|READ_FTP_USER | user | Username |\n+----------------------------+------------------------+--------------------------+\n|READ_FTP_PORT | 22 | Port the server uses |\n+----------------------------+------------------------+--------------------------+\n|READ_AWS_ACCESS_KEY_ID | accessKey1 | Access key for S3 |\n+----------------------------+------------------------+--------------------------+\n|READ_AWS_S3_BUCKET_NAME | aws-ingest | Bucket name |\n+----------------------------+------------------------+--------------------------+\n|READ_AWS_S3_HOST | http://localhost:8000 | URL of S3 |\n+----------------------------+------------------------+--------------------------+\n|READ_AWS_S3_REGION | eu-west-1 | region for s3 bucket |\n+----------------------------+------------------------+--------------------------+\n\nTarget / write settings\n\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\n\nProvide the connection settings for either sFTP or S3. You only need to\nconfigure the settings associated with the target storage type.\n\n+----------------------------+-----------------------+-------------------------+\n|Environment Variable | Example | Description |\n+============================+=======================+=========================+\n|WRITE_FTP_HOST | localhost | Hostname or IP of server|\n+----------------------------+-----------------------+-------------------------+\n|WRITE_FTP_USER | user | Username |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_FTP_PASSWORD | pass | Password |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_FTP_PORT | 22 | Port for server |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_AWS_ACCESS_KEY_ID | accesskey1 | Access key for S3 |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_AWS_SECRET_ACCESS_KEY | verysecret | Secrey key |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_AWS_S3_BUCKET_NAME | aws-ingest | Bucket name |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_AWS_S3_HOST | http://localhost:8000 | URL of S3 |\n+----------------------------+-----------------------+-------------------------+\n|WRITE_AWS_S3_REGION | eu-west-1 | region for s3 bucket |\n+----------------------------+-----------------------+-------------------------+\n\nRunning the application\n-----------------------\n\nTo run the application from the command line:\n\nFor pip installed versions::\n\n data-transfer\n\nCalling the application directly::\n\n python bin/data-transfer\n\nFor production use we recommend running the application using PM2, please ensure\nthat PM2 is installed globally before running this command::\n\n pm2 start ecosystem.config.js --only data-transfer\n\nEnvirnment variables required should be changed in the ecosystem file before\nrunning PM2. It is also recommended to run pm2 from within a python virtual env.\n\nRunning Multi-Instances\n-----------------------\n\nTo run more that one instance of the application with different config settings,\nyou will need to change/add additional services into the ecosystem config file.\n\nSee here for examples:\n\n\n\n\nWindows\n-------\n\nThe application is portable between linux and windows, however when running the\napp on windows there are some specifics you may want to take into account:\n\n1. If you are running the microservice using a batch file or other mechanism\nother than PM2, you will need to ensure that the environment variables are\nset without quotes.\n\n2. The file paths for FolderStorage should be Windows paths, for FTP,sFTP and\nS3 these can be unix format.\n\nFor sFTP, and Folder storage ensure paths are absolute without a trailing slash\n /path/to/something\n\nFor S3 the path is used with the URL so can be relative, but without a trailing slash\n path/to/something\n\n\nAWS\n---\n\nIf you are running the app on a AWS instance that has anIAM policy you can set\nthe USE_IAM_CREDS var to True and the application will use IAM policies. You must\nhowever ensure that the bucket name is set correctly.\n\n\nContributing\n\"\"\"\"\"\"\"\"\"\"\"\"\n\nThis project is Open source and we welcome ocntributions to and suggestions to\nimprove the application. Please raise issues in the usual way on Github and for\ncontributing code:\n\n* Fork the repo github\n* Clone the project locally\n* Commit your changes to your own branch\n* Push your work back to your fork\n* Submit a Pull Request so that we can review the changes\n\n\nLicensing\n\"\"\"\"\"\"\"\"\"\n\nThis application is released under the `BSD license`_.\n\n.. _BSD license: LICENSE.txt\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/UKHomeOffice/data-transfer", "keywords": "data movement ftp sftp s3 buckets ingest", "license": "BSD", "maintainer": "", "maintainer_email": "", "name": "data-transfer", "package_url": "https://pypi.org/project/data-transfer/", "platform": "", "project_url": "https://pypi.org/project/data-transfer/", "project_urls": { "Homepage": "https://github.com/UKHomeOffice/data-transfer" }, "release_url": "https://pypi.org/project/data-transfer/1.5.0/", "requires_dist": null, "requires_python": "", "summary": "A data movement app that can use different source/targets tomove data around.", "version": "1.5.0" }, "last_serial": 3599473, "releases": { "1.0.4": [ { "comment_text": "", "digests": { "md5": "7521ed4e7e5288d517192f1f45dee445", "sha256": "8a608596b94465bb2d9ed465cd121229b399841f87044eeff30f4b6f97386878" }, "downloads": -1, "filename": "data-transfer-1.0.4.tar.gz", "has_sig": false, "md5_digest": "7521ed4e7e5288d517192f1f45dee445", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12200, "upload_time": "2017-12-20T17:33:47", "url": "https://files.pythonhosted.org/packages/36/0d/0274721f705cb382ee0d1f7a07c99b59575ac715e99da26b1feac9f4a15c/data-transfer-1.0.4.tar.gz" } ], "1.0.5": [ { "comment_text": "", "digests": { "md5": "39e00d0173106f946075d34052d3bc10", "sha256": "4154dadea22a64eea84a6d388caba9b7028345a953287dad60e7ffc626d0aa6a" }, "downloads": -1, "filename": "data-transfer-1.0.5.tar.gz", "has_sig": false, "md5_digest": "39e00d0173106f946075d34052d3bc10", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12199, "upload_time": "2017-12-21T09:39:45", "url": "https://files.pythonhosted.org/packages/b3/08/c9fc62369475fed461b8b5931ce2b40eb4997da048adec78effd33def135/data-transfer-1.0.5.tar.gz" } ], "1.0.8": [ { "comment_text": "", "digests": { "md5": "751850462d4081c4dcd8c81b578fa467", "sha256": "5cc04d4458c44c6fee821a02065b5388da77126a58b9589c8b5ff067327cc11d" }, "downloads": -1, "filename": "data-transfer-1.0.8.tar.gz", "has_sig": false, "md5_digest": "751850462d4081c4dcd8c81b578fa467", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12193, "upload_time": "2017-12-21T10:07:04", "url": "https://files.pythonhosted.org/packages/fd/7b/60c7fd56a6532d881ec79bed5024774a8920712a3d1b0f030895cafc6b70/data-transfer-1.0.8.tar.gz" } ], "1.0.9": [ { "comment_text": "", "digests": { "md5": "98abbccc5a7e6512d2eea79ece1366af", "sha256": "a77e5eb4102cc542459d6debe4b8c85e09d362d2579c7593f71654c66bc215c9" }, "downloads": -1, "filename": "data_transfer-1.0.9-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "98abbccc5a7e6512d2eea79ece1366af", "packagetype": "bdist_wheel", "python_version": "3.5", "requires_python": null, "size": 13409, "upload_time": "2017-12-28T11:50:05", "url": "https://files.pythonhosted.org/packages/42/97/1e875b334ee5537f87c27ac081f721e528973c43646f43ad9d29052c9629/data_transfer-1.0.9-py2.py3-none-any.whl" } ], "1.2.0": [ { "comment_text": "", "digests": { "md5": "157c9b1759331360b20ad1374f54f463", "sha256": "3cfe5e1b54f45a379c319dadd8689c0a2857aa221ba1018aae3c0418b2f65426" }, "downloads": -1, "filename": "data-transfer-1.2.0.tar.gz", "has_sig": false, "md5_digest": "157c9b1759331360b20ad1374f54f463", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14472, "upload_time": "2018-01-11T11:33:11", "url": "https://files.pythonhosted.org/packages/06/5e/79d13ee1f77efaa93ffca8e706f894c8b50164b2e5a448af6d5aa737fa53/data-transfer-1.2.0.tar.gz" } ], "1.5.0": [ { "comment_text": "", "digests": { "md5": "a3e8919d28523dd77ad4975b97c50bf4", "sha256": "ed9829f18a706bc7c8d4eba345a9c6c5c42d0d08ffd29926f00e2e60ca77e842" }, "downloads": -1, "filename": "data-transfer-1.5.0.tar.gz", "has_sig": false, "md5_digest": "a3e8919d28523dd77ad4975b97c50bf4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16002, "upload_time": "2018-02-20T15:23:32", "url": "https://files.pythonhosted.org/packages/d6/9a/9d7093580dfd9427b0d0f38ea8ceeb91a05586a6f02efd0593191016ef88/data-transfer-1.5.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a3e8919d28523dd77ad4975b97c50bf4", "sha256": "ed9829f18a706bc7c8d4eba345a9c6c5c42d0d08ffd29926f00e2e60ca77e842" }, "downloads": -1, "filename": "data-transfer-1.5.0.tar.gz", "has_sig": false, "md5_digest": "a3e8919d28523dd77ad4975b97c50bf4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16002, "upload_time": "2018-02-20T15:23:32", "url": "https://files.pythonhosted.org/packages/d6/9a/9d7093580dfd9427b0d0f38ea8ceeb91a05586a6f02efd0593191016ef88/data-transfer-1.5.0.tar.gz" } ] }