{
    "info": {
        "author": "Department for International Trade - WebOps",
        "author_email": "webops@digital.trade.gov.uk",
        "bugtrack_url": null,
        "classifiers": [
            "Framework :: AsyncIO",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Topic :: Internet :: File Transfer Protocol (FTP)"
        ],
        "description": "# aioftps3 [![CircleCI](https://circleci.com/gh/uktrade/aioftps3.svg?style=svg)](https://circleci.com/gh/uktrade/aioftps3) [![Maintainability](https://api.codeclimate.com/v1/badges/4a9332f4782f6b4bf35a/maintainability)](https://codeclimate.com/github/uktrade/aioftps3/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/4a9332f4782f6b4bf35a/test_coverage)](https://codeclimate.com/github/uktrade/aioftps3/test_coverage)\n\nFTP in front of AWS S3, using [asyncio](https://docs.python.org/3/library/asyncio.html), and [aiohttp](https://github.com/aio-libs/aiohttp). Only a subset of the FTP protocol is supported, with implicit TLS and PASV mode; connections will fail otherwise.\n\n## Installation\n\n```bash\npip install aioftps3\n```\n\nAn SSL key and certificate must be present `$HOME/ssl.key` and `$HOME/ssl.crt` respectively. To create a self-signed certificate, you can use openssl.\n\n```bash\nopenssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj /CN=selfsigned \\\n    -keyout $HOME/ssl.key \\\n    -out $HOME/ssl.crt\n```\n\n## Running\n\n```bash\npython -m aioftps3.server_main\n```\n\n## Configuration\n\nConfiguration is through environment variables\n\n| Varaiable | Description | Example |\n| --- | --- | --- |\n| `AWS_AUTH_MECHANISM` | How requests to AWS are authenticated. Can be `secret_access_key` or `ecs_role`. If `ecs_role` it is expected that the server runs in an ECS container. | `secret_access_key` |\n| `AWS_ACCESS_KEY_ID` | The ID of the  AWS access key, if `AWS_AUTH_MECHANISM` is `secret_access_key`. | _ommitted_ |\n| `AWS_SECRET_ACCESS_KEY` | The secret part of the  AWS access key, if `AWS_AUTH_MECHANISM` is `secret_access_key` | _ommitted_ |\n| `AWS_S3_BUCKET_REGION` | The region of the S3 bucket that stores the files. | `eu-west-1` |\n| `AWS_S3_BUCKET_HOST` | The hostname used to communicate with S3. | `s3-eu-west-1.amazonaws.com` |\n| `AWS_S3_BUCKET_NAME` | The name of the bucket files are stored in. | `my-bucket-name` |\n| `AWS_S3_BUCKET_DIR_SUFFIX` | The suffix of the keys created in order to simulate a directory. Must start with a forward slash, but does not need to be longer.  | `/` |\n| `FTP_USERS__i__LOGIN` | For `i` any integer, the username of an FTP user that can login. | `my-user` |\n| `FTP_USERS__i__PASSWORD_HASHED` | For `i` any integer, the hash, as generated by [create_password.py](https://github.com/uktrade/aioftps3/blob/master/create_password.py), of the password of an FTP user that can login, using the salt in `FTP_USERS__i__PASSWORD_SALT` | _ommitted_ |\n| `FTP_USERS__i__PASSWORD_SALT` | See `FTP_USERS__i__PASSWORD_HASHED` | _ommitted_ |\n| `FTP_COMMAND_PORT` | The port that the server listens on for command connections. | `8021` |\n| `FTP_DATA_PORTS_FIRST` | The first data port in the range for PASV mode data transfers. | `4001` |\n| `FTP_DATA_PORTS_COUNT` | The number of ports used after `FTP_DATA_PORTS_FIRST`. | `30` |\n| `FTP_DATA_CIDR_TO_DOMAINS__i__CIDR` | For `i` any integer, a CIDR range used to match the IP of incoming command connections. If a match is found, the IP of the corresponding domain or IP address in `FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN` is returned to the client in response to PASV mode requests. Some clients will respond to `FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN` being `0.0.0.0` by making PASV mode data connections to the same IP as the original command connection, but not all. | `0.0.0.0/0` |\n| `FTP_DATA_CIDR_TO_DOMAINS__i__DOMAIN` | See `FTP_DATA_CIDR_TO_DOMAINS__i__CIDR`. | `ftp.my-domain.com` |\n| `HEALTHCHECK_PORT` | The port the server listens on for healthcheck requests, such as from an AWS network load balancer. | `8022` |\n\n\n## Advanced usage\n\nThe code in [aioftps3.server_main](https://github.com/uktrade/aioftps3/blob/master/aioftps3/server_main.py) satisfies a very particular use case, which may not be useful to most. However, the bulk of the code can be used for other cases: you will have to write your own aioftps3.server_main-equivalent, using the functions [aioftps3.server.on_client_connect](https://github.com/uktrade/aioftps3/blob/master/aioftps3/server.py) and [aioftps3.server_socket.server](https://github.com/uktrade/aioftps3/blob/master/aioftps3/server_socket.py). For example, you could\n\n- Store credentials, appropriately hashed, differently, .e.g. in a database.\n- Have the credentials hashed differently.\n- Allow/deny PASV mode data connections based on some condition.\n\nSee the source of [aioftps3.server_main](https://github.com/uktrade/aioftps3/blob/master/aioftps3/server_main.py) for how these functions can be used.\n\n\n## Creating a password and salt\n\n```bash\npython ./create_password.py\n```\n\n## Running tests\n\nCertificates must be created, and Minio, which emulates S3 locally, must be started\n\n```bash\n./certificates-create.sh && ./minio-start.sh\n```\n\nand then to run the tests themselves.\n\n```bash\n./tests.sh\n```\n\n\n## Features / Design / Limitations\n\n- Can upload files bigger than 2G: uses [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingRESTAPImpUpload.html) under the hood.\n\n- Does not store uploading files in memory before uploading them to S3: i.e. it is effectively a streaming upload. However, it's not completely streaming: each part of multipart upload is stored in memory before it begins to transfer to S3, in order to be able to hash its content and determine its length.\n\n- For uploading files, hashes are computed incrementally as data comes in in order to not block the event loop just before uploads to S3.\n\n- As few dependencies as is reasonable: aiohttp and its dependencies. Boto 3 is _not_ used.\n\n- May not behave well if upload to the server is faster than its upload to S3.\n\n- There is some locking to deal with the same files being operated on concurrently. However...\n\n- .... it does nothing to deal with [eventual consistency of S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel), and so some operations may appear to not have an immediate effect.\n\n\n## Building and running locally\n\n```bash\ndocker build -t ftps-s3 . && \\\ndocker run --rm -p 8021-8042:8021-8042 \\\n  -e AWS_AUTH_MECHANISM=secret_access_key \\\n  -e AWS_ACCESS_KEY_ID=ommitted \\\n  -e AWS_SECRET_ACCESS_KEY=ommitted \\\n  -e AWS_S3_BUCKET_REGION=eu-west-1 \\\n  -e AWS_S3_BUCKET_HOST=s3-eu-west-1.amazonaws.com \\\n  -e AWS_S3_BUCKET_NAME=my-bucket-name \\\n  -e AWS_S3_BUCKET_DIR_SUFFIX=/ \\\n  -e FTP_USERS__1__LOGIN=user \\\n  -e FTP_USERS__1__PASSWORD_HASHED=ommitted \\\n  -e FTP_USERS__1__PASSWORD_SALT=ommitted \\\n  -e FTP_COMMAND_PORT=8021 \\\n  -e FTP_DATA_PORTS_FIRST=4001 \\\n  -e FTP_DATA_PORTS_COUNT=2 \\\n  -e FTP_DATA_CIDR_TO_DOMAINS__1__CIDR=0.0.0.0/0 \\\n  -e FTP_DATA_CIDR_TO_DOMAINS__1__DOMAIN=0.0.0.0 \\\n  -e HEALTHCHECK_PORT=8022\n  ftps-s3\n```\n\n\n## Building and pushing to Quay\n\n```bash\ndocker build -t ftps-s3 . && \\\ndocker tag ftps-s3:latest quay.io/uktrade/ftps-s3:latest && \\\ndocker push quay.io/uktrade/ftps-s3:latest\n```\n\n\n## Building and pushing healthcheck application to Quay\n\n```bash\ndocker build -t ftps-s3-healthcheck . -f Dockerfile-healthcheck && \\\ndocker tag ftps-s3-healthcheck:latest quay.io/uktrade/ftps-s3-healthcheck:latest && \\\ndocker push quay.io/uktrade/ftps-s3-healthcheck:latest\n```\n\n## Building and pushing Minio, used for testing, to Quay\n\n```bash\ndocker build -t ftps-s3-minio . -f Dockerfile-minio && \\\ndocker tag ftps-s3-minio:latest quay.io/uktrade/ftps-s3-minio:latest && \\\ndocker push quay.io/uktrade/ftps-s3-minio:latest\n```\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/uktrade/aioftps3",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "aioftps3",
        "package_url": "https://pypi.org/project/aioftps3/",
        "platform": "",
        "project_url": "https://pypi.org/project/aioftps3/",
        "project_urls": {
            "Homepage": "https://github.com/uktrade/aioftps3"
        },
        "release_url": "https://pypi.org/project/aioftps3/0.0.6/",
        "requires_dist": [
            "aiohttp"
        ],
        "requires_python": "",
        "summary": "FTP in front of AWS S3, powered by asyncio and aiohttp",
        "version": "0.0.6"
    },
    "last_serial": 4520149,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c9085b9525e77beee91f813ebc405543",
                    "sha256": "eeb77919190a45602aebe7dd0f02ba416d9430ad24aa3d3e430dedc9dbacad64"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c9085b9525e77beee91f813ebc405543",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 7832,
                "upload_time": "2018-10-27T12:05:29",
                "url": "https://files.pythonhosted.org/packages/3f/9c/4f5de823413484c5e3923c0c189eea55f4b94575bd167c0ecd8fb190c731/aioftps3-0.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a8074aa7f8a117caa147b7fba50d209f",
                    "sha256": "2f2a2a3d412d1144d02eebe984f8914ec66536fa3667e93640b36f5607fc4657"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "a8074aa7f8a117caa147b7fba50d209f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6792,
                "upload_time": "2018-10-27T12:05:30",
                "url": "https://files.pythonhosted.org/packages/4a/01/cc823a6c8048a7e1c1678c6d655e9b84c66354669645d3ee90e20c2acf83/aioftps3-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d337f344864c1fe2e28b34fa16b55540",
                    "sha256": "33749100e005e46896de24796a930291889de57cde163ab1e89bc99b4d71e2d6"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d337f344864c1fe2e28b34fa16b55540",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9526,
                "upload_time": "2018-10-29T14:07:18",
                "url": "https://files.pythonhosted.org/packages/32/75/28b3ca3d7e52333362ac65f3c6ca5f1e9c98da5480fca0ee18d81d6cb9f2/aioftps3-0.0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "44c0f25d04c087739f787004ff91d332",
                    "sha256": "e958d3b06e165e56b9316d08593b1e5c8b8ae206a4066a224d3137957d29a494"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "44c0f25d04c087739f787004ff91d332",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8598,
                "upload_time": "2018-10-29T14:07:20",
                "url": "https://files.pythonhosted.org/packages/19/69/0ae5a72941c27c3e4dda61306dd7609cf7fb46ca05fb4b936b8e1b8febce/aioftps3-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e3827b36f41d51aac0f7a12ff9690f75",
                    "sha256": "05beb938e3225f63c9ae076566cd343baaac5e049099b5e9547a4c1f08fc5d63"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e3827b36f41d51aac0f7a12ff9690f75",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9800,
                "upload_time": "2018-10-30T09:43:59",
                "url": "https://files.pythonhosted.org/packages/fe/2b/92a6c1b84a27e669b83bea3a666fbd90f5dadc994ccc382903c32df5f7fe/aioftps3-0.0.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e0cb2eabb9f9160a526416aededf3a0b",
                    "sha256": "0921b355146e1a2fe79fb9cec2bdad8e9bf1839b76e84dce491bc2ca2be893d0"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "e0cb2eabb9f9160a526416aededf3a0b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8867,
                "upload_time": "2018-10-30T09:44:00",
                "url": "https://files.pythonhosted.org/packages/42/de/88ce0f8b3581330e904c0e834e1efbca7c099448ac0b5f3df6a43988eb41/aioftps3-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d2ed1973122f92705ea263c711795fb8",
                    "sha256": "85264127c67ae4ce27521b9890f7fdf9461c7a4a23684b61ffb8a065fd85a774"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d2ed1973122f92705ea263c711795fb8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 10967,
                "upload_time": "2018-11-13T11:04:10",
                "url": "https://files.pythonhosted.org/packages/74/df/5feaff8b63f596273e41a495b4da520a2c90d3c08331f23d30cd16c9e23d/aioftps3-0.0.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "aca6650909cc8a0893359c345595514d",
                    "sha256": "758b91083c10c12c2f6dd9a5c32c93960aa57d3eacb009a9c3719846e01ba07d"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "aca6650909cc8a0893359c345595514d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3578,
                "upload_time": "2018-11-13T11:04:11",
                "url": "https://files.pythonhosted.org/packages/e2/33/44ec9e4134046d5081aa6fa9462d02897150788c502805d2046d161ffd6f/aioftps3-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "065347fac885b354b885084212950499",
                    "sha256": "11d104d0a7fc596fac3eb30878f685634124d4ba68d3679ea55fb51947f0942c"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "065347fac885b354b885084212950499",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11115,
                "upload_time": "2018-11-13T11:13:00",
                "url": "https://files.pythonhosted.org/packages/63/b5/1149d787aa96c805996ba3c2bf647eac3452c99ad41f7c03dd81e141ba56/aioftps3-0.0.5-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "4062c35206e41d1902fef13981b4cad8",
                    "sha256": "d078d4c69815c266be4020afba31038fd74d8aa7693a25ffca9513baa624d796"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "4062c35206e41d1902fef13981b4cad8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3787,
                "upload_time": "2018-11-13T11:13:01",
                "url": "https://files.pythonhosted.org/packages/b7/42/9e1fc77e13ebfcfa0aa11148dde61ef7139dfda45036d1ec574a5b02e510/aioftps3-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5d6a3b25c114e1bc81b2f2bc3295493f",
                    "sha256": "762a63a56dc7335d04d2ad74df920b757d7bd48494d6567a21264b4fd1bf1b43"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5d6a3b25c114e1bc81b2f2bc3295493f",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11523,
                "upload_time": "2018-11-23T10:46:43",
                "url": "https://files.pythonhosted.org/packages/e8/27/9f54a0b0dbf66b217840b2f4e85563d7ba611600cddd05e50ba3a0035712/aioftps3-0.0.6-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "041eac2ab3c43be314fedc6e902823e4",
                    "sha256": "e40d731f24d3b176e45bad94e4dbe6813b34e21310ce645a843869fc138bb4e0"
                },
                "downloads": -1,
                "filename": "aioftps3-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "041eac2ab3c43be314fedc6e902823e4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4494,
                "upload_time": "2018-11-23T10:46:45",
                "url": "https://files.pythonhosted.org/packages/52/bc/c481fe2ecf1ece0363f4311e6c5abe7760f90788d52662e3e55e9a742ecc/aioftps3-0.0.6.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5d6a3b25c114e1bc81b2f2bc3295493f",
                "sha256": "762a63a56dc7335d04d2ad74df920b757d7bd48494d6567a21264b4fd1bf1b43"
            },
            "downloads": -1,
            "filename": "aioftps3-0.0.6-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "5d6a3b25c114e1bc81b2f2bc3295493f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 11523,
            "upload_time": "2018-11-23T10:46:43",
            "url": "https://files.pythonhosted.org/packages/e8/27/9f54a0b0dbf66b217840b2f4e85563d7ba611600cddd05e50ba3a0035712/aioftps3-0.0.6-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "041eac2ab3c43be314fedc6e902823e4",
                "sha256": "e40d731f24d3b176e45bad94e4dbe6813b34e21310ce645a843869fc138bb4e0"
            },
            "downloads": -1,
            "filename": "aioftps3-0.0.6.tar.gz",
            "has_sig": false,
            "md5_digest": "041eac2ab3c43be314fedc6e902823e4",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 4494,
            "upload_time": "2018-11-23T10:46:45",
            "url": "https://files.pythonhosted.org/packages/52/bc/c481fe2ecf1ece0363f4311e6c5abe7760f90788d52662e3e55e9a742ecc/aioftps3-0.0.6.tar.gz"
        }
    ]
}