{
    "info": {
        "author": "Marko Ristin",
        "author_email": "marko@parquery.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3.5"
        ],
        "description": "gcloudwrap\n==========\n\n``gcloudwrap`` provides a wrapper around the Google API client to help you manage your Google cloud (*i.e.* its\n'compute' component). The original client provides all the needed functionality and ``gcloudwrap`` adds little in terms\nof extras. However, we found it easier to have a thin wrapper around the cloud management to give us a domain language\ncloser to the tasks we had to repeatedly execute.\n\nWe initially found it hard to develop using original Google API client since it lacked type annotations and its dynamic\napproach to object creation allowed no code inspection in an IDE such as PyCharm. Since\nscripts involving cloud management usually take longer to execute, we found our development iterations to be fairly\nlong and often broke due to minor errors related to incorrect types. Therefore, we added type annotations so that the\nstatic analyzer (such as mypy) would panic and catch the type errors early. This substantially facilitated the\ndevelopment of the deployment scripts.\n\nMind that we have not wrapped all of Google Compute API. We focused only on the parts necessary for the deployment.\nPlease create an issue if you would like more functionality to be wrapped.\n\nScenarios\n=========\n\nBatch Jobs\n----------\nWriting a deployment script for large batch jobs is straight-forward with ``gcloudwrap``:\n\n* You spin up a couple of machines.\n* Use ``google-cloud-storage`` module to copy the necessary data from the storage.\n* Use an SSH Python module (such as spurplus_) to install the dependencies, copy the released executables and initiate\n  the processes within ``tmux`` (so that they stay alive after you exit the shell).\n\nWith this setup, no extra virtualization layer (such as Docker) is necessary, and you do not need to understand how to\nset up and maintain an extra cluster management tool. It worked very well for us for different scales\n(from trivial one-off batch jobs to large-scale deployments). Additionally, we found it much simpler to debug (and\nresume afterwards) directly on a machine than going through an extra virtualization layer.\n\nDeployment\n----------\nApart from batch jobs, we intensively use ``gcloudwrap`` for the deployment in the cloud. Our workflow includes:\n\n* Reserving an external address,\n* Creating the instance,\n* Tagging the instance (usually in order to modify the firewall rules),\n* Authorizing the SSH keys,\n* Creating and attaching persistent disks,\n* Formatting the disks (if necessary) and mounting them and\n* Running an initialization & installation script (based on an SSH module like spurplus_).\n\n``gcloudwrap`` is intended to substantially simplify the process compared to the original Google Python API client.\n\n(In case you need to deploy on other premises, we would suggest you to explicitly separate the\ninitialization script from the deployment script. The initialization scripts should accept user name and host name as\narguments and execute the commands remotely *via* SSH.\n\nWhen you deploy on Google Cloud, create the instance and authorize your SSH key with it. Once the instance is up,\nrun the initialization script against its external IP address as host name and whatever user you authorized your\nSSH key with.\n\nIf you need to deploy on other premises, ask them to authorize you your SSH key. Now you can simply run the same\ninitialization script with a different pair of user and host name.)\n\n.. _spurplus: https://pypi.org/project/spurplus/\n\nRelated Projects\n================\n\n* https://github.com/google/google-api-python-client/ -- original Google API client whih we wrapped\n* https://googlecloudplatform.github.io/google-cloud-python/ -- client for Google Cloud Platform services; it still\n  lacked Compute support at the time we started developing ``gcloudwrap`` (2018-05-01)\n* https://pypi.org/project/gwrappy/ -- user-friendly wrapper for Google APIs. This wrapper covers way more services\n  that we had in mind. Since it does not focus on the deployment, it did not match with our idea to have a simple\n  deployment language.\n* https://libcloud.readthedocs.io/en/latest/index.html -- abstracts differences between cloud providers and provides\n  a unified API. If you plan to use different cloud providers, this is a better alternative. In case you only want to\n  use Google cloud, we found that such an abstraction library introduces unnecessary complexity.\n\n\nUsage\n=====\n\nWe tried to follow the naming of the API as much as possible. The definition and description of the API can be found\nhere: `Google Compute API`_. However, we complied to Python naming convention, and renamed all camelCase arguments into snake_case.\n\n.. _`Google Compute API`: https://developers.google.com/resources/api-libraries/documentation/compute/v1/python/latest/\n\nThe following code snippet shows a common deployment which creates an instance with a persistent disk and a\nreserved external IP address. The instance is tagged with 'default-allow-http' to open a HTTP port in its firewall.\nFinally, we authorize a public SSH key with the instance.\n\n.. code-block:: python\n\n    import pathlib\n\n    import gcloudwrap\n\n    instance = 'some-instance'\n    address = 'some-address'\n    disk = 'some-persistent-disk'\n    device_name = 'persistency'\n\n    service_account = 'some-service-account@some-project-221984.iam.gserviceaccount.com\n\n    public_ssh_key = pathlib.Path(\n        '/some/path/to/ssh/id_rsa.pub').read_text()\n\n    gce = gcloudwrap.Gce()\n\n    gce.addresses.insert(name=address)\n    static_ip = gce.addresses.static_ip(\n        address=address)\n\n    gce.disks.insert(\n        name=disk,\n        disk_type='pd-standard',\n        size_gb=5)\n\n    gce.instances.insert(\n        name=instance,\n        machine_type='f1-micro',\n        address=static_ip,\n        service_account=service_account)\n\n    gce.instances.attach_disk(\n        instance=instance,\n        disk=disk,\n        device_name=device_name)\n\n    # open up HTTP port\n    tags = gce.instances.tags(instance=instance)\n    tags.items.add('default-allow-http')\n    gce.instances.set_tags(\n        instance=instance, tags=tags)\n\n    # authorize the SSH key\n    keys = gcloudwrap.SSHKey(\n        user='some-user',\n        public_key=public_key)\n\n    metadata = gce.instances.metadata(instance=instance)\n    metadata.set_ssh_keys(keys=[key])\n\n    gce.instances.set_metadata(\n        instance=instance,\n        metadata=metadata)\n\n    # format the persistent disk and mount it\n    ssh = gce.instances.ssh(\n        instance=instance,\n        user=\"some-devop-user\")\n\n    operator = gcloudwrap.Operator(call_fn=ssh.call)\n\n    operator.format_disk(\n        device_name=device_name)\n\n    operator.mount_disk(\n        device_name=device_name,\n        path=pathlib.Path('/mnt/disks/persistency'))\n\nSometimes it is convenient to store the list of authorized SSH keys in a file and re-use this list when\ndeploying the instance. We provide a shortcut function ``gcloudwrap.ssh_keys_from_file`` to achieve that:\n\n.. code-block:: python\n\n    import gcloudwrap\n\n    instance = 'some-instance'\n    keys_path = '/path/to/some/keys.txt'\n\n    keys = gcloudwrap.ssh_keys_from_file(\n        path=keys_path,\n        default_user='some-default-user')\n\n    metadata = gce.instances.metadata(instance=instance)\n    metadata.set_ssh_keys(keys=keys)\n\n    gce.instances.set_metadata(\n        instance=instance,\n        metadata=metadata)\n\n\nInstallation\n============\n\n* Create a virtual environment:\n\n.. code-block:: bash\n\n    python3 -m venv venv3\n\n* Activate it:\n\n.. code-block:: bash\n\n    source venv3/bin/activate\n\n* Install ``gcloudwrap`` with pip:\n\n.. code-block:: bash\n\n    pip3 install gcloudwrap\n\n* Set up the application-default credentials\n\n.. code-block:: bash\n\n    gcloud auth application-default login\n\nDevelopment\n===========\n\n* Check out the repository.\n\n* In the repository root, create the virtual environment:\n\n.. code-block:: bash\n\n    python3 -m venv venv3\n\n* Activate the virtual environment:\n\n.. code-block:: bash\n\n    source venv3/bin/activate\n\n* Install the development dependencies:\n\n.. code-block:: bash\n\n    pip3 install -e .[dev]\n\n* We provide a set of live tests. You need to set up your environment such that the credentials can be directly\n  inferred by the tests. Apart from the credentials, you can also use the following environment variables:\n\n    * ``TEST_GCLOUDWRAP_SERVICE_ACCOUNT`` to specify the service account attached to the instances created during the\n      test. If unspecified, default service account of the GCE project is used.\n    * ``TEST_GCLOUDWRAP_PREFIX`` to specify the prefix of all the created Google cloud resources; if not specified,\n      equals \"test-gcloudwrap\"\n    * ``TEST_GCLOUDWRAP_SSH_PUBLIC_KEY_PATH`` to specify the path to the SSH public key; if not specified,\n      equals ~/.ssh/id_rsa.pub (where \"~\" is expanded to the user home directory)\n\n  Mind that the live tests will use Google Cloud resources for which you will be billed. Always check that no resources\n  are used after the tests finished so that you don't incur an unnecessary cost!\n\n* We use tox for testing and packaging the distribution. Assuming that the virtual environment has been activated and\n  the development dependencies have been installed, run:\n\n.. code-block:: bash\n\n    tox\n\n* We also provide a set of pre-commit checks that lint and check code for formatting. Run them locally from an activated\n  virtual environment with development dependencies:\n\n.. code-block:: bash\n\n    ./precommit.py\n\n* The pre-commit script can also automatically format the code:\n\n.. code-block:: bash\n\n    ./precommit.py  --overwrite\n\nVersioning\n==========\nWe follow `Semantic Versioning <http://semver.org/spec/v1.0.0.html>`_. The version X.Y.Z indicates:\n\n* X is the major version (backward-incompatible),\n* Y is the minor version (backward-compatible), and\n* Z is the patch version (backward-compatible bug fix).",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/Parquery/gcloudwrap",
        "keywords": "Google cloud gcloud deployment wrap",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "gcloudwrap",
        "package_url": "https://pypi.org/project/gcloudwrap/",
        "platform": "",
        "project_url": "https://pypi.org/project/gcloudwrap/",
        "project_urls": {
            "Homepage": "https://github.com/Parquery/gcloudwrap"
        },
        "release_url": "https://pypi.org/project/gcloudwrap/1.0.5/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "wraps Google cloud client to facilitate deployment.",
        "version": "1.0.5"
    },
    "last_serial": 5410385,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "34d901ed8a1f06d7526ee00263f7f21b",
                    "sha256": "3c7b8ecf733a75858b9246535ffbff200e66736fd77887739edefa0c85168a2e"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "34d901ed8a1f06d7526ee00263f7f21b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19383,
                "upload_time": "2018-05-30T14:52:21",
                "url": "https://files.pythonhosted.org/packages/8c/69/da6261876b70b23d0f235018cf647020d229206dc6eba13e277c9d6f1f5f/gcloudwrap-1.0.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "70d1acfdb06f8a085495be0f339f3330",
                    "sha256": "ac9a3ec120b7c4d73eb1a6225ca3f2d92f4b8b3dafa80a347adc4f6ced194e4f"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "70d1acfdb06f8a085495be0f339f3330",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19423,
                "upload_time": "2018-08-03T14:45:26",
                "url": "https://files.pythonhosted.org/packages/aa/e8/aa7722f583ee6ed9f3dc16d435d53191e82b5246575df29fada3e201edfb/gcloudwrap-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d702b9fdc18068eb2f5b0951f396b843",
                    "sha256": "cf3317fc30dbcbcb7a21f6d3774941f02d158077ab768d9d01f074a11385a816"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "d702b9fdc18068eb2f5b0951f396b843",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19445,
                "upload_time": "2018-08-17T10:37:10",
                "url": "https://files.pythonhosted.org/packages/cf/d5/46fa1042cf1d805791320d07a422a6a5b237790297536ca1b9802f9536a9/gcloudwrap-1.0.2.tar.gz"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "971595243573488f371d87c8f14e9653",
                    "sha256": "607a0be12a8f644d5e6a15e19dab9c4e2659d865782c2e606f4b6c5c1d818c73"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "971595243573488f371d87c8f14e9653",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19515,
                "upload_time": "2018-08-17T13:00:48",
                "url": "https://files.pythonhosted.org/packages/c2/f8/eecf2c3b8ff1346536254179f08067b4d67283a80cc25ea4e660e1d4362e/gcloudwrap-1.0.3.tar.gz"
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "205cf358399e93857b10a35fd5944f9d",
                    "sha256": "8600e175da18cb2aa1093f519842237b989650c9a944937f0bf46c22371b84d9"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "205cf358399e93857b10a35fd5944f9d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19728,
                "upload_time": "2018-09-24T15:27:23",
                "url": "https://files.pythonhosted.org/packages/e1/6c/83d323b7332f9ebb8d1f31713cbc4a48e5c941f200d2f51b0b84430c790f/gcloudwrap-1.0.4.tar.gz"
            }
        ],
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "938d6a9bff6be8ee159e60c0d998b003",
                    "sha256": "a745a12f38aacd6f1463932b0324553ac58d6839076fb0d826975b758b6b5552"
                },
                "downloads": -1,
                "filename": "gcloudwrap-1.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "938d6a9bff6be8ee159e60c0d998b003",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19754,
                "upload_time": "2018-10-23T12:10:48",
                "url": "https://files.pythonhosted.org/packages/21/83/d74a3873ad6bd35ece087808e204e43b7cb7086499859aeaa3eaeb9ae3de/gcloudwrap-1.0.5.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "938d6a9bff6be8ee159e60c0d998b003",
                "sha256": "a745a12f38aacd6f1463932b0324553ac58d6839076fb0d826975b758b6b5552"
            },
            "downloads": -1,
            "filename": "gcloudwrap-1.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "938d6a9bff6be8ee159e60c0d998b003",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 19754,
            "upload_time": "2018-10-23T12:10:48",
            "url": "https://files.pythonhosted.org/packages/21/83/d74a3873ad6bd35ece087808e204e43b7cb7086499859aeaa3eaeb9ae3de/gcloudwrap-1.0.5.tar.gz"
        }
    ]
}