{
    "info": {
        "author": "Firefox Operations Security Team (foxsec)",
        "author_email": "foxsec+frost@mozilla.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
            "Natural Language :: English",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# pytest-services\n\nClients and [pytest](https://docs.pytest.org/en/latest/index.html)\ntests for checking that third party services the @foxsec team uses are\nconfigured correctly.\n\nWe trust third party services to return their status correctly, but\nwant to answer questions whether they are configured properly such as:\n\n* Are our AWS DB snapshots publicly accessible?\n* Is there a dangling DNS entry in Route53?\n* Will someone get paged when an alert goes off?\n\n## Usage\n\n### Requirements\n\n* [GNU Make 3.81](https://www.gnu.org/software/make/)\n* [Python 3.6.2](https://www.python.org/downloads/)\n\nNote: other versions may work too these are the versions @g-k used for development\n\n### Installing\n\nFrom the project root run:\n\n```console\nmake install\n```\n\nThis will:\n\n* create a Python [virtualenv](https://docs.python.org/3/library/venv.html) to isolate it from other Python packages\n* install Python requirements in the virtualenv\n\n### Running\n\nActivate the venv in the project root:\n\n```console\nsource venv/bin/activate\n```\n\nTo fetch RDS resources from the cache or AWS API and check that\nbackups are enabled for DB instances for [the configured aws\nprofile](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html)\nnamed `default` in the `us-west-2` region we can run:\n\n```console\npytest --ignore aws/s3 --ignore aws/ec2 -k test_rds_db_instance_backup_enabled -s --aws-profiles default --debug-calls\n```\n\nThe options include pytest options:\n\n* [`--ignore`](https://docs.pytest.org/en/latest/example/pythoncollection.html#ignore-paths-during-test-collection) to skip fetching resources for non-RDS resources\n* [`-k`](https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name) for selecting tests matching the substring `test_rds_db_instance_backup_enabled` for the one test we want to run\n* [`-m`](https://docs.pytest.org/en/latest/example/markers.html#marking-test-functions-and-selecting-them-for-a-run) not used but the marker filter can be useful for selecting all tests for specific services (e.g. `-m rds`)\n* [`-s`](https://docs.pytest.org/en/latest/capture.html) to disable capturing stdout so we can see the progress fetching AWS resources\n\nand options pytest-services adds:\n\n* `--debug-calls` for printing (with `-s`) API calls we make\n* `--aws-profiles` for selecting one or more AWS profiles to fetch resources for or the AWS default profile / `AWS_PROFILE` environment variable\n* `--offline` a flag to tell HTTP clients to not make requests and return empty params\n* [`--config`](#custom-test-config) path to test custom config file\n\nand produces output like the following showing a DB instance with backups disabled:\n\n```console\n=========================================================== test session starts ===========================================================\nplatform darwin -- Python 3.6.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0\nmetadata: {'Python': '3.6.2', 'Platform': 'Darwin-15.6.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.3.2', 'py': '1.5.2', 'pluggy': '0.6.\n0'}, 'Plugins': {'metadata': '1.5.1', 'json': '0.4.0', 'html': '1.16.1'}}\nrootdir: /Users/gguthe/mozilla-services/pytest-services, inifile:\nplugins: metadata-1.5.1, json-0.4.0, html-1.16.1\ncollecting 0 items                                                                                                                        c\nalling AWSAPICall(profile='default', region='us-west-2', service='rds', method='describe_db_instances', args=[], kwargs={})\ncollecting 4 items\n...\naws/rds/test_rds_db_instance_backup_enabled.py ...F                                                                                 [100%]\n\n================================================================ FAILURES =================================================================\n_______________________________________ test_rds_db_instance_backup_enabled[test-db] ________________________________________\n\nrds_db_instance = {'AllocatedStorage': 50, 'AutoMinorVersionUpgrade': True, 'AvailabilityZone': 'us-west-2c', 'BackupRetentionPeriod': 0, .\n..}\n\n    @pytest.mark.rds\n    @pytest.mark.parametrize('rds_db_instance',\n                             rds_db_instances(),\n                             ids=lambda db_instance: db_instance['DBInstanceIdentifier'])\n    def test_rds_db_instance_backup_enabled(rds_db_instance):\n>       assert rds_db_instance['BackupRetentionPeriod'] > 0, \\\n            'Backups disabled for {}'.format(rds_db_instance['DBInstanceIdentifier'])\nE       AssertionError: Backups disabled for test-db\nE       assert 0 > 0\n\naws/rds/test_rds_db_instance_backup_enabled.py:12: AssertionError\n=========================================================== 72 tests deselected ===========================================================\n============================================ 1 failed, 3 passed, 72 deselected in 3.12 seconds ============================================\n```\n\n#### IAM Policy for pytest-services\n\nThe below policy will allow you to run all AWS tests in pytest-services against all resources in your account.\n\n```json\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"PytestServicesReadOnly\",\n      \"Action\": [\n        \"autoscaling:DescribeLaunchConfigurations\",\n        \"cloudtrail:DescribeTrails\",\n        \"ec2:DescribeFlowLogs\",\n        \"ec2:DescribeInstances\",\n        \"ec2:DescribeSecurityGroups\",\n        \"ec2:DescribeSnapshotAttribute\",\n        \"ec2:DescribeSnapshots\",\n        \"ec2:DescribeVolumes\",\n        \"ec2:DescribeVpcs\",\n        \"elasticache:DescribeCacheClusters\",\n        \"elasticloadbalancing:DescribeLoadBalancers\",\n        \"es:DescribeElasticsearchDomains\",\n        \"es:ListDomainNames\",\n        \"iam:GenerateCredentialReport\",\n        \"iam:GetCredentialReport\",\n        \"iam:GetLoginProfile\",\n        \"iam:ListAccessKeys\",\n        \"iam:ListAttachedGroupPolicies\",\n        \"iam:ListAttachedRolePolicies\",\n        \"iam:ListAttachedUserPolicies\",\n        \"iam:ListGroupPolicies\",\n        \"iam:ListGroupsForUser\",\n        \"iam:ListMFADevices\",\n        \"iam:ListRolePolicies\",\n        \"iam:ListRoles\",\n        \"iam:ListUserPolicies\",\n        \"iam:ListUsers\",\n        \"rds:DescribeDbInstances\",\n        \"rds:DescribeDbSecurityGroups\",\n        \"rds:DescribeDbSnapshotAttributes\",\n        \"rds:DescribeDbSnapshots\",\n        \"rds:ListTagsForResource\",\n        \"redshift:DescribeClusterSecurityGroups\",\n        \"redshift:DescribeClusters\",\n        \"s3:GetBucketAcl\",\n        \"s3:GetBucketCORS\",\n        \"s3:GetBucketLogging\",\n        \"s3:GetBucketPolicy\",\n        \"s3:GetBucketVersioning\",\n        \"s3:GetBucketWebsite\",\n        \"s3:ListAllMyBuckets\",\n        \"s3:ListBucket\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"*\"\n    }\n  ]\n}\n```\n\n#### Setting up GCP tests\n\n##### Enabling required API's for your project\n\n```\ngcloud [--project <project name>] services enable bigquery-json.googleapis.com\ngcloud [--project <project name>] services enable cloudresourcemanager.googleapis.com\ngcloud [--project <project name>] services enable compute.googleapis.com\ngcloud [--project <project name>] services enable sqladmin.googleapis.com\n```\n\n#### Setting up GSuite tests\n\nMake sure to have an OAuth2 app created and have the `client_secret.json` file in `~/.credentials` and then run:\n```\nmake setup_gsuite\n```\n\n### Caching\n\nThe AWS client will use AWS API JSON responses when available and save them using AWS profile, region, service name, service method, [botocore](http://botocore.readthedocs.io/) args and kwargs in the cache key to filenames with the format `.cache/v/pytest_aws:<aws profile>:<aws region>:<aws service>:<service method>:<args>:<kwargs>.json` e.g.\n\n```\nhead .cache/v/pytest_aws:cloudservices-aws-stage:us-west-2:rds:describe_db_instances::.json\n{\n    \"DBInstances\": [\n        {\n            \"AllocatedStorage\": 5,\n            \"AutoMinorVersionUpgrade\": true,\n            \"AvailabilityZone\": \"us-west-2c\",\n            \"BackupRetentionPeriod\": 1,\n            \"CACertificateIdentifier\": \"rds-ca-2015\",\n            \"CopyTagsToSnapshot\": false,\n            \"DBInstanceArn\": \"arn:aws:rds:us-west-2:123456678901:db:test-db\",\n```\n\nThese files can be removed individually or all at once with [the pytest --cache-clear](https://docs.pytest.org/en/latest/cache.html#usage) option.\n\n## Custom Test Config\n\npytest-services adds a `--config` cli option for passing in a custom config file specific to tests within pytest-services.\n\nThe example config in repo (`config.yaml.example`):\n```\nexemptions:\n  - test_name: test_ec2_instance_has_required_tags\n    test_param_id: i-0123456789f014c162\n    expiration_day: 2019-01-01\n    reason: ec2 instance has no owner\n  - test_name: test_ec2_security_group_opens_specific_ports_to_all\n    test_param_id: '*HoneyPot'\n    expiration_day: 2020-01-01\n    reason: purposefully insecure security group\nseverities:\n  - test_name: test_ec2_instance_has_required_tags\n    severity: INFO\n  - test_name: '*'\n    severity: ERROR\nregressions:\n  - test_name: test_ec2_security_group_opens_all_ports_to_all\n    test_param_id: '*mycustomgroup'\n    comment: this was remediated by ops team\naws:\n  admin_groups:\n    - \"Administrators\"\n  admin_policies:\n    - \"AWSAdminRequireMFA\"\n  user_is_inactive:\n    no_activity_since:\n      years: 1\n      months: 0\n    created_after:\n      weeks: 1\n  access_key_expires_after:\n    years: 1\n    months: 0\n  required_tags:\n    - Name\n    - Type\n    - App\n    - Env\n  required_amis:\n    - ami-00000000000000000\n    - ami-55555555555555555\n  whitelisted_ports_global:\n    - 25\n  whitelisted_ports:\n    - test_param_id: '*bastion'\n      ports:\n        - 22\n        - 2222\ngsuite:\n  domain: 'example.com'\n  user_is_inactive:\n    no_activity_since:\n      years: 1\n      months: 0\npagerduty:\n  users_with_remote_access_monitoring: 'pd_users.json'\n  bastion_users: 'hierahash/*hierahash.json'\n  alternate_usernames: 'alternate_usernames.json'\n```\n\n### Test Exemptions\n\npytest-services custom config format adds support for\nmarking test and test resource IDs as expected failures.\n\nThe keys for each exemption rule is:\n* test_name - Name of the test\n* test_param_id - test ID (usually an AWS resource ID) (prefix with `*` to turn into a regex matcher)\n* expiration_day - exception expiration day (as YYYY-MM-DD)\n* reason - exception reason\n\nThe config looks like:\n```\n...\nexemptions:\n  - test_name: test_ec2_instance_has_required_tags\n    test_param_id: i-0123456789f014c162\n    expiration_day: 2019-01-01\n    reason: ec2 instance has no owner\n  - test_name: test_ec2_security_group_opens_specific_ports_to_all\n    test_param_id: '*HoneyPot'\n    expiration_day: 2020-01-01\n    reason: purposefully insecure security group\n...\n```\n\n#### Enabling regex for test ID\n\nYou can prefix the test ID with a `*` to enable regex matching for the test ID. The `*` prefix will be stripped\noff, and the rest will be used as a regex.\n\nFor example:\n - `*foobar` becomes `foobar`\n - `*foo\\w+` becomes `foo\\w+`\n\nFor more information on Python's regex syntax see: [Regular Expression HOWTO](https://docs.python.org/3.4/howto/regex.html#regex-howto).\n\n**Note:** All regex rules are applied first. As well, the ordering of both regex and non-regex rules is top to bottom and the first one wins.\n\n\nWhen a json report is generated, the exemptions will show up in the\njson metadata as serialized markers:\n\n```json\npython -m json.tool report.json | grep -C 20 xfail\n...\n                        \"markers\": {\n                            \"ec2\": {\n                                \"name\": \"ec2\",\n                                \"args\": [],\n                                \"kwargs\": {}\n                            },\n                            \"parametrize\": {\n                                \"name\": \"parametrize\",\n                                \"args\": [\n                                    \"...skipped...\"\n                                ],\n                                \"kwargs\": [\n                                    \"...skipped...\"\n                                ]\n                            },\n                            \"xfail\": {\n                                \"name\": \"xfail\",\n                                \"args\": [],\n                                \"kwargs\": {\n                                    \"reason\": \"ec2 instance has no owner\",\n                                    \"strict\": true,\n                                    \"expiration\": \"2019-01-01\"\n                                }\n                            }\n                        },\n...\n```\n\n\n#### Test Severity\n\npytest-services custom config format adds support for marking the severity of a certain test. A severity can be `INFO`, `WARN`, or `ERROR`.\n\nThese do not modify pytest results (pass, fail, xfail, skip, etc.).\n\nThe config looks like:\n\n```\n...\nseverities:\n  - test_name: test_ec2_instance_has_required_tags\n    severity: INFO\n  - test_name: '*'\n    severity: ERROR\n...\n```\n\nAnd results in a severity and severity marker being included in the\njson metadata:\n\n```console\npytest --ignore aws/s3 --ignore aws/rds --ignore aws/iam -s --aws-profiles stage --aws-require-tags Name Type App Stack -k test_ec2_instance_has_required_tags --config config.yaml.example --json=report.json\n...\n```\n\n```json\npython -m json.tool report.json\n{\n    \"report\": {\n        \"environment\": {\n            \"Python\": \"3.6.2\",\n            \"Platform\": \"Darwin-15.6.0-x86_64-i386-64bit\"\n        },\n        \"tests\": [\n            {\n...\n                \"metadata\": [\n                    {\n...\n                    \"markers\": {\n...\n                            \"severity\": {\n                                \"name\": \"severity\",\n                                \"args\": [\n                                    \"INFO\"\n                                ],\n                                \"kwargs\": {}\n                            }\n                        },\n...\n                        \"severity\": \"INFO\",\n                        \"unparametrized_name\": \"test_ec2_instance_has_required_tags\"\n                    }\n...\n```\n\n### Test Regressions\n\npytest-services custom config format adds support for marking specific tests on specific resources as regressions.\nAs with `severity` this does not modify the pytest results, but rather adds a marker that can be used when analyzing the results.\n\nThe config looks like:\n```\n...\nregressions:\n  - test_name: test_ec2_security_group_opens_all_ports_to_all\n    test_param_id: '*mycustomgroup'\n    comment: this was remediated by ops team\n...\n```\n\n### AWS Config\n\npytest-services has a suite of AWS tests. This section of the custom config includes configuration options specific\nto these tests.\n\nThe config looks like:\n```\n...\naws:\n  # Relative time delta for test_iam_user_is_inactive. no_activity_since will be used as the failure marker,\n  # so in this example any user that hasn't had any activity for a year will be marked as a \"failure\". created_after\n  # is used as a grace period, so in this case any user that was created within the last week will be automatically\n  # pass this test.\n  user_is_inactive:\n    no_activity_since:\n      years: 1\n      months: 0\n    created_after:\n      weeks: 1\n  # Required tags used within the test_ec2_instance_has_required_tags test\n  required_tags:\n    - Name\n    - Type\n    - App\n    - Env\n  # Whitelsited ports for the test_ec2_security_group_opens_specific_ports_to_all\n  # test for all instances\n  whitelisted_ports_global:\n    - 25\n  # Whitelsited ports for the test_ec2_security_group_opens_specific_ports_to_all\n  # test for specific instances. In this example, we are whitelisting ports 22\n  # and 2222 for all security groups that include the word 'bastion' in them.\n  whitelisted_ports:\n    - test_param_id: '*bastion'\n      ports:\n        - 22\n        - 2222\n...\n```\n\n### GSuite Config\n\npytest-services has a suite of GSuite tests. This section of the\ncustom config includes configuration options specific to these tests.\n\n**Make sure to [setup GSuite](#setting-up-gsuite-tests) before running GSuite tests**\n\nThe config looks like:\n```\ngsuite:\n  # The specific GSuite domain to test.\n  domain: 'example.com'\n  # Relative time delta for test_admin_user_is_inactive. no_activity_since will be used as the failure marker,\n  # so in this example any user that hasn't had any activity for a year will be marked as a \"failure\".\n  user_is_inactive:\n    no_activity_since:\n      years: 1\n      months: 0\n```\n\n### Pagerduty Config\n\npytest-services does not query the pagerduty API, but can run tests against output from it.\n\nThe config looks like:\n```\npagerduty:\n  users_with_remote_access_monitoring: 'pd_users.json'\n  bastion_users: 'hierahash/*hierahash.json'\n  alternate_usernames: 'alternate_usernames.json'\n```\n\nWhere `users_with_remote_access_monitoring` and `bastion_users` are\nglobs for multiple files relative to the current working directory and\n`alternate_usernames` is the path to a single file.\n\nThe files have examples formats as follows:\n\n* `users_with_remote_access_monitoring`:\n\n```json\n[\n  {\n    \"avatar_url\": \"https://secure.gravatar.com/avatar/...\",\n    \"billed\": true,\n    \"color\": \"sea-green\",\n    \"contact_methods\": [],\n    \"description\": null,\n    \"email\": \"example@example.com\",\n    \"html_url\": \"https://example.pagerduty.com/users/AAA0999\",\n    \"id\": \"AAA0999\",\n    \"invitation_sent\": false,\n    \"job_title\": null,\n    \"name\": \"Example Examplerton\",\n    \"notification_rules\": [],\n    \"role\": \"user\",\n    \"self_\": \"https://api.pagerduty.com/users/AAA0999\",\n    \"summary\": \"C. Hobbes\",\n    \"teams\": [],\n    \"time_zone\": \"America/New_York\",\n    \"type\": \"user\"\n  },\n  ...\n]\n```\n\n* `bastion_users`:\n\n```json\n{\n  \"chobbes\": {\n    \"groups\": [\"\"],\n    \"root_ssh\": true\n  },\n  \"movedon\": {\n    \"ensure\": \"absent\"\n  },\n  ...\n}\n```\n\n* `alternate_usernames`:\n\n```json\n{\n  \"chobbes\": [\"calvin\", \"spacemanspiff\"]\n}\n```\n\n### Test Accuracy\n\nThere are two important things to note about `pytest-services` tests that may be different from your expectations.\n\nFirst, the focus is on \"actionable results\". This plays out as an attempt to reduce false\npositives by trying to filter out unused resources. An example of this can be seen by looking at\nany of the security group tests, where we are skipping any security groups that are not attached to a resource.\n\nSecond, there are some tests that make naive assumptions instead of trying to capture the complexities\nof the system. The current best example of this is all IAM tests that relate to \"admin\" users. How we\nare determining what an user or role is an admin is based simply off substring matching on the policies\nattached. This obviously has a high chance of false negatives.\n\n## Development\n\n### Goals\n\n1. replace one-off scripts for each check\n1. share checks with other organizations\n1. consolidate bugs in one place (i.e. one thing to update)\n1. in pytest use a known existing framework for writing checks\n1. be vendor agnostic e.g. support checks across cloud providers or in hybrid environments or competing services\n1. cache and share responses to reduce third party API usage (i.e. lots of tests check AWS security groups so fetch them once)\n1. provide a way to run a single test or subset of tests\n1. focus on actionable results (see [test accuracy](#test-accuracy) for more information)\n\n### Non-Goals\n\n1. Invent a new DSL for writing expectations (use pytest conventions)\n1. Verify how third party services or their client libraries work\n   (e.g. don't answer \"Does GET / on the CRUD1 API return 400 when\n   query param `q` is `$bad_value`?\")\n\n### Design\n\nCurrently this is a monolithic pytest package, but should eventually\n[be extracted into a pytest plugin](#3) and with [separate dependent\npytest plugins for each service](#4).\n\nAPI responses should fit on disk and in memory (i.e. don't use this\nfor log processing or checking binaries for malware), and be safe to\ncache for minutes, hours, or days (i.e. probably don't use this for\nmonitoring a streaming API) (NB: [bug for specifying data\nfreshness](#5)).\n\nAdditionally we want:\n\n* data fetching functions in a `resources.py`\n* data checking and test helpers in a `helpers.py`\n* prefix test files with `test_`\n* doctests for non test files (e.g. `helpers.py`, `resources.py`, `client.py`)\n  * tests that depend on external IO or the runtime environment (env vars, file system, HTTP) to use the prefix `meta_test_` (and probably `mock` or `pytest.monkeypatch`)\n    * JSON fixtures for anonymized cached http call in `example_cache/v/`\n* tests to have pytest markers for any services they depend on for data\n* HTTP clients should be read only and use read only credentials\n* running a test should not modify services\n\n#### File Layout\n\n```console\npytest-services\n...\n\u251c\u2500\u2500 example_cache\n\u2502\u00a0\u00a0 \u2514\u2500\u2500 v\n\u2502\u00a0\u00a0     \u251c\u2500\u2500 cache\n\u2502\u00a0\u00a0     \u2502\u00a0\u00a0 \u2514\u2500\u2500 lastfailed\n\u2502\u00a0\u00a0     \u251c\u2500\u2500 pytest_aws:example-account:us-east-1:ec2:describe_instances::.json\n\u2502\u00a0\u00a0     \u251c\u2500\u2500 pytest_aws:example-account:us-east-1:ec2:describe_security_groups::.json\n...\n\u251c\u2500\u2500 <third party service A>\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 client.py\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 meta_test_client.py\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 <subservice A (optional)>\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 helpers.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 resources.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ...\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 test_ec2_security_group_all_ports.py\n\u2502\u00a0\u00a0 \u251c\u2500\u2500 <subservice b (optional)>\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 resources.py\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 ...\n\u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500 test_s3_bucket_web_hosting_disabled.py\n\u2514\u2500\u2500 <third party service B>\n \u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n \u00a0\u00a0 \u251c\u2500\u2500 helpers.py\n \u00a0\u00a0 \u251c\u2500\u2500 resources.py\n \u00a0\u00a0 \u2514\u2500\u2500 test_user_has_escalation_policy.py\n```\n\n### Adding an example test\n\nLet's write a test to check that http://httpbin.org/ip returns an AWS IP:\n\n1. create a file `httpbin/test_httpbin_ip.py` with the contents:\n\n```python\nimport itertools\nimport ipaddress\nimport pytest\nimport json\nimport urllib.request\n\n\ndef get_httpbin_ips():\n    # IPs we always want to test\n    ips = [\n        '127.0.0.1',\n        '13.58.0.0',\n    ]\n\n    req = urllib.request.Request('http://httpbin.org/ip')\n\n    with urllib.request.urlopen(req) as response:\n        body = response.read().decode('utf-8')\n        ips.append(json.loads(body).get('origin', None))\n\n    return ips\n\n\ndef get_aws_ips():\n    req = urllib.request.Request('https://ip-ranges.amazonaws.com/ip-ranges.json')\n\n    with urllib.request.urlopen(req) as response:\n        body = response.read().decode('utf-8')\n        return json.loads(body)['prefixes']\n\n\n@pytest.mark.httpbin\n@pytest.mark.aws_ip_ranges\n@pytest.mark.parametrize(\n    ['ip', 'aws_ip_ranges'],\n    zip(get_httpbin_ips(), itertools.repeat(get_aws_ips())))\ndef test_httpbin_ip_in_aws(ip, aws_ip_ranges):\n    for aws_ip_range in aws_ip_ranges:\n        assert ipaddress.IPv4Address(ip) not in ipaddress.ip_network(aws_ip_range['ip_prefix']), \\\n          \"{0} is in AWS range {1[ip_prefix]} region {1[region]} service {1[service]}\".format(ip, aws_ip_range)\n```\n\nNotes:\n\n* we add two data fetching functions that return lists that we can zip into tuples for [the pytest parametrize decorator](https://docs.pytest.org/en/latest/parametrize.html#pytest-mark-parametrize-parametrizing-test-functions)\n* we add markers for the services we're fetching data from\n\n\n1. Running it we see that one of the IPs is an AWS IP:\n\n```console\npytest --ignore aws/\nplatform darwin -- Python 3.6.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0\nmetadata: {'Python': '3.6.2', 'Platform': 'Darwin-15.6.0-x86_64-i386-64bit', 'Packages': {'pytest': '3.3.2', 'py': '1.5.2', 'pluggy': '0.6.0'}, 'Plugins': {'metadata': '1.5.1', 'json': '0.4.0', 'html': '1.16.1'}}\nrootdir: /Users/gguthe/mozilla-services/pytest-services, inifile:\nplugins: metadata-1.5.1, json-0.4.0, html-1.16.1\ncollected 3 items\n\nhttpbin/test_httpbin_ip_in_aws.py .F.                                                                                               [100%]\n\n================================================================ FAILURES =================================================================\n____________________________________________ test_httpbin_ip_in_aws[13.58.0.0-aws_ip_ranges1] _____________________________________________\n\nip = '13.58.0.0'\naws_ip_ranges = [{'ip_prefix': '13.32.0.0/15', 'region': 'GLOBAL', 'service': 'AMAZON'}, {'ip_prefix': '13.35.0.0/16', 'region': 'GLOB...on': 'us-west-1', 'service': 'AMAZON'}, {'ip_prefix': '13.57.0.0/16', 'region': 'us-west-1', 'service': 'AMAZON'}, ...]\n\n    @pytest.mark.httpbin\n    @pytest.mark.aws_ip_ranges\n    @pytest.mark.parametrize(\n        ['ip', 'aws_ip_ranges'],\n        zip(get_httpbin_ips(), itertools.repeat(get_aws_ips())),\n        # ids=lambda ip: ip\n        )\n    def test_httpbin_ip_in_aws(ip, aws_ip_ranges):\n        for aws_ip_range in aws_ip_ranges:\n>           assert ipaddress.IPv4Address(ip) not in ipaddress.ip_network(aws_ip_range['ip_prefix']), \\\n              \"{0} is in AWS range {1[ip_prefix]} region {1[region]} service {1[service]}\".format(ip, aws_ip_range)\nE           AssertionError: 13.58.0.0 is in AWS range 13.58.0.0/15 region us-east-2 service AMAZON\nE           assert IPv4Address('13.58.0.0') not in IPv4Network('13.58.0.0/15')\nE            +  where IPv4Address('13.58.0.0') = <class 'ipaddress.IPv4Address'>('13.58.0.0')\nE            +    where <class 'ipaddress.IPv4Address'> = ipaddress.IPv4Address\nE            +  and   IPv4Network('13.58.0.0/15') = <function ip_network at 0x107cf66a8>('13.58.0.0/15')\nE            +    where <function ip_network at 0x107cf66a8> = ipaddress.ip_network\n\nhttpbin/test_httpbin_ip_in_aws.py:43: AssertionError\n=================================================== 1 failed, 2 passed in 15.69 seconds ===================================================\n```\n\nNote: marking tests as expected failures with `@pytest.mark.xfail` can hide data fetching errors\n\nTo improve this we could:\n\n1. Add parametrize ids so it's clearer which parametrize caused test failures\n1. Add directions about why it's an issue and how to fix it or what the associated risks are\n\nAs we add more tests we can:\n\n1. Move the JSON fetching functions to `<service name>/resources.py` files and import them into the test\n1. Move the fetching logic to a shared library `<service name>/client.py` and save to the pytest cache",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/mozilla-services/pytest-services",
        "keywords": "",
        "license": "MPL2",
        "maintainer": "",
        "maintainer_email": "",
        "name": "frost",
        "package_url": "https://pypi.org/project/frost/",
        "platform": "",
        "project_url": "https://pypi.org/project/frost/",
        "project_urls": {
            "Homepage": "https://github.com/mozilla-services/pytest-services"
        },
        "release_url": "https://pypi.org/project/frost/0.3.1/",
        "requires_dist": null,
        "requires_python": ">=3.6",
        "summary": "tests for checking that third party services the Firefox Operations Security or foxsec team uses are configured correctly",
        "version": "0.3.1"
    },
    "last_serial": 5805364,
    "releases": {
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a809666651e63c75610999383188e670",
                    "sha256": "d5f4a4539b6d484062f4efbaa7c6078b6ba61e8ddc9faec2514153c9baf705de"
                },
                "downloads": -1,
                "filename": "frost-0.3.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a809666651e63c75610999383188e670",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 65938,
                "upload_time": "2019-09-09T20:10:37",
                "url": "https://files.pythonhosted.org/packages/06/87/b1558befdc0a1937f9fb1ded8e8d6b6cdb26c0a11429e2b5a9548caa2a15/frost-0.3.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "af94fa31fbb8c1141fde0b47e5018a4a",
                    "sha256": "b9000141f380555c4f782cc6b432a6da1997bc526d5ee1d4d52df1738027c634"
                },
                "downloads": -1,
                "filename": "frost-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "af94fa31fbb8c1141fde0b47e5018a4a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 53249,
                "upload_time": "2019-09-09T20:10:26",
                "url": "https://files.pythonhosted.org/packages/af/6f/0b6c60d24c13fba434eff859895cb8c454d039b9aa0e38b716650db66738/frost-0.3.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "a809666651e63c75610999383188e670",
                "sha256": "d5f4a4539b6d484062f4efbaa7c6078b6ba61e8ddc9faec2514153c9baf705de"
            },
            "downloads": -1,
            "filename": "frost-0.3.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a809666651e63c75610999383188e670",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 65938,
            "upload_time": "2019-09-09T20:10:37",
            "url": "https://files.pythonhosted.org/packages/06/87/b1558befdc0a1937f9fb1ded8e8d6b6cdb26c0a11429e2b5a9548caa2a15/frost-0.3.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "af94fa31fbb8c1141fde0b47e5018a4a",
                "sha256": "b9000141f380555c4f782cc6b432a6da1997bc526d5ee1d4d52df1738027c634"
            },
            "downloads": -1,
            "filename": "frost-0.3.1.tar.gz",
            "has_sig": false,
            "md5_digest": "af94fa31fbb8c1141fde0b47e5018a4a",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 53249,
            "upload_time": "2019-09-09T20:10:26",
            "url": "https://files.pythonhosted.org/packages/af/6f/0b6c60d24c13fba434eff859895cb8c454d039b9aa0e38b716650db66738/frost-0.3.1.tar.gz"
        }
    ]
}