{
    "info": {
        "author": "Geographica",
        "author_email": "hello@geographica.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# Glutemulo\n\nA HA geo socio demo data ingestor\n\n## Usage\n\nRead de [examples files](examples).\n\nWe use environ vars. See [Environ vars file example](.env.example) for complete list,\nand examples.\n\n### Using producer to upload data to kafka\n\nSee python examples bellow. You must produce a dict with column_mame:value\n\n### Using the ingestor consumer\nUse `gluto` docker and fill enviroment vars.\n\nSelect the backend using `GLUTEMULO_BACKEND` and specific vars for it (database, host, etc).\nYou can select 2 backends: `postgres` or `carto`\nSee [Environ vars file example](.env.example) for complete list.\n\nThen set:\n\n1. `GLUTEMULO_INGESTOR_DATASET`  \nTable to upload data\n2. `GLUTEMULO_INGESTOR_DATASET_COLUMNS`  \nComma separted list of column names\n\nNow, create the table on backend or set `GLUTEMULO_INGESTOR_DATASET_DDL` and `GLUTEMULO_INGESTOR_DATASET_AUTOCREATE=False`\n\nThen configure ingestor for kafka.\nFirst read the [python-kafka doc](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html)\nand then use the following vars:\n\n1. `GLUTEMULO_INGESTOR_TOPIC`  \nTopic to use\n2. `GLUTEMULO_INGESTOR_BOOTSTRAP_SERVERS`  \nList of servers to connect\n3. `GLUTEMULO_INGESTOR_GROUP_ID`  \nGroup id.\n4. `GLUTEMULO_INGESTOR_AUTO_OFFSET_RESET`  \nlatest or earliest.\n5. `GLUTEMULO_INGESTOR_MAX_POLL_RECORDS`  \nThe maximum number of records returned in a batch of messages\n6. `GLUTEMULO_INGESTOR_FETCH_MIN_BYTES`  \nMinimum amount of data the server should return for a fetch request, otherwise wait up to fetch_max_wait_ms for more data to accumulate. Default: 1\n\nFor the docker, we include a [example docker-compose file](docker-compose.yml).\nRemember you can scale with same group_id\n\n```bash\ndocker-compose scale gluto=3\n```\n\n## Run flask demo\n\n```bash\n$ FLASK_ENV=development flask run\n * Environment: development\n * Debug mode: on\n * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)\n * Restarting with stat\n * Debugger is active!\n * Debugger PIN: 194-409-049\n```\n\n## Test\n\n```bash\n$ http -j POST localhost:5000/v1/ uno=1 dos=2`\nHTTP/1.0 201 CREATED\nContent-Length: 13\nContent-Type: text/html; charset=utf-8\nDate: Thu, 02 May 2019 14:56:07 GMT\nServer: Werkzeug/0.15.2 Python/3.7.2\n\nDATA Received\n```\n\n## Producer / Consumer\n\n### Kafka + json\n\nAsync producer:\n\n```python\nfrom glutemulo.kafka.producer import JsonKafka\nproductor = JsonKafka(bootstrap_servers=\"localhost:9092\")\nfuture = productor.produce('simple-topic', dict(dos='BB'))\n```\n\nConsumer in batches:\n```python\nfrom glutemulo.kafka.consumer import JsonKafka\nconsumer = JsonKafka('simple-topic', bootstrap_servers=\"localhost:9092\")\nfor msg in consumer.consume():\n    for msg in messages:\n        print(msg)\n```\n\n### Kafka + Avro\n\nsync producer:\n\n```python\nSCHEMA = {\n    \"type\": \"record\",\n    \"name\": \"simpledata\",\n    \"doc\": \"This is a sample Avro schema to get you started.\",\n    \"fields\": [\n        {\"name\": \"name\", \"type\": \"string\"},\n        {\"name\": \"number1\", \"type\": \"int\"},\n    ],\n}\nSCHEMA_ID = 1\n```\n\n```python\nfrom glutemulo.kafka.producer import AvroKafka as Producer\nproductor = Producer(SCHEMA, SCHEMA_ID,bootstrap_servers=\"localhost:9092\")\nfuture = productor.produce('simple-topic-avro', dict(name='Un nombre', number1=10))\n```\n\nConsumer:\n```python\nfrom glutemulo.kafka.consumer import AvroKafka as Consumer\nconsumer = Consumer('simple-topic-avro', SCHEMA, SCHEMA_ID, bootstrap_servers=\"localhost:9092\")\nfor messages in consumer.consume():\n    for msg in messages:\n        print(msg)\n```\n\n## For testing\n\nYou can setup a Kafka Consumer using the kafka-console-consumer script that comes with Kafka.\n\n```bash\n$ bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.240:9092 --topic pylog --from-beginning\n\nthis is an awsome log\n```\n\n### Testing With KafkaCat\n\nYou ca use an application called KafkaCat.\n\nAfter the application is installed we will run it in consumer mode (which is the default).\n\n```bash\nkafkacat -b 192.168.240.41:9092 -t one-test\n```\n\nThis should not show anything yet because we haven't sent anything to our topic yet...\n\nTo send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.\n\n```bash\n$ cat README | kafkacat -b 192.168.240.41 -t one-test\n```\nYou should see the output in the first window which has KafkaCat still running in consumer mode.\n\n\n## Links\n\n- [Zoonavigator](http://localhost:8004). Use 'zoo1' as connection string\n- [schema-registry-ui](http://localhost:8001)\n- [Rebrow](http://localhost:5001)\n- [kafka topics ui](http://localhost:8000)\n- [kafka rest proxy](http://localhost:8082)\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "MIT",
        "maintainer": "Geographica",
        "maintainer_email": "hello@geographica.com",
        "name": "geolibs-glutemulo",
        "package_url": "https://pypi.org/project/geolibs-glutemulo/",
        "platform": "",
        "project_url": "https://pypi.org/project/geolibs-glutemulo/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/geolibs-glutemulo/0.1.2/",
        "requires_dist": [
            "kafka-python (>=1.4,<2.0)",
            "environs (>=4.1,<5.0)",
            "fastavro (>=0.21.23,<0.22.0)",
            "carto (>=1.4,<2.0); extra == \"carto\"",
            "psycopg2-binary (>=2.8,<3.0); extra == \"postgres\"",
            "redis (>=3.2,<4.0); extra == \"redis\""
        ],
        "requires_python": ">=3.6,<4.0",
        "summary": "GeoLibs data ingestor",
        "version": "0.1.2"
    },
    "last_serial": 5729812,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ae67af190b88ac94a37cfc87a05c271b",
                    "sha256": "d21717bee5300791a0bd414b86a671485813f6f4896737445875b59dadcb5120"
                },
                "downloads": -1,
                "filename": "geolibs_glutemulo-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ae67af190b88ac94a37cfc87a05c271b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 13365,
                "upload_time": "2019-05-23T14:00:11",
                "url": "https://files.pythonhosted.org/packages/15/a7/d2b9642bfa52b02138bb66f88c89c9036f38c7315f52c2cc1e4f3628551a/geolibs_glutemulo-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2185ff69f2a483d6d71d28638386cffa",
                    "sha256": "5230221faedbbeffb1f949af15ec2ebac76736842a45133923812c7780e15ed3"
                },
                "downloads": -1,
                "filename": "geolibs-glutemulo-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2185ff69f2a483d6d71d28638386cffa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 11669,
                "upload_time": "2019-05-23T14:00:09",
                "url": "https://files.pythonhosted.org/packages/1a/5f/20afe441bdd9d9d5e0d88badd627744f332a7f39844164657a6a7a7d2916/geolibs-glutemulo-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "35077e8aa5b4ea501edb3a3d4f577949",
                    "sha256": "3f55dd7dc7c9efc041b2f4c5a1aa0557a55aa34cf6334e415cff87ea6a033007"
                },
                "downloads": -1,
                "filename": "geolibs_glutemulo-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "35077e8aa5b4ea501edb3a3d4f577949",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 13450,
                "upload_time": "2019-05-24T08:31:26",
                "url": "https://files.pythonhosted.org/packages/b4/3d/4bdedca535279d8c0c820675d2485ab92076d9cf6eb06cf0525810f421bf/geolibs_glutemulo-0.1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "53cae36c67b52567c309a304f7bd1de2",
                    "sha256": "c0e4cfcb2e15dc9607dedd56d06df7fecd42ac1731744a436ed830758b3dcc32"
                },
                "downloads": -1,
                "filename": "geolibs-glutemulo-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "53cae36c67b52567c309a304f7bd1de2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 11721,
                "upload_time": "2019-05-24T08:31:25",
                "url": "https://files.pythonhosted.org/packages/52/ee/a01dedc94a32d933bfc39317bd134765c65c792fcd0938685d9b455b0ba4/geolibs-glutemulo-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d585be84442ac79f38124af3c8ad1417",
                    "sha256": "7ea345257f9e869f71baddc9d6591d7869f0156a484598282be4a0b7ecdda24c"
                },
                "downloads": -1,
                "filename": "geolibs_glutemulo-0.1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d585be84442ac79f38124af3c8ad1417",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 14920,
                "upload_time": "2019-08-26T09:28:51",
                "url": "https://files.pythonhosted.org/packages/5a/05/92d5c4bfb1db18e9f96aa97e8cbc4c2845ed86ba9742fdfc046a9dfc2efb/geolibs_glutemulo-0.1.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e593d64161333806ab263e83ac41e064",
                    "sha256": "7e5e20104f1d7907cc00d0df1d68af9ccbf91e066ca0b3d8f8d5f5774f27689a"
                },
                "downloads": -1,
                "filename": "geolibs-glutemulo-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "e593d64161333806ab263e83ac41e064",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 12797,
                "upload_time": "2019-08-26T09:28:49",
                "url": "https://files.pythonhosted.org/packages/a4/54/1bec341b5c16633f06cea8344377c2958bfb09c93bdcb2fa4870a07b2a86/geolibs-glutemulo-0.1.2.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "d585be84442ac79f38124af3c8ad1417",
                "sha256": "7ea345257f9e869f71baddc9d6591d7869f0156a484598282be4a0b7ecdda24c"
            },
            "downloads": -1,
            "filename": "geolibs_glutemulo-0.1.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "d585be84442ac79f38124af3c8ad1417",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6,<4.0",
            "size": 14920,
            "upload_time": "2019-08-26T09:28:51",
            "url": "https://files.pythonhosted.org/packages/5a/05/92d5c4bfb1db18e9f96aa97e8cbc4c2845ed86ba9742fdfc046a9dfc2efb/geolibs_glutemulo-0.1.2-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "e593d64161333806ab263e83ac41e064",
                "sha256": "7e5e20104f1d7907cc00d0df1d68af9ccbf91e066ca0b3d8f8d5f5774f27689a"
            },
            "downloads": -1,
            "filename": "geolibs-glutemulo-0.1.2.tar.gz",
            "has_sig": false,
            "md5_digest": "e593d64161333806ab263e83ac41e064",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6,<4.0",
            "size": 12797,
            "upload_time": "2019-08-26T09:28:49",
            "url": "https://files.pythonhosted.org/packages/a4/54/1bec341b5c16633f06cea8344377c2958bfb09c93bdcb2fa4870a07b2a86/geolibs-glutemulo-0.1.2.tar.gz"
        }
    ]
}