{
    "info": {
        "author": "Evgeny Medvedev",
        "author_email": "evge.medvedev@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# EOS ETL\n\n[![Join the chat at https://gitter.im/ethereum-eth](https://badges.gitter.im/ethereum-etl.svg)](https://gitter.im/ethereum-etl/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n[![Build Status](https://travis-ci.org/blockchain-etl/eos-etl.png)](https://travis-ci.org/blockchain-etl/eos-etl)\n[Join Telegram Group](https://t.me/joinchat/GsMpbA3mv1OJ6YMp3T5ORQ)\n\nInstall EOS ETL:\n\n```bash\npip install eos-etl\n```\n\nExport blocks, transactions and actions ([Schema](#schema), [Reference](#export_blocks)):\n\n```bash\n> eosetl export_blocks --start-block 1 --end-block 500000 \\\n--provider-uri http://api.main.alohaeos.com \\\n--blocks-output blocks.json --transactions-output transactions.json --actions-output actions.json\n```\n\nStream blockchain data continually to console:\n\n```bash\n> pip install eos-etl[streaming]\n> eosetl stream -p http://api.main.alohaeos.com --start-block 500000\n```\n\nFor the latest version, check out the repo and call \n```bash\n> pip install -e .[streaming] \n> python eosetl.py\n```\n\n## Table of Contents\n\n- [Schema](#schema)\n- [Exporting the Blockchain](#exporting-the-blockchain)\n  - [Running in Docker](#running-in-docker)\n  - [Command Reference](#command-reference)\n- [Public Datasets in BigQuery](#public-datasets-in-bigquery)\n\n## Schema\n\n- [blocks](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/raw/schemas/blocks.json)\n- [transactions](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/raw/schemas/transactions.json)\n- [actions](https://github.com/blockchain-etl/eos-etl-airflow/tree/master/dags/resources/stages/enrich/raw/actions.json)\n\n## Exporting the Blockchain\n\n1. Install python 3.6.0+ https://www.python.org/downloads/\n\n1. Install EOS node or get access to EOS node maintained by someone else (because running your own node is not so easy).\nSome docs:\n- [https://developers.eos.io/eosio-nodeos/docs/](https://developers.eos.io/eosio-nodeos/docs/)\n- [https://eosnode.tools/](https://eosnode.tools/)\n- [https://github.com/CryptoLions/EOS-MainNet](https://github.com/CryptoLions/EOS-MainNet)\n\n1. Make sure it downloaded the blocks that you need by executing in the terminal:\n```bash\ncurl --request POST \\\n  --url https://localhost:8080/v1/chain/get_info \\\n  --header 'accept: application/json'\n```\nYou can export blocks below `last_irreversible_block_num`, there is no need to wait until the full sync\n\n1. Install EOS ETL:\n\n    ```bash\n    > pip install eos-etl\n    ```\n\n1. Export blocks, transactions and actions:\n\n    ```bash\n    > eosetl export_all --start 1 --end 499999  \\\n    --provider-uri http://api.main.alohaeos.com\n    ```\n    \n    In case `eosetl` command is not available in PATH, use `python -m eosetl` instead.\n  \n    The result will be in the `output` subdirectory, partitioned in Hive style:\n\n    ```bash\n    output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv\n    output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv\n    ...\n    output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv\n    ...\n    output/actions/start_block=00000000/end_block=00000099/actions_00000000_00000099.csv\n    ...\n    ```\n    \n### Running in Docker\n\n1. Install Docker https://docs.docker.com/install/\n\n1. Build a docker image\n    ```bash\n    > docker build -t eos-etl:latest .\n    > docker image ls\n    ```\n\n1. Run a container out of the image\n    ```bash\n    > MSYS_NO_PATHCONV=1 docker run -v $HOME/output:/eos-etl/output eos-etl:latest \\\n        export_blocks --max-workers 50 --start-block 30000000 \\\n        --end-block 30000100 --provider-uri http://your_eos_node:node_port \\\n        --blocks-output ./output/blocks.csv --transactions-output ./output/transactions.csv \\\n        --actions-output ./output/actions.csv\n    ```\n    \n1. Run streaming to console or Pub/Sub\n    ```bash\n    > MSYS_NO_PATHCONV=1 docker build -t eos-etl:latest-streaming -f Dockerfile_with_streaming .\n    > echo \"Stream to console\"\n    > MSYS_NO_PATHCONV=1 docker run eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000\n    > echo \"Stream to Pub/Sub\"\n    > MSYS_NO_PATHCONV=1 docker run -v /path_to_credentials_file/:/eos-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/eos-etl/credentials_file.json eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000 --output projects/your-project/topics/crypto_eos\n    ```\n\n1. Refer to https://github.com/blockchain-etl/blockchain-etl-streaming for deploying the streaming app to \nGoogle Kubernetes Engine.\n\n### Command Reference\n\n- [export_blocks](#export_blocks)\n- [get_block_range_for_date](#get_block_range_for_date)\n- [export_all](#export_all)\n- [stream](#stream)\n\nAll the commands accept `-h` parameter for help, e.g.:\n\n```bash\n> python eosetl.py export_blocks --help\nUsage: eosetl.py export_blocks [OPTIONS]\n\n  Export blocks, transactions and actions.\n\nOptions:\n  -s, --start-block INTEGER   Start block\n  -e, --end-block INTEGER     End block  [required]\n  -p, --provider-uri TEXT     The URI of the remote EOS node\n  -w, --max-workers INTEGER   The maximum number of workers.\n  --blocks-output TEXT        The output file for blocks. If not provided\n                              blocks will not be exported. Use \"-\" for stdout\n  --transactions-output TEXT  The output file for transactions. If not\n                              provided transactions will not be exported. Use\n                              \"-\" for stdout\n  --actions-output TEXT       The output file for actions. If not provided\n                              transactions will not be exported. Use \"-\"\n                              for stdout\n  --help                      Show this message and exit.\n```\n\nFor the `--output` parameters the supported type is json. The format type is inferred from the output file name.\n\n#### export_blocks\n\n```bash\n> python eosetl.py export_blocks --start-block 1 --end-block 500000 \\\n  --provider-uri http://api.main.alohaeos.com \\\n  --blocks-output blocks.json --transactions-output transactions.json --actions-output actions.json\n```\n\nOmit `--blocks-output` or `--transactions-output` or `--actions-output` options if you want to export only transactions/blocks/actions.\n\nYou can tune `--max-workers` for performance.\n\n#### get_block_range_for_date\n\n```bash\n> python eosetl.py get_block_range_for_date --provider-uri http://api.main.alohaeos.com --date=2018-06-09\n```\n\n#### export_all\n\n```bash\n> python eosetl.py export_all --provider-uri http://api.main.alohaeos.com --start 2018-06-08 --end 2018-06-09\n```\n\nYou can tune `--export-batch-size`, `--max-workers` for performance.\n\n#### stream\n\n```bash\n> python eosetl.py stream --provider-uri http://api.main.alohaeos.com --start-block 500000\n```\n\n- This command outputs blocks and transactions to the console by default.\n- Use `--output` option to specify the Google Pub/Sub topic where to publish blockchain data, \ne.g. `projects/your-project/topics/eos_blockchain`.\n- The command saves its state to `last_synced_block.txt` file where the last synced block number is saved periodically.\n- Specify either `--start-block` or `--last-synced-block-file` option. `--last-synced-block-file` should point to the \nfile where the block number, from which to start streaming the blockchain data, is saved.\n- Use the `--lag` option to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to \nhandle chain reorganizations - they are less likely the further a block from the head.\n- You can tune `--period-seconds`, `--batch-size`, `--max-workers` for performance.\n \n\n### Running Tests\n\n```bash\n> pip install -e .[dev]\n> echo \"The below variables are optional\"\n> export EOSETL_PROVIDER_URI=http://api.main.alohaeos.com\n> pytest -vv\n```\n\n### Running Tox Tests\n\n```bash\n> pip install tox\n> tox\n```\n\n### Public Datasets in BigQuery\n\nTODO: https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/blockchain-etl/eos-etl",
        "keywords": "EOS",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "eos-etl",
        "package_url": "https://pypi.org/project/eos-etl/",
        "platform": "",
        "project_url": "https://pypi.org/project/eos-etl/",
        "project_urls": {
            "Bug Reports": "https://github.com/blockchain-etl/eos-etl/issues",
            "Chat": "https://gitter.im/ethereum-etl/Lobby",
            "Homepage": "https://github.com/blockchain-etl/eos-etl",
            "Source": "https://github.com/blockchain-etl/eos-etl"
        },
        "release_url": "https://pypi.org/project/eos-etl/1.0.0/",
        "requires_dist": null,
        "requires_python": ">=3.6.0,<3.8.0",
        "summary": "Tools for exporting EOS blockchain data to JSON",
        "version": "1.0.0"
    },
    "last_serial": 5649848,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d6b9e9a1c84b920115af5b7921575297",
                    "sha256": "09a2c1fdfac8d11d3461c5a5a1bfb7c6e5190482804147371e58d412134c5874"
                },
                "downloads": -1,
                "filename": "eos-etl-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d6b9e9a1c84b920115af5b7921575297",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0,<3.8.0",
                "size": 17450,
                "upload_time": "2019-08-08T12:28:17",
                "url": "https://files.pythonhosted.org/packages/f3/3e/e34a6bd5192351230251c0203075c5a990925f53fcf47456d431e62e369a/eos-etl-1.0.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "d6b9e9a1c84b920115af5b7921575297",
                "sha256": "09a2c1fdfac8d11d3461c5a5a1bfb7c6e5190482804147371e58d412134c5874"
            },
            "downloads": -1,
            "filename": "eos-etl-1.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "d6b9e9a1c84b920115af5b7921575297",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6.0,<3.8.0",
            "size": 17450,
            "upload_time": "2019-08-08T12:28:17",
            "url": "https://files.pythonhosted.org/packages/f3/3e/e34a6bd5192351230251c0203075c5a990925f53fcf47456d431e62e369a/eos-etl-1.0.0.tar.gz"
        }
    ]
}