{ "info": { "author": "", "author_email": "", "bugtrack_url": null, "classifiers": [], "description": "# Attendedsysupgrade Server for OpenWrt (GSoC 2017)\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/d0a6edfd64ef41b6bb44a01ba9b8d7d7)](https://app.codacy.com/app/aparcar/attendedsysupgrade-server?utm_source=github.com&utm_medium=referral&utm_content=aparcar/attendedsysupgrade-server&utm_campaign=Badge_Grade_Dashboard)\n[![Build Status](https://travis-ci.com/aparcar/attendedsysupgrade-server.svg?branch=master)](https://travis-ci.com/aparcar/attendedsysupgrade-server)\n[![PyPI version](https://badge.fury.io/py/asu.svg)](https://badge.fury.io/py/asu)\n[![Docker Pulls](https://img.shields.io/docker/pulls/aparcar/asu-base.svg)](https://cloud.docker.com/repository/docker/aparcar/asu-base)\n\nThis project intends to simplify the sysupgrade process of devices running OpenWrt or\ndistributions based on the former like LibreMesh. The provided tools here offer an easy\nway to reflash the router with a new version or package upgrades, without the need of\n`opkg` installed.\n\nAdditionally it offers an API (covered below) to request custom images with any\nselection of packages pre-installed, allowing to create firmware images without the need\nof setting up a build environment, even from mobile devices.\n\n## Clients\n\n### [`luci-app-attendedsysupgrade`](https://github.com/openwrt/luci/tree/master/applications/luci-app-attendedsysupgrade)\n\nAdd a view to the Luci system tab called \"Attended Sysupgrade\". Offers a button to search for updates and if found, to flash the image created by the update server.\n\n![luci-app-attendedsysupgrade-screenshot](https://user-images.githubusercontent.com/16000931/53075790-57b74480-34ee-11e9-9955-68e9c00dc3bb.png)\n\n### [`auc`](https://github.com/openwrt/packages/tree/master/utils/auc)\n\nAdd CLI to perform sysupgrades. Makes use of `ucert` to verify images are from a trusted source.\n\n### [Chef Online Builder](https://github.com/libremesh/chef)\n\n- \n\n![Chef](https://user-images.githubusercontent.com/16000931/54073419-21681c00-4287-11e9-8592-3648431f1b41.png)\n\n## Server\n\nThe server listens to update and image requests and images are automatically generated\nif the requests was valid. This is done by automatically setting up OpenWrt\nImageBuilders and cache images in a database. This allows to quickly respond to\nrequests without rebuilding exiting images again.\n\n### Active server\n\n- [chef.libremesh,org](https://chef.libremesh.org)\n- [as-test.stephen304.com](https://as-test.stephen304.com) **unstable dev server**\n\nYou can set this server in `/etc/config/attendedsysupgrade` after installation of a\nclient.\n\n## Run your own server\n\nIt's fairly easy to run your own _asu_ server! You can test it locally via Docker,\nVagrant or Ansible. The following steps except you are familiar with either Docker,\nVagrant or Ansible.\n\n### via Docker\n\nMake sure to have `docker` and `docker-compose` installed. Simply execute the server via\nthe following command:\n\n docker-compose up\n\nThis will start a postgres container preseeded with the required database schema.\nAfterwards a server is started which performs an initial download of available versions\nand target/subtarget combinations. Once this is done the server itself is started via\n`gunicorn3`.\n\nA worker container waits for the server to come up (on port 8000) and will start\nbuilders, garbage collectors and an updater.\n\nThe folders `worker` and `updater` are created, caching downloaded ImageBuilders. You\ncan change this behaviour in the `docker-compose.yml` file.\n\n### via Ansible\n\nCopy the configuration file from `./asu/utils/config.yml.default` to\n`./ansible/host_vars/.yml`. Add the Ansible variables `ansible_host` and\n`ansible_user` to the top of the config file.\n\nChange all settings as you like, the config file is automatically copied to the host\nfolder `/config.yml`.\n\n### via Vagrant\n\nMake sure your vagrant environment is setup and supports the used Debian 9 image\n(virtualbox/libvirt). Also [Ansible](https://ansible.com) is requred to setup the\nservice. To start vagrant simply run the following command:\n\n vagrant up\n\nAnsible automatically starts to setup the postgres database, server and worker. Once\ninstalled two systemd services are running, called `asu-server` and `asu-worker`. Check\ntheir well beeing via `journalct -fu asu-*`.\n\nAnsible takes the configuration file from `./asu/utils/config.yml.default` or a specific\none, if exists, from `./ansible/host_vars/.yml`.\n\n## Development\n\nTo hack on the server, please install it manually. The following steps give an (may\nincomplete) overview on the required steps. It's focused on Debian based system, feel\nfree to add documentation for other systems.\n\n### Required packages\n\nThe server requires the following packages\n\n apt install python3-pip odbc-postgresql unixodbc-dev gunicorn3 git \\\n bash wget postgresql\n\nTo run the worker addiditonal packages are required, based on the [official\nwiki](https://openwrt.org/docs/guide-developer/quickstart-build-images)\n\n apt install subversion g++ zlib1g-dev build-essential git python rsync \\\n man-db libncurses5-dev gawk gettext unzip file libssl-dev wget zip\n\n### Install the server package\n\nRun `pip3` to install the package\n\n pip3 install -e .\n\nThis allows `gunicorn3` and `flask` to find the package.\n\n### Setting up PostgreSQL\n\nAs `asu` uses [ODBC](https://en.wikipedia.org/wiki/Open_Database_Connectivity) you have\nto add the servers database to your `.odbc.ini` or `/etc/odbc.ini`. See an example\nconfiguration [here](odbc.ini). Once added setup the PostgreSQL user account. From a\nroot shell login as postgres, create the `asu` database and change the password!\n\n su postgres\n createdb asu\n psql\n create role asu with password 'changeme' nosuperuser nocreatedb nocreaterole noinherit login noreplication nobypassrls;\n grant all privileges on database asu to asu;\n\n\nNow let flask initiate the database and load available targets.\n\n export FLASK_APP=asu\n flask initdb\n flask loaddb\n\nThe server and worker(s) are now ready to run!\n\n### Starting the server\n\nEither start the server in single thread mode via `flask` or via `gunicorn3`\n\n # runs on localhost:5000\n FLASK_APP=asu\n flask run\n\n # runs on localhost:8000\n gunicorn3 asu:app\n\n### Starting the worker\n\nSimply run the following command to run the worker, it will start multiple threads for\nupdating, cleaning and building firmware images:\n\n FLASK_APP=asu\n flask run-worker\n\n## API\n\n### Upgrade check `/api/upgrade-check`\n\nSends information about the device to the server to see if a new distribution version or\npackage upgrades are available. An _upgrade check_ could look like this:\n\n| key | value | information |\n| ----------- | -------------------------------------------------------------- | --------------------------- |\n| `distro` | `OpenWrt` | installed distribution |\n| `version` | `17.01.0` | installed version |\n| `target` | `ar71xx/generic` | installed target |\n| `installed` | `{ \"libuci-lua\": \"2017-04-12-c4df32b3-1\", \"cgi-io\": \"3\", ...}` | all user installed packages |\n\nMost information can be retrieved via `ubus call system board`. Missing information can\nbe gathered via the `rpcd-mod-rpcsys` package. `packages` contains all user installed\npackages plus version. Packages installed as an dependence are excluded as they've been\nautomatically and dependencies may change between versions.\n\nIt's also possible to check for a new version without sending packages by removing\n`installed` from the request.\n\n### Response `status 200`\n\nThe server validates the request. Below is a possible response for a new version:\n\n| key | value | information |\n| ---------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------ |\n| `version` | `18.06.2` | newest version |\n| `upgrades` | `{ \"luci-lib-jsonc\": [ \"git-17.230.25723-2163284-1\", \"git-17.228.56579-209deb5-1\" ], ... }` | Package updates `[new_version, current_version]` |\n| `packages` | `[ \"libuci-lua\", \"cgi-io\", ... ]` | All packages for the new image |\n\nSee [other status codes](#response-status-codes)\n\nAn version upgrade does not ignore package upgrades for the following reason.\nBetween versions it possible that package names may change, packages are dropped\nor merged. The _response_ contains all packages included changed ones.\n\nThe _upgrade check response_ should be shown to the user in a readable way.\n\n### Upgrade request `/api/upgrade-request`\n\nOnce the user decides to perform the sysupgrade a new request is send to the\nserver called _upgrade request_.\n\n#### POST\n\n| key | value | information |\n| ---------- | --------------------------------- | ---------------------------------------- |\n| `distro` | `OpenWrt` | installed distribution |\n| `version` | `18.06.2` | installed version |\n| `target` | `ar71xx/genreic` | installed target |\n| `board` | `tl-wdr4300-v1` | `board_name` of `ubus call system board` |\n| `packages` | `[ \"libuci-lua\", \"cgi-io\", ... ]` | All packages for the new image |\n\nThe _upgrade request_ is nearly the same as the _upgrade check_ before, except only\ncontaining package names without version and adding `board` and possibly `model`. While\nthe server builds the requested image the clients keeps polling the server sending a\n`request_hash` via `GET` to the server.\n\n#### GET\n\nIf the `request_hash` was retrieved the client should switch to `GET` requests with the\nhash to save the server from validating the request again.\n\n`api/upgrade-request/`\n\n### Response `status 200`\n\n| key | value | information |\n| -------------- | -------------------------------------------------------------- | --------------------------- |\n| `files` | `/json/openwrt/18.06.2/ar71xx/generic/ap121f/812744035616fc9/` | path where files are stored |\n| `sysupgrade` | `openwrt-18.06.2-8127440...ric-ap121f-squashfs-sysupgrade.bin` | name of sysupgrade file |\n| `log` | `/download/openwrt/18.06...16fc9/buildlog-347d575ed2ca2f1.txt` | path to build log |\n| `image_hash` | `347d575ed2ca2f1` | hash of the image |\n| `request_hash` | `f6560b451837` | hash of the request |\n\nSee [other status codes](#response-status-codes)\n\n### Build request `/api/build-request`\n\nIt's also possible to request to build an image. The request is nearly the same as for\n`upgrade-request`. The response only contains a link to the created `files` or\n`upgrade-request` parameters if available.\n\nAn additional parameter is the `defaults` parameter which allows to set the content of\n`/etc/uci-defaults/99-server-defaults` within the image. This allows to set custom\noptions for the resulting image. To distinguish between custom images the name will\ncontain a hash of the requested `defaults` value and is stored in a different place,\nonly visible if the full hash (32bit) is known.\n\nThis is a special case for clients that do not necessary require a sysupgrade compatible\nimage. An example is the [LibreMesh Chef](https://chef.libremesh.org) firmware builder.\n\n### Response status codes\n\nThe client should check the status code:\n\n| status | meaning | information |\n| ------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |\n| 200 | build finish / upgrade available | see parameters of `upgrade-check`, `upgrade-request` or `build-request` |\n| 202 | building, queued, imagebuilder setup | building right now, in build queue, imagebuilder not ready. Details are in header `X-Imagebuilder-Status` and `X-Build-Queue-Position` |\n| 204 | no updates | device is up to date. Contains `request_hash` |\n| 400 | bad request | see `error` parameter |\n| 409 | manifest fail | selection of requested packages caused a conflict |\n| 413 | imagesize fail | produced image too big for device |\n| 420 | defaults size fail | requested defaults exceeds maximum size (10kB) |\n| 422 | unknown package | unknown package in request |\n| 500 | build failed | see `log` for build log |\n| 501 | no sysupgrade | image build successful but no sysupgrade image created |\n| 502 | proxy backend down | nginx runs but python part is down, likely maintenance |\n| 503 | server overload | please wait ~5 minutes |\n\n### Request data\n\nIt's also possible to receive information about build images or package versions,\navailable devices and more. All responses are in `JSON` format.\n\n- `/api/image/` Get information about an image. This contains various\n information stored about the image.\n\n- `/api/manifest/` Get packages and versions of a manifest. The\n manifest contains all installed packages of an image. The `manifest_hash` can be\n received by the api call `/api/image`.\n\n- `/api/distros` Get all supported distros with latest version and a short description\n if available.\n\n- `/api/versions[?distro=]` Get all supported versions with short\n description (of a singele distribution if given).\n\n- `/api/models?distro=&version=&model_search=` Get all supported\n devices of distro/version that contain the `model_search` string\n\n- `/api/packages_image?distro=&version=&target=&subtarget=&profile=` Get all default\n packages installed on an image\n\n### Request stats\n\n| request | answer |\n| -------------------------------- | ----------------------------------- |\n| `/api/v1/stats/popular_packages` | Get list of most installed packages |\n| `/api/v1/stats/popular_targets` | Get list of most created targets |\n| `/api/v1/stats/images` | Return image build information |\n| `/api/v1/stats/packages` | Return number of known packages |\n\n## Donations\n\nThis project cooperates with [LibreMesh][0], please consider a small donation at [open\ncollective][1], directly supporting this project as well!\n\n[0]: https://libremesh.org\n\n[1]: https://opencollective.com/libremesh\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/aparcar/attendedsysupgrade-server", "keywords": "", "license": "", "maintainer": "Paul Spooren", "maintainer_email": "mail@aparcar.org", "name": "asu", "package_url": "https://pypi.org/project/asu/", "platform": "", "project_url": "https://pypi.org/project/asu/", "project_urls": { "Homepage": "https://github.com/aparcar/attendedsysupgrade-server" }, "release_url": "https://pypi.org/project/asu/0.2.4/", "requires_dist": [ "flask", "pyodbc", "pyyaml" ], "requires_python": "", "summary": "Create sysupgrade images for OpenWrt on demand", "version": "0.2.4" }, "last_serial": 4926074, "releases": { "0.2.0": [ { "comment_text": "", "digests": { "md5": "19813147cf17f79a965257694724e2d7", "sha256": "a8f9a43a98fe5135361c81d8d20052be961d34d12bdf8ac478a06a870e4fc999" }, "downloads": -1, "filename": "asu-0.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "19813147cf17f79a965257694724e2d7", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 36893, "upload_time": "2019-03-07T19:13:48", "url": "https://files.pythonhosted.org/packages/1a/83/f052b920a2e3abfc07aa6ad45d19cae961d488dc9e6de1c4dae88920a2cb/asu-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bbb75ffb7d61df3bd1e68b7271495c63", "sha256": "e5936cb44b735e84c41e8c6a0858f79497d6cc5be3d703f359b96f496d9e8c55" }, "downloads": -1, "filename": "asu-0.2.0.tar.gz", "has_sig": false, "md5_digest": "bbb75ffb7d61df3bd1e68b7271495c63", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 35352, "upload_time": "2019-03-07T19:13:51", "url": "https://files.pythonhosted.org/packages/7f/f3/b31f8e97d5a25015516a032695216d8466c955afd2356c93aac2883ed766/asu-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "d8b64c5291692b03d428d453b457c29c", "sha256": "f88f539e846f79c99b4d587bf8e79a91d79191533ab2b2dbf6e60060d289ddcc" }, "downloads": -1, "filename": "asu-0.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d8b64c5291692b03d428d453b457c29c", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 37507, "upload_time": "2019-03-07T23:25:45", "url": "https://files.pythonhosted.org/packages/88/a0/4abd4673f998bb1adc7838d722bb7f2a9a7316d732fe04a893f57a1aca45/asu-0.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c6156a200990a2c51fa61fe70cfa431a", "sha256": "5c787bb983c1189b01151aee939acd9f56376d0bb9e6e51cb038ccd103518007" }, "downloads": -1, "filename": "asu-0.2.1.tar.gz", "has_sig": false, "md5_digest": "c6156a200990a2c51fa61fe70cfa431a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36011, "upload_time": "2019-03-07T23:25:47", "url": "https://files.pythonhosted.org/packages/05/2a/860081557600642ea4f2c09caeb324d1b28beeafa18e6f9a3f08176994bb/asu-0.2.1.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "63a3d2ce715007f2ba9f43b86a3f8bad", "sha256": "acbf0e59af0760c009c3b47d618791919e938c5d2c9e0d7d10de4584b4d84634" }, "downloads": -1, "filename": "asu-0.2.2-py3-none-any.whl", "has_sig": false, "md5_digest": "63a3d2ce715007f2ba9f43b86a3f8bad", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 37627, "upload_time": "2019-03-08T01:40:25", "url": "https://files.pythonhosted.org/packages/ef/e8/8e0c406db4647b79edf980b52aaffaa78fd57c0b5045ab2f8cdbc6b9b68d/asu-0.2.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "864cf5db8a13e4a765d8af64e2300281", "sha256": "bfbe4e7e56eb22a3be2b5798bcb4ed0aed48caaf94fe595372c4b3c795b6dc51" }, "downloads": -1, "filename": "asu-0.2.2.tar.gz", "has_sig": false, "md5_digest": "864cf5db8a13e4a765d8af64e2300281", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36141, "upload_time": "2019-03-08T01:40:26", "url": "https://files.pythonhosted.org/packages/17/58/4990d293977a394d6a25e40bc820c8afca8cdc8acd21812009456737e733/asu-0.2.2.tar.gz" } ], "0.2.3": [ { "comment_text": "", "digests": { "md5": "9c6c82c5391df3d85f86aedf99dbedce", "sha256": "2bca211f49aa8d3507f2f47458502e8ee371d955d12c289d5fe7322936393d7c" }, "downloads": -1, "filename": "asu-0.2.3-py3-none-any.whl", "has_sig": false, "md5_digest": "9c6c82c5391df3d85f86aedf99dbedce", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 37736, "upload_time": "2019-03-08T02:35:05", "url": "https://files.pythonhosted.org/packages/59/ef/8230941d305b4197028a1324dc47f2a0591374c5df6a025b7678e0f42226/asu-0.2.3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "64e3559e37fa9b81760d729ae6bbae4d", "sha256": "e60027cd531cc5b9b20d3321acc06fdf0cdd894004919800575b8235343ba8ef" }, "downloads": -1, "filename": "asu-0.2.3.tar.gz", "has_sig": false, "md5_digest": "64e3559e37fa9b81760d729ae6bbae4d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36251, "upload_time": "2019-03-08T02:35:06", "url": "https://files.pythonhosted.org/packages/6d/1f/90c86cb1bfd3fc5cb231076b36a811d6da45e9d9fb613b5ed71821e044d2/asu-0.2.3.tar.gz" } ], "0.2.4": [ { "comment_text": "", "digests": { "md5": "2e76af970ffa1f1591346301f4fba05d", "sha256": "94dea0f35e6962c140905f1f8f6ed4831a0ab2e67238491c6fd1214f9a6e8b3e" }, "downloads": -1, "filename": "asu-0.2.4-py3-none-any.whl", "has_sig": false, "md5_digest": "2e76af970ffa1f1591346301f4fba05d", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 37623, "upload_time": "2019-03-11T17:07:54", "url": "https://files.pythonhosted.org/packages/bf/8f/264adbe5d077673ba4a37918aa6b307ee737ff54ef3d99abb6b34fac98d3/asu-0.2.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "0aae6fcff29ad9e0e13b5feb17fe0af0", "sha256": "6c4fa3b52bf826ae403f964fb0a8551dded24104c3290be25b1288b88dda8b2b" }, "downloads": -1, "filename": "asu-0.2.4.tar.gz", "has_sig": false, "md5_digest": "0aae6fcff29ad9e0e13b5feb17fe0af0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36281, "upload_time": "2019-03-11T17:07:56", "url": "https://files.pythonhosted.org/packages/4a/83/a7cddad442fbd8f58e0828912f335f895ecd90de144f3800db28deb9da10/asu-0.2.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "2e76af970ffa1f1591346301f4fba05d", "sha256": "94dea0f35e6962c140905f1f8f6ed4831a0ab2e67238491c6fd1214f9a6e8b3e" }, "downloads": -1, "filename": "asu-0.2.4-py3-none-any.whl", "has_sig": false, "md5_digest": "2e76af970ffa1f1591346301f4fba05d", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 37623, "upload_time": "2019-03-11T17:07:54", "url": "https://files.pythonhosted.org/packages/bf/8f/264adbe5d077673ba4a37918aa6b307ee737ff54ef3d99abb6b34fac98d3/asu-0.2.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "0aae6fcff29ad9e0e13b5feb17fe0af0", "sha256": "6c4fa3b52bf826ae403f964fb0a8551dded24104c3290be25b1288b88dda8b2b" }, "downloads": -1, "filename": "asu-0.2.4.tar.gz", "has_sig": false, "md5_digest": "0aae6fcff29ad9e0e13b5feb17fe0af0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36281, "upload_time": "2019-03-11T17:07:56", "url": "https://files.pythonhosted.org/packages/4a/83/a7cddad442fbd8f58e0828912f335f895ecd90de144f3800db28deb9da10/asu-0.2.4.tar.gz" } ] }