{ "info": { "author": "Alex Boonstra, Walter Doekes, OSSO B.V.", "author_email": "wjdoekes+planb@osso.nl", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Web Environment", "Framework :: Django", "Framework :: Django :: 1.11", "Intended Audience :: System Administrators", "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)", "Operating System :: POSIX :: Linux", "Programming Language :: Python", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Topic :: System :: Archiving :: Backup" ], "description": "|PlanB|\n=======\n\nPlanB backs up your remote SSH-accessible files using rsync to a local ZFS\nstorage. Manage many hosts and host groups. Automate daily, weekly, monthly and\nyearly backups with snapshots.\n\n\n------------\nHow it looks\n------------\n\nAt the moment, the interface is just a Django admin interface:\n\n.. image:: assets/example_hosts.png\n :alt: A list of hosts configured in PlanB with most recent backup status\n\nThe files are stored on ZFS storage, using snapshots to keep earlier versions\nof tiles. See this example shell transscript::\n\n # zfs list | grep mongo2\n rpool/BACKUP/experience-mongo2 9,34G 1,60T 855M /srv/backups/experience-mongo2\n\n # ls -l /srv/backups/experience-mongo2/data/srv/mongodb\n total 646610\n -rw------- 1 planb nogroup 67108864 jun 17 17:03 experience.0\n -rw------- 1 planb nogroup 134217728 jun 9 16:01 experience.1\n ...\n\nThose are the \"current\" files in the workspace. But you can go back in time::\n\n # zfs list -r -t all rpool/BACKUP/experience-mongo2 | head -n4\n NAME USED AVAIL REFER MOUNTPOINT\n rpool/BACKUP/experience-mongo2 9,34G 1,60T 855M /srv/backups/experience-mongo2\n rpool/BACKUP/experience-mongo2@daily-201706031147 0 - 809M -\n rpool/BACKUP/experience-mongo2@monthly-201706031147 0 - 809M -\n\n # cd /srv/backups/experience-mongo2/.zfs/\n # ls -1\n daily-201706031147\n daily-201706031211\n daily-201706040001\n daily-201706050002\n ...\n\n # ls daily-201706031147/data/srv/mongodb -l\n total 581434\n -rw------- 1 planb nogroup 67108864 jun 2 18:21 experience.0\n -rw------- 1 planb nogroup 134217728 mei 29 14:38 experience.1\n ...\n\n\n--------------------\nRequirements / setup\n--------------------\n\nPlanB can be installed as a standalone Django_ application, or it can be\nintegrated in another Django project.\n\nSee `requirements.txt`_ or `setup.py`_ for up-to-date dependencies/requirements.\n\nBasically, you'll need: ZFS storage, ssh and rsync, a webserver (nginx), python\nhosting (uwsgi), a database (mysql), a communication/cache bus (redis) and a\nfew python packages.\n\nFor more detailed steps, see `Setting it all up`_ below.\n\n.. _Django: https://www.djangoproject.com/\n.. _`requirements.txt`: ./requirements.txt\n.. _`setup.py`: ./setup.py\n\n\n----\nTODO\n----\n\n* Fix logrotate sample.\n* Add uwsgi-uid==djangoq-uid check?\n* Re-add some form of \"list-stale-mounts\" (!).\n # contrib/list-stale-mounts | mail -E -s \"[$HOSTNAME] Stale ZFS mounts?\"\n ^-- document this in FAQ below..\n* Re-add non-INFO output from planb_custom.daily...\n # run_backupinfo | grep -vFB1 INFO/ /var/log/osso-backup/billing.log |\n # mail -E -s \"[$HOSTNAME] Backup billing push\"\n* Alter HostGroup:\n - use fs-name and human-name\n - use asciifield for fs-name?\n* Alter HostConfig:\n - use fs-name and optionally human-name\n - use asciifield for fs-name?\n* Replace the exception mails for common errors (like failing rsync) to\n use mail_admins style mail.\n* After using mail_admins style mail, we can start introducing mail digests\n instead: daily summary of backup successes and failures.\n* Fix admin \"Planb\" name as \"PlanB\".\n* Split off the subparts of the HostConfig to separate configs:\n - include-config\n - transport-config\n - retention-config\n - host-status (use this as main enqueue-view?)\n* Use hostgroup+hostname in more places. Right now the friendly_name is\n too short. Also, use unique_together, so the friendlyname can be reused.\n* BUG: Items added to /exclude list are not deleted from destination if\n they have already been backed up once.\n* Replace the \"daily report\" hack with a signal-receiver.\n\n\n-------\nWARNING\n-------\n\nThe Django-Q task scheduler is highly configurable from the\n``/admin/``-view. With a little effort it will run user-supplied python\ncode directly. Any user with access to the schedulers will have\ntremendous powers\n\n**Recommendation**: don't give your users powers to edit the schedulers.\nUse the fine-grained permissions of the Django-admin systems to limit\nthem to Hosts and HostGroups only.\n\n*Perhaps we should disable web-access to it altogether.*\n\n\n-----------------\nSetting it all up\n-----------------\n\nIf you follow the HOWTO below, you'll set up PlanB as a standalone\nproject. Those familiar with Django_ will know how to integrate it into\ntheir own project.\n\nThe setup below assumes you'll be using the ``planb`` user. You're free\nto change that consistently of course.\n\n\nSetting up a ZFS pool\n~~~~~~~~~~~~~~~~~~~~~\n\nTODO: Document this briefly.\n\n\nSetting up the project\n~~~~~~~~~~~~~~~~~~~~~~\n\nSetting up a virtualenv (optional)::\n\n mkdir -p /srv/virtualenvs\n echo 'WORKON_HOME=/srv/virtualenvs' >>~/.bashrc\n apt-get install python3-virtualenv python3-pip virtualenvwrapper\n # you may need to log in/out once after this\n\n mkvirtualenv planb --python=$(which python3) --system-site-packages\n\n mkdir /etc/planb\n cd /etc/planb\n pwd >$VIRTUAL_ENV/.project\n\n workon planb\n\nInstalling PlanB using pip::\n\n apt-get install mysql-server redis-server\n pip3 install planb\n\nInstalling PlanB without pip::\n\n apt-get install mysql-server redis-server python3-mysqldb python3-redis \\\n python3-setproctitle\n pip install git+https://github.com/ossobv/planb.git@master\n\nSetting up a local ``planb`` user::\n\n adduser planb --disabled-password --home=/var/spool/planb \\\n --shell=/bin/bash --system\n\n sudo -H -u planb ssh-keygen -b 8192\n\n.. note:: *You may want to back that ssh key up somewhere.*\n\nSetting up the local environment::\n\n cat >/etc/planb/envvars </etc/sudoers.d/planb <>~remotebackup/.ssh/authorized_keys <&2 2>&1 1>&3 \\\n | mail -E -s 'ERROR: planb.discovery (zabbix)' root ) 2>&1\n\n\n----------------\nDoing daily jobs\n----------------\n\nA quick hack to get daily reports up and running is by placing something\nlike this in ``/etc/planb/planb_custom.py``::\n\n from planb.contrib.billing import BossoBillingPoster, daily_hostgroup_report\n\n def daily_billing_report():\n \"\"\"\n This function is added into: Home >> Task Queue >> Scheduled task\n As: \"Report to Billing\" \n \"\"\"\n daily_hostgroup_report(BossoBillingPoster('http://my.url.here/'))\n\n\n------\nF.A.Q.\n------\n\nCan I use the software and customize it to my own needs?\n It is licensed under the GNU GPL version 3.0 or higher. See the LICENSE\n file for the full text. That means: probably yes, but you may be required to\n share any changes you make. But you were going to do that anyway, right?\n\n\nThe ``uwsgi`` log complains about *\"No module named site\"*.\n If your uwsgi fails to start, and the log looks like this::\n\n Python version: 2.7.12 (default, Nov 19 2016, 06:48:10)\n Set PythonHome to /srv/virtualenvs/planb\n ImportError: No module named site\n\n Then your uWSGI is missing the Python 3 module. Go install\n ``uwsgi-plugin-python3``.\n\n\nThe ``mkvirtualenv`` said ``locale.Error: unsupported locale setting``.\n You need to install the right locales until ``perl -e setlocale`` is\n silent. How depends on your system and your config. See ``locale`` and\n e.g. ``locale-gen en_US.UTF-8``.\n\n\nRsync complains about ``Invalid or incomplete multibyte or wide character``.\n If rsync returns with code 23 and says this::\n\n rsync: recv_generator: failed to stat \"...\\#351es-BCS 27-09-11.csv\":\n Invalid or incomplete multibyte or wide character (84)\n\n Then you might be backing up old hosts with legacy Latin-1 encoding\n on the filesystem. Adding ``--iconv=utf8,latin1`` to the hostconfig\n flags should fix it.\n\n You may need rsync version 3 or higher for that.\n\n Right now we opt to *not* implement any of these workarounds:\n\n * Patch rsync to cope with ``EILSEQ`` (84) \"Illegal byte sequence\".\n * Cope with error code 23 and pretend that everything went fine.\n\n Instead, you should install a recent rsync and/or fix the filenames\n on your remote filesystem.\n\n\nRsync complains about ``failed to stat`` or ``mkdir failed``.\n If rsync returns these messages::\n\n rsync: recv_generator: failed to stat \"...\": Permission denied (13)\n rsync: recv_generator: mkdir \"...\" failed: Permission denied (13)\n\n Then you may be looking at parent directories with crooked\n permissions, like 077. Fix the permissions on the remote end.\n\n However, many of these problems have likely been fixed by the\n addition of the --chmod=Du+rwx rsync option.\n\n\nBackup success mail are sent, but failure mails are not.\n Check the ``DEBUG`` setting. At the moment, error-mails are sent\n through the logging subsystem and that is disabled when running in\n debug-mode.\n\n\n-------\nAuthors\n-------\n\nPlanB was started in 2013 as \"OSSO backup\" by Alex Boonstra at OSSO B.V. Since\nthen, it has been evolved into *PlanB*. When it was Open Sourced by Walter\nDoekes in 2017, the old commits were dropped to ensure that any private company\ninformation was not disclosed.\n\n\n.. |PlanB| image:: assets/planb_head.png\n :alt: GoCollect\n\n\n\n-------\nChanges\n-------\n\nv1.6.post1 - *2019-03-20*\n~~~~~~~~~~~~~~~~~~~~~~~~~\n\n**Web interface**\n\n- Show last snapshot size distribution, and first backup success date,\n in hostconfig edit-view.\n- Show real-used-size instead of apparent-used-size in snapshot data\n distribution. If you have ZFS compression enabled, you'll see a drop\n in the snapshot size summaries (not in the total disk usage).\n- Hide last-error message in hostconfig edit-view for hosts that are\n disabled.\n\n**Other**\n\n- Single Sign-On (Discourse style) can be enabled (using the optional\n kleides-dssoclient dependency). See KLEIDES_DSSO_ENDPOINT option.\n- Tweak permissions so you don't need is_superuser powers anymore.\n- Update BossoBillingPoster for posting backup data counts to (internal)\n Bosso system.\n\n\nv1.5 - *2018-06-13*\n~~~~~~~~~~~~~~~~~~~\n\n**Web interface**\n\n- Show \"time since last backup\" in listing, instead of just OK/FAIL.\n\n**Tasks**\n\n- Add locking to dutree scanner, so the filesystem isn't raped. Only do\n one dutree scan at a time.\n- Change scheduler: jobs are now scheduled ahead of time by deducting\n the expected duration.\n- Fix rsync issue when remote directory permissions are wrong\n (unreadable by user). In that case, the download (as root user) would\n succeed, but later changes would fail locally (planb user).\n\n**Other**\n\n- Change schema, removing unnecessary weekly/monthly booleans.\n- Fix ``total_size_mb`` in report, which was too large.\n- Improve ``breport`` command to output to stdout by default.\n\n\nv1.4 - *2018-04-06*\n~~~~~~~~~~~~~~~~~~~\n\n**Web interface**\n\n- Show job failures in hostconfig detail view.\n- Reduce clutter in hostconfig list view, using smaller items and less\n clutter.\n- Show average run time, instead of last run time.\n\n**Other**\n\n- Fix bug with sending of breport emails.\n- Use git version for pip-install if available; add makefile for quick\n commands.\n- Update qcluster argv so it's still considered busy while doing the\n dutree scan.\n\n\nv1.3 - *2018-03-19*\n~~~~~~~~~~~~~~~~~~~\n\n**Web interface**\n\n- Disallow deletion of non-empty host groups.\n\n**CLI**\n\n- Add ``breport`` command to send out backup reports. See the template\n in templates/planb/report_email_body.txt. Note that the report is\n still in alpha stage. NOTE: To get e-mail reports as well, you need\n to have ``rst2html`` installed.\n- Add ``--with-disabled`` to ``confexport`` command to get complete\n exports.\n- Fix that planb runserver can be used for development (through\n PYTHONPATH propagation).\n\n**Other**\n\n- Dependency updates to Django 2.0+.\n- Add backup history record keeping, for better logging and averages.\n\n\nv1.2 - *2017-09-18*\n~~~~~~~~~~~~~~~~~~~\n\n- Fix release, this time without pyc files and with wheel package.\n Run this for upload: python setup.py sdist bdist_wheel upload\n\n\nv1.1 - *2017-09-18*\n~~~~~~~~~~~~~~~~~~~\n\n**Settings**\n\n- Add ``PLANB_DEFAULT_INCLUDES``.\n- Rename ``ZFS_BIN``, ``SUDO_BIN`` and ``RSYNC_BIN`` to ``PLANB_``.\n- Fix allowing use of alternate ``DJANGO_SETTINGS_MODULE``.\n\n**Web interface**\n\n- Add hosts to hostgroup listing.\n- Allow ordering hosts by enabled/queued/running.\n\n**CLI**\n\n- Add \"stale mounts\" listing (planb slist).\n- Create \"hostconfig\" export in YAML or JSON format (planb confexport).\n\n**Queue**\n\n- Fix so long running jobs don't suffer from lost DB connections.\n\n**Other**\n\n- Misc refactoring/cleanup.\n\n\nv1.0 - *2017-07-11*\n~~~~~~~~~~~~~~~~~~~\n\n- Initial release.", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/ossobv/planb", "keywords": "", "license": "GPLv3+", "maintainer": "", "maintainer_email": "", "name": "planb", "package_url": "https://pypi.org/project/planb/", "platform": "linux", "project_url": "https://pypi.org/project/planb/", "project_urls": { "Homepage": "https://github.com/ossobv/planb" }, "release_url": "https://pypi.org/project/planb/1.6.post1/", "requires_dist": null, "requires_python": "", "summary": "PlanB automates remote SSH+rsync backups", "version": "1.6.post1" }, "last_serial": 4963493, "releases": { "1.0": [ { "comment_text": "", "digests": { "md5": "e19916e53a27e8e623feb14843e57f74", "sha256": "cbc4ad0f1b08493b5d42f826861b3e3042aa299091433f95bd7ef587377b68bb" }, "downloads": -1, "filename": "planb-1.0.tar.gz", "has_sig": false, "md5_digest": "e19916e53a27e8e623feb14843e57f74", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 46130, "upload_time": "2017-07-11T13:22:05", "url": "https://files.pythonhosted.org/packages/c9/5c/0584f5a52c908f710e169ea84443c63b2d68bd3a57eacf3dd0c6935a1bec/planb-1.0.tar.gz" } ], "1.1": [ { "comment_text": "", "digests": { "md5": "e13b19a7d1438413b307493906633649", "sha256": "c739f25d0819a60746e1239eb8da4035faf012946906f8be274d163e1e5d42fd" }, "downloads": -1, "filename": "planb-1.1.tar.gz", "has_sig": false, "md5_digest": "e13b19a7d1438413b307493906633649", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 78422, "upload_time": "2017-09-18T10:25:46", "url": "https://files.pythonhosted.org/packages/e7/d0/db7544875b26df6cb8d9fe8bccec79115b47a84a56780fdd54487dbcf043/planb-1.1.tar.gz" } ], "1.1.1": [ { "comment_text": "", "digests": { "md5": "3300339f34513679ed37f32e125186e1", "sha256": "4e1d8970a564ab87cbf98338a8c7f7587e1d32427f74baf8db825334722187ba" }, "downloads": -1, "filename": "planb-1.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "3300339f34513679ed37f32e125186e1", "packagetype": "bdist_wheel", "python_version": "3.5", "requires_python": null, "size": 70012, "upload_time": "2017-09-18T10:47:13", "url": "https://files.pythonhosted.org/packages/3c/94/44148deee71a1b1b2f6896f49104c0f293e1c71a4854c7e033015d2fd4a7/planb-1.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1771de5059a09bbf418648ce058629fa", "sha256": "2d66a1f5f8ab31d9603c2e1b41c02cd59aff15104b2a6e1945e0fef6f92174ca" }, "downloads": -1, "filename": "planb-1.1.1.tar.gz", "has_sig": false, "md5_digest": "1771de5059a09bbf418648ce058629fa", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47172, "upload_time": "2017-09-18T10:47:11", "url": "https://files.pythonhosted.org/packages/9a/c8/933959100102828795e8d46a660fb923f0dde1d3332225822744b9b7e82f/planb-1.1.1.tar.gz" } ], "1.2": [ { "comment_text": "", "digests": { "md5": "4d3c44e6400359c0242647a71008f909", "sha256": "6aa916786f0bcbc62d1d82e2e25d66a14207c6f5e83d856756be8b559a47cf44" }, "downloads": -1, "filename": "planb-1.2-py3-none-any.whl", "has_sig": false, "md5_digest": "4d3c44e6400359c0242647a71008f909", "packagetype": "bdist_wheel", "python_version": "3.5", "requires_python": null, "size": 70091, "upload_time": "2017-09-18T10:52:57", "url": "https://files.pythonhosted.org/packages/6b/6b/beb8cf7bbc906dbeef5b32fc7c82a9eab025264fd9933f1717ee37340ed5/planb-1.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "52237d36ef78b7ca69960a5d4475c01a", "sha256": "3cd17a083d66208fdacd83766e026900e4f962a5aaa54d92706e6e25627da757" }, "downloads": -1, "filename": "planb-1.2.tar.gz", "has_sig": false, "md5_digest": "52237d36ef78b7ca69960a5d4475c01a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47427, "upload_time": "2017-09-18T10:52:55", "url": "https://files.pythonhosted.org/packages/49/42/de97a4c927f819fdfcb9c7b4db769464faa0100ad86578cc3491410cda76/planb-1.2.tar.gz" } ], "1.3": [ { "comment_text": "", "digests": { "md5": "329b603ea8955caf3c386419fb930ecc", "sha256": "69250195e790483b8c012430509808857ddf3fbf5b36333f4cbbd0f884678a34" }, "downloads": -1, "filename": "planb-1.3-py3-none-any.whl", "has_sig": false, "md5_digest": "329b603ea8955caf3c386419fb930ecc", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 77040, "upload_time": "2018-03-19T10:14:25", "url": "https://files.pythonhosted.org/packages/78/f3/dcfdd5016e7d3671e2877c3307acfee8b94490f51e2ac67181fb6b6a1f71/planb-1.3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3fccefbc503038ff5d68b2ed60a17922", "sha256": "5c91d62327fca9850a2ffbc45f7026a5400d4e838dd39285f93c8eaa8d28ec41" }, "downloads": -1, "filename": "planb-1.3.tar.gz", "has_sig": false, "md5_digest": "3fccefbc503038ff5d68b2ed60a17922", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 57921, "upload_time": "2018-03-19T10:14:28", "url": "https://files.pythonhosted.org/packages/c4/88/0a2cc705922d254e0d21590d8a1a4d54df930b5e1c388320d819544db1b8/planb-1.3.tar.gz" } ], "1.4": [ { "comment_text": "", "digests": { "md5": "41d7f83259d778a9eb85f6866a02b007", "sha256": "cab2029ffd700ea6301c7559762843129a23bfbecb35572b498ad8399f9bef40" }, "downloads": -1, "filename": "planb-1.4-py3-none-any.whl", "has_sig": false, "md5_digest": "41d7f83259d778a9eb85f6866a02b007", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 81722, "upload_time": "2018-04-06T15:51:09", "url": "https://files.pythonhosted.org/packages/90/c7/94a8437ecba42555899c4ff77fcd3fbc4f618f6e9432f19077a7d0b2e22b/planb-1.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "5ef22672027389f583b83fc1028979a9", "sha256": "1538654af56175db43e70d0b56214e4c9996cc20b649172e49fa79d307310896" }, "downloads": -1, "filename": "planb-1.4.tar.gz", "has_sig": false, "md5_digest": "5ef22672027389f583b83fc1028979a9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 62035, "upload_time": "2018-04-06T15:51:10", "url": "https://files.pythonhosted.org/packages/4e/d2/9e3c5848aad57cc7420ae44e86ce3fabad6bbe4e865e8ff4b8b034ef13e0/planb-1.4.tar.gz" } ], "1.5": [ { "comment_text": "", "digests": { "md5": "afec56378874820d609cea7871691051", "sha256": "11a0fb633d8094314cd886c39c90ae4da7122ea2cb931d08e25835fb54d84ee7" }, "downloads": -1, "filename": "planb-1.5.tar.gz", "has_sig": false, "md5_digest": "afec56378874820d609cea7871691051", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 57593, "upload_time": "2018-06-13T09:26:35", "url": "https://files.pythonhosted.org/packages/bd/a0/1cab0b133d2724afa3057f224b3c3d2d24a2e9d4818b14a2783a3cec74cd/planb-1.5.tar.gz" } ], "1.6.post1": [ { "comment_text": "", "digests": { "md5": "be25d4d899f9add9c8b29ddcbc2d674f", "sha256": "6e423954a4c20f4b2ce56b396dc818fada73936166c08ba1bf615763263696ca" }, "downloads": -1, "filename": "planb-1.6.post1.tar.gz", "has_sig": false, "md5_digest": "be25d4d899f9add9c8b29ddcbc2d674f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 66721, "upload_time": "2019-03-20T12:51:08", "url": "https://files.pythonhosted.org/packages/4a/be/d1349a3c74c2ead231c313a865e1c51c85def1dc1cb939eb2a108ccb836a/planb-1.6.post1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "be25d4d899f9add9c8b29ddcbc2d674f", "sha256": "6e423954a4c20f4b2ce56b396dc818fada73936166c08ba1bf615763263696ca" }, "downloads": -1, "filename": "planb-1.6.post1.tar.gz", "has_sig": false, "md5_digest": "be25d4d899f9add9c8b29ddcbc2d674f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 66721, "upload_time": "2019-03-20T12:51:08", "url": "https://files.pythonhosted.org/packages/4a/be/d1349a3c74c2ead231c313a865e1c51c85def1dc1cb939eb2a108ccb836a/planb-1.6.post1.tar.gz" } ] }