{
"info": {
"author": "Ciprian Tarta",
"author_email": "me@cipriantarta.ro",
"bugtrack_url": null,
"classifiers": [
"Environment :: Web Environment",
"Framework :: Django",
"Framework :: Django :: 1.7",
"Framework :: Django :: 1.8",
"Framework :: Django :: 1.9",
"Intended Audience :: Developers",
"License :: OSI Approved :: BSD License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
"Topic :: Software Development :: Libraries :: Python Modules"
],
"description": ".. role:: python(code)\n :language: python\n\nDjango Autoshard\n================\n\nA `Django `_ library that makes sharding easy, using the `Consistent Hashing `_ algorithm.\n\n.. image:: https://badge.fury.io/py/django-autoshard.svg\n :target: https://badge.fury.io/py/django-autoshard\n\n.. image:: https://travis-ci.org/cipriantarta/django-autoshard.svg?branch=master\n :alt: Build Status\n :target: https://travis-ci.org/cipriantarta/django-autoshard\n\n.. image:: https://coveralls.io/repos/github/cipriantarta/django-autoshard/badge.svg?branch=master\n :alt: Coverage Status\n :target: https://coveralls.io/github/cipriantarta/django-autoshard?branch=master\n\n\nNotes\n=====\nWriting a general sharding library for all business models is practically impossible, but there are common particularities that apply to most of them.\n\nIf you are not familiar with sharding, you should first document yourself on what sharding means and how you could apply it to your own business model. You can start `here `_.\n\nThis library was written with the following business model in mind.\n\nAn application usually has a user account that must be different from other user accounts either by using a unique email address, or username, or other information that can be considered unique across the application.\n\nEach user account will be sharded using that unique constraint and all data that belongs to that user will live on the same shard that the user account does.\n\nYou can perhaps customize this business model to fit your own requirements, but the idea of this library was to add sharding to a Django app, with a minimum amount of effort.\n\nThe sharding algorithm used by this library is inspired by `Instagram's sharding solution `_, but instead of an auto-increment ID and stored procedures, this library uses a RNG for the local ID part. This probably means that when the stars align, you might get a collision if an insert is made in the same millisecond and the RNG gives you the same number.\n\n\nInstallation\n============\n\n:python:`pip install django-autoshard`\n\n1. Add :python:`django_autoshard` to your INSTALLED_APPS settings like this:\n\n.. code-block:: python\n\n INSTALLED_APPS = (\n ...\n 'django_autoshard',\n )\n\n2. Create a new user model that extends the :python:`ShardedModel` and extend all related models from :python:`ShardRelatedModel`\n\n.. code-block:: python\n\n from django.contrib.auth.models import AbstractUser\n from django_autoshard.models import ShardedModel, ShardRelatedModel\n from django_autoshard.managers import ShardedManager\n\n\n class User(ShardedModel, AbstractUser):\n SHARD_KEY = 'email'\n objects = ShardedManager()\n\n\n class Book(ShardRelatedModel):\n user = models.ForeignKey(User)\n\n3. Use this model as the default auth model in your :python:`settings.py` file.\n :python:`AUTH_USER_MODEL='.User'`\n\n4. Make sure you have set up the :python:`settings.DATABASES` correctly(any Django supported database back-end will work) and add the following to your settings file. The range() will create the logical shards, so in the example below, range(10) will create 10 logical shards on the NODE \"192.168.0.100\" using the default database name, user and password:\n .. code-block:: python\n\n DJANGO_AUTOSHARD = {\n \"NODES\": [\n {\n \"HOST\": \"192.168.0.100\", # DB MACHINE 1\n \"RANGE\": range(10)\n },\n {\n \"HOST\": \"192.168.0.101\", # DB MACHINE 2\n \"RANGE\": range(10, 20)\n }\n # and so on ...\n ]\n }\n\n5. Run :python:`python manage.py create_shards`\n\n6. Run :python:`python manage.py migrate`\n\n7. Run :python:`python manage.py migrate_shards`\n\n8. Run :python:`python manage.py drop_constraints`\n\nCommands\n========\nManagement Commands that come with this library:\n\n 1. create_shards:\n - this command will create all the logical shards(new databases) on all of the configured shard nodes in :python:`settings.DJANGO_AUTOSHARD`\n\n 2. migrate_shards:\n - this command will migrate all your application's models to all of the logical shards created with \"create_shards\"\n\n 3. drop_constraints:\n - this command will drop all the foreign key constraints from the \"default\" database that have a relation with your \"ShardedModel\"\n\nSettings\n========\nThe settings are isolated into a single dict in your settings.py file like so:\n\n.. code-block:: python\n\n DJANGO_AUTOSHARD = {\n 'EPOCH': '2016-01-01',\n 'MAX_SHARDS': 1000,\n 'NODES': {\n ...\n }\n }\n\n:python:`EPOCH` - defaults to :python:`'2016-01-01'`. Must be in :python:`'%Y-%m-%d'` format.\n\n:python:`MAX_SHARDS` - defaults to :python:`8192`. This should NEVER be changed after initial setup, unless you want to rehash all your sharded data.\n\nCaveats\n=======\n- you will no longer be able to use database joins between your sharded models, but you can still use joins on models that are related to your sharded model(models on the same shard as the user)\n- models that come from third party apps that are related to your sharded model and you don't have any control over, will need to have their foreign key dropped(use :python:`drop_constraints` command).\n- instead of using :python:`Book.objects.create(...)` you will have to use :python:`book = Book(...)` and then :python:`book.save()`. This is because of how Django model managers work.\n- if your business model requires to do searches on shard related models, or other fields of the sharded model besides the configured :python:`SHARD_KEY`, for example text based search, you will need to use tools like Elasticsearch, where you will store your text info and the shard id of tha object that this text info belongs to, in a single Elasticsearch document.\n- :python:`ShardedModel` does not support :python:`count()` and :python:`all()`\n- :python:`django.contrib.admin` will not work with sharded models\n\nTODO\n====\n\n- Add replicas support\n- Create shard migration script\n- Create a benchmarking script\n- Add more tests\n\nChange Log\n==========\n\n1.2.0 [2016-06-30]\n------------------\n- Changed the way shards are built, using `settings.DJANGO_AUTOSHARD['NODES']`. See INSTALLATION\n- added support for Django 1.7\n- removed support for python 3.3, because it only worked with Django 1.8\n- django_autoshard User model is only created when testing now.\n\n1.1.2 [2016-06-27]\n------------------\n- allow `all()` and `count()` if `using` is passed.\n\n1.1 [2016-05-21]\n----------------\n- fixes management commands for python2 and python3 < 3.5\n- raise :python:`NotImplementedError` when trying to use :python:`count()` or :python:`all()` on a :python:`ShardedModel`\n- Update documentation\n\n1.0(alpha) [2016-04-02]\n-----------------------\n- Initial release.",
"description_content_type": null,
"docs_url": null,
"download_url": "UNKNOWN",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/cipriantarta/django-autoshard",
"keywords": "django shard sharding",
"license": "BSD",
"maintainer": null,
"maintainer_email": null,
"name": "django-autoshard",
"package_url": "https://pypi.org/project/django-autoshard/",
"platform": "UNKNOWN",
"project_url": "https://pypi.org/project/django-autoshard/",
"project_urls": {
"Download": "UNKNOWN",
"Homepage": "https://github.com/cipriantarta/django-autoshard"
},
"release_url": "https://pypi.org/project/django-autoshard/1.2.1/",
"requires_dist": null,
"requires_python": null,
"summary": "A Django library that makes sharding easy.",
"version": "1.2.1"
},
"last_serial": 2218167,
"releases": {
"1.0": [
{
"comment_text": "",
"digests": {
"md5": "5e48503ae729614743acc04e1b0115fb",
"sha256": "6aceb517ef4a141caebf87d0452d56b670b03c073630913da9d5ed94aaf847ee"
},
"downloads": -1,
"filename": "django-autoshard-1.0.tar.gz",
"has_sig": false,
"md5_digest": "5e48503ae729614743acc04e1b0115fb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10406,
"upload_time": "2016-04-02T07:00:27",
"url": "https://files.pythonhosted.org/packages/7c/83/d8df3a6d4598fe336f9241d94359408212e16e792209a06601ee114d090e/django-autoshard-1.0.tar.gz"
}
],
"1.1": [
{
"comment_text": "",
"digests": {
"md5": "0d02557887d20634e8e95bf6e6506d37",
"sha256": "a8d37d597601ea782266094e43f9600e72f15b4ed91ee0a2152b5a8f47a9aeb9"
},
"downloads": -1,
"filename": "django-autoshard-1.1.tar.gz",
"has_sig": false,
"md5_digest": "0d02557887d20634e8e95bf6e6506d37",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10325,
"upload_time": "2016-05-21T08:07:56",
"url": "https://files.pythonhosted.org/packages/c6/56/0c0764d4142edcc65fdf4cb417beb5ee1b121b5da91fd741033975f9c0cd/django-autoshard-1.1.tar.gz"
}
],
"1.1.2": [
{
"comment_text": "",
"digests": {
"md5": "456405d5cd57611c749d046d82ebc99c",
"sha256": "8502a372ff683cdfceb70c82ab712e1f46a9cbd907cd983b1c27b69c6f63b0e8"
},
"downloads": -1,
"filename": "django-autoshard-1.1.2.tar.gz",
"has_sig": false,
"md5_digest": "456405d5cd57611c749d046d82ebc99c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10499,
"upload_time": "2016-06-27T07:33:18",
"url": "https://files.pythonhosted.org/packages/0f/60/22bf6c01aefdf47f056f59fc65ef3616152b6ca2cbb135f4130ce8d5e549/django-autoshard-1.1.2.tar.gz"
}
],
"1.2.0": [
{
"comment_text": "",
"digests": {
"md5": "0a4e947405b63b9734bcc517c094ad9b",
"sha256": "59fd25c105de129a1e45543f37c838061f8e5d75d549c50954df5dc8d91cef42"
},
"downloads": -1,
"filename": "django-autoshard-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "0a4e947405b63b9734bcc517c094ad9b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10546,
"upload_time": "2016-06-30T17:07:28",
"url": "https://files.pythonhosted.org/packages/c9/40/bfd44a6b95d96074a5e08d2a50e524a2c9e2a2084144353d647686360474/django-autoshard-1.2.0.tar.gz"
}
],
"1.2.1": [
{
"comment_text": "",
"digests": {
"md5": "7fd542d56620e74a892c4968dcd4038f",
"sha256": "fa664a43c6627dc2e299f729ef59ec924e3c6a15ebd31477fbae69b619975e3a"
},
"downloads": -1,
"filename": "django-autoshard-1.2.1.tar.gz",
"has_sig": false,
"md5_digest": "7fd542d56620e74a892c4968dcd4038f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10684,
"upload_time": "2016-07-13T05:08:54",
"url": "https://files.pythonhosted.org/packages/48/13/c95ab61e18b30de0268ffceef4a9c531b2e8346ddddf1ad8a9d8c56245db/django-autoshard-1.2.1.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "7fd542d56620e74a892c4968dcd4038f",
"sha256": "fa664a43c6627dc2e299f729ef59ec924e3c6a15ebd31477fbae69b619975e3a"
},
"downloads": -1,
"filename": "django-autoshard-1.2.1.tar.gz",
"has_sig": false,
"md5_digest": "7fd542d56620e74a892c4968dcd4038f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 10684,
"upload_time": "2016-07-13T05:08:54",
"url": "https://files.pythonhosted.org/packages/48/13/c95ab61e18b30de0268ffceef4a9c531b2e8346ddddf1ad8a9d8c56245db/django-autoshard-1.2.1.tar.gz"
}
]
}