{ "info": { "author": "David Davis", "author_email": "davisd50@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7" ], "description": "sparc.cache\n===========\n\nData caching components for the SPARC platform.\n\nSQL Cache Area\n----------------\nSQL implementation of ICacheArea, providing a mechanism to store items \nimplementing either ICachableItem (a single item) or ICachableSource\n(many items).\n\nThis implementation utilizes SQLAlchemy to provide abstraction from the \nspecific SQL technology (MySQL, Oracle, SQL Server, etc) to be used. Also,\nthis is implemented as a ZCA multi-adapter...allowing easy lookup given\nan instance of ISqlAlchemySession and ICachedItemMapper.\n\nAlthough we provide a *very* flexible implementation for caching information,\nthis flexibility comes at the cost of some added complexities. We use ZCA as a \nframework to help us manage the complexity.\n\nEven though, technically, this class can be used outside of ZCA...our examples\nwill use ZCA because it helps to simplify the explanation.\n\nCachable Item\n----------------\nOf course, as a first step, we need some information that is to be cached.\nWe need structured information that our SQL Cache Area object can understand.\nWe have the ICachableItem interface to help us here. So as a first step,\nwe'll create an implementation of this interface.\n\n >>> from zope.interface import implements\n >>> from sparc.cache import ICachableItem\n >>> class myItem(object):\n ... implements(ICachableItem)\n ... def __init__(self, attributes = None):\n ... self.key = 'ENTRY #'\n ... self.attributes = attributes\n ... def getId(self):\n ... return self.attributes[self.key]\n ... def validate(self):\n ... if not self.attributes.has_key(self.key):\n ... raise KeyError(\"expected item's attributes to have entry for key field: %s\", self.key)\n ... if not self.attributes[self.key]:\n ... raise ValueError(\"expected item's key attribute to have a non-empty value\")\n\nNotice...we did not need to explicitly add members in the class that represent\nindividually cachable information (i.e. column definitions). Instead we\nhave single myItem.attributes which will contain a Dictionary of the\ninformation that is to be cached in key, value pairs.\n\nAlso notice...we had to specify the key name of the attribute entry which will\nbe used to uniquely identify the item within a group of items.\n\nAlthough the above code shows how to create the implementation, mostly it is \nboiler plate so we have a readily available mixin class to help simplify \nthings for us.\n\n >>> del(myItem)\n >>> from sparc.cache.item import cachableItemMixin\n >>> class myItem(cachableItemMixin):\n ... def __init__(self, attributes=None):\n ... super(myItem, self).__init__('ENTRY #', attributes)\n\n\nFinally, we create a factory...whose responsibility is simply to generate new \ninstances of our implementation.\n\n >>> from zope.component.factory import IFactory, Factory\n >>> myCachableItemFactory = Factory(myItem, 'myCachableItemFactory')\n\nCached Item\n----------------\nInformation that has been cached may require some subtle translations from the\noriginal information source (ICachableItem). For example, if the cachable \nitem has a date string then it would be nice to store that information in\nthe DB as a datetime type...therefore allowing date searches. IF we just\nstored the information as a string-type...we couldn't utilze the DB's full\ncapabilities for type-specific searching.\n\nFor this, and other reasons, we need a new data structure (class) that will\nrepresent the information after it has been cached. For this task, we\nhave the ICachedItem interface. Again, we require an implementation of this\ninterface to do anything useful.\n\n >>> import sqlalchemy.ext.declarative\n >>> class myBaseMixin(object):\n ... @sqlalchemy.ext.declarative.declared_attr\n ... def __tablename__(cls):\n ... return cls.__name__.lower()\n ... __mapper_args__= {'always_refresh': True}\n ... def __repr__(self):\n ... return \"<\" + self.__class__.__name__ + \"(id: '%s'')>\" % (self.getId())\n >>> Base = sqlalchemy.ext.declarative.declarative_base()\n \n >>> from sparc.cache import ICachedItem\n >>> import sqlalchemy\n >>> from datetime import date, timedelta\n >>> class myCachedItem(myBaseMixin, Base):\n ... implements(ICachedItem)\n ... entry_number = sqlalchemy.Column(sqlalchemy.BigInteger(), primary_key=True)\n ... logged_date = sqlalchemy.Column(sqlalchemy.DateTime(),nullable=True)\n ... def getId(self):\n ... return self.entry_number\n ... def __eq__(self, instance):\n ... if not isinstance(instance, self.__class__):\n ... return False\n ... if self.entry_number != instance.entry_number:\n ... return False\n ... if self.logged_date - instance.logged_date != timedelta(0):\n ... return False\n ... return True\n ... def __ne__(self, instance):\n ... return not self.__eq__(instance)\n\nThis implementation is a bit more tricky than our myItem object from above.\nFirst thing to note is our dependency on sqlalchmeny. This is an \nimplementation-specific dependency...driven by our base requirement to allow\ninformation to be stored in a DB via the sqlalchemy abstraction layer. Other\nthan conformance to the ICachedItem interface requirements, this acts like a\nnormal SQL Alchemy persistent object class\n\nBut once again, the above implementation is not that fun to write out, so we \nhave a ready-to-use mixin class to help things out.\n\n >>> del(myCachedItem)\n >>> from sparc.cache.item import cachedItemMixin\n >>> Base = sqlalchemy.ext.declarative.declarative_base() # we need to reset this, cause myCachedItem is already defined above\n >>> class myCachedItem(cachedItemMixin, myBaseMixin, Base):\n ... _key = 'entry_number'\n ... entry_number = sqlalchemy.Column(sqlalchemy.BigInteger(), primary_key=True)\n ... logged_date = sqlalchemy.Column(sqlalchemy.DateTime(),nullable=True)\n\nFinally (like above), we created factory...whose responsibility is simply to \ngenerate new instances of our implementation.\n\n >>> from zope.component.factory import IFactory, Factory\n >>> myCachedItemFactory = Factory(myCachedItem, 'myCachedItemFactory')\n\nCached Item Mapper\n-------------------\nNow that we have two different classes, one representing information we want to\ncache and the other representing the information after it has been cached, we need \na way to relate the 2 different classes. For this task, we have the\nICachedItemMapper interface. This class provides a mechanism to populate\nan ICachedItem from information stored in a ICachableItem.\n\n >>> from zope.component import adapts\n >>> from sparc.cache.interfaces import ICachedItemMapper, ICachableSource\n >>> from sparc.cache.sources import normalizedDateTime, normalizedDateTimeResolver\n >>> class myItemCacheMapper(object):\n ... implements(ICachedItemMapper)\n ... adapts(ICachableSource)\n ... mapper = {\n ... 'entry_number' :'ENTRY #', \n ... 'logged_date' :normalizedDateTime('LOGGED DATE')\n ... }\n ... def __init__(self, CachableSource):\n ... self.cachableSource = CachableSource\n ... def key(self):\n ... for _key, _value in self.mapper.iteritems():\n ... if _value == self.cachableSource.key():\n ... return _key\n ... raise LookupError(\"expected to find matching cache key for given source key via map lookup\")\n ... def factory(self):\n ... return myCachedItem()\n ... def get(self, sourceItem):\n ... _cachedItem = self.factory()\n ... _cachedItem.entry_number = int(sourceItem.attributes['ENTRY #'])\n ... _cachedItem.logged_date = normalizedDateTimeResolver(self.logged_date).manage(sourceItem.attributes['LOGGED DATE'])\n ... return _cachedItem\n\nFor the purpose of illustration, the above class fully implements the \nICachedItemMapper interface. However, since it's not fun to create this\ntype of class, there's an available mixin class to make things a bit simpler\nto read/write\n\n >>> del(myItemCacheMapper)\n >>> from sparc.cache.sql import SqlObjectMapperMixin\n >>> class myItemCacheMapper(SqlObjectMapperMixin):\n ... mapper = {\n ... 'entry_number' :'ENTRY #', \n ... 'logged_date' :normalizedDateTime('LOGGED DATE')\n ... }\n\nMuch nicer to look at. Also note the normalizedDateTime() reference. This\nclass is an implementation of the IManagedCachedItemMapperAttributeKeyWrapper\ninterface. The interface is simple...it needs to supply a __call__() method\nthat returns the string key name.\n\nConfiguration\n----------------\nWe've now created classes that represent information in its pre and post\ncached state, and also a mapper for the two states. Things are about to get\ninteresting. Before we move forward, we will configure the ZCA registry\nwith both our components above and also components that are available in\nsparc (defined in the various *.zcml files in the framework).\n\n >>> from sparc.common import Configure\n >>> import sparc.cache\n >>> Configure([sparc.cache])\n\nWe now have populated the global ZCA registry with all available components\nin the sparc.cache package. It's a good start, but we also need to register\nthe ICachableItemMapper adapter implementation components above.\nRemember, registration allows these components to be looked up via ZCA calls.\n\n >>> from zope.component import getGlobalSiteManager\n >>> gsm = getGlobalSiteManager()\n >>> gsm.registerAdapter(myItemCacheMapper, (ICachableSource, IFactory), ICachedItemMapper)\n\nData Source\n----------------\nUp until now, we've really only dealt with data definitions. You might be \nasking...\"so where's the data?\". We have the ICachableSource interface to \ndeal with data sources. But first, we need to start with the actual data.\nWe'll use the test_csvdata.csv file located under the \nsparc.cache.sources.tests directory.\n\n >>> import os\n >>> csv_file = os.path.dirname(os.path.abspath(__file__)) + os.sep + 'sources' + os.sep + 'tests' + os.sep + 'test_csvdata.csv'\n\nWe can now create instances of sparc.cache.sources.csvdata.CSVSource via \nits factory. This object will be used to generate ICachableItem objects for\neach valid CSV row entry.\n\n >>> from zope.component import createObject\n >>> myCSVSource = createObject('cache.sources.CSVSourceFactory', csv_file, myCachableItemFactory)\n\nLet's run a few quick test to make sure the data source is sane\n\n >>> item = myCSVSource.first()\n >>> item.attributes['ENTRY #']\n '9098328463'\n >>> item.attributes['LOGGED DATE']\n '6/18/2014 16:28'\n >>> item.getId()\n '9098328463'\n >>> item.validate()\n\nDatabase Connection\n----------------\nTo work with a database, we need to connect to it per SQLAlchemy instructions.\nWe'll use a simple SQLite memory database for this example.\n \n >>> from sqlalchemy import create_engine\n >>> engine = create_engine('sqlite:///:memory:')\n >>> from sqlalchemy.orm import sessionmaker\n >>> Session = sessionmaker(bind=engine)\n >>> session = Session()\n\nWe need to provide the ISqlAlchemySession marker interface to the session \nobject to enable easy ZCA based component lookup for adapters.\n\n >>> from zope.interface import directlyProvides\n >>> from sparc.db import ISqlAlchemySession\n >>> directlyProvides(session, ISqlAlchemySession)\n\nLoad the Cache!!!\n----------------\nOK, it was a bit complicated to get here...but now we'll see the fruits of \nour labor. Our actual caching interface is defined by ICacheArea. We've\nimplemented this interface via an adapter located at\nsparc.cache.sql.SqlObjectCacheArea. This adapter is already registered, so\nwe can look it up.\n\n >>> from sparc.cache import ICacheArea\n >>> from zope.component import getMultiAdapter\n >>> myMapper = getMultiAdapter((myCSVSource, myCachedItemFactory), ICachedItemMapper) # get our mapper via our adapter implementation\n >>> mySqlObjectCacheArea = getMultiAdapter((session, myMapper), ICacheArea)\n\nOne more final activity...we need to initialize (i.e. create the DB tables) \nthe storage area.\n\n >>> mySqlObjectCacheArea.initialize(Base)\n\nwhew...that was a lot of work...now we can finally use it. let's start\nby getting one of our cachable items in hand.\n\n >>> item = myCSVSource.first()\n >>> item.getId()\n '9098328463'\n\nLet's make sure our mapper works ok.\n\n >>> myMapper.get(item).getId()\n 9098328463\n \nThis item hasn't been cached yet\n\n >>> mySqlObjectCacheArea.get(item)\n >>> mySqlObjectCacheArea.isDirty(item)\n True\n >>> cached = mySqlObjectCacheArea.cache(item)\n >>> cached.getId()\n 9098328463\n >>> mySqlObjectCacheArea.isDirty(item)\n False\n >>> item.attributes['LOGGED DATE'] = '6/18/2014 16:28' # same as orig\n >>> mySqlObjectCacheArea.isDirty(item)\n False\n >>> mySqlObjectCacheArea.cache(item)\n False\n >>> item.attributes['LOGGED DATE'] = '6/25/2014 16:28' # new date\n >>> mySqlObjectCacheArea.isDirty(item)\n True\n >>> myMapper.get(item).getId()\n 9098328463\n \nLet's test the transactional features\n\n >>> mySqlObjectCacheArea.rollback()\n >>> mySqlObjectCacheArea.isDirty(item)\n True\n >>> cached = mySqlObjectCacheArea.cache(item)\n >>> cached.getId()\n 9098328463\n >>> mySqlObjectCacheArea.isDirty(item)\n False\n >>> mySqlObjectCacheArea.commit()\n >>> mySqlObjectCacheArea.isDirty(item)\n False\n >>> item = myCSVSource.first() # this resets the date from above\n >>> mySqlObjectCacheArea.isDirty(item)\n True\n >>> cached = mySqlObjectCacheArea.cache(item)\n >>> cached.getId()\n 9098328463\n >>> mySqlObjectCacheArea.commit()\n\n Let's get the item directly from the DB and make sure it acts like a\n ICachedItem\n \n >>> mySqlObjectCacheArea.session.expunge(cached)\n >>> cached = mySqlObjectCacheArea.get(item)\n >>> cached.getId()\n 9098328463\n \nNow lets do a bulk import...when we import it will bring in 1 less\nthan the total CSV source (because the first entry already done and does not\nrequire an update)\n\n >>> mySqlObjectCacheArea.import_source(myCSVSource)\n 3\nsparc.cache\n==========\n\n0.0.3\n++++++++++++++++++\n\n* pypi packaging update\n\n0.0.2\n++++++++++++++++++\n\n* include zcml configuration files in packaging\n\n0.0.1\n++++++++++++++++++\n\n* initial release", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/davisd50/sparc.cache", "keywords": "zca", "license": "MIT", "maintainer": null, "maintainer_email": null, "name": "sparc.cache", "package_url": "https://pypi.org/project/sparc.cache/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/sparc.cache/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/davisd50/sparc.cache" }, "release_url": "https://pypi.org/project/sparc.cache/0.0.3/", "requires_dist": null, "requires_python": null, "summary": "Data caching components for the SPARC platform", "version": "0.0.3" }, "last_serial": 1792294, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "f3d4665f8fe73a541be0d4faba64b47b", "sha256": "8a23a7bfa612bcf32cf4356d77870b0e7267ba6b12f5eebb953cf8f4aafa1f1d" }, "downloads": -1, "filename": "sparc.cache-0.0.1.tar.gz", "has_sig": false, "md5_digest": "f3d4665f8fe73a541be0d4faba64b47b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17399, "upload_time": "2015-10-16T17:09:25", "url": "https://files.pythonhosted.org/packages/3f/5c/8f9ecd04d43c407c3571531164400297f96cfa20254fea46816a661cb0c5/sparc.cache-0.0.1.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "603cefa5e4f6be446d5f2a2481872619", "sha256": "a7403cfd6f7f1eff984428b64a9907a78bec085ff006fecf799ac3927c5a196b" }, "downloads": -1, "filename": "sparc.cache-0.0.3.tar.gz", "has_sig": false, "md5_digest": "603cefa5e4f6be446d5f2a2481872619", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20050, "upload_time": "2015-10-29T18:25:41", "url": "https://files.pythonhosted.org/packages/2a/7d/3ecdd4737426a498f51432de6719bc18a467c74e806f3b14ff5c9ef83f3a/sparc.cache-0.0.3.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "603cefa5e4f6be446d5f2a2481872619", "sha256": "a7403cfd6f7f1eff984428b64a9907a78bec085ff006fecf799ac3927c5a196b" }, "downloads": -1, "filename": "sparc.cache-0.0.3.tar.gz", "has_sig": false, "md5_digest": "603cefa5e4f6be446d5f2a2481872619", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20050, "upload_time": "2015-10-29T18:25:41", "url": "https://files.pythonhosted.org/packages/2a/7d/3ecdd4737426a498f51432de6719bc18a467c74e806f3b14ff5c9ef83f3a/sparc.cache-0.0.3.tar.gz" } ] }