Supporting Persistent Objects In Python Jeremy Hylton

(C) 2002 Zope Corp. What is persistence? Data lives longer than programs –Files, pipes, relational databases, etc. –But, discontinuity of representation Orthogonal persistence –Automatic management of program state –Independent of data type or longevity –Allow programmer to focus on application data model

(C) 2002 Zope Corp. Python approach Goals –Minimize changes to existing programs –Work with standard interpreter Zope Object Database (ZODB) –Support Zope application server –Originally targeted for web development Separate database and storage layers

(C) 2002 Zope Corp. Not on todays agenda Benefits of persistence Discussion of related work –Early work PS-algol, Napier88 –Much recent work on Java PJava, ObjectStore PSE

(C) 2002 Zope Corp. ZODB key ideas Persistence by reachability –Any reachable object is saved Transactions to control updates –Safe sharing among clients Implement with Persistent mixin –Provides hook for persistence –Compromise on transparency

(C) 2002 Zope Corp. Persistent base class C extension type that defines –Four _p_ attributes –Custom attribute accessors –Efficient C API Marks serialization boundaries Interacts with database –Load objects on demand –Register modifications with TM

(C) 2002 Zope Corp. Persistence by reachability Any object reachable from ZODB root is persistent

(C) 2002 Zope Corp. A simple example from Persistence import Persistent from Transaction import get_transaction from ZODB.FileStorage import DB class Counter(Persistent): _value = 0 def inc(self): self._value += 1 def main(): fs = DB(data.fs) conn = db.open(); root = conn.root() obj = root[myobj] = Counter() get_transaction().commit() obj.inc() get_transaction().commit()

(C) 2002 Zope Corp. Object serialization Standard pickle library –Serializes arbitrary object graph –Raises TypeError for sockets, files, &c. –Instance vars serialized via dictionary Hooks to define custom state –__getstate__() / __setstate__() –Persistent mixin ignores _v_ attributes

(C) 2002 Zope Corp. Pickling persistent objects Stores objects in separate records –Persistent objs pickled as oid + class –Works with cache to maintain identity Handling non-persistent objects –Copied into record of containing object –Sharing by persistent objs is problematic

(C) 2002 Zope Corp. Object identity / caching Cache maintains oid obj mapping –Guarantees only one copy of object –Unpickler loads all referenced objects Ghost objects –Only Persistent header initialized –No instance state loaded –State loaded on first object access LRU cache of recent objects

(C) 2002 Zope Corp. Attribute access handlers Persistent implements C wrappers –Override tp_getattro, tp_setattro slots –Mediate access to instance variables –Crucial Python feature

(C) 2002 Zope Corp. Example __getattribute__() hook class Example(object): _p_state = False def _p_activate(self): print "activate" self._p_state = True def __getattribute__(self, attr): print "intercept", attr if not attr.startswith("_p_") and not self._p_state: self._p_activate() return super(Example, self).__getattribute__(attr) >>> obj = Example(); obj.value = "test" >>> print obj.value intercept value intercept _p_state intercept _p_activate activate test

(C) 2002 Zope Corp. Transactions Supports multiple threads, processes –Independent database connections –Updates visible at transaction boundaries Optimistic concurrency control –When conflict occurs, abort and retry On error, abort to restore consistency –Reverts to last saved state

(C) 2002 Zope Corp. Concurrency and conflicts Invalidations sent at commit time –Clients process at transaction boundaries Conflicting transactions aborted –Write conflict at commit time –Read conflict on object access Application must retry on conflict –Can use generic wrapper –Can define conflict resolution method

(C) 2002 Zope Corp. Retrying conflicts Example wrapper for retries def transact(f, retries=3): def wrapper(*args, **kwargs): n = retries while n: try: try: return f(*args, **kwargs) finally: get_transaction().commit() except ConflictError: n -= 1 if n == 0: raise except: get_transaction().abort() raise return wrapper

(C) 2002 Zope Corp. Other features Undo support –Storage stores multiple revisions –Transactional undo reverts to earlier state BTrees: efficient persistent containers Storing code in database

(C) 2002 Zope Corp. Limitations Schema evolution –Must code manually in __setstate__() Database management –Manual pack() to remove revisions, do GC Sharing of non-persistent objects Integration with legacy code –Multiple inheritance helps –Factory classes

(C) 2002 Zope Corp. Getting the software –info central for ZODB ZODB 3.1 released Oct –Uses ExtensionClass ZODB4 alpha planned for Dec –Based on Python 2.2 type support –Fewer onions in the varnish