[pybsddb] File-like interface for get/put?

David Wolever wolever at cs.toronto.edu
Mon Aug 11 17:53:00 CEST 2008


Hey,
I'm considering using bsddb as a storage backend for a storage  
service.  One of the requirements, however, is that the data being  
stored should never be in memory all at once because it might be  
"very big".  The obvious solution to this is to return a file-like  
object which can be read() in small hunks.

So, back to bsddb.  I've noticed that get/put accept dlen/doff  
(http://www.jcea.es/programacion/pybsddb_doc/db.html#put) arguments  
which can do this sort of thing.

Now, my questions:
First, is there a pre-existing file-like interface to reading/writing  
data to/from the DB (for example, "output = db.streaming_put(key);  
shutil.copyfileobj(input, output)")

Second, if one does not exist, will code like the following be Really  
Bad?

     def put(self, object):
         offset = 0
         while True:
             data = object.read(16*1024)
             if not data: break
             len = len(data)
             self.db.put(self.key, data,
                                 dlen=len, doff=offset)
             offset += len

For example, having to call db.put() a bunch of times suggests that  
it needs to find the correct insertion point for each call... Is it  
some how cached?  Is there some way that the insertion point (in the  
case of a b-tree) could be cached?  Is there anything else I should  
worry about?

Thanks in advance,
David




More information about the pybsddb mailing list