[pybsddb] File-like interface for get/put?
David Wolever
wolever at cs.toronto.edu
Mon Aug 11 17:53:00 CEST 2008
Hey,
I'm considering using bsddb as a storage backend for a storage
service. One of the requirements, however, is that the data being
stored should never be in memory all at once because it might be
"very big". The obvious solution to this is to return a file-like
object which can be read() in small hunks.
So, back to bsddb. I've noticed that get/put accept dlen/doff
(http://www.jcea.es/programacion/pybsddb_doc/db.html#put) arguments
which can do this sort of thing.
Now, my questions:
First, is there a pre-existing file-like interface to reading/writing
data to/from the DB (for example, "output = db.streaming_put(key);
shutil.copyfileobj(input, output)")
Second, if one does not exist, will code like the following be Really
Bad?
def put(self, object):
offset = 0
while True:
data = object.read(16*1024)
if not data: break
len = len(data)
self.db.put(self.key, data,
dlen=len, doff=offset)
offset += len
For example, having to call db.put() a bunch of times suggests that
it needs to find the correct insertion point for each call... Is it
some how cached? Is there some way that the insertion point (in the
case of a b-tree) could be cached? Is there anything else I should
worry about?
Thanks in advance,
David
More information about the pybsddb
mailing list