[pybsddb] Bulk Data load into a DB

[Jon Kerr Nilsen]

On 29. mars 2010, at 19.54, Jesus Cea wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 03/26/2010 11:38 PM, Joepaul Puthenangadi wrote:
>> Was reading through the documentation and found something on the oracle
>> website about the possibility of loading bulk key and values into a hash
>> or btree db using the flag = |DB_MULTIPLE. Not trying to be picky or
>> anything just wanted to know whether that is something yet to be
>> implemented to |bsddb3 module.?
> 
> pybsddb is fairly incomplete, yet. It is a work in progress.
> 
> DB_MULTIPLE is not supported, currently. In fact using it will crash the
> application, corrupt memory, etc.
> 
> I will support "bulk" interface sooner or later, depending of "popular
> demand" :). Are you demanding it? :))
> 
> The details are tricky because I want to support arbitrary iterators,
> and I don't want to iterate twice over the data. And I need to iterate
> twice to know the lenght of the data and allocate appropiate memory
> size, trying to avoid "realloc()'s" in the process.
> 
> I have had this interface in mind for years. Implementation will be
> "faster" if somebody demands it. I don't have personal need for it.

Hi!

We're working on this storage system Chelonia[1] and we use pybsddb for the replicated meta-database (and hey, it works :)). Anyway, we access the database over WAN, which is quite high in latency, so a bulk interface would be really cool. We practically demand it :)

BTW, great product this pybsddb!

cheers,
Jon

[1]  http://www.nordugrid.org/chelonia/