[pybsddb] Problem scanning large hashed database

[andrew]

Hi All,

I have a large-ish production database (1.6M objects) that usually gets
accessed only for reads on writes on specific keys, for which it is nice
and fast. However, I need to be able to do some adhoc queries on this
data occasionally and I seem to be running into some problems doing
that.

My approach is simply to fetch all keys and then iterate through the
keys, fetching the data for each key and then writing some of it to an
output file. I want to do this while the database is still "up", i.e.,
while a server process that allows access to the database via Pyro is
still running. Unfortunately, when I run this scanning script (which
takes minutes just to fetch all the keys) it seems to kill the Pyro
process with:

bsddb.db.DBRunRecoveryError: (-30978, 'DB_RUNRECOVERY: Fatal error, run
database recovery -- PANIC: fatal region error detected; run recovery',
'This error occured remotely (Pyro). Remote traceback is available.')

My questions are:

1. Is there a better way to do this ?
2. Could it be some sort of locking problem ?
3. Has anyone else done anything like this successfully ?

Thanks for your help.

Cheers, Andrew.