[pybsddb] Making calls to keys() more efficient (or, how to do inequality gets on keys?)

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/06/10 21:46, Denis Papathanasiou wrote:
> I know that using keys() can be inefficient, as per the warnings in the
> docs.
> 
> I do need to calls keys() in the following scenario, though, so I was
> wondering if there was anything I could do to make it run faster:
> 
> A db that stores temporary session data has as its key a guid, prefixed
> by a unix timestamp. The associated value is just another guid
> identifier (which is unrelated to the one in the key).
> 
> Since the session data is temporary, I have a function which examines
> all the keys, and removes every key/value pair older than n seconds
> (it's a simple calc using the unix timestamp prefix in the key).
> 
> I realize that I could restructure the db so that the unix timestamp is
> part of the value, and by adding a secondary index on it, I could get
> all the keys older than n.
> 
> But, restructuring it and doing all the tests will take time, so in the
> interim, is there anything I can do to make using keys() more effective?
> 
> Alternatively, is there any way to simply get all the keys greater than n?
> 
> It seems beyond the scope of bdb, but I was curious if I'd missed anything.

If your prefix is big endian (the timestamp) and you store the data in a
btree with the standard comparison function, iterating (with a cursor)
over the database would give you the keys in order.

You could do something like:

1. Create cursor
2. Get the first element
3. Delete the cursor.
3. Check the timestamp in that first element. If it is recent, wait a
bit and try later.
4. If not, delete that record and go to step 1 again.

If you use this, document *CLEARLY* that that database MUST be a BTREE.

Hope it helps.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTBJQZplgi5GaxT1NAQI7VAQAgIHrRPtZbub8s/i/lIZYQoMgMfrfizYD
vgtUCHAFic6Cb+fs6Qler4bj6o53ArX5DjxZH7N55KmRfXw7JKXzsxTPem02BuGW
P/0uKXJonQQz8YWhyOo4XMIPrYStarJ8m78VhAB7oQbCLZvat2zO7sdG6jkyK7+k
VNh3KLILX/w=
=qhdi
-----END PGP SIGNATURE-----