<meta http-equiv="content-type" content="text/html; charset=utf-8"><br><blockquote class="gmail_quote" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; ">
Maybe you can use memcached (an LRU) and not bdb (a key-value store).</blockquote><br><div>FYI, memcached specifically limits the number of keys you can query from a bucket in one go (to a 2MB response), and does not support range queries (on keys), so it is often not possible to dump the entire cache's keys.<div>
<br></div><div>e.g. `stats cachedump`ing keys from the buckets of a table with 3643788 entries yields only 163452 keys</div><div><br></div><blockquote class="webkit-indent-blockquote" style="margin: 0 0 0 40px; border: none; padding: 0px;">
<div><div><font class="Apple-style-span" face="'courier new', monospace"><div><div>$ memcached-tool localhost dump | grep -a ^add | wc -l</div><div>Dumping memcache contents</div><div> Number of buckets: 5</div><div>
Number of items : 3643788</div><div>Dumping bucket 1 - 2248149 total items</div><div>Dumping bucket 2 - 1094868 total items</div><div>Dumping bucket 3 - 275682 total items</div><div>Dumping bucket 4 - 24911 total items</div>
<div>Dumping bucket 5 - 178 total items</div><div>163452</div></div><div><br></div></font></div></div></blockquote><div><div>Cheers</div><div> - Jeremy<br><br><div class="gmail_quote">On Thu, Jun 10, 2010 at 1:05 AM, lasizoillo <span dir="ltr"><<a href="mailto:lasizoillo@gmail.com">lasizoillo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">2010/6/9 Denis Papathanasiou <<a href="mailto:denis.papathanasiou@gmail.com">denis.papathanasiou@gmail.com</a>>:<br>
<div class="im">> I know that using keys() can be inefficient, as per the warnings in the<br>
> docs.<br>
<br>
</div>The docs says:<br>
Warning: this method traverses the entire database so it can possibly<br>
take a long time to complete.<br>
<br>
possibly take long time != inefficient<br>
<br>
But is a interesting warning.<br>
<br>
If you read all database with read locks:<br>
* You are blocking all writing operarion for a *possible long time*<br>
* You can exhaust the lock space.<br>
<br>
Relax ACID schema can cause data corruption. Full ACID schema can<br>
cause *possible long time* delays.<br>
<div class="im"><br>
><br>
> I do need to calls keys() in the following scenario, though, so I was<br>
> wondering if there was anything I could do to make it run faster:<br>
><br>
> A db that stores temporary session data has as its key a guid, prefixed by a<br>
> unix timestamp. The associated value is just another guid identifier (which<br>
> is unrelated to the one in the key).<br>
><br>
> Since the session data is temporary, I have a function which examines all<br>
> the keys, and removes every key/value pair older than n seconds (it's a<br>
> simple calc using the unix timestamp prefix in the key).<br>
><br>
> I realize that I could restructure the db so that the unix timestamp is part<br>
> of the value, and by adding a secondary index on it, I could get all the<br>
> keys older than n.<br>
><br>
> But, restructuring it and doing all the tests will take time, so in the<br>
> interim, is there anything I can do to make using keys() more effective?<br>
><br>
<br>
</div>Take all keys => O(n)<br>
Take a key in btree => O(log n)<br>
<div class="im"><br>
> Alternatively, is there any way to simply get all the keys greater than n?<br>
><br>
<br>
</div>Using cursors.<br>
<div class="im"><br>
> It seems beyond the scope of bdb, but I was curious if I'd missed anything.<br>
<br>
</div>Take a look to the bdb reference guide:<br>
<a href="http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/index.html" target="_blank">http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/index.html</a><br>
<br>
You can search for LRU papers for more implementations details.<br>
<br>
Maybe you can use memcached (an LRU) and not bdb (a key-value store).<br>
<br>
Regards,<br>
<br>
Javi<br>
<div><div></div><div class="h5">_______________________________________________<br>
pybsddb mailing list<br>
<a href="mailto:pybsddb@jcea.es">pybsddb@jcea.es</a><br>
<a href="https://mailman.jcea.es:28443/listinfo/pybsddb" target="_blank">https://mailman.jcea.es:28443/listinfo/pybsddb</a><br>
<a href="http://www.jcea.es/programacion/pybsddb.htm" target="_blank">http://www.jcea.es/programacion/pybsddb.htm</a><br>
</div></div></blockquote></div><br></div></div></div>