[pybsddb] IO wait up to 100% after put 2 million keys

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/04/2009 11:01 AM, Nî wrote:
> Hi, guys:
>       
>          I used pybsddb to operate bdb, but everytime after I put 2
> million keys , IO wail would up to 100%, but it's ok when I use C API to
> operate BDB.
>          Any help would be appreciate.
>        
>        dbenv.set_cachesize(0, cachesize64*1024*104, 0)  #  I had
> tried  256K, 1M, 2M , 16M, 64M
>        dbenv.open(dir, DB_PRIVATE | DB_CREATE | DB_INIT_LOCK |
> DB_INIT_MPOOL | DB_THREAD | DB_INIT_LOG)
>        bdb = DB(dbenv)
>        bdb.set_pagesize(4096)  # some to os block
>        for i in xrange(3500000):
>             tmp['%s_/root/2000/dsaf/aaa/aa' % i] = (1, 1) 
>        for k, v in tmp.iteritems():
>             db.put(k, phpserialize.dumps(v))   # phpserialize.dumps just
> change (1, 1) to a serialize string

Sorry for the late reply.

I think you could be hitting the SWAP. How much memory your machine
has?. You are generating 3.5 millions of objects in memory, and then
dumping them in a row. Storing in disk 3.5 million objects would take a
while, too.

As generated in the code, 3.5 million objects would take 100MB. And in
the creation process, Python's garbage collector would hit a few times.

Your code doesn't work, so any conclusion based on it would be
questionable. If you are still seeing this issue, let us know. And
include code for a reproductible case, please, describing your
environment, python, pybsddb and BDB versions, etc.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS3wgMplgi5GaxT1NAQKMZgP+L4oJiRFTspp4ao4f13RuMDF2TykF5UMA
brXlLwLK46gT9fbenCdfn1cBbkc+6jAB2dDoFtHdVTGqtdjMoCHSK3WyZnJgqZqD
WuFPHUegtstoQ1DHw/7+Sn9PV4UcxbhAa/fWQFStFkObwdYb2jDszlpeYaqRU4Op
SjKeJVlBQQY=
=voLn
-----END PGP SIGNATURE-----