[pybsddb] Batch import is slowing down

[Jesus Cea]

On 27/06/15 18:30, Amirouche Boubekki wrote:

>> without sync with gc
>>> first: 175 seconds
>>> second: 368
>>> third: 603
> 
>> without sync without gc
>>> first: 172 seconds
>>> second: 359
>>> third: 581

Ok, your issue is not Garbage Collection.

My next step now would be to time your BSDDB3 calls.

Profile your code, locate the slow routine and post it here.

>> txn less environment without gc
>>> first: 72 seconds
>>> second: 87
>>> third: 101

In my experience ACI (with durability disabled and async writes to the
log) is quite performant. Not the 1:5 degradation you are seeing, for sure.

> Configuration of the ACID environment:

Have you tried the ACI configuration I proposed a few days ago?

> Primary tables use hash backend. In the worst case an entry will do:

What are the indexes?. If you have locality, a btree could be better.
Probably not an issue since everything you are doing so far fits in the
cache.

-- 
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <https://mailman.jcea.es/pipermail/pybsddb/attachments/20150627/00f3942d/attachment.asc>