[pybsddb] Batch import is slowing down

[Amirouche Boubekki]

On 2015-06-24 22:30, Amirouche Boubekki wrote:
> On 2015-06-24 18:56, Jesus Cea wrote:
>> On 24/06/15 12:09, Amirouche Boubekki wrote:
>>> Indeed the documentation is very good, you did a very good job with 
>>> that.
>> 
>> I didn't write it :-). In fact neither Oracle did. It was mostly 
>> written
>> in the SleepyCat era :), before Oracle bought them.
>> 
>> Anyway I am not related to Oracle in any way at all.
>> 
>>> - The first set of 10 000 entries took 3 seconds to load.
>>> - 49th set took 2 minutes
>> 
>> If could be a cache size issue. I can't really tell without checking 
>> it
>> myself. What does "db_stat -m" show?. Initialize the stats with 
>> "db_stat
>> -mZ" from time to time and compare results between the beginning and 
>> the
>> slow part.
> 
> I see nothing suspicious in the log. They are attached. I'll be very
> happy if you can help me.
> 
> Here are the timings:
> 
> 10000 0:00:03.643761
> 20000 0:00:04.169674
> 30000 0:00:05.834176
> 40000 0:00:05.220522
> 50000 0:00:06.411978
> 60000 0:00:08.391819
> 70000 0:00:12.561121
> 80000 0:00:06.480804
> 90000 0:00:07.154480
> 100000 0:00:07.106232

Seems like; if I do checkpoints more often, it faster here is a new 
timings with checkpoints every 1000 entries, cut to make appear only the 
ten thousands

10000 0:00:00.623182
20000 0:00:00.987063
30000 0:00:01.568719
40000 0:00:01.659039
50000 0:00:02.171126
60000 0:00:01.960296
70000 0:00:02.301340
80000 0:00:02.216895
90000 0:00:02.549782
100000 0:00:03.089665