[pybsddb] How to manage logs

[Jesus Cea]

On 18/06/15 11:58, Amirouche Boubekki wrote:
> The problem I have is that even when I'm not using transactions (passing
> txn=None) my database fills the disk with log files. There is 2.3 Go of
> database files (including __db.* files) out of 429 Go total disk space
> used by the database directory (du -h .).

Logfiles can be deleted (even "autodeleted" with the DB_LOG_AUTO_REMOVE
flag) after a database checkpoint.

I really recommend everybody to read the Oracle Berkeley DB
documentation. It is really good. Berkeley DB is very flexible but that
flexibility means that you need to learn about the inner working and
details of implementation. Skipping this will be frustrating. Invest
some time reading the docs.

<https://docs.oracle.com/cd/E17076_04/html/programmer_reference/index.html>.

In particular you need to know how logfiles works: everychange done in
the database is stored first in the logfile and later -eventually- in
the real DB. You can not delete the logfile until you are absolutely
sure that the real DB was modified. You do a "checkpoint" for that. When
you do a checkpoint you force every modified data in memory (cache) to
be stored on disk too. After a checkpoint you know that the database
files have the data, you don't need the logfiles anymore (you need the
last one always).

Details:
<https://docs.oracle.com/cd/E17076_04/html/programmer_reference/transapp_checkpoint.html>.

Your case:

You can not remove the logfiles because you need to do a checkpoint
first. You can force a checkpoint "now" or you can ask Berkeley DB to do
a real checkpoint after so many minutes or so many Megabytes:
<https://docs.oracle.com/cd/E17076_04/html/api_reference/C/txncheckpoint.html>.

Beware though that checkpointing can be a costly operation (relatively),
so I do a conditional checkpoint after let say 100 Mbytes or 15 minutes
is more efficient that checkpointing after every transaction.

If you do a "txn_checkpoint()" after 100MB, for example, your logfiles
will be automatically deleted.

-- 
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <https://mailman.jcea.es/pipermail/pybsddb/attachments/20150621/973b9c93/attachment.asc>