[pybsddb] Snapshotting / replicating bdb

[Jesus Cea]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

andrew wrote:
> The
> production databases are under constant load, and when I try to run a
> table scan of one of them (2M objects) it tends to kill the database
> for some unknown reason. Hence I want to try doing this analysis
> against a separate, readonly version.

If you are scanning the database using transactions (you should), it
will lock pages until you close the transaction. You can reduce your
isolation level to avoid it, at the cost of inconsistences.

Read
<http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html>

If you are running a recent Berkeley DB version, you can use MVCC
(Multiversion concurrency control), but it is delicate and impose some
restrictions in your cache size and such.

andrew wrote:
> Thanks, qyb. I'm using Python 2.4 which looks to be running against BSD
> 3 so I can't find test_replication.py - looks like I might have to
> upgrade :-)

Yes, you should upgrade bsddb3. And, in the process, install a recent
version of Berkeley DB :).

Remember, nevertheless, that if your database is big, replication from
the ground up will be slow and will penalice your main application a
lot. Moreover, you have to implement replication in your code, and that
is not trivial.

I would use the "copy" approach :).

If you have a "decent" operating system, like Solaris 10 under ZFS, you
can snapshot&clone your database live, with zero overhead and only a
couple of seconds, independently of the size of it. Then you run your
read-only code in the clone, and when done you delete the clone and the
snapshot, all in seconds and with your main application running live.
This is what I actually use daily.

If you are using Linux with LVM2, I think you can do something similar.

I would only use the replication approach if your read-only code
executes frequently and you can pay the code complexity, etc., of
supporting replication.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBShaHQJlgi5GaxT1NAQI1mwQAhfCHpGqIBB6G/vpVFHso1P+28qflBL5N
aCN8tH5boj1Z2og2St6TVnLsiMtVMA9VwVopOIqbH28UqSN92FVG6E8r5QHMZ7p7
UYQuS4Hr2Ye3hYrCS7ECez1cveRHlkD5bim+GsUvA57hkPdolENyBenM6blT4O7U
6rO6TteJihg=
=q6z3
-----END PGP SIGNATURE-----