[pybsddb] Windows support for berkeleydb

[Brian Matthews]

Hi, thanks for responding.

1. Windows build
      If you just revert the change you made it will still not build on Windows.  There are some minor syntax changes needed to make that happen. It has been like this for a long time!

  # Supported Oracle Berkeley DB versions, in order of preference.
  db_ver_list = ((18, 1), (6, 2), (5, 3), (4, 8))
  db_ver = None

   the block above is defined inside " if os.name == 'posix':"  so the windows block cannot see it. I moved it in front of the posix compare.

   The section in " elif os.name == 'nt':" did not correctly build the libname's

  The link does not need runtime_library_dirs.

  The changes I sent you are the changes you cut out restored, but with these modifications


2. Binary key

The bsddb3 implementation used a str() variable which allowed to have binary characters. No NULL's as Berkekey will truncate it as you say. We use a randomly generated binary sequence.
But the point is we have 000's of systems out there for 10-15 years that use this string or ones like it. And I cannot use it in berkeleydb python3 ports because it expects a python3 str() which is actually Unicode.
It tries to convert it.
I need to be able to pass a bytes string b''. as the key. Then it is backwards compatible. BerkeleyDB assumes it is a char* string null terminated. Which does not preclude non-ascii and non-unicode characters.
This functionality needs to be restored I believe.

The changes I made are for bytes strings. It could easily handle both Unicode and bytes. We surely cant be the only system hitting this problem.

Thanks

Brian


-----Original Message-----
From: Jesus Cea <jcea at jcea.es>
Sent: Sunday, December 15, 2024 5:08 PM
To: Brian Matthews <Brian.Matthews at vaisala.com>
Cc: Python bindings for Oracle Berkeley DB <pybsddb at jcea.es>
Subject: Re: [pybsddb] Windows support for berkeleydb

[You don't often get email from jcea at jcea.es. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

On 26/11/24 20:16, Brian Matthews wrote:
> It also contains a change for src/Module/berkeleydb.c This changes
> allow the encryption password to be either a string, or a byte string.
> Bsddb3 used to allow binary keys. This restores that ability.

Brian, that patch is wrong. Berkeley DB password must be a string with no null characters. A generic binary key is unsafe if it contains "\0", because it would be truncated to the first null.

The code enforces a string for a good reason.

Legacy "bsddb3" accepts binary, but it is wrong, the key could be silently truncated.

I added some details in the documentation. See:

https://docs.jcea.es/berkeleydb/latest/db.html#set_encrypt
https://docs.jcea.es/berkeleydb/latest/dbenv.html#set_encrypt

If you already has a binary key with "\0" inside, that key will actually be truncated up to that "\0". So you can simply truncate the key there by yourself, check that it actually can access the database just fine and then maybe export/import the database with an actually strong key, with no \0 inside. Just to be safe and be sure it uses a full length key.

If you require being able to use a random key, I would suggest you to encode it using base16, base64 or something like that, to transform it to a regular string able to encode "\0" values.

If you have a "legacy" binary key with no "\0" inside, just encode it to a string as described in the links above.

Let me know if this explanation is enough for you and if I can help you with anything else.

I am still thinking how to integrate your patches for windows support.
Apparently they mostly restore code I deleted in 18.1.10. Reverting that deletion would be enough for you?

Thanks for the compiled binary. Would be nice to be able to use a Windows compilation service somewhere, for free. I am not familiar with that area, though. Can somebody help here?

--
Jesús Cea Avión                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - https://www.jcea.es/    _/_/    _/_/  _/_/    _/_/  _/_/
Twitter: @jcea                        _/_/    _/_/          _/_/_/_/_/
jabber / xmpp:jcea at jabber.org  _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz