Getting optional LMDB support into Cytoplasm

lda commented

2024-08-10 08:59:03 +00:00

Contributor

No description provided.

lda added 10 commits 2024-08-10 08:59:04 +00:00

[ADD/WIP] Start adding a lmdb flag to configure 87d9421f11

Start of my work to get out LMDB support. I want to make it optional, as
some environments just can't use LMDB(due to mapped RAM limits, or
places where mmap is unavailable (a rather cursed platform!)).

Next up: Start separating Db to allow multiple subimplementations
instead of being expressly for nested-dir JSON ops.

[MOD/WIP] Start separating the main DB API b87979e9a2

Not everything is available as of now, I'm still working on it, but it
builds and seems to work, and its 9PM, so that's worthapush.

[MOD/WIP] Separate all DB operations 59dbfae1ae

Somewhat untested, but I fail to see how it could fail, right?

Next up: Getting the basics of LMDB up and running.

[ADD/WIP] Start to add LMDB operations 5cb51a4d58

[ADD/WIP] Most DB operations for LMDB 0743401955

Somewhat untested. I want to go on a test run soon.

Next up: aargh... listing... The one thing LMDB may suck at.

[ADD/WIP] Listing in LMDB 004c53a028

I need to start entering a LMDB test run.

[MOD/WIP] Disable thread-local storage for LMDB a90c66736c

DB locks ought to be enough...

Merge branch 'master' of https://git.telodendria.io/Telodendria/Cytoplasm into lmdbwerk 2b061f1226

[FIX/WIP] Temporary fix to the database system f6af2cd782

It used to crash, my bad.

[FIX/WIP] Fix mutex issues around LMDB 20bb7a20ad

Currently doing a test run on another project of mine to find out how
stable it is.

Next up(more long-termed): Faster JSON parsing than just plaintext!

lda added 1 commit 2024-08-10 22:02:00 +00:00

[MOD/WIP] Mark listing transactions as readonly f32cdb7d89

May want to sprinkle in "hinting" on the nature of operations done the
database, which could allow LMDB to deal with those far more efficiently
(for example, a read-only transaction can just be done as soon as the
JSON itself is parsed out, as we don't really need the former anymore!)

lda added 3 commits 2024-08-11 13:30:03 +00:00

[ADD/WIP] Start rolling in intents 3df1e4ab7b

[MOD/WIP] Use one shared DBI e133eebef3

Yeah, I was stupid for not doing that earlier.

Merge branch 'lmdbwerk' of https://git.telodendria.io/lda/Cytoplasm into lmdbwerk cf1b78b224

lda changed title from ~~WIP: Getting optional LMDB support into Cytoplasm~~ to Getting optional LMDB support into Cytoplasm

2024-08-16 20:32:54 +00:00

jordan commented

2024-08-18 23:27:36 +00:00

Owner

Can you give some sort of rationale statement for this? I'm not opposed to it, and if you say it works I can go ahead and merge it, but a little comment here as to why this is necessary might be nice.

I should also do some research of my own on LMDB, I'm not super familiar with it.

Can you give some sort of rationale statement for this? I'm not opposed to it, and if you say it works I can go ahead and merge it, but a little comment here as to *why* this is necessary might be nice. I should also do some research of my own on LMDB, I'm not super familiar with it.

lda commented

2024-08-19 10:33:07 +00:00

Author

Contributor

Can you give some sort of rationale statement for this? I'm not opposed to it, and if you say it works I can go ahead and merge it, but a little comment here as to why this is necessary might be nice.

I should also do some research of my own on LMDB, I'm not super familiar with it.

It's mostly out of reliability and speed. I've seen some scenarios where the JSON-based backend could sometimes just corrupt(by not writing all the way, but then again, it was on rather large JSON objects, but I can think of at least one place where that could happen (like PDUs, which can potentially be up to 64KB)) after crashes. LMDB on the otherhand does avoid the problem by design (c.f) (c.f 2).
Also, since it is memory-mapped, it can just avoid doing open/read/close system calls.

> Can you give some sort of rationale statement for this? I'm not opposed to it, and if you say it works I can go ahead and merge it, but a little comment here as to *why* this is necessary might be nice. > > I should also do some research of my own on LMDB, I'm not super familiar with it. It's mostly out of reliability and speed. I've seen some scenarios where the JSON-based backend could sometimes just corrupt(by not writing all the way, but then again, it was on rather large JSON objects, but I can think of at least one place where that could happen (like PDUs, which can potentially be up to 64KB)) after crashes. LMDB on the otherhand does avoid the problem by design [(c.f)](https://en.m.wikipedia.org/wiki/Lightning_Memory-Mapped_Database#Technical_description) [(c.f 2)](https://www.symas.com/lmdb). Also, since it is *memory-mapped*, it can just avoid doing `open`/`read`/`close` system calls.

jordan commented

2024-08-21 18:35:11 +00:00

Owner

That's true, the JSON backend isn't the most robust at the moment, but I would like it to be a viable alternative for those that don't want to use LMDB, so my only hesitation is that if we merge this, the JSON backend will be neglected and thus no longer work at all. As long as we make sure we maintain both so that both are reliable (even if the JSON is much slower) then I'm okay with this.

jordan merged commit b6b915530c into master

2024-08-23 20:19:59 +00:00

jordan referenced this pull request from a commit

2024-08-23 20:19:59 +00:00

Merge pull request 'Getting optional LMDB support into Cytoplasm' (#43) from lda/Cytoplasm:lmdbwerk into master

Getting optional LMDB support into Cytoplasm #43