based on feedback that the current mechanism is overly complicated
and doesn't add any real security, and it's better to stick with
something simple e.g. just a tried-and-true hash of the basics
was mixing deletions and updates and because they are in arbitrary
order due to the hashset structure, sometimes i'd delete an entry
after it was updated.
this is now found and fixed.
not converting core structures from HashSet to BTreeSet though --
the idea is that in production the mappings should be fully random,
but we are just trying to get reproducibility in hosted mode so
that we can debug some hairy edge cases more easily.
Trying to find a new approach to making dialog boxes that
are cleaner and don't require as much code repetition. Probably
about to go on a long tangent.
the formatting of the listing output is a bit messed up but
i'm too tired and too late to deal with that. at least the
core logic seems functionally correct!
should be able to get away with "just" regenerating the fast space
log only if that's what we've run out of; however, doing that optimization
we run into out of space errors. maybe we need some tuning of the thresholds
for when to reclaim space
it's one of those bugs where you're like oh...right. it's doing
exactly what i told it to. I just told it to do the wrong thing.
Right. If you're updating data at the end of a file, you kind of
don't want to delete the file entirely then do the update.
Fix the code so that in the case of a file extension of a small
key, you're re-inserting the key in its entirety (not just the
updated portion) on the extension case.
Also handled an issue with large files where key extension
should not lead to overwrite of the neighboring data, and put
a cap of the large file limit of 32GiB. See comments for thoughts
on the trade-off of the large file limit versus the allocation
limit of the number of files and the physically realistic
outcomes for the PDDB. Basically as written the PDDB can handle
32GiB files (big enough to hold a blu-ray movie, but not a 4k/8k
movie, uncompressed movies, and/or large virtual machine images),
and can handle creation of files at an average rate of one per
second for 10 years without stop before running out of virtual
memory space.
This isn't "Web scale" as large servers will
definitely require files bigger than 32GiB these days, and
can easily create files at a rate of thousands per second. But,
the PDDB isn't performant enough nor targeted toward large
scale server applications. It should, however, be adequate for
pretty much any user data on a mobile phone if it were recompiled
for a 64-bit target. And, if you needed to scale up to bigger
files and/or more write cycles, the problem can be solved
with some simple memory allocation tricks on the large pool,
but because Precursor only has 16MiB of its own RAM to play
with, we don't implement that as the RAM overhead is both
detrimental and unnecessary at our scale of operation.
us to have a full 64-bit address space upon expansion of the page
number (turns out this is probably important because of the crappy
increment-only allocator used for the large blocks)
need to figure out how to compute if it passes; it of course, breaks
the checksum attached for CI purposes. Maybe it's just a matter
of reading back the data and patching it again -- works things even harder!
also, ran the manual test case on real hardware tonight, after 389 hours
of simulation-only development. happy to report that it worked as expected,
so there's no huge obvious gaps between simulation and reality.
can create/delete dicts, create/delete/update keys.
basis deletion is...just forgetting the password. that's the point,
at least. could maybe extend the basis AAD to include some small
amount of derived data that could be deleted to facilitate basis
deletion...
anyways. next up is...lots and lots of testing.
trying to allocate key indicies within the dictionary pool in a late-binding
way is hard to do in rust, because of various interior mutability problems.
i guess it could be done but it turned out to be easier to pro-actively
track everything and generate them on demand.
allocations are hard in rust. i guess that's why the alloc thing is
still not stable.
see comments in code for more details, but, the Rust borrow checker
found a problem where the virtual to physical map could be concurrently
modified with additional entries during an allocation, so now the code
works on an "look-before-you-leap" allocation basis (where it's assumed
you either ensured sufficient space or allocated as you ran along previously),
versus a "oops we need memory let's stop the world and redo everything
while in the middle of anything" situation. The latter is obviously
more convevient, but, alas, I have to agree with the borrow checker --
probably not the safest way to do things.
this probably isn't new to most other people but this has been
a very visceral exercise of how tricky it is to use Repr(c)
for data layouts in anything but the most trivial data structures.
Unfortunately the Basis root data structure needs to be fairly rich,
so we're playing around with using rkyv to help with the serialization
to disk but I'm also not convinced this is a good idea yet. This
checkpoints commits a "working" rkyv implementation, but I'm not
entirely happy with the notion that rkyv can shuffle around data
bits based on its particular version or decisions in the serializer
crate. I'd like deterministic, repeatable serialization that will
be solid through lots of revs...
- hosted mode uses an awful, horrific hack to simulate FLASH:
it makes a global singleton that is freely mutable. Oh but wait!
that's actually how the hardware behaves. It's up to the code
around the hardware to create those primitives. So, maybe it makes
sense that the simulation model for the hardware is terribly
unsafe and unprotected.
- the format routine at least doesn't crash and creates some
semblance of a result that looks right-ish. Need to write a
utility that helps to explore the "disk image" so we can
understand what's going on.
- TrngPool is going to be more efficient if I have a single pool that's
shared among all the functions of the PDDB. It's split out into its own
object that is refcell-counted.
- Trying to make the Basis handle encryption and decryption with iterator
traits. Let's see how that works out.
- reworked fast space allocation
- shifted order of events around slightly
- formatting is now using "generic" calls to create
the free space structures instead of bespoke single-use calls
left off at getting the system basis written. need to rework
the data structure to be orthogonal with the page table remap
on a page-by-page basis. have to think about it a bit...