Commit graph

10340 commits

Author SHA1 Message Date
bunnie
572cc8285f Merge branch 'bunnie-dev' into main
This is a partial commit of the PDDB development items. The PDDB server
doesn't do anything yet in this commit, but in order to get the documentation
links correct the docs need to be pushed to main. So, consider this a
"push documentation" commit. There will be some build warnings resulting
from this that will be fixed as the PDDB becomes more finalized.
2021-11-23 15:49:47 +08:00
bunnie
49d24f9fae revert vscode settings for a merge to main 2021-11-23 15:49:25 +08:00
bunnie
dc05b2b610 add a README and prepare for a merge to main so docs are discoverable 2021-11-23 15:47:53 +08:00
bunnie
cbf7ee032f reorganize documentation 2021-11-23 15:47:40 +08:00
bunnie
9c54152f4d patching test written
need to figure out how to compute if it passes; it of course, breaks
the checksum attached for CI purposes. Maybe it's just a matter
of reading back the data and patching it again -- works things even harder!

also, ran the manual test case on real hardware tonight, after 389 hours
of simulation-only development. happy to report that it worked as expected,
so there's no huge obvious gaps between simulation and reality.
2021-11-23 04:36:58 +08:00
bunnie
d716fea17a now passing dictionary add/remove test 2021-11-23 02:57:13 +08:00
bunnie
77d3f8a646 fix bug in python aes-gcm-siv reference implementation 2021-11-23 02:56:54 +08:00
bunnie
eda60a5d9c base case test done, with CI checking 2021-11-22 18:26:19 +08:00
bunnie
61b5aeba98 add ability to re-run pddb test cases in hosted mode for more test automation 2021-11-22 16:28:26 +08:00
bunnie
26736c09b6 start to build out tougher test cases 2021-11-22 16:04:50 +08:00
bunnie
151e054314 clean up a whole bunch of warnings 2021-11-22 16:04:31 +08:00
bunnie
62059a9f93 relocate images to a directory 2021-11-22 14:00:43 +08:00
bunnie
352c54eeb3 relocate images to a directory 2021-11-22 14:00:25 +08:00
bunnie
1874c8e72d most of the core routines are now in place
can create/delete dicts, create/delete/update keys.
basis deletion is...just forgetting the password. that's the point,
at least. could maybe extend the basis AAD to include some small
amount of derived data that could be deleted to facilitate basis
deletion...

anyways. next up is...lots and lots of testing.
2021-11-22 04:46:36 +08:00
bunnie
2859e54bb9 that was a long walk to fix a bad architectural decision
trying to allocate key indicies within the dictionary pool in a late-binding
way is hard to do in rust, because of various interior mutability problems.
i guess it could be done but it turned out to be easier to pro-actively
track everything and generate them on demand.

allocations are hard in rust. i guess that's why the alloc thing is
still not stable.
2021-11-22 01:46:53 +08:00
bunnie
e4e5e6c390 more refactoring 2021-11-22 01:19:57 +08:00
bunnie
00339b0c5f refactoring to simplify free key index tracking. 2021-11-22 00:23:53 +08:00
bunnie
193f563782 very basic small key additions working as expected now 2021-11-21 02:21:25 +08:00
bunnie
05b79f066c framework for adding keys in place. now to debug it... 2021-11-20 18:25:50 +08:00
bunnie
23caf88526 prelim code to update/add keys
next up: sync the small key alloc area to disk, and then we can
start testing...
2021-11-20 01:12:10 +08:00
bunnie
6da984e51e slow but steady progress toward implemeting key allocation and access 2021-11-19 04:00:33 +08:00
bunnie
14f4beef98 refactor data structures into smaller files for easier nav 2021-11-18 15:38:12 +08:00
bunnie
8186930e94 commit some thoughts on key allocation 2021-11-18 04:04:03 +08:00
bunnie
e1d00ef57c fix clean state bookkeeping on the basisroot cache record 2021-11-18 02:13:05 +08:00
bunnie
a5da30de37 retire the barque BasisRoot encryption mechanism for a much simpler implementation 2021-11-18 02:10:58 +08:00
bunnie
c62b2a25fb dictionary adding now seems to work 2021-11-18 02:01:30 +08:00
bunnie
507d99ee41 moving toward refactoring the basis sync function 2021-11-18 00:09:55 +08:00
bunnie
70a478795d refactor structure; fix mutability issue with fast space allocations.
see comments in code for more details, but, the Rust borrow checker
found a problem where the virtual to physical map could be concurrently
modified with additional entries during an allocation, so now the code
works on an "look-before-you-leap" allocation basis (where it's assumed
you either ensured sufficient space or allocated as you ran along previously),
versus a "oops we need memory let's stop the world and redo everything
while in the middle of anything" situation. The latter is obviously
more convevient, but, alas, I have to agree with the borrow checker --
probably not the safest way to do things.
2021-11-17 22:29:47 +08:00
bunnie
6fa2332ff5 stash an intermediate set before refactoring to fix a borrow issue 2021-11-17 04:07:11 +08:00
bunnie
bbbaa6b810 clean up the structural definition of the dictionaries and keys 2021-11-16 18:45:24 +08:00
bunnie
49d00c783e basic mounting of the PDDB & readout of system basis 2021-11-16 02:58:33 +08:00
bunnie
d7cc518552 add mbbb mechanism 2021-11-15 19:05:03 +08:00
bunnie
65bbc82ef5 commit to Repr(C), include doc comment for rationale and tests 2021-11-15 16:38:57 +08:00
bunnie
ce51205155 checkpoint with memory placement and disk storage
this probably isn't new to most other people but this has been
a very visceral exercise of how tricky it is to use Repr(c)
for data layouts in anything but the most trivial data structures.

Unfortunately the Basis root data structure needs to be fairly rich,
so we're playing around with using rkyv to help with the serialization
to disk but I'm also not convinced this is a good idea yet. This
checkpoints commits a "working" rkyv implementation, but I'm not
entirely happy with the notion that rkyv can shuffle around data
bits based on its particular version or decisions in the serializer
crate. I'd like deterministic, repeatable serialization that will
be solid through lots of revs...
2021-11-15 16:00:01 +08:00
bunnie
ba9b136f87 debugging alignment issues on serialization of compound structures 2021-11-15 04:03:55 +08:00
bunnie
696e99b173 ignore the pycache that comes about from running the pddbdbg tools 2021-11-14 19:04:12 +08:00
bunnie
4a7bdab5ea various debug messages and minor edits 2021-11-14 19:03:07 +08:00
bunnie
94a86c542d add an entrails parser for the PDDB 2021-11-14 19:02:34 +08:00
bunnie
93fd991dff fix paths and ignores for the image dump files 2021-11-13 17:34:34 +08:00
bunnie
1ed606fa7d very rudimentary pddb dumper for debugging.
yah, I know. :-/

anyways.
2021-11-12 22:57:37 +08:00
bunnie
f5a5211851 hosted mode simulation; basic disk formatting doesn't crash.
- hosted mode uses an awful, horrific hack to simulate FLASH:
it makes a global singleton that is freely mutable. Oh but wait!
that's actually how the hardware behaves. It's up to the code
around the hardware to create those primitives. So, maybe it makes
sense that the simulation model for the hardware is terribly
unsafe and unprotected.

- the format routine at least doesn't crash and creates some
semblance of a result that looks right-ish. Need to write a
utility that helps to explore the "disk image" so we can
understand what's going on.
2021-11-12 21:15:30 +08:00
bunnie
4b33069ac6 starting on the simulated test/debug of the initial formatting 2021-11-11 23:48:05 +08:00
bunnie
55f2314e7c first pass at a format + passing trivial built-in tests 2021-11-11 23:33:01 +08:00
bunnie
d45369a19d basis root now iterates itself into an encrypted set of blocks 2021-11-11 18:54:17 +08:00
bunnie
62ef3b9414 refactor trngpool to a shared refcell-protected object. encryption iterator for basis.
- TrngPool is going to be more efficient if I have a single pool that's
shared among all the functions of the PDDB. It's split out into its own
object that is refcell-counted.
- Trying to make the Basis handle encryption and decryption with iterator
traits. Let's see how that works out.
2021-11-11 16:30:45 +08:00
bunnie
79d32e29f6 still grinding through the disk format procedure.
- reworked fast space allocation
- shifted order of events around slightly
- formatting is now using "generic" calls to create
the free space structures instead of bespoke single-use calls

left off at getting the system basis written. need to rework
the data structure to be orthogonal with the page table remap
on a page-by-page basis. have to think about it a bit...
2021-11-10 03:42:54 +08:00
bunnie
a4a2d12d57 refactoring the order of format events; consolidate fscb init op
try to condense ops that we anticipate calling in other places
into helper functions.

add AAD to the fscb.
2021-11-09 04:57:12 +08:00
bunnie
da9fcb24ff initial draft of fastspace collection compiles 2021-11-08 18:41:31 +08:00
bunnie
7858aaad0e free_pool scanning first cut 2021-11-08 05:41:39 +08:00
bunnie
81d9fb3fda more incremental progress 2021-11-07 05:34:07 +08:00