Commit graph

392 commits

Author SHA1 Message Date
bunnie
6d655c8085 convert salt to use an HMAC instead of SHA to harden against length extension attacks
(as reccommended by reviewer)
2022-01-02 19:37:30 +08:00
bunnie
9910547e21 modify salt generation scheme
based on feedback that the current mechanism is overly complicated
and doesn't add any real security, and it's better to stick with
something simple e.g. just a tried-and-true hash of the basics
2022-01-02 17:53:00 +08:00
bunnie
0965c4ccc6 remove suspend/resume opcode, we shouldn't need to be aware of it... 2022-01-02 16:44:44 +08:00
bunnie
350276c08a clean up more warnings and errors 2022-01-02 15:31:56 +08:00
bunnie
421a3814d8 clean up some warnings before moving on to further integration 2022-01-02 15:07:48 +08:00
bunnie
c2ee9f7c7a refactor tests and CI into a feature 2022-01-02 05:21:05 +08:00
bunnie
33b0593f9a fx fspace cache entry space check issue
was counting total number of pages, not actual free pages.
2022-01-02 01:41:55 +08:00
bunnie
fef355423b fix CI red herrings; add more debug granularty 2022-01-01 15:14:44 +08:00
bunnie
656f17f621 add extensive CI test 2021-12-29 05:49:07 +08:00
bunnie
6990513278 fixed a subtle bug in the pte sync algorithm
was mixing deletions and updates and because they are in arbitrary
order due to the hashset structure, sometimes i'd delete an entry
after it was updated.

this is now found and fixed.
2021-12-28 06:47:33 +08:00
bunnie
713c0d6e73 add a multi-iteration CI loop around the pddb test
this allows us to capture seeds and corners of interest
2021-12-27 16:15:51 +08:00
bunnie
dd14dfc4c6 retrofit the seed into an atomicu64 so we can set it elsewhere 2021-12-27 02:49:08 +08:00
bunnie
809dbbdbc2 fix a bug in the reserved amount of data in large keys 2021-12-26 16:29:35 +08:00
bunnie
0bd110c163 determinism achieved for testing. now to squash some bugs. 2021-12-26 01:50:30 +08:00
bunnie
37b46fae98 make determinism an optional feature 2021-12-26 01:42:36 +08:00
bunnie
d4027fa8a2 sort all the lists in the test code so the vectors are deterministic
not converting core structures from HashSet to BTreeSet though --
the idea is that in production the mappings should be fully random,
but we are just trying to get reproducibility in hosted mode so
that we can debug some hairy edge cases more easily.
2021-12-25 16:44:24 +08:00
bunnie
66606ed3e0 initial attempt to make hosted mode more deterministic
replace all threadRng with a seeded Rng
2021-12-25 15:58:27 +08:00
bunnie
c1c57edfd6 some mods to the logging infra to chase down a "rare" issue 2021-12-22 07:09:30 +08:00
bunnie
5d226a002a prepare to start using the modals server for the pddb 2021-12-17 03:11:06 +08:00
bunnie
725d943198 add interlocking so multiple threads don't interfere with each other 2021-12-17 01:55:36 +08:00
bunnie
dcad15284d refactor tests into a separate file 2021-12-16 01:48:11 +08:00
bunnie
60d6d08933 basic framework for modal API wrapping in place 2021-12-15 13:46:24 +08:00
bunnie
de4e84abf5 starting in on the larger system integration of the PDDB
Trying to find a new approach to making dialog boxes that
are cleaner and don't require as much code repetition. Probably
about to go on a long tangent.
2021-12-13 14:24:14 +08:00
bunnie
c84ed61ecd basis overlay test done & passing
the formatting of the listing output is a bit messed up but
i'm too tired and too late to deal with that. at least the
core logic seems functionally correct!
2021-11-30 00:38:29 +08:00
bunnie
8066b8fb5e fixed the bug with the pt map encryption, now we hvae two basis!
more testing needed, but looking good so far.
2021-11-29 00:25:09 +08:00
bunnie
c15841cfdc test framework in place to look at multiple basis
found one bug already, need to refactor how PT encryptor is
handled
2021-11-28 23:54:58 +08:00
bunnie
5a12e83ff2 add first cut at basis key derivation function 2021-11-28 20:42:26 +08:00
bunnie
af9f4cbde3 back out changes for stress testing the fastspace mechanims 2021-11-27 21:23:42 +08:00
bunnie
185d629b98 stress test fastspace allocation and regeneration
- confirmed that log entries regenerate on trigger
- confirmed that fastspace cache regenerates and contains a
new subset of blank pages.
2021-11-27 21:17:41 +08:00
bunnie
93ee050079 fastspace regen debugging -- passes test, but needs optimization
should be able to get away with "just" regenerating the fast space
log only if that's what we've run out of; however, doing that optimization
we run into out of space errors. maybe we need some tuning of the thresholds
for when to reclaim space
2021-11-27 02:19:57 +08:00
bunnie
e02df18374 fix reporting error in directory listings 2021-11-26 03:36:36 +08:00
bunnie
468e372bfa most tests passing, also, be a little smarter about cache fills. 2021-11-26 03:08:08 +08:00
bunnie
0df5f1c926 fix key de-allocation bug
forgot to reduce key count for the dictionary record when
a key is removed.
2021-11-25 22:03:37 +08:00
bunnie
85e48c80ed small keys had their data missing because i deleted it :-/
it's one of those bugs where you're like oh...right. it's doing
exactly what i told it to. I just told it to do the wrong thing.

Right. If you're updating data at the end of a file, you kind of
don't want to delete the file entirely then do the update.

Fix the code so that in the case of a file extension of a small
key, you're re-inserting the key in its entirety (not just the
updated portion) on the extension case.

Also handled an issue with large files where key extension
should not lead to overwrite of the neighboring data, and put
a cap of the large file limit of 32GiB. See comments for thoughts
on the trade-off of the large file limit versus the allocation
limit of the number of files and the physically realistic
outcomes for the PDDB. Basically as written the PDDB can handle
32GiB files (big enough to hold a blu-ray movie, but not a 4k/8k
movie, uncompressed movies, and/or large virtual machine images),
and can handle creation of files at an average rate of one per
second for 10 years without stop before running out of virtual
memory space.

This isn't "Web scale" as large servers will
definitely require files bigger than 32GiB these days, and
can easily create files at a rate of thousands per second. But,
the PDDB isn't performant enough nor targeted toward large
scale server applications. It should, however, be adequate for
pretty much any user data on a mobile phone if it were recompiled
for a 64-bit target. And, if you needed to scale up to bigger
files and/or more write cycles, the problem can be solved
with some simple memory allocation tricks on the large pool,
but because Precursor only has 16MiB of its own RAM to play
with, we don't implement that as the RAM overhead is both
detrimental and unnecessary at our scale of operation.
2021-11-25 17:56:39 +08:00
bunnie
ca485d5541 fix off-by-one in small pool to dictionary storage mappings
consolidate index math into helper functions to reduce errors.
2021-11-25 03:14:29 +08:00
bunnie
9d83b37818 unit tests for the free key index tracker 2021-11-24 14:50:44 +08:00
bunnie
8de0d70f43 debugging the patch-extend test.
probably should make a bunch of unit tests for the free dictionary
key tracking logic, it's too tricky to test out.
2021-11-24 03:57:50 +08:00
bunnie
800fcbe3bf fix page table consistency issue -- needs more testing to see if we licked it 2021-11-24 01:50:15 +08:00
bunnie
66a5b97705 revise pagetable entry to be 7 bytes wide -- this will allow
us to have a full 64-bit address space upon expansion of the page
number (turns out this is probably important because of the crappy
increment-only allocator used for the large blocks)
2021-11-24 01:49:22 +08:00
bunnie
42e926fcab fixes for patch update test
lots of corner cases found and ifxed
2021-11-23 23:33:06 +08:00
bunnie
472d60ae95 update a comment in the readme. 2021-11-23 15:55:27 +08:00
bunnie
cbf7ee032f reorganize documentation 2021-11-23 15:47:40 +08:00
bunnie
9c54152f4d patching test written
need to figure out how to compute if it passes; it of course, breaks
the checksum attached for CI purposes. Maybe it's just a matter
of reading back the data and patching it again -- works things even harder!

also, ran the manual test case on real hardware tonight, after 389 hours
of simulation-only development. happy to report that it worked as expected,
so there's no huge obvious gaps between simulation and reality.
2021-11-23 04:36:58 +08:00
bunnie
d716fea17a now passing dictionary add/remove test 2021-11-23 02:57:13 +08:00
bunnie
eda60a5d9c base case test done, with CI checking 2021-11-22 18:26:19 +08:00
bunnie
61b5aeba98 add ability to re-run pddb test cases in hosted mode for more test automation 2021-11-22 16:28:26 +08:00
bunnie
26736c09b6 start to build out tougher test cases 2021-11-22 16:04:50 +08:00
bunnie
151e054314 clean up a whole bunch of warnings 2021-11-22 16:04:31 +08:00
bunnie
1874c8e72d most of the core routines are now in place
can create/delete dicts, create/delete/update keys.
basis deletion is...just forgetting the password. that's the point,
at least. could maybe extend the basis AAD to include some small
amount of derived data that could be deleted to facilitate basis
deletion...

anyways. next up is...lots and lots of testing.
2021-11-22 04:46:36 +08:00
bunnie
2859e54bb9 that was a long walk to fix a bad architectural decision
trying to allocate key indicies within the dictionary pool in a late-binding
way is hard to do in rust, because of various interior mutability problems.
i guess it could be done but it turned out to be easier to pro-actively
track everything and generate them on demand.

allocations are hard in rust. i guess that's why the alloc thing is
still not stable.
2021-11-22 01:46:53 +08:00
bunnie
e4e5e6c390 more refactoring 2021-11-22 01:19:57 +08:00
bunnie
00339b0c5f refactoring to simplify free key index tracking. 2021-11-22 00:23:53 +08:00
bunnie
193f563782 very basic small key additions working as expected now 2021-11-21 02:21:25 +08:00
bunnie
05b79f066c framework for adding keys in place. now to debug it... 2021-11-20 18:25:50 +08:00
bunnie
23caf88526 prelim code to update/add keys
next up: sync the small key alloc area to disk, and then we can
start testing...
2021-11-20 01:12:10 +08:00
bunnie
6da984e51e slow but steady progress toward implemeting key allocation and access 2021-11-19 04:00:33 +08:00
bunnie
14f4beef98 refactor data structures into smaller files for easier nav 2021-11-18 15:38:12 +08:00
bunnie
8186930e94 commit some thoughts on key allocation 2021-11-18 04:04:03 +08:00
bunnie
e1d00ef57c fix clean state bookkeeping on the basisroot cache record 2021-11-18 02:13:05 +08:00
bunnie
a5da30de37 retire the barque BasisRoot encryption mechanism for a much simpler implementation 2021-11-18 02:10:58 +08:00
bunnie
c62b2a25fb dictionary adding now seems to work 2021-11-18 02:01:30 +08:00
bunnie
507d99ee41 moving toward refactoring the basis sync function 2021-11-18 00:09:55 +08:00
bunnie
70a478795d refactor structure; fix mutability issue with fast space allocations.
see comments in code for more details, but, the Rust borrow checker
found a problem where the virtual to physical map could be concurrently
modified with additional entries during an allocation, so now the code
works on an "look-before-you-leap" allocation basis (where it's assumed
you either ensured sufficient space or allocated as you ran along previously),
versus a "oops we need memory let's stop the world and redo everything
while in the middle of anything" situation. The latter is obviously
more convevient, but, alas, I have to agree with the borrow checker --
probably not the safest way to do things.
2021-11-17 22:29:47 +08:00
bunnie
6fa2332ff5 stash an intermediate set before refactoring to fix a borrow issue 2021-11-17 04:07:11 +08:00
bunnie
bbbaa6b810 clean up the structural definition of the dictionaries and keys 2021-11-16 18:45:24 +08:00
bunnie
49d00c783e basic mounting of the PDDB & readout of system basis 2021-11-16 02:58:33 +08:00
bunnie
d7cc518552 add mbbb mechanism 2021-11-15 19:05:03 +08:00
bunnie
65bbc82ef5 commit to Repr(C), include doc comment for rationale and tests 2021-11-15 16:38:57 +08:00
bunnie
ce51205155 checkpoint with memory placement and disk storage
this probably isn't new to most other people but this has been
a very visceral exercise of how tricky it is to use Repr(c)
for data layouts in anything but the most trivial data structures.

Unfortunately the Basis root data structure needs to be fairly rich,
so we're playing around with using rkyv to help with the serialization
to disk but I'm also not convinced this is a good idea yet. This
checkpoints commits a "working" rkyv implementation, but I'm not
entirely happy with the notion that rkyv can shuffle around data
bits based on its particular version or decisions in the serializer
crate. I'd like deterministic, repeatable serialization that will
be solid through lots of revs...
2021-11-15 16:00:01 +08:00
bunnie
ba9b136f87 debugging alignment issues on serialization of compound structures 2021-11-15 04:03:55 +08:00
bunnie
4a7bdab5ea various debug messages and minor edits 2021-11-14 19:03:07 +08:00
bunnie
93fd991dff fix paths and ignores for the image dump files 2021-11-13 17:34:34 +08:00
bunnie
1ed606fa7d very rudimentary pddb dumper for debugging.
yah, I know. :-/

anyways.
2021-11-12 22:57:37 +08:00
bunnie
f5a5211851 hosted mode simulation; basic disk formatting doesn't crash.
- hosted mode uses an awful, horrific hack to simulate FLASH:
it makes a global singleton that is freely mutable. Oh but wait!
that's actually how the hardware behaves. It's up to the code
around the hardware to create those primitives. So, maybe it makes
sense that the simulation model for the hardware is terribly
unsafe and unprotected.

- the format routine at least doesn't crash and creates some
semblance of a result that looks right-ish. Need to write a
utility that helps to explore the "disk image" so we can
understand what's going on.
2021-11-12 21:15:30 +08:00
bunnie
4b33069ac6 starting on the simulated test/debug of the initial formatting 2021-11-11 23:48:05 +08:00
bunnie
55f2314e7c first pass at a format + passing trivial built-in tests 2021-11-11 23:33:01 +08:00
bunnie
d45369a19d basis root now iterates itself into an encrypted set of blocks 2021-11-11 18:54:17 +08:00
bunnie
62ef3b9414 refactor trngpool to a shared refcell-protected object. encryption iterator for basis.
- TrngPool is going to be more efficient if I have a single pool that's
shared among all the functions of the PDDB. It's split out into its own
object that is refcell-counted.
- Trying to make the Basis handle encryption and decryption with iterator
traits. Let's see how that works out.
2021-11-11 16:30:45 +08:00
bunnie
79d32e29f6 still grinding through the disk format procedure.
- reworked fast space allocation
- shifted order of events around slightly
- formatting is now using "generic" calls to create
the free space structures instead of bespoke single-use calls

left off at getting the system basis written. need to rework
the data structure to be orthogonal with the page table remap
on a page-by-page basis. have to think about it a bit...
2021-11-10 03:42:54 +08:00
bunnie
a4a2d12d57 refactoring the order of format events; consolidate fscb init op
try to condense ops that we anticipate calling in other places
into helper functions.

add AAD to the fscb.
2021-11-09 04:57:12 +08:00
bunnie
da9fcb24ff initial draft of fastspace collection compiles 2021-11-08 18:41:31 +08:00
bunnie
7858aaad0e free_pool scanning first cut 2021-11-08 05:41:39 +08:00
bunnie
81d9fb3fda more incremental progress 2021-11-07 05:34:07 +08:00
bunnie
fabc84aaf6 working through an initial Flash format routine as a pipecleaner
clean compile checkpoint, not much else.
2021-11-07 03:16:32 +08:00
bunnie
778fa8d271 just journaling intermediate work 2021-11-06 04:22:34 +08:00
bunnie
56e82f37d3 more thoughts on the pddb.
this is definitely a rough-draft implementation and is going to
need a v2 or v3 to plug all the serious holes.
2021-11-05 03:58:58 +08:00
bunnie
629803fe72 basic front-end APIs sketched out 2021-11-03 01:42:25 +08:00
bunnie
337afbbe47 bump loader version number 2021-11-02 03:26:16 +08:00
bunnie
edb43a17a3 some structure on a more refined version of the pddb api.
doesn't compile tho, need to figure out what the status of
Path is on Xous.
2021-11-01 04:30:19 +08:00
bunnie
46f1f60cda commit to journal of bad ideas issue #243: a misguided attempt at a PDDB API. 2021-10-31 22:58:20 +08:00
bunnie
b43eb56346 add some skeleton for a net crate; testing COM interrupts 2021-10-08 20:01:13 +08:00
bunnie
e4c79a5f47 start to lay out skeleton for pddb 2021-10-04 14:50:30 +08:00