Commit graph

392 commits

Author SHA1 Message Date
bunnie
3d992cedf4 fix an import error + reduce log noise on a normal path 2022-08-02 01:38:07 +08:00
bunnie
0d61cbb838 improve debug output 2022-08-02 01:37:45 +08:00
bunnie
13b366345c clean up/optimize the key listing routine
provides some marginal improvement on the vault UX
2022-08-01 23:12:26 +08:00
bunnie
826aa43f20 Merge branch 'pddb-flamegraph' into main 2022-08-01 15:46:57 +08:00
bunnie
5be62e372d chore: move AES-GCM-SIV off of branch to 0.11.1 official
this patch moves us off the branch we had to hop on for cipher
API compatibility to the one that's officially published by
RustCrypto. This closes #168.
2022-08-01 15:44:16 +08:00
bunnie
7d62ca85b8 some preliminary schwags at making flamegraphs
turns out, they're pretty useless. I mean, for debugging actual
running code. it tells us basically what we know already which is
99.99% of the time is spent waiting on inter-thread messaging
in hosted mode.
2022-08-01 01:32:19 +08:00
bunnie
e469736278 stagger the flush timing a little bit
noticed it's hitting exactly when NTP updates because it also
happens on a 24-hour schedule, staggering a bit to try and keep
everything from hitting exactly at the same time once a day.
2022-07-29 15:00:44 +08:00
bunnie
46c53d9612 add telemetry for better PDDB CI 2022-07-28 18:46:56 +08:00
bunnie
ac1f746566 add fscb/spaceupdate journal flushing & periodic flush
- fix journal number bug in basis root record
- fix conflict PTE resolution to also consider basis root record,
which has a different encryption form.
2022-07-28 04:54:35 +08:00
bunnie
ec5cc29ccd hotfix for incorrect AES-KWP implementation
The crate we originally grabbed for AES-KWP seems to be non-compliant
(or I'm an idiot and I can't figure out how to call it in a way
that is compliant).

See https://github.com/jedisct1/rust-aes-keywrap/issues/9

Turns out that `aes_kw` *is* compliant, but unfortunately, it didn't
exist back when the PDDB was being written. The PDDB started in October
2021, and `aes_kw` was pushed in Jan 2022. *just* barely missed it.

Anyways, caught this error while trying to implement some CI for
backups. Hurray for testing!

This patch removes the encryption path using the incorrect version
of the algorithm, but leaves the decryption path in place. The
system will try to decrypt initially with the correct version,
but if that fails, it will fall back and try the incorrect version.
If that succeeds, it returns an error code with the correct key and
an updated wrapped key, which the caller can store so that future
calls do not fail. Thus, we have a seamless migration from the
incorrect version to the correct version.

Have verified that the aes_kw version *does* work with the CI
script, insofar as it has been written (we just got to that point
and got stuck). So, full speed ahead. Users shouldn't notice this
change.

Also, fwiw, I did take a look at the "incorrect" implementation,
and it doesn't look obviously insecure. Printing out the various
rounds of the wrapping function looks like the first couple rounds
match, but it diverges somewhere probably due to an off-by-one
in a counter or something like that. I didn't chase it out -- just
wanted to make sure that nothing looked like a huge data compromise.

Good news is this comes right on the heels of v0.9.9 release, so
this will be pushed as a hotfix right away.
2022-07-25 02:56:20 +08:00
bunnie
6f10061653 reset "don't ask again" flag for password creation during factory test
this facility is necessary in case a worker accidentally picks
the wrong setting, disabling the feature for the device. When
the factory test completes, a call is made to reset the parameter.

This was a slightly complicated affair because the the don't ask
flag is contained within the rootkeys enclave itself, and we can't
just give access to rootkeys willy-nilly. So the flag is passed
through the PDDB and then to the rootkeys enclave. It's not harmful
to have it reset by an unauthorized actor; the pop-up will not
happen if passwords are, in fact, already set.
2022-07-19 23:57:25 +08:00
bunnie
e40a109f6c add UX flow dedicated to the 'churn' operation 2022-07-19 16:57:47 +08:00
bunnie
9fc74e3291 strip out debug info 2022-07-19 15:56:16 +08:00
bunnie
e8188309cf add rekey operation for PDDB migrations to new devices 2022-07-19 03:16:37 +08:00
bunnie
a89ae13463 move pddb mounter to status thread 2022-07-11 04:12:44 +08:00
Sean Cross
d6ac900748 pddb: backend: use Instant for access time
Instead of using the ticktimer server directly, use
`std::time::Instant::now()`. This enables tests to be run in hosted
mode.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-06 13:59:35 +08:00
Sean Cross
086c014de2 pddb: close all related filehandles when a file is deleted
Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 23:56:31 +08:00
Sean Cross
ee18fe889c pddb: initial libstd support
Add support for libstd calls to pddb.

The pddb uses `:` as the path separator, and grafts the concept of paths
onto dicts.

Currently implemented features:

    * mkdir
    * rmdir
    * create file
    * open file
    * read file
    * write file
    * unlink file
    * seek file

Other calls may be added as needed.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 22:50:58 +08:00
Sean Cross
6ae91d8e46 pddb: make select_basis and latest_basis non-mut
These do not need to clone data and do not need to be mut. This improves
access time considerably.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 17:23:40 +08:00
bunnie
e26106e334 fix warnings and clean up configs
tighten up the scope of some features to reduce warning messages
2022-07-04 18:13:11 +08:00
bunnie
8565c7efe1 fix FSCB generation issues
- fix problem where we were generating a size based off of total disk
size, instead of fraction of available space

- enumerate FSCB entries into the page table scan. This is because
the page table cache could hold used entries that aren't reflected
in the low-level disk scan, and there is no way to trigger a
flush at the point of re-allocation.
2022-07-04 17:56:54 +08:00
bunnie
baa5097313 fix basis FSCB generation on full - basic fix in place and tested 2022-07-04 15:19:40 +08:00
bunnie
7162b05a7f add routines for testing secret basis
When the "autobasis" feature is selected, the hardware block
will track all basis that have been opened and unlock them when
it comes time to scan/generate more FSCB space. It also bypasses
all UX normally invoked.

This allows us to create tests that open and close Bases automatically,
but of course, totally destroys the notion of plausible deniability.
So, don't turn the feature on for production.
2022-07-04 04:06:53 +08:00
bunnie
8a81954cf0 clean up the page table enumeration
since all the keys are now disclosed, the routine has been
renamed accordingly and the re-scan of existing bases is
redundant.
2022-07-03 01:12:12 +08:00
bunnie
559229d85e fix key enumeration to include keys that we already know 2022-07-03 01:06:56 +08:00
bunnie
e8140f257c make sure the basis list is displayed at least once
need to cover the case that no enumeration was done, and just
show a "null" list confirming basically nothing other than
the system basis is disclosed.
2022-07-03 00:42:16 +08:00
bunnie
b10cf878db fix large key read; and cursor positions in read() and write()
1. the large key read algorithm was broken. this is now fixed. we only
saw this triggered when the large key was much larger than 8kiB (and
the original CI tests just use 8kiB keys for "large").

2. the cursor position in read() and write() were being incremented
by the expected read amount, not the actual read amount.
2022-07-03 00:35:11 +08:00
bunnie
587f7ef39c some enhanced debugging 2022-07-02 00:09:38 +08:00
bunnie
ae37c2cd02 don't truncate key allocations that have an alloc_hint specified 2022-07-02 00:09:21 +08:00
bunnie
a1950a4c00 make a default alloc hint that's not 0
set to 8 bytes initially, could be adjusted up or down.
2022-07-01 19:45:55 +08:00
bunnie
5008c7afac first draft of all basis unlock
This has worked in testing, but:

- There seems to be a new error unrelated to this with allocating
very large blocks of memory, which needs to be traced down.
- Because this operation will typically fully block the caller,
UX redraws can't happen (unless the caller has a fully multithreaded
redraw routine). This needs some thought because:

1. If you have several basis you're trying to unlock, eventually,
you'll overflow the application incoming queue with redraw requests,
and the system will crash.

2. It's cosmetically disturbing to have the dialog boxes piling up
on top of each other during a critical operation like this.

I *think* the right thing to do is to refactor the GAM a bit
to handle pop-up modals that block a bit more gracefully. But
I think it's more urgent to trace down the problem with large blocks
of memory being allocated, since it is a functional issue and not
a cosmetic one.
2022-07-01 17:09:51 +08:00
bunnie
23a31c2cf0 fix PDDB initial stack size and increase heap limit on boot
I had thought there was an issue with heap size limit increases
and multithreading but I think actually the problem is the
PDDB runs pretty close to the stack size limit. This increases
the stack size limit for the PDDB on boot and also bumps up
the heap limit so caching can work more flawlessly.
2022-06-28 19:58:20 +08:00
Sean Cross
7aae6dcab9 pddb: change Option<Opcode> -> Opcode in main
Instead of writing `Some(Opcode::*)` repeatedly, change it to simply
match against `Opcode::*`. This is accomplished by defining a new opcode
`Invalid` that will be matched against as a default.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-27 09:24:11 +08:00
Sean Cross
748d4ce87d Merge remote-tracking branch 'origin/main' into libstd-fs 2022-06-27 09:14:04 +08:00
Sean Cross
8cd04c3415 pddb: rework PddbBuf ordering to be non-packed
The new ordering remains the same size, but is no longer
`#[repr(packed)]`. This allows it to be `#[repr(align(4096))]` which
means we can get away with using a stack page for this struct instead of
allocating a new object every time.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-27 09:13:00 +08:00
bunnie
fd63a33078 add PDDB heap limit increase based on passing a threshold 2022-06-27 04:03:33 +08:00
Sean Cross
1388da7680 Merge remote-tracking branch 'origin/vault' into libstd-fs 2022-06-26 16:48:20 +08:00
bunnie
88ea0145da clean up some debug spew 2022-06-26 03:55:26 +08:00
bunnie
9f409fda2d fix basis search order issues
at least, it passes some simple tests. much more testing is needed.
2022-06-26 03:52:27 +08:00
bunnie
e62c9bf3d3 add basis management functions and debug some basis issues
there's some problems with the automatic basis unionization
feature. now debugging...
2022-06-26 02:11:13 +08:00
Sean Cross
19532c5634 pddb: add explicit discriminants to enums
In order to ensure the API is stable, add explicit opcodes to enums
within pddb.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-25 11:00:41 +08:00
bunnie
4c18662430 add a periodic scrub to the PDDB
this periodically polls the PDDB and has it check the memory allocation
levels, and if too high, invokes a scrub attempt.

This bit is probably the most poorly tested of all the code, but
also some of the simpler logic. The more important part this code
brings is analytics on actual heap and cache usage, which will help
inform us on how urgent/bad things are getting in terms of PDDB
cache usage.
2022-06-17 02:02:11 +08:00
bunnie
dc0a4e5400 add backpressure triggers to the CI tests 2022-06-17 02:01:59 +08:00
bunnie
87efc267f1 add PDDB backpressure items (#109 partial) 2022-06-17 02:01:30 +08:00
bunnie
66dc39c2a8 pddb sync does not require the handle to be mutable 2022-06-06 18:25:33 +08:00
bunnie
17446588bb fix read_all bug in PDDB 2022-06-05 01:45:05 +08:00
bunnie
21116a1693 fix another errant mut that wasn't needed in pddb 2022-06-01 00:06:55 +08:00
bunnie
bf535bd0e1 use interior mutability to handle callbacks
this eases a mutability requirement for the PDDB object
2022-05-31 23:54:16 +08:00
Sean Cross
3ae9d8abf0 apps/services: remove xous::xous_main macro usage
This macro will be removed, since it is no longer necessary. Remove this
macro across the project and replace it with a standard `main()`
function.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-05-31 13:33:30 +08:00
bunnie
1946d38a69 work around https://github.com/RustCrypto/AEADs/issues/410
We're trying to bring the AES version up so we can do CBC
(required for FIDO2) but unfortunately the aes-gcm-siv crate
is locked to an outdated version of AES, and this is blocking
on a type that is missing in the `block-ciphers` crate.

For now, we're using a patched version of aes-gcm-siv from
rozbb's repo, which allows us to move forward, but this needs
to be reverted once a new version of the official crate is patched.
2022-05-30 04:36:33 +08:00
bunnie
2cd4e80d9c bring PDDB to AES 0.8 compatibility 2022-05-30 03:08:41 +08:00
bunnie
e14e6206ed include a getrandom implementation that allows us to use modern randcore
A lot of packages are depending on 0.6 api, which requires a crate
that has OS specific TRNG calls. This patch keeps the 0.5 calls
inside the TRNG crate as direct API calls, and migrates to support
a 0.6 api to a patched version of getrandom that supports Xous.
2022-05-29 02:47:51 +08:00
bunnie
f4a18401e5 expose is_efuse_secured() API via the PDDB
is_efuse_secured() comes from rootkeys, but we don't want
to expose the full rootkeys API to the world.

PDDB already has an approved capability for accessing rootkeys,
so this single API is passed through via the PDDB object here,
allowing other procesess to query it.
2022-05-28 18:21:07 +08:00
bunnie
62cbc86aeb add automation strings 2022-05-24 23:21:26 +08:00
bunnie
8af8926ad1 make disk sizing errors non-fatal in hosted mode
it can be innocuous in the case that you're changing sizes
for debugging
2022-05-10 17:37:08 +08:00
bunnie
ea6cf27ed9 fix PDDB length field -- somehow this got flipped from the debug branch 2022-05-06 14:43:29 +08:00
bunnie
ee0f443883 hoist net thread exhaustion check to before we check the poll interval 2022-05-06 14:17:29 +08:00
nworbnhoj
0e561802ce enable notification modal text + qrcode
change the notification modal to accept the primary message and an
Option of independent text to display as a qrcode

pub fn show_notification(&self, notification: &str, qrtext:Option<&str>) -> Result<(), xous::Error> {}
2022-04-24 21:29:15 +10:00
bunnie
34c2ffe1a3 fix password entry to work with the new return type
also requires a call to `reset_action_payloads` for the
number of field elements to be recorded correctly!
2022-04-22 17:43:26 +08:00
bunnie
4ddd414ed6 fix modals API in the PDDB migration code 2022-04-22 15:55:15 +08:00
bunnie
2c3143f790 Merge branch 'main' into multi-text-modals 2022-04-22 15:54:18 +08:00
bunnie
38d6765586 fix remaining bugs in migration, tested and passing on CI + real hardware 2022-04-20 03:48:05 +08:00
bunnie
b3c89e5f09 add migration of secret basis for v1->v2; fix some interior mutability issues
Probably nobody is really using secret Bases right now, but it's
probably a good idea to lay out all the infrastructure and start
testing it for when it comes time to do it with data that really
matters.
2022-04-19 04:50:15 +08:00
bunnie
4ba34bbd58 intermediate work on v1->v2 migration, need to set aside to fix a libstd problem 2022-04-19 00:45:10 +08:00
bunnie
c8a38b4bf5 fix representation of va in pagetable so we're not burning 12 bits on 0's
this is a v2 fix, incompatible with v1
2022-04-19 00:44:45 +08:00
bunnie
7553053f29 v0.0.2.1 initial implementation - still needs migration code
Previous version was v0.0.1.1
New version is v0.0.2.1 (don't ask me why I skipped v0.0.2.0)

The most significant change is that the key derivation for the basis
is now split into two keys. One key is used for the page table ECB
encoding, and a different key is used for the data AES-GCM-SIV encoding.

This is because someone much smarter than me looked at my scheme
and said it is very dangerous to use the same key for two different
types of ciphers. This person is correct, so it had to be rectified.

For the System Basis, the keys are encrypted using the root keys
wrapper API. In this case, we simply create two random keys, one for
the page table, and one for the data region, and wrap them using
the root keys API.

For user secret Basis, the derived keys are sent through HKDF
with two different "info" strings to derive a pair of keys from
a common password.

In case the cryptographer who looked at this is reading along,
note that the basis name is encoded as a fixed-length, 0-padded
record, so, we don't encode the length of the basis in the hash
used to derive the keys. When I explained the scheme I thought
I was using a variable-length basis name to derive the key, but
when I checked the code, it's always 64 bytes long, and bytes
that are not used are 0-padded, so, I think we don't need the
length encoding field.
2022-04-18 15:59:15 +08:00
bunnie
cd8c762c55 patch up QR code PR, change method signature as discussed
now any modal notification can also render as a QR code
2022-04-08 15:19:56 +08:00
Gianguido Sorà
8123393a7d Merge branch 'main' into feature/multi-text-modals 2022-04-07 16:39:03 +02:00
bunnie
d49c35dc0c verify that unmount callbacks are working 2022-04-07 17:58:37 +08:00
bunnie
52d56f39f1 wire up PDDB key unmount callbacks (oops) 2022-04-07 17:37:25 +08:00
bunnie
6ac4ef292b add basis mount/unmount to pddb shell commands, fix some UX formatting 2022-04-07 15:32:47 +08:00
bunnie
1ad7c43f81 refactor pddb ux thread to use deferred response messaging 2022-04-07 14:46:44 +08:00
bunnie
92c6a93147 eliminate yield loop on blocking mount poller 2022-04-07 00:04:25 +08:00
bunnie
b75df1efb3 annotate rationale on multiple variants of pddb mount calls 2022-04-06 23:46:39 +08:00
Gianguido Sorà
83e61f5f1b feat: add wiring for multiple TextEntryPayloads drawing 2022-03-31 17:54:22 +02:00
bunnie
8b5ad2657b an aborted attempt at doing a clean-shutdown for hosted mode
This doesn't work because the shutdown call destroys all the servers then terminates
the processes. But, the problem is that the processes are still
running, but servers are disappearing. So what you end up with is a
panic when the first message gets delivered to a server that no longer
exists, and then you end up with the program hanging on that panic
because some servers haven't been destroyed/exited yet and are now
just spinning waiting to be scheduled. so the call works in the tidy
kernel test cases but to get it to work on a full OS you'd need to
have a way to put every server into an eternal blocking state before
making the shutdown call (to prevent new messages from being issued
while servers are going away), which is more work than i'm willing to
put into the problem right now.
2022-03-31 01:10:58 +08:00
bunnie
4f1f4a3907 optimize journal resolution code
We have the journals tracked in a HashSet, so, use the hashing
properties instead of an O(N) search.
2022-03-29 16:39:54 +08:00
bunnie
0bdf21b3ac retire debugging infos 2022-03-29 16:33:40 +08:00
bunnie
c542807958 fix PDDB bug where journal numbers on pages were incorrectly restored
An edge case was missed where a page which was allocated out of
the FSCB and then later written to disk was having its journal
number set to 0.

The root cause of the logic issue is that the page table itself
does not store a journal number. It's assumed that the page table
is ground truth, and is unjournaled.

As long as the pages don't exist in the FSCB, that's not a problem,
the journal number doesn't matter.

But, let's say this sequence happens:

1. FSCB allocates some free space
2. process takes that free space allocation and commits it to its own page table
3. FSCB update log gets a note that free space is now allocated
4. system reboots
5. process reads its page table from disk. There are no journals associated
with the page table on disk, so, it assigns a journal number of 0 by default.
6. we now have a conflict between the Process' view of the journal age
of that page table entry, and the FSCB's view.

The fix to this is during the page table scan, to check and see if
any candidate pages happen to already exist inside the FSCB. If it does,
then you adopt the latest journal number for that page in the FSCB.
This keeps the two views consistent.

A second bug was also identified while fixing this where the "Random"
order we were hoping for in visiting the free space cache as an
iterator is less random than we had hoped. So, there is a patch to
extract the free pages and pull one random one from the pool, using
entropy coming from the TRNG. This is less computationally efficient,
but at least it is correct and has more transparent properties
than relying on the indeterminism of the HashSet implementation.
2022-03-29 16:26:55 +08:00
bunnie
bdd194ea7b retire rust issue 90195 work-around in PDDB 2022-03-29 16:25:33 +08:00
bunnie
4eec54620c fix warnings and debug messages 2022-03-29 04:39:33 +08:00
bunnie
9a41b18400 fix bug where dictionaries with 1 entry fail to sync
also ensure that at least 1 byte is reserved for any key
that is allocated; this helps avoid edge cases dealing with
zero-length key entries later on.
2022-03-29 04:38:15 +08:00
bunnie
55b05cfd80 add debugging routines; fix zero-length key creation case
- add a "dangerous debug" command that only exists when the feature
"pddbtest" is enabled. The feature can't work on real hardware, but
still, it enables things like dumping of the database and storing
the keys for inspection, so it's isolated in a feature that prevents
the code from being compiled in the first place when it's not selected.

- fix a bug where keys that have zero length aren't being created
correctly. this is a common use case for reserving some space for
data that isn't yet available.
2022-03-29 01:47:28 +08:00
bunnie
18c202dcee start of testing code for the PDDB 2022-03-28 22:56:32 +08:00
bunnie
c69b9e451d unbreak hosted mode 2022-03-28 04:04:38 +08:00
bunnie
ae0b1f1288 fix bug in dict delete.
also soften a journal fault message to help with debugging.
that it no longer panics does not mean it is right...
2022-03-28 03:36:27 +08:00
bunnie
9555287273 remove a debug print 2022-03-27 17:07:48 +08:00
bunnie
727135b347 throttle debug messages 2022-03-27 16:44:26 +08:00
bunnie
fc4a598a9b add a command to reset RTC after a PDDB format 2022-03-27 16:44:13 +08:00
bunnie
515d49db08 add a sync operation after every write
I did observe some PDDB corruption that happened today that looks
ostensibly linked to a record that didn't sync up with the disk,
so, let's be safer and sync everything. We can work on optimizing
performance after correctness is guaranteed.
2022-03-27 02:55:19 +08:00
bunnie
a103536b58 add a non-blocking mount poller to the PDDB
The problem with the current is_mounted() call is that it blocks
while the PDDB is requesting a password or actively formatting
(otherwise it doesn't block). It's actually just...partially blocking.

That doesn't cut it for libstd-linked system services that need to run but
before the PDDB comes up. In particular, we need to be able to respond
to time queries *somehow*, even before our timezone and offsets are
loaded from the PDDB.

I considered doing a callback mechanism for mounting, but this
is kind of complex and probably not worth it. Instead, a separate
thread that can return the value of a single boolean revealing
the mount state of the PDDB is created, and you can create an
object that's capable of inspecting that boolean.
2022-03-27 02:52:29 +08:00
bunnie
91d0511fa4 add Seek trait to the PDDB 2022-03-26 02:34:57 +08:00
bunnie
0abd4a1db6 add a response code for dictionary not found 2022-03-25 23:50:26 +08:00
bunnie
92f1552d94 Merge branch 'main' into bunnie-dev 2022-03-24 03:32:36 +08:00
bunnie
0bcc93c78e attempt to add persistent storage to hosted mode PDDB
This will create a 100MiB file in tools/ppdb-images/hosted.bin
(sorry!!) which is persistent storage for hosted mode.

Delete the file if you want to start over from scratch.

It's...pretty brittle. I expect problems, but maybe it will at
least get things moving.
2022-03-24 03:21:00 +08:00
bunnie
bfb8a17cd5 strip out legacy RTC calls in preparation for refactor 2022-03-24 02:19:26 +08:00
bunnie
36ea69f6df add a one-liner API call to block execution until PDDB is mounted 2022-03-22 22:18:23 +08:00
bunnie
86e86df68c add a one-liner API call to block execution until PDDB is mounted 2022-03-22 22:16:46 +08:00
bunnie
9f6e444e3f back out testing structures 2022-03-10 23:45:49 +08:00
bunnie
e8b13fcb44 integrate TTS into more dialog boxes and modals
also tighten up the `blocking` method for TTS for prompts that
absolutely should not be truncated or overriden.
2022-03-10 23:40:35 +08:00
bunnie
06e391e541 fix key commitment check to be constant time
thanks again to Michael Rosenberg for catching this bug and
teaching about about how to use the Subtle/.ct_eq() traits to
implement constant time equality checks.
2022-03-02 19:00:52 +08:00
bunnie
dcbe9f122a bump the PDDB version field to demarcate the breaking change 2022-02-28 04:07:45 +08:00
bunnie
39f9e656cf suppress a debug message
key commit seems to work on real hardware. will promote for
release status with a bit more testing
2022-02-28 03:57:16 +08:00
bunnie
0b454cb7a8 evict salamanders from AES-GCM-SIV in the PDDB
So you'd like to think that an authenticating cipher would like,
authenticate stuff on decryption. And AES-GCM-SIV was billed as sort
of AES with the pointy edges taken off.

Well, there's always one more bug, actually this time a salamander:
https://eprint.iacr.org/2020/1456.pdf Apparently one can forge
authentication if the attacker can control keys and ciphertext. In the
case of the PDDB I think the attack is largely theoretical, because
you'd have to forge through both the AES-GCM-SIV and the rather
insufficient nonce/murmur hash check on the PTE that locates the page
to be decrypted.

But it's never good to have cryptographic properties depend upon
something that's known to be insufficient, so let's assume the attacker
can simply forge keys and stick some data on the device.

This wouldn't be a problem for confidentiality but it could lead to
the ability to plant evidence, but more importantly could lead to code
paths that were thought to be protected against fuzzing being fuzzed
as you could conceivably create a Basis root record that "checked out"
and then the larger body of the PDDB code would try to chomp on other
pages that are forged, leading to a bigger attack surface. Plus who
knows what else might exist beyond salamanders...

This patch does a "key commitment" on the Basis root page, so that it
is now cryptographically hard to create a salamander.  Basically
instead of using the system root key directly in AES-GCM-SIV, the key
goes through a KDF that expands it with yet another nonce plus a bit
of header data. The KDF is SHA512/256 and it expands it twice,
changing one bit in the header data (as prescribed by the paper cited
above), to create a version of the key that is written to disk (the
commitment), and a version of the key that is used to
decrypt/authenticate the AES-GCM-SIV data. Only if both the commitment
and the authentication check out, do we accept a Basis root page as
valid.

I had some concerns about the slightly different structure of the
Basis Root page being detectable, but I have been assured that the key
commitment form and the MAC in AES-GCM-SIV satisfy a property known as
IND$-CPA ("indistinguishable from uniform randomness by a
chosen-plaintext attacker") by Rogaway[1], and, more strongly, "Secure
nonce misuse-resistant authenticated encryption (mrAE)" by Gueron and
Lindell (see appendix for def)[2]. Still, just out of total paranoia
and nothing else, I organize the basis root page such that the MAC is
in the identical location as the other non-committed pages...just in
case there is some mathematical property of the MAC that could be
leveraged to leak info about the existence of the basis root record.
The commitment nonce and sha512/256 result I understand better and
feel pretty comfortable that they are going to be strongly random and
indistinguishable from AES ciphertext.

Thanks to Michael Rosenberg for taking the time to explain all this to
me and to guide me through the process of making the fixes. It is a
major PDDB API-breaking change and I'm glad it's able to be rolled out
before most users have received a device.

[1] https://web.cs.ucdavis.edu/~rogaway/papers/ad.pdf
[2] https://eprint.iacr.org/2015/102
2022-02-28 03:13:26 +08:00
bunnie
5ef0d9f159 narrow up the time for pddb to request password box
now that excess redraws on boot have been ferretted out
2022-02-05 19:00:34 +08:00
bunnie
fd5c6568d1 trim out some excess redraw calls that are slowing down boot 2022-02-05 18:55:46 +08:00
bunnie
2a2e305c7b visually better with the delay, but need to figure out why it takes so long now. 2022-02-05 02:46:24 +08:00
bunnie
3b6a696fc4 minor fixes, push for CI check 2022-02-05 02:40:41 +08:00
bunnie
b4a34bd9ec add busy-wait queuing for alerts in case multiple alerts pop up
this case happens when e.g. you're asking for a password but
then the connection manager throws an error.
2022-02-05 02:03:56 +08:00
bunnie
08421999ad part-way through a refactor of trust levels to prevent alerts from popping over password boxes 2022-02-04 23:59:20 +08:00
bunnie
b35f2a1d3e catch and handle fat-finger UX events better
they used to panic, now they just print a warning
2022-02-04 18:22:17 +08:00
bunnie
b498669003 use NIST SP800-38F (AES-KWP) to wrap the system key for the PDDB
this is instead of our lolfail ECB-based method with a check block.
2022-02-03 18:24:06 +08:00
bunnie
13de52fa8d TcpListener debugging, just a little bit
Spent a couple hours chasing down a "bug" which is acutally just
something about how http works that I don't understand. TCP was
working just fine but there is ... idk some sort of keep alive
thing that occupies a socket so the "web server" blocks waiting
for a read. Anyways, I guess the "right" way to solve that would
be to make a multi-threaded web server

OH GOD STOP

right. it's just a demo to prove that TCP works. done.

Open an issue if you find a problem with TCP kthx!
2022-02-02 00:23:14 +08:00
bunnie
827888ccbb fix up the REFCOUNT on connections
Was doing disjoint loads and stores on an Atomic, really, should
be using the fetch_add, fetch_sub variants to be proper.

Didn't seem to impact anything but maybe there's a future version
of me that high fives me for doing this.
2022-01-31 23:13:10 +08:00
bunnie
6653d42114 cleanup UX context registry
it's now a much smaller nightmare to maintain the registry, as all
the items are concentrated into a single location
2022-01-23 14:31:21 +08:00
bunnie
2642e6c075 various UX fixes that take advantage of the new font rendering engine 2022-01-22 00:10:31 +08:00
bunnie
356d95a266 yet another refactor of the word wrapping
Really clear up the roles and structures of word wrapping.

Significantly, we have now figured out that the word wrapper
should be the word wrapper, and that the text bounds should
be computed by a text bounds computation.

Conflating flexible text bounds with word wrapping makes
the math really hard.

So, the word wrapper always works with a bounding area starting
at 0,0, and will wrap until it hits a Point designated as its
absolute max bounds.

The returned typeset block is referenced to 0,0, and then it
is shifted to the correct location by the rendering function.

This externalizes the computation of the final location plus
the rendering bounds to the outer functions that need to care
about this, thus simplifying the word wrapper significantly.
2022-01-20 18:30:13 +08:00
bunnie
18130cddae fix suspend/resume regression
had to pin down the correct order of events to avoid lockups on suspend
2022-01-17 22:46:18 +08:00
bunnie
2c711036a9 add a submenu for PDDB operations 2022-01-17 01:59:38 +08:00
bunnie
cf06d68cf7 add the pddb menu into the source tree 2022-01-16 18:27:43 +08:00
bunnie
b7b0bdc852 convert menu infrastructure to the "menumatic" API call
hopefully this makes menus a bit easier
2022-01-16 17:30:32 +08:00
bunnie
f53f86077d add a blank check step before issuing a sector erase
this should significantly speed up the initial on-boarding
for first-time users
2022-01-12 02:33:09 +08:00
bunnie
49110a9176 add a comment about some directives 2022-01-11 12:16:25 +08:00
bunnie
994e3d7cfb refactor llio crate (phase 1): organize files into sub-systems 2022-01-11 12:02:27 +08:00
bunnie
eb0c1c4377 fix some hosted mode brokeness that came with pinning crates
the rand api was downgraded to be in line with the ed25519 crate
which removed some "modern" conveniences. Small price to pay for
less code to review.
2022-01-10 17:13:00 +08:00
bunnie
c5fe1bc99b add sequencing to suspend/resume order of operations
this is necessary because we want to make sure that the SPINOR
service does not shut down before the PDDB service finishes syncing
its disks.
2022-01-10 13:45:56 +08:00
bunnie
dd18d94a03 e2e demo of pddb working now, we can store an ssid/password pair...finally. 2022-01-09 13:12:36 +08:00
bunnie
5523d81669 add commands to check mount status and to initiate mount manually 2022-01-08 23:33:10 +08:00
bunnie
00ce905088 clean up hosted mode flow 2022-01-07 23:33:53 +08:00
bunnie
904847fc89 improve error message 2022-01-07 23:25:53 +08:00
bunnie
aa5447cb4a add missing feature: basis retention policy 2022-01-07 22:51:02 +08:00
bunnie
3f4fea1274 clean up some easy warnings 2022-01-07 22:19:39 +08:00
bunnie
b4bfaefa64 start a cleanup pass on warnings 2022-01-07 22:10:54 +08:00
bunnie
11c22b6d19 add listing routines 2022-01-07 21:03:40 +08:00
bunnie
7d660a3592 add key attribute query, clean up some warning 2022-01-07 16:18:44 +08:00
bunnie
fa7a9db348 add more key manipulation APIs 2022-01-07 15:43:50 +08:00
bunnie
2118233657 diagnose alignment bug in CI code 2022-01-06 23:16:01 +08:00
bunnie
c046422746 tighten up the on-boarding password flow
- check the password with a double-entry on the first boot
- if we see a blank disk, bypass the usual checks and allow the mount
to fail, then, enter the setup routine.
2022-01-06 22:13:00 +08:00
bunnie
4c927f2c99 fix name records so they encode a length
This is one of the truly redeeming features of Rust is that a
major refactor of a core structure like this doesn't leave me
shaking in my boots -- I feel pretty good that this works
since everything passes the CI tests.

The original FS implementation did the "stupid C thing" which
just creates a fixed lenth field and uses null-terminated C
strings. Which actually restricts PDDB entry names to be ascii
as valid UTF-8 can encode single-byte nulls.

The intention was for names to be fully utf-8 compliant, but in
order to do that we have to burn one byte of potential filename
length and encode an overall length field for the name strings.

This change does that, which both makes the code a bit safer
at the expense of losing a character off of the maximum filename
length.
2022-01-06 18:53:01 +08:00
bunnie
233a7cb7d3 getting started on dict management, but just realized we did strings in a dumb way
we really should have coded a length field in all the name string
stored in the PDDB (derp, old C habits die hard).

let's fix that before we move on.
2022-01-06 17:12:30 +08:00
bunnie
d318e31666 flesh out the basis management opcodes 2022-01-06 16:36:19 +08:00
bunnie
20e28395d8 start to lay in the end user API, beginning with basis management commands 2022-01-06 14:50:56 +08:00
bunnie
489a26dd11 fix an issue where we don't decrement the connection counter on drop 2022-01-05 20:01:54 +08:00
bunnie
8b37303b06 add a check on correct password entry
The rootkey enclave actually doesn't check if your password is
correct -- it has no way of knowing because it doesn't know how
the keys will be used. Password checking is delegated to the
requesting process because also the password retry method and
policy depends upon the application, and is a significant amount
of UX plumbing (in fact the rootkeys should have less UX plumbing
than it has right now - this is something to do in the near future).

Add a mechanism in the PDDB to check if a password was entered
correctly, and retry if it isn't. Without this, an incorrect
password would actually trigger a reformat of the PDDB, since
it can't find the system basis structure with a bad password.
2022-01-05 11:17:13 +08:00
bunnie
c6bba1b13e increase PDDB size to the full disk size; test on real hardware
takes about 15 minutes to fully initialize a 100MiB PDDB.

time is dominated by the sector erase time (which is fundamental
to the device) and data write rate (also fundamental to the device).
2022-01-04 21:59:26 +08:00
bunnie
5b66e692d3 fix cipher sync bug; reduce logging rate
- logging rate reduced for UX for significant speedup
- fix issue where on AES key re-generation a pre-keyed cipher
was not being retired.
2022-01-04 11:11:36 +08:00
bunnie
de991df43c fix slicing to match that of the Xous environment 2022-01-04 10:50:56 +08:00
bunnie
0844b3e961 add formatting check and UX flow for formatting; redo salt (again)
probably should have split this into two commits. oh well.

first part introduces a check to see if root keys are initialized,
and if so, tries to mount the PDDB. If it can't mount, it asks
the user if they want to format it. It also updates the progress
of the fromatting.

the second part redoes the salt once again. After some discussion
SHA512/256 is also resistant to length extension attacsk so we
don't need to use the HMAC technique suggested. Would prefer to
reduce the number of algos to analyze, so we're going to a simple
SHA512/256 hash of the salt/basis name/plaintext password to generate
the salt that goes into the bcrypt routine that ultimately generates
the final key used by the system.
2022-01-03 16:51:51 +08:00
bunnie
2317c71e3a perhaps an ill-fated exercise to randomize journal number starting points
trying to avoid a situation where inspection of the system basis
can lead to conclusions about how old the system "should" be.

This is of course slightly upset by the fact that we need to track `Age`
numbers within keys to implement LRU caching algorithms (coming real soon
now), but at least this decouples the pages' low-level journal entry
count from the age parameter.
2022-01-02 20:26:36 +08:00
bunnie
7094c00590 update comment 2022-01-02 19:43:44 +08:00
bunnie
6d655c8085 convert salt to use an HMAC instead of SHA to harden against length extension attacks
(as reccommended by reviewer)
2022-01-02 19:37:30 +08:00
bunnie
9910547e21 modify salt generation scheme
based on feedback that the current mechanism is overly complicated
and doesn't add any real security, and it's better to stick with
something simple e.g. just a tried-and-true hash of the basics
2022-01-02 17:53:00 +08:00
bunnie
0965c4ccc6 remove suspend/resume opcode, we shouldn't need to be aware of it... 2022-01-02 16:44:44 +08:00
bunnie
350276c08a clean up more warnings and errors 2022-01-02 15:31:56 +08:00
bunnie
421a3814d8 clean up some warnings before moving on to further integration 2022-01-02 15:07:48 +08:00
bunnie
c2ee9f7c7a refactor tests and CI into a feature 2022-01-02 05:21:05 +08:00
bunnie
33b0593f9a fx fspace cache entry space check issue
was counting total number of pages, not actual free pages.
2022-01-02 01:41:55 +08:00
bunnie
fef355423b fix CI red herrings; add more debug granularty 2022-01-01 15:14:44 +08:00
bunnie
656f17f621 add extensive CI test 2021-12-29 05:49:07 +08:00
bunnie
6990513278 fixed a subtle bug in the pte sync algorithm
was mixing deletions and updates and because they are in arbitrary
order due to the hashset structure, sometimes i'd delete an entry
after it was updated.

this is now found and fixed.
2021-12-28 06:47:33 +08:00
bunnie
713c0d6e73 add a multi-iteration CI loop around the pddb test
this allows us to capture seeds and corners of interest
2021-12-27 16:15:51 +08:00
bunnie
dd14dfc4c6 retrofit the seed into an atomicu64 so we can set it elsewhere 2021-12-27 02:49:08 +08:00
bunnie
809dbbdbc2 fix a bug in the reserved amount of data in large keys 2021-12-26 16:29:35 +08:00
bunnie
0bd110c163 determinism achieved for testing. now to squash some bugs. 2021-12-26 01:50:30 +08:00
bunnie
37b46fae98 make determinism an optional feature 2021-12-26 01:42:36 +08:00
bunnie
d4027fa8a2 sort all the lists in the test code so the vectors are deterministic
not converting core structures from HashSet to BTreeSet though --
the idea is that in production the mappings should be fully random,
but we are just trying to get reproducibility in hosted mode so
that we can debug some hairy edge cases more easily.
2021-12-25 16:44:24 +08:00
bunnie
66606ed3e0 initial attempt to make hosted mode more deterministic
replace all threadRng with a seeded Rng
2021-12-25 15:58:27 +08:00
bunnie
c1c57edfd6 some mods to the logging infra to chase down a "rare" issue 2021-12-22 07:09:30 +08:00
bunnie
5d226a002a prepare to start using the modals server for the pddb 2021-12-17 03:11:06 +08:00
bunnie
725d943198 add interlocking so multiple threads don't interfere with each other 2021-12-17 01:55:36 +08:00
bunnie
dcad15284d refactor tests into a separate file 2021-12-16 01:48:11 +08:00
bunnie
60d6d08933 basic framework for modal API wrapping in place 2021-12-15 13:46:24 +08:00
bunnie
de4e84abf5 starting in on the larger system integration of the PDDB
Trying to find a new approach to making dialog boxes that
are cleaner and don't require as much code repetition. Probably
about to go on a long tangent.
2021-12-13 14:24:14 +08:00
bunnie
c84ed61ecd basis overlay test done & passing
the formatting of the listing output is a bit messed up but
i'm too tired and too late to deal with that. at least the
core logic seems functionally correct!
2021-11-30 00:38:29 +08:00
bunnie
8066b8fb5e fixed the bug with the pt map encryption, now we hvae two basis!
more testing needed, but looking good so far.
2021-11-29 00:25:09 +08:00
bunnie
c15841cfdc test framework in place to look at multiple basis
found one bug already, need to refactor how PT encryptor is
handled
2021-11-28 23:54:58 +08:00
bunnie
5a12e83ff2 add first cut at basis key derivation function 2021-11-28 20:42:26 +08:00
bunnie
af9f4cbde3 back out changes for stress testing the fastspace mechanims 2021-11-27 21:23:42 +08:00
bunnie
185d629b98 stress test fastspace allocation and regeneration
- confirmed that log entries regenerate on trigger
- confirmed that fastspace cache regenerates and contains a
new subset of blank pages.
2021-11-27 21:17:41 +08:00
bunnie
93ee050079 fastspace regen debugging -- passes test, but needs optimization
should be able to get away with "just" regenerating the fast space
log only if that's what we've run out of; however, doing that optimization
we run into out of space errors. maybe we need some tuning of the thresholds
for when to reclaim space
2021-11-27 02:19:57 +08:00
bunnie
e02df18374 fix reporting error in directory listings 2021-11-26 03:36:36 +08:00
bunnie
468e372bfa most tests passing, also, be a little smarter about cache fills. 2021-11-26 03:08:08 +08:00
bunnie
0df5f1c926 fix key de-allocation bug
forgot to reduce key count for the dictionary record when
a key is removed.
2021-11-25 22:03:37 +08:00
bunnie
85e48c80ed small keys had their data missing because i deleted it :-/
it's one of those bugs where you're like oh...right. it's doing
exactly what i told it to. I just told it to do the wrong thing.

Right. If you're updating data at the end of a file, you kind of
don't want to delete the file entirely then do the update.

Fix the code so that in the case of a file extension of a small
key, you're re-inserting the key in its entirety (not just the
updated portion) on the extension case.

Also handled an issue with large files where key extension
should not lead to overwrite of the neighboring data, and put
a cap of the large file limit of 32GiB. See comments for thoughts
on the trade-off of the large file limit versus the allocation
limit of the number of files and the physically realistic
outcomes for the PDDB. Basically as written the PDDB can handle
32GiB files (big enough to hold a blu-ray movie, but not a 4k/8k
movie, uncompressed movies, and/or large virtual machine images),
and can handle creation of files at an average rate of one per
second for 10 years without stop before running out of virtual
memory space.

This isn't "Web scale" as large servers will
definitely require files bigger than 32GiB these days, and
can easily create files at a rate of thousands per second. But,
the PDDB isn't performant enough nor targeted toward large
scale server applications. It should, however, be adequate for
pretty much any user data on a mobile phone if it were recompiled
for a 64-bit target. And, if you needed to scale up to bigger
files and/or more write cycles, the problem can be solved
with some simple memory allocation tricks on the large pool,
but because Precursor only has 16MiB of its own RAM to play
with, we don't implement that as the RAM overhead is both
detrimental and unnecessary at our scale of operation.
2021-11-25 17:56:39 +08:00
bunnie
ca485d5541 fix off-by-one in small pool to dictionary storage mappings
consolidate index math into helper functions to reduce errors.
2021-11-25 03:14:29 +08:00
bunnie
9d83b37818 unit tests for the free key index tracker 2021-11-24 14:50:44 +08:00
bunnie
8de0d70f43 debugging the patch-extend test.
probably should make a bunch of unit tests for the free dictionary
key tracking logic, it's too tricky to test out.
2021-11-24 03:57:50 +08:00
bunnie
800fcbe3bf fix page table consistency issue -- needs more testing to see if we licked it 2021-11-24 01:50:15 +08:00
bunnie
66a5b97705 revise pagetable entry to be 7 bytes wide -- this will allow
us to have a full 64-bit address space upon expansion of the page
number (turns out this is probably important because of the crappy
increment-only allocator used for the large blocks)
2021-11-24 01:49:22 +08:00
bunnie
42e926fcab fixes for patch update test
lots of corner cases found and ifxed
2021-11-23 23:33:06 +08:00
bunnie
472d60ae95 update a comment in the readme. 2021-11-23 15:55:27 +08:00
bunnie
cbf7ee032f reorganize documentation 2021-11-23 15:47:40 +08:00
bunnie
9c54152f4d patching test written
need to figure out how to compute if it passes; it of course, breaks
the checksum attached for CI purposes. Maybe it's just a matter
of reading back the data and patching it again -- works things even harder!

also, ran the manual test case on real hardware tonight, after 389 hours
of simulation-only development. happy to report that it worked as expected,
so there's no huge obvious gaps between simulation and reality.
2021-11-23 04:36:58 +08:00
bunnie
d716fea17a now passing dictionary add/remove test 2021-11-23 02:57:13 +08:00
bunnie
eda60a5d9c base case test done, with CI checking 2021-11-22 18:26:19 +08:00
bunnie
61b5aeba98 add ability to re-run pddb test cases in hosted mode for more test automation 2021-11-22 16:28:26 +08:00
bunnie
26736c09b6 start to build out tougher test cases 2021-11-22 16:04:50 +08:00
bunnie
151e054314 clean up a whole bunch of warnings 2021-11-22 16:04:31 +08:00
bunnie
1874c8e72d most of the core routines are now in place
can create/delete dicts, create/delete/update keys.
basis deletion is...just forgetting the password. that's the point,
at least. could maybe extend the basis AAD to include some small
amount of derived data that could be deleted to facilitate basis
deletion...

anyways. next up is...lots and lots of testing.
2021-11-22 04:46:36 +08:00
bunnie
2859e54bb9 that was a long walk to fix a bad architectural decision
trying to allocate key indicies within the dictionary pool in a late-binding
way is hard to do in rust, because of various interior mutability problems.
i guess it could be done but it turned out to be easier to pro-actively
track everything and generate them on demand.

allocations are hard in rust. i guess that's why the alloc thing is
still not stable.
2021-11-22 01:46:53 +08:00