Commit graph

521 commits

Author SHA1 Message Date
bunnie
78866199ea fix zero-length key readback error in bulk reads
zero-length keys return a data section of None, instead
of Some(<empty vec>), as the latter representation deserializes
to a null pointer panic.
2023-01-25 23:17:30 +08:00
bunnie
b7d5eca720 fix missing corner case in cache refill 2023-01-25 13:50:06 +08:00
bunnie
c083a9ca85 chore: fix typos 2023-01-25 00:55:32 +08:00
bunnie
71a1dddf15 fix some more edge cases in PDDB key deletion 2023-01-20 18:07:24 +08:00
bunnie
ea913a6fb4 fix initialization order so that CI can run 2023-01-20 18:07:09 +08:00
bunnie
fb648b4e21 Merge branch 'maybe-fix-delete' into main 2023-01-19 23:53:49 +08:00
bunnie
0d65d9a7ca chore: fix another typo 2023-01-19 15:04:00 +08:00
bunnie
71294f7591 add a missing sync after key deletion 2023-01-19 14:46:25 +08:00
Sean Cross
dc4ca79834 services: pddb: use an enum for file types
Rather than hardcoding numbers, use an enum with the constant values.
This will make it easier to reason about what the constants mean.

Signed-off-by: Sean Cross <sean@xobs.io>
2023-01-18 09:06:16 +08:00
bunnie
232a4b6c6c bump xous version 2023-01-18 00:02:46 +08:00
bunnie
41d4837551 bump versions to absorb WDT change 2023-01-06 17:48:12 +08:00
bunnie
de231cae54 explain to users that they will need to create a PIN code for logins
Users are confused when they are prompted to enter a PIN code for
the first time with no preparation. Update the format request message
to prime users for the task.
2023-01-05 16:42:12 +08:00
bunnie
bb5d7b3adf bump versions to absorb hosted mode fix 2023-01-05 14:07:24 +08:00
bunnie
36398810dd bump version numbers to absorb new syscalls 2022-12-30 15:24:36 +08:00
bunnie
f71278b422 fix issue #109 - refactor key fill/scan to not crash after pruning
After a prune, key fill/scan can crash because there is an assumption
that data is always resident in the cache for a small key.

This code handles the case where the small key data was evicted during
a prune, and reads the data back in just for that key on demand.

Furthermore, during a `sync` call, it will pull in the key data
that corresponds to the page containing just the small key, allowing
that page to be re-composed and replaced in memory. caching is used
on the decryption of the small key page so the expensive bit
of the call should only happen once per small key bank.

tested with pddb-ci and manually in hosted mode.

A portion of code was re-written to make it clear that only the
length of allocated data is copied to/from disk and memory, instead
of the reserved space as the previous code would indicate. The reserved
space is guaranteed to be zero because the RAM-based page for composing
the small key block is zero-ized prior to operations; left comments
reminding me of this in the future so I don't accidentally optimize
something out and cause data leakage between small keys because I'm
allowing the reserved areas to contain stale data.
2022-12-29 22:04:55 +08:00
bunnie
e25da3a720 add implemenation of the prune debug call
this allows us to trigger the cache prune manually, instead of
in response to memory backpressure only
2022-12-29 22:04:15 +08:00
bunnie
1e56598b10 chore: fix some typos 2022-12-29 00:53:32 +08:00
bunnie
5370c333da add callback for monitoring basis lock/unlock events 2022-12-29 00:10:42 +08:00
bunnie
ad79ebf8ba chore: more typos. no really, i can stop at any time. 2022-12-13 23:02:09 +08:00
bunnie
b1b6d46d27 chore: more typos
probably should not have tried to spell check this, now i'm wasting
my time.
2022-12-13 22:56:37 +08:00
bunnie
5370ce396d bump all revs due to SoC artifact update 2022-12-12 16:07:20 +08:00
bunnie
52955681d5 reflect unmount status after tryunmount
this fixes a problem where backups fail when autobacklight is
enabled, because autobacklight is constantly querying the PDDB
for the length of the autobacklight timeout, even after the
PDDB is unmounted.
2022-12-02 20:10:10 +08:00
bunnie
1d1b3a35b7 commit version bump 2022-11-26 10:00:44 +08:00
bunnie
aa9977cc58 bump all the things because we changed utralib again 2022-11-17 22:13:40 +08:00
bunnie
dabbfd6c35 bump all the things to absorb the SoC change 2022-11-17 19:39:18 +08:00
bunnie
c3ba30d55d expand on errors reported 2022-11-09 14:25:26 +08:00
bunnie
a7162fd03c bump xous-names dependents 2022-11-07 22:19:38 +08:00
bunnie
0646ec23ba track "attempted" mounts in PDDB
this is to allow a UX process to clear the "waiting for boot"
indicator once an attempt has completed.

right now, it responds *after* the mount process is done,
so the "please waiting" text persists even doing a format.

I think that is...appropriate?
2022-11-07 00:21:52 +08:00
bunnie
3ee36f41ee remove UX delays on PDDB path
retain just one of them going into the erase routine, because
it consumes enough CPU that the dialog box takes a bit too long to
pop up.
2022-11-06 18:45:17 +08:00
bunnie
30583bdf07 bump versions with xous change 2022-11-06 03:48:12 +08:00
bunnie
fa90e74a10
Merge pull request #270 from gsora/feat/pddb-init-notification
feat: show boot process notifications for PDDB
2022-11-06 03:40:17 +08:00
bunnie
7eecf2b1ce remove the initial wait from PDDB; move the pop-up modal location
The initial wait message is now rendered by the first app
that owns the screen (currently shellchat).

The location where the pop-up modal is rendered is also now bracketed
directly around the mount call to avoid interference with other
edge cases that may pop up modals.
2022-11-06 03:07:35 +08:00
Gianguido Sorà
d277ae9551 feat: show boot process notifications for PDDB
This commit shows two notifications:
 1. as soon as the device boots, a notification greets the user and
 tells them everything is fine, the device is booting
 2. as soon as the user enters (or doesn't) enter the password, a
 notification tell the user Xous is loading and mounting PDDB

I spent too much time on this but I believe it's a nice UX improvement.
2022-11-04 17:31:27 +01:00
bunnie
5de44a9502 don't fail if keyattribute not found in current basis when no basis specified
This is not actually an error, we just need to try another basis
down the list. This is a copypasta bug that was only caught now. :-/
2022-11-04 17:05:51 +08:00
bunnie
0d9a39a349 print debug information when a key goes missing in bulk read
this helps debug problems down the road
2022-11-04 17:05:32 +08:00
bunnie
d74673ab29 allow main menu to be used if there are no root keys 2022-11-04 03:55:57 +08:00
bunnie
80a1bd3654 resolve issue #267 2022-11-03 23:18:13 +08:00
bunnie
ff2a7f8240 some more debugging to help trace through dict sync issues 2022-11-03 21:06:04 +08:00
bunnie
b04dcc21ef add smalldb flag, which builds a PDDB with a size of 4MiB
do not interchange this on devices with the full sized
PDDB, you may corrupt the database
2022-11-03 21:04:14 +08:00
bunnie
719020b08b don't allow polls to the unmount to turn true until other callbacks have been allowed 2022-11-03 17:23:11 +08:00
bunnie
7af04b7000 a bit more debugging around dictionary creation 2022-11-03 01:38:18 +08:00
bunnie
ba28dd8795 bump to absorb utralib straggler configs 2022-10-28 15:11:17 +08:00
bunnie
6b296c9b48 fix a cargo file that got nuked in the merge 2022-10-28 15:04:56 +08:00
bunnie
e60f7cc855 Merge branch 'i18n/adds-french-localization' into main 2022-10-28 15:00:27 +08:00
bunnie
dfb10ed7f2 bump crate numbers to publish lang-fr change into xous-rs 2022-10-28 14:01:09 +08:00
Tom Marble
006a69105c
Completes the i18n_helper.py tool
Added 'fr' translation
Added *.json to .gitattributes

Signed-off-by: Tom Marble <tmarble@info9.net>
2022-10-27 16:07:05 -05:00
bunnie
d63c452fb6 catch serialization buffer edge case
if the buffer fills up to the point where there isn't enough
space for the 8-bytes of header, a split_at() call will fail before
the routine even gets to decide if the record can fit.

abort the loop if the buffer is that full.
2022-10-27 23:00:10 +08:00
bunnie
c82a79dab9 add performance monitoring in for bulk reads 2022-10-27 16:45:04 +08:00
bunnie
475e464efe bump versions due to svd2utra change 2022-10-27 03:04:22 +08:00
bunnie
bef734553e clarify some comments and clean up debug messages 2022-10-27 02:53:38 +08:00
bunnie
6992b31104 fix the bufserializer approach
this is much more efficient than the alloc serializer
2022-10-27 02:41:46 +08:00
bunnie
6272a94604 intermediate attempt to extract a length parameter 2022-10-27 01:57:42 +08:00
bunnie
ac01088ec7 Merge branch 'pddb-bulk' into pddb-bulk-bufserializer 2022-10-27 01:30:27 +08:00
bunnie
3d1885e165 add test code
results are slightly better on the first run (3700ms instead of 4900ms)
but there seems to be really excessive allocations using the heap
allocated serializer by rkyv. We really need to use the bufserializer
I think for this to work.
2022-10-27 01:10:17 +08:00
bunnie
721dbc82d7 add an attempt to use bufserializer that didn't work 2022-10-26 23:29:43 +08:00
bunnie
d9bffb3556 add some notes about making this more efficient 2022-10-26 22:06:00 +08:00
bunnie
665e7dce51 fix minor bugs, now passing basic test 2022-10-26 15:16:40 +08:00
bunnie
bffe92cab9 add first draft of bulk serialize/deserialize functions 2022-10-26 05:29:44 +08:00
bunnie
2e768dd19e Merge branch 'main' of github.com:betrusted-io/xous-core into pddb-bulk 2022-10-25 23:15:51 +08:00
bunnie
0ee9ec1d80 bump version numbers for crate publication 2022-10-25 22:38:58 +08:00
Jean-Pierre De Jesus DIAZ
f5176ea380
Remove xous-rs unused features
The rationale is that xous-rs should have only architecture
specific code, e.g.: riscv, arm, x86, etc. To make other
crates that depend on it not specify the features.

This could be useful when using xous-rs with `libstd`
for example.

Signed-off-by: Jean-Pierre De Jesus DIAZ <me@jeandudey.tech>
2022-10-25 13:29:15 +02:00
bunnie
9e39c432bd Merge branch 'main' into pddb-bulk 2022-10-25 17:41:58 +08:00
bunnie
f5289be6fe experiment with 'std' in rkyv for pddb crate 2022-10-25 17:41:12 +08:00
bunnie
0ec75861e6 cleanup a normalization that escaped due to merge conflict 2022-10-25 17:40:18 +08:00
bunnie
30ea893c74
Merge branch 'main' into xous-rs-os-selection 2022-10-25 17:28:33 +08:00
bunnie
d331d405ee bump version numbers for publication 2022-10-25 17:23:40 +08:00
bunnie
67fc21fbdd harmonize hosted mode selection, but retain an explicit hosted flag
In general, hosted mode is assumed "by default" if the target
is not Xous. This allows packages to be built cleanly on command
lines that don't know to pass a --feature flag to specify a target.

However, an explicit "hosted" flag is still retained so that we
can confirm the build system intent. This saves us from the failure
mode that we thought we were building for target A, but accidentally
ommitted the flag, and instead we've been silently building for
"hosted" mode without any warning. The main consumer of the "hosted"
flag is now just the build.rs script inside of the utralib.
2022-10-25 17:21:02 +08:00
bunnie
97f0d34daa more analytics on PDDB debugging 2022-10-24 02:35:55 +08:00
bunnie
83d31ff8b9 fix non-perf builds for pddb 2022-10-24 00:37:05 +08:00
bunnie
e515092636 add performance profiling options to PDDB 2022-10-23 23:14:57 +08:00
bunnie
fc8f2a25e8 expand on "internal error" for .get() results from PDDB 2022-10-20 17:40:11 +08:00
bunnie
805a7d528c flip back to using remote references 2022-10-20 16:30:06 +08:00
bunnie
8e05925cfa somehow the transition to local paths got missed in the merge 2022-10-18 00:39:01 +08:00
bunnie
0584ad2f2c switch to local paths to see if this helps the build 2022-10-18 00:26:56 +08:00
bunnie
4d1da235a9 bump revs that depend on susres to absorb changes 2022-10-17 05:11:09 +08:00
bunnie
d7392d3704 improve comments and add count to the user-failed mount case 2022-10-17 03:51:36 +08:00
bunnie
de41e5ecff rework password login to have a count and reboot/suspend after N retries
The implementation is a bit clumsy because the original
premise of the login loop is that it should be infalliable, and
because of the isolation of the password mechanism there isn't
a great way to pass number of failures back and forth across
that firewall (and I don't think there should be).

The application calling the password routine tracks failures
but the mounting routine has to kick that up to the main loop.
However, the PDDB main loop can't get stuck on the mount attempt
if you want to suspend, because it has to respond to the suspend
request.

So the failure message has to get kicked all the way back to the
status thread, which is what kicks off the mount, and then handled
there.
2022-10-17 02:40:51 +08:00
bunnie
ec1d1b4444 don't try to unmount anything if nothing was mounted 2022-10-17 00:32:06 +08:00
bunnie
4b368d71db bump versions of crates 2022-10-15 01:31:54 +08:00
bunnie
63a87ca1bc bump all the versions 2022-10-14 16:54:43 +08:00
bunnie
ff30131f43 absorb more fixes into the kernel 2022-10-13 04:08:56 +08:00
bunnie
3503965f98 bump versions to absorb xous-rs changes 2022-10-13 03:41:16 +08:00
bunnie
f8a2ea51bb remove rkyv warnings from xous-api-names 2022-10-10 03:16:40 +08:00
bunnie
02b80425f5 bump xous-rs to absorb change in language specification 2022-10-10 03:07:44 +08:00
bunnie
f815ed85b5 fix rust 1.64.0 rkyv warning regressions on unused enum types
rkyv-derived enums throw warnings that they are unused.
Not sure how to get rid of them; I've tried surrounding them in
`#[allow(dead_code)]` directives, and also explicitly creating
each enum variant that I'm aware of through some dummy code and
some log prints of the dummy code calls to ensure the results are
used, and the warnings still persist.

Significantly, the warnings only happen for enums that take
a value. Simple rkyv enums do not throw a warning.

These warnings overwhelm the build logs, so they need to be isolated
in order for us to ensure that the build is clean of other more
important warnings.

The method chosen to isolate them is to take the enums that are
problematic, carve them into their own special file, and add
a file-wide `#![allow(dead_code)]` directive. This only happens
in 7 service crates, and this change commits all of the modifications
in a single big commit that could be undone later if this turns out
to be a compiler regression or something similar.

I also opened an issue with the rkyv maintainer to see what his
opinion is on the situation.

https://github.com/rkyv/rkyv/issues/318
2022-10-09 17:17:20 +08:00
bunnie
2ff9163789 add checksums to PDDB backup
every 1MiB block is checksummed with a sha512/256 truncated to
128 bits. This is used to confirm the media and transport integrity
only. It is not intended to be a cryptographic check, it just confirms
that the backup data was transferred correctly assuming no intentionally
malicious actors in the middle.
2022-10-09 01:10:42 +08:00
bunnie
62af36560b checks for cache coherence issues
Most of the changes introduced in this commit were rolled back,
but, there is now a test facility that's hanging out for the PDDB
to help with future debug efforts.

The patch was motivated by a problem observed in FSCB consistency
when it's updated. Ultimately the root cause was traced to a
cache coherence problem and it's fixed by a patch in the spinor
crate.

However, this patch still retains a check that the data is coherent,
in case the flushing routine isn't complete or some other factor
was missed. It also includes a large amount of commented-out code
that keeps a bit of history used to debug the problem -- eventually,
we can just cull that but it's kept around in case we got the
root cause wrong (or there were multiple causes and coherence is
just one of them).
2022-10-07 18:52:57 +08:00
bunnie
3bca9de2a0 bump version numbers for xous-kernel patch
this fixes tests in CI
2022-10-05 23:51:35 +08:00
bunnie
969f93b001 cross-repo update versions from xous-rs bump 2022-10-05 18:32:44 +08:00
bunnie
9dcc0a3386 absorb ticktimer api bump, propagated from susres bump 2022-10-03 04:40:06 +08:00
bunnie
d2b2383d1a bump susres version 2022-10-03 04:36:32 +08:00
bunnie
b184bfbe6a move ticktimer to remote api 2022-10-03 03:37:38 +08:00
bunnie
89ca75c4bf move over to remote susres api 2022-10-03 02:24:44 +08:00
bunnie
6123e8ff9b xous-names conversion to crates.io 2022-10-03 01:18:20 +08:00
bunnie
de514ac66f rename packages in prep for upload to crates.io 2022-10-03 00:24:45 +08:00
bunnie
4e40ae6c5d move from target selectors to feature selectors for back-ends
This will allow for e.g. xous on riscv32-imac but not a precursor
target, for example: by using feature selectors we get to have
a bigger range of targets that we support, such as for implementations
that haven't been upstreamed or have official target-triples.

The downsides are:

- patch override libraries don't absorb feature flags. so, for
example, the AES crate that is used to patch other crates won't
be aware of the target features without modifying the referencing
crates.

- features are normally meant to be additive, but target features
are mutually exclusive. The mutex is enforced by a check in
utralib that throws a panic on build if two targets are accidentally
selected at the same time.
2022-10-02 02:56:44 +08:00
bunnie
fb29c9b2c1 refactor utra target line to use explicit feature targets
still need to rework the crate feature selection
2022-10-02 02:20:03 +08:00
bunnie
ecab0ba045 fix default-features in services 2022-09-27 02:44:46 +08:00
bunnie
d667d7e1f5 bump xous-ipc version 2022-09-27 01:14:43 +08:00
bunnie
f188e0f866 re-bump xous-ipc to 0.9.9 2022-09-26 16:50:38 +08:00
bunnie
507ab42836 downgrade xous-ipc to 0.9.7 until i have permissions to push a new version 2022-09-26 16:11:26 +08:00
bunnie
3579dfdd17 update xous-ipc and xous versions 2022-09-26 16:07:35 +08:00
bunnie
114d60bc53 migrate logserver API to external package 2022-09-26 14:06:46 +08:00
bunnie
85279566f0 fix #77
This adds some UX feedback in the edge case that a suspend could not
happen due to a blocking sever, e.g. a sha512 hash pending or
a SPINOR flash write/erase pending.

The UI will tell the user to retry again; usually the ops in
a normally functioning system for hashes or SPI are very short,
so we could almost just retry automatically but I think for now
I want to see if this edge case actually happens.
2022-09-24 02:40:11 +08:00
bunnie
9225cbaba3 resolve issue #206 by adding check for existing basis 2022-09-20 23:06:22 +08:00
bunnie
4ae3308fbe add PIN change to the PDDB 2022-08-12 23:10:18 +08:00
bunnie
2b4ce52a10 fix some regressions on empty key listings 2022-08-02 02:52:04 +08:00
bunnie
3d992cedf4 fix an import error + reduce log noise on a normal path 2022-08-02 01:38:07 +08:00
bunnie
0d61cbb838 improve debug output 2022-08-02 01:37:45 +08:00
bunnie
13b366345c clean up/optimize the key listing routine
provides some marginal improvement on the vault UX
2022-08-01 23:12:26 +08:00
bunnie
513fe519a7 profiling should not be enabled, somehow that slipped through 2022-08-01 16:12:38 +08:00
bunnie
826aa43f20 Merge branch 'pddb-flamegraph' into main 2022-08-01 15:46:57 +08:00
bunnie
5be62e372d chore: move AES-GCM-SIV off of branch to 0.11.1 official
this patch moves us off the branch we had to hop on for cipher
API compatibility to the one that's officially published by
RustCrypto. This closes #168.
2022-08-01 15:44:16 +08:00
bunnie
7d62ca85b8 some preliminary schwags at making flamegraphs
turns out, they're pretty useless. I mean, for debugging actual
running code. it tells us basically what we know already which is
99.99% of the time is spent waiting on inter-thread messaging
in hosted mode.
2022-08-01 01:32:19 +08:00
bunnie
e469736278 stagger the flush timing a little bit
noticed it's hitting exactly when NTP updates because it also
happens on a 24-hour schedule, staggering a bit to try and keep
everything from hitting exactly at the same time once a day.
2022-07-29 15:00:44 +08:00
bunnie
46c53d9612 add telemetry for better PDDB CI 2022-07-28 18:46:56 +08:00
bunnie
3a4f5d07e8 refer to Xous Book in PDDB readme 2022-07-28 16:00:30 +08:00
bunnie
ac1f746566 add fscb/spaceupdate journal flushing & periodic flush
- fix journal number bug in basis root record
- fix conflict PTE resolution to also consider basis root record,
which has a different encryption form.
2022-07-28 04:54:35 +08:00
bunnie
ec5cc29ccd hotfix for incorrect AES-KWP implementation
The crate we originally grabbed for AES-KWP seems to be non-compliant
(or I'm an idiot and I can't figure out how to call it in a way
that is compliant).

See https://github.com/jedisct1/rust-aes-keywrap/issues/9

Turns out that `aes_kw` *is* compliant, but unfortunately, it didn't
exist back when the PDDB was being written. The PDDB started in October
2021, and `aes_kw` was pushed in Jan 2022. *just* barely missed it.

Anyways, caught this error while trying to implement some CI for
backups. Hurray for testing!

This patch removes the encryption path using the incorrect version
of the algorithm, but leaves the decryption path in place. The
system will try to decrypt initially with the correct version,
but if that fails, it will fall back and try the incorrect version.
If that succeeds, it returns an error code with the correct key and
an updated wrapped key, which the caller can store so that future
calls do not fail. Thus, we have a seamless migration from the
incorrect version to the correct version.

Have verified that the aes_kw version *does* work with the CI
script, insofar as it has been written (we just got to that point
and got stuck). So, full speed ahead. Users shouldn't notice this
change.

Also, fwiw, I did take a look at the "incorrect" implementation,
and it doesn't look obviously insecure. Printing out the various
rounds of the wrapping function looks like the first couple rounds
match, but it diverges somewhere probably due to an off-by-one
in a counter or something like that. I didn't chase it out -- just
wanted to make sure that nothing looked like a huge data compromise.

Good news is this comes right on the heels of v0.9.9 release, so
this will be pushed as a hotfix right away.
2022-07-25 02:56:20 +08:00
bunnie
6f10061653 reset "don't ask again" flag for password creation during factory test
this facility is necessary in case a worker accidentally picks
the wrong setting, disabling the feature for the device. When
the factory test completes, a call is made to reset the parameter.

This was a slightly complicated affair because the the don't ask
flag is contained within the rootkeys enclave itself, and we can't
just give access to rootkeys willy-nilly. So the flag is passed
through the PDDB and then to the rootkeys enclave. It's not harmful
to have it reset by an unauthorized actor; the pop-up will not
happen if passwords are, in fact, already set.
2022-07-19 23:57:25 +08:00
bunnie
e40a109f6c add UX flow dedicated to the 'churn' operation 2022-07-19 16:57:47 +08:00
bunnie
0a4702d6c3 clean up dialog messages to conform to the latest lexicon 2022-07-19 16:30:00 +08:00
bunnie
9fc74e3291 strip out debug info 2022-07-19 15:56:16 +08:00
bunnie
e8188309cf add rekey operation for PDDB migrations to new devices 2022-07-19 03:16:37 +08:00
bunnie
a89ae13463 move pddb mounter to status thread 2022-07-11 04:12:44 +08:00
bunnie
1da8578459 remove v1 PDDB migration in prep for min version setting 2022-07-08 18:07:10 +08:00
Sean Cross
d6ac900748 pddb: backend: use Instant for access time
Instead of using the ticktimer server directly, use
`std::time::Instant::now()`. This enables tests to be run in hosted
mode.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-06 13:59:35 +08:00
Sean Cross
086c014de2 pddb: close all related filehandles when a file is deleted
Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 23:56:31 +08:00
Sean Cross
ee18fe889c pddb: initial libstd support
Add support for libstd calls to pddb.

The pddb uses `:` as the path separator, and grafts the concept of paths
onto dicts.

Currently implemented features:

    * mkdir
    * rmdir
    * create file
    * open file
    * read file
    * write file
    * unlink file
    * seek file

Other calls may be added as needed.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 22:50:58 +08:00
Sean Cross
6ae91d8e46 pddb: make select_basis and latest_basis non-mut
These do not need to clone data and do not need to be mut. This improves
access time considerably.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-07-05 17:23:40 +08:00
bunnie
e26106e334 fix warnings and clean up configs
tighten up the scope of some features to reduce warning messages
2022-07-04 18:13:11 +08:00
bunnie
c46b5179ff commit various project configs 2022-07-04 17:58:52 +08:00
bunnie
8565c7efe1 fix FSCB generation issues
- fix problem where we were generating a size based off of total disk
size, instead of fraction of available space

- enumerate FSCB entries into the page table scan. This is because
the page table cache could hold used entries that aren't reflected
in the low-level disk scan, and there is no way to trigger a
flush at the point of re-allocation.
2022-07-04 17:56:54 +08:00
bunnie
baa5097313 fix basis FSCB generation on full - basic fix in place and tested 2022-07-04 15:19:40 +08:00
bunnie
7162b05a7f add routines for testing secret basis
When the "autobasis" feature is selected, the hardware block
will track all basis that have been opened and unlock them when
it comes time to scan/generate more FSCB space. It also bypasses
all UX normally invoked.

This allows us to create tests that open and close Bases automatically,
but of course, totally destroys the notion of plausible deniability.
So, don't turn the feature on for production.
2022-07-04 04:06:53 +08:00
bunnie
8c5a475f49 some whitespace auto-changes 2022-07-03 01:13:31 +08:00
bunnie
8a81954cf0 clean up the page table enumeration
since all the keys are now disclosed, the routine has been
renamed accordingly and the re-scan of existing bases is
redundant.
2022-07-03 01:12:12 +08:00
bunnie
559229d85e fix key enumeration to include keys that we already know 2022-07-03 01:06:56 +08:00
bunnie
95055c30d7 add some whitespace formatting to improve readability of basis listing 2022-07-03 00:43:27 +08:00
bunnie
e8140f257c make sure the basis list is displayed at least once
need to cover the case that no enumeration was done, and just
show a "null" list confirming basically nothing other than
the system basis is disclosed.
2022-07-03 00:42:16 +08:00
bunnie
b10cf878db fix large key read; and cursor positions in read() and write()
1. the large key read algorithm was broken. this is now fixed. we only
saw this triggered when the large key was much larger than 8kiB (and
the original CI tests just use 8kiB keys for "large").

2. the cursor position in read() and write() were being incremented
by the expected read amount, not the actual read amount.
2022-07-03 00:35:11 +08:00
bunnie
587f7ef39c some enhanced debugging 2022-07-02 00:09:38 +08:00
bunnie
ae37c2cd02 don't truncate key allocations that have an alloc_hint specified 2022-07-02 00:09:21 +08:00
bunnie
a1950a4c00 make a default alloc hint that's not 0
set to 8 bytes initially, could be adjusted up or down.
2022-07-01 19:45:55 +08:00
bunnie
5008c7afac first draft of all basis unlock
This has worked in testing, but:

- There seems to be a new error unrelated to this with allocating
very large blocks of memory, which needs to be traced down.
- Because this operation will typically fully block the caller,
UX redraws can't happen (unless the caller has a fully multithreaded
redraw routine). This needs some thought because:

1. If you have several basis you're trying to unlock, eventually,
you'll overflow the application incoming queue with redraw requests,
and the system will crash.

2. It's cosmetically disturbing to have the dialog boxes piling up
on top of each other during a critical operation like this.

I *think* the right thing to do is to refactor the GAM a bit
to handle pop-up modals that block a bit more gracefully. But
I think it's more urgent to trace down the problem with large blocks
of memory being allocated, since it is a functional issue and not
a cosmetic one.
2022-07-01 17:09:51 +08:00
bunnie
3aeec85e06 add a note about the overhead of the migration flag 2022-06-30 22:58:36 +08:00
bunnie
23a31c2cf0 fix PDDB initial stack size and increase heap limit on boot
I had thought there was an issue with heap size limit increases
and multithreading but I think actually the problem is the
PDDB runs pretty close to the stack size limit. This increases
the stack size limit for the PDDB on boot and also bumps up
the heap limit so caching can work more flawlessly.
2022-06-28 19:58:20 +08:00
Sean Cross
7aae6dcab9 pddb: change Option<Opcode> -> Opcode in main
Instead of writing `Some(Opcode::*)` repeatedly, change it to simply
match against `Opcode::*`. This is accomplished by defining a new opcode
`Invalid` that will be matched against as a default.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-27 09:24:11 +08:00
Sean Cross
748d4ce87d Merge remote-tracking branch 'origin/main' into libstd-fs 2022-06-27 09:14:04 +08:00
Sean Cross
8cd04c3415 pddb: rework PddbBuf ordering to be non-packed
The new ordering remains the same size, but is no longer
`#[repr(packed)]`. This allows it to be `#[repr(align(4096))]` which
means we can get away with using a stack page for this struct instead of
allocating a new object every time.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-27 09:13:00 +08:00
bunnie
fd63a33078 add PDDB heap limit increase based on passing a threshold 2022-06-27 04:03:33 +08:00
Sean Cross
1388da7680 Merge remote-tracking branch 'origin/vault' into libstd-fs 2022-06-26 16:48:20 +08:00
bunnie
88ea0145da clean up some debug spew 2022-06-26 03:55:26 +08:00
bunnie
9f409fda2d fix basis search order issues
at least, it passes some simple tests. much more testing is needed.
2022-06-26 03:52:27 +08:00
bunnie
e62c9bf3d3 add basis management functions and debug some basis issues
there's some problems with the automatic basis unionization
feature. now debugging...
2022-06-26 02:11:13 +08:00
Sean Cross
19532c5634 pddb: add explicit discriminants to enums
In order to ensure the API is stable, add explicit opcodes to enums
within pddb.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-06-25 11:00:41 +08:00
bunnie
8bf36f98a4 partial progress on #168
at least we're off a fork and back onto the main repo.
2022-06-20 14:15:53 +08:00
bunnie
4c18662430 add a periodic scrub to the PDDB
this periodically polls the PDDB and has it check the memory allocation
levels, and if too high, invokes a scrub attempt.

This bit is probably the most poorly tested of all the code, but
also some of the simpler logic. The more important part this code
brings is analytics on actual heap and cache usage, which will help
inform us on how urgent/bad things are getting in terms of PDDB
cache usage.
2022-06-17 02:02:11 +08:00
bunnie
dc0a4e5400 add backpressure triggers to the CI tests 2022-06-17 02:01:59 +08:00
bunnie
87efc267f1 add PDDB backpressure items (#109 partial) 2022-06-17 02:01:30 +08:00
bunnie
e32ea4d3f5 fix #174 2022-06-10 21:36:53 +08:00
bunnie
66dc39c2a8 pddb sync does not require the handle to be mutable 2022-06-06 18:25:33 +08:00
bunnie
17446588bb fix read_all bug in PDDB 2022-06-05 01:45:05 +08:00
bunnie
21116a1693 fix another errant mut that wasn't needed in pddb 2022-06-01 00:06:55 +08:00
bunnie
bf535bd0e1 use interior mutability to handle callbacks
this eases a mutability requirement for the PDDB object
2022-05-31 23:54:16 +08:00
Sean Cross
3ae9d8abf0 apps/services: remove xous::xous_main macro usage
This macro will be removed, since it is no longer necessary. Remove this
macro across the project and replace it with a standard `main()`
function.

Signed-off-by: Sean Cross <sean@xobs.io>
2022-05-31 13:33:30 +08:00
bunnie
1946d38a69 work around https://github.com/RustCrypto/AEADs/issues/410
We're trying to bring the AES version up so we can do CBC
(required for FIDO2) but unfortunately the aes-gcm-siv crate
is locked to an outdated version of AES, and this is blocking
on a type that is missing in the `block-ciphers` crate.

For now, we're using a patched version of aes-gcm-siv from
rozbb's repo, which allows us to move forward, but this needs
to be reverted once a new version of the official crate is patched.
2022-05-30 04:36:33 +08:00
bunnie
2cd4e80d9c bring PDDB to AES 0.8 compatibility 2022-05-30 03:08:41 +08:00
bunnie
e14e6206ed include a getrandom implementation that allows us to use modern randcore
A lot of packages are depending on 0.6 api, which requires a crate
that has OS specific TRNG calls. This patch keeps the 0.5 calls
inside the TRNG crate as direct API calls, and migrates to support
a 0.6 api to a patched version of getrandom that supports Xous.
2022-05-29 02:47:51 +08:00
bunnie
f4a18401e5 expose is_efuse_secured() API via the PDDB
is_efuse_secured() comes from rootkeys, but we don't want
to expose the full rootkeys API to the world.

PDDB already has an approved capability for accessing rootkeys,
so this single API is passed through via the PDDB object here,
allowing other procesess to query it.
2022-05-28 18:21:07 +08:00
bunnie
62cbc86aeb add automation strings 2022-05-24 23:21:26 +08:00
bunnie
8af8926ad1 make disk sizing errors non-fatal in hosted mode
it can be innocuous in the case that you're changing sizes
for debugging
2022-05-10 17:37:08 +08:00
bunnie
ea6cf27ed9 fix PDDB length field -- somehow this got flipped from the debug branch 2022-05-06 14:43:29 +08:00
bunnie
ee0f443883 hoist net thread exhaustion check to before we check the poll interval 2022-05-06 14:17:29 +08:00
nworbnhoj
0e561802ce enable notification modal text + qrcode
change the notification modal to accept the primary message and an
Option of independent text to display as a qrcode

pub fn show_notification(&self, notification: &str, qrtext:Option<&str>) -> Result<(), xous::Error> {}
2022-04-24 21:29:15 +10:00
bunnie
34c2ffe1a3 fix password entry to work with the new return type
also requires a call to `reset_action_payloads` for the
number of field elements to be recorded correctly!
2022-04-22 17:43:26 +08:00
bunnie
4ddd414ed6 fix modals API in the PDDB migration code 2022-04-22 15:55:15 +08:00
bunnie
2c3143f790 Merge branch 'main' into multi-text-modals 2022-04-22 15:54:18 +08:00
bunnie
38d6765586 fix remaining bugs in migration, tested and passing on CI + real hardware 2022-04-20 03:48:05 +08:00
bunnie
b3c89e5f09 add migration of secret basis for v1->v2; fix some interior mutability issues
Probably nobody is really using secret Bases right now, but it's
probably a good idea to lay out all the infrastructure and start
testing it for when it comes time to do it with data that really
matters.
2022-04-19 04:50:15 +08:00
bunnie
4ba34bbd58 intermediate work on v1->v2 migration, need to set aside to fix a libstd problem 2022-04-19 00:45:10 +08:00
bunnie
c8a38b4bf5 fix representation of va in pagetable so we're not burning 12 bits on 0's
this is a v2 fix, incompatible with v1
2022-04-19 00:44:45 +08:00
bunnie
7553053f29 v0.0.2.1 initial implementation - still needs migration code
Previous version was v0.0.1.1
New version is v0.0.2.1 (don't ask me why I skipped v0.0.2.0)

The most significant change is that the key derivation for the basis
is now split into two keys. One key is used for the page table ECB
encoding, and a different key is used for the data AES-GCM-SIV encoding.

This is because someone much smarter than me looked at my scheme
and said it is very dangerous to use the same key for two different
types of ciphers. This person is correct, so it had to be rectified.

For the System Basis, the keys are encrypted using the root keys
wrapper API. In this case, we simply create two random keys, one for
the page table, and one for the data region, and wrap them using
the root keys API.

For user secret Basis, the derived keys are sent through HKDF
with two different "info" strings to derive a pair of keys from
a common password.

In case the cryptographer who looked at this is reading along,
note that the basis name is encoded as a fixed-length, 0-padded
record, so, we don't encode the length of the basis in the hash
used to derive the keys. When I explained the scheme I thought
I was using a variable-length basis name to derive the key, but
when I checked the code, it's always 64 bytes long, and bytes
that are not used are 0-padded, so, I think we don't need the
length encoding field.
2022-04-18 15:59:15 +08:00
bunnie
cd8c762c55 patch up QR code PR, change method signature as discussed
now any modal notification can also render as a QR code
2022-04-08 15:19:56 +08:00
Gianguido Sorà
8123393a7d Merge branch 'main' into feature/multi-text-modals 2022-04-07 16:39:03 +02:00
bunnie
d49c35dc0c verify that unmount callbacks are working 2022-04-07 17:58:37 +08:00
bunnie
52d56f39f1 wire up PDDB key unmount callbacks (oops) 2022-04-07 17:37:25 +08:00
bunnie
6ac4ef292b add basis mount/unmount to pddb shell commands, fix some UX formatting 2022-04-07 15:32:47 +08:00
bunnie
1ad7c43f81 refactor pddb ux thread to use deferred response messaging 2022-04-07 14:46:44 +08:00
bunnie
92c6a93147 eliminate yield loop on blocking mount poller 2022-04-07 00:04:25 +08:00
bunnie
b75df1efb3 annotate rationale on multiple variants of pddb mount calls 2022-04-06 23:46:39 +08:00
Gianguido Sorà
83e61f5f1b feat: add wiring for multiple TextEntryPayloads drawing 2022-03-31 17:54:22 +02:00
bunnie
8b5ad2657b an aborted attempt at doing a clean-shutdown for hosted mode
This doesn't work because the shutdown call destroys all the servers then terminates
the processes. But, the problem is that the processes are still
running, but servers are disappearing. So what you end up with is a
panic when the first message gets delivered to a server that no longer
exists, and then you end up with the program hanging on that panic
because some servers haven't been destroyed/exited yet and are now
just spinning waiting to be scheduled. so the call works in the tidy
kernel test cases but to get it to work on a full OS you'd need to
have a way to put every server into an eternal blocking state before
making the shutdown call (to prevent new messages from being issued
while servers are going away), which is more work than i'm willing to
put into the problem right now.
2022-03-31 01:10:58 +08:00
bunnie
4f1f4a3907 optimize journal resolution code
We have the journals tracked in a HashSet, so, use the hashing
properties instead of an O(N) search.
2022-03-29 16:39:54 +08:00
bunnie
0bdf21b3ac retire debugging infos 2022-03-29 16:33:40 +08:00
bunnie
c542807958 fix PDDB bug where journal numbers on pages were incorrectly restored
An edge case was missed where a page which was allocated out of
the FSCB and then later written to disk was having its journal
number set to 0.

The root cause of the logic issue is that the page table itself
does not store a journal number. It's assumed that the page table
is ground truth, and is unjournaled.

As long as the pages don't exist in the FSCB, that's not a problem,
the journal number doesn't matter.

But, let's say this sequence happens:

1. FSCB allocates some free space
2. process takes that free space allocation and commits it to its own page table
3. FSCB update log gets a note that free space is now allocated
4. system reboots
5. process reads its page table from disk. There are no journals associated
with the page table on disk, so, it assigns a journal number of 0 by default.
6. we now have a conflict between the Process' view of the journal age
of that page table entry, and the FSCB's view.

The fix to this is during the page table scan, to check and see if
any candidate pages happen to already exist inside the FSCB. If it does,
then you adopt the latest journal number for that page in the FSCB.
This keeps the two views consistent.

A second bug was also identified while fixing this where the "Random"
order we were hoping for in visiting the free space cache as an
iterator is less random than we had hoped. So, there is a patch to
extract the free pages and pull one random one from the pool, using
entropy coming from the TRNG. This is less computationally efficient,
but at least it is correct and has more transparent properties
than relying on the indeterminism of the HashSet implementation.
2022-03-29 16:26:55 +08:00
bunnie
bdd194ea7b retire rust issue 90195 work-around in PDDB 2022-03-29 16:25:33 +08:00
bunnie
4eec54620c fix warnings and debug messages 2022-03-29 04:39:33 +08:00
bunnie
9a41b18400 fix bug where dictionaries with 1 entry fail to sync
also ensure that at least 1 byte is reserved for any key
that is allocated; this helps avoid edge cases dealing with
zero-length key entries later on.
2022-03-29 04:38:15 +08:00
bunnie
55b05cfd80 add debugging routines; fix zero-length key creation case
- add a "dangerous debug" command that only exists when the feature
"pddbtest" is enabled. The feature can't work on real hardware, but
still, it enables things like dumping of the database and storing
the keys for inspection, so it's isolated in a feature that prevents
the code from being compiled in the first place when it's not selected.

- fix a bug where keys that have zero length aren't being created
correctly. this is a common use case for reserving some space for
data that isn't yet available.
2022-03-29 01:47:28 +08:00