this fixes a problem where backups fail when autobacklight is
enabled, because autobacklight is constantly querying the PDDB
for the length of the autobacklight timeout, even after the
PDDB is unmounted.
this is to allow a UX process to clear the "waiting for boot"
indicator once an attempt has completed.
right now, it responds *after* the mount process is done,
so the "please waiting" text persists even doing a format.
I think that is...appropriate?
The initial wait message is now rendered by the first app
that owns the screen (currently shellchat).
The location where the pop-up modal is rendered is also now bracketed
directly around the mount call to avoid interference with other
edge cases that may pop up modals.
This commit shows two notifications:
1. as soon as the device boots, a notification greets the user and
tells them everything is fine, the device is booting
2. as soon as the user enters (or doesn't) enter the password, a
notification tell the user Xous is loading and mounting PDDB
I spent too much time on this but I believe it's a nice UX improvement.
if the buffer fills up to the point where there isn't enough
space for the 8-bytes of header, a split_at() call will fail before
the routine even gets to decide if the record can fit.
abort the loop if the buffer is that full.
results are slightly better on the first run (3700ms instead of 4900ms)
but there seems to be really excessive allocations using the heap
allocated serializer by rkyv. We really need to use the bufserializer
I think for this to work.
In general, hosted mode is assumed "by default" if the target
is not Xous. This allows packages to be built cleanly on command
lines that don't know to pass a --feature flag to specify a target.
However, an explicit "hosted" flag is still retained so that we
can confirm the build system intent. This saves us from the failure
mode that we thought we were building for target A, but accidentally
ommitted the flag, and instead we've been silently building for
"hosted" mode without any warning. The main consumer of the "hosted"
flag is now just the build.rs script inside of the utralib.
The implementation is a bit clumsy because the original
premise of the login loop is that it should be infalliable, and
because of the isolation of the password mechanism there isn't
a great way to pass number of failures back and forth across
that firewall (and I don't think there should be).
The application calling the password routine tracks failures
but the mounting routine has to kick that up to the main loop.
However, the PDDB main loop can't get stuck on the mount attempt
if you want to suspend, because it has to respond to the suspend
request.
So the failure message has to get kicked all the way back to the
status thread, which is what kicks off the mount, and then handled
there.
rkyv-derived enums throw warnings that they are unused.
Not sure how to get rid of them; I've tried surrounding them in
`#[allow(dead_code)]` directives, and also explicitly creating
each enum variant that I'm aware of through some dummy code and
some log prints of the dummy code calls to ensure the results are
used, and the warnings still persist.
Significantly, the warnings only happen for enums that take
a value. Simple rkyv enums do not throw a warning.
These warnings overwhelm the build logs, so they need to be isolated
in order for us to ensure that the build is clean of other more
important warnings.
The method chosen to isolate them is to take the enums that are
problematic, carve them into their own special file, and add
a file-wide `#![allow(dead_code)]` directive. This only happens
in 7 service crates, and this change commits all of the modifications
in a single big commit that could be undone later if this turns out
to be a compiler regression or something similar.
I also opened an issue with the rkyv maintainer to see what his
opinion is on the situation.
https://github.com/rkyv/rkyv/issues/318
every 1MiB block is checksummed with a sha512/256 truncated to
128 bits. This is used to confirm the media and transport integrity
only. It is not intended to be a cryptographic check, it just confirms
that the backup data was transferred correctly assuming no intentionally
malicious actors in the middle.
Most of the changes introduced in this commit were rolled back,
but, there is now a test facility that's hanging out for the PDDB
to help with future debug efforts.
The patch was motivated by a problem observed in FSCB consistency
when it's updated. Ultimately the root cause was traced to a
cache coherence problem and it's fixed by a patch in the spinor
crate.
However, this patch still retains a check that the data is coherent,
in case the flushing routine isn't complete or some other factor
was missed. It also includes a large amount of commented-out code
that keeps a bit of history used to debug the problem -- eventually,
we can just cull that but it's kept around in case we got the
root cause wrong (or there were multiple causes and coherence is
just one of them).
This will allow for e.g. xous on riscv32-imac but not a precursor
target, for example: by using feature selectors we get to have
a bigger range of targets that we support, such as for implementations
that haven't been upstreamed or have official target-triples.
The downsides are:
- patch override libraries don't absorb feature flags. so, for
example, the AES crate that is used to patch other crates won't
be aware of the target features without modifying the referencing
crates.
- features are normally meant to be additive, but target features
are mutually exclusive. The mutex is enforced by a check in
utralib that throws a panic on build if two targets are accidentally
selected at the same time.
This adds some UX feedback in the edge case that a suspend could not
happen due to a blocking sever, e.g. a sha512 hash pending or
a SPINOR flash write/erase pending.
The UI will tell the user to retry again; usually the ops in
a normally functioning system for hashes or SPI are very short,
so we could almost just retry automatically but I think for now
I want to see if this edge case actually happens.
this patch moves us off the branch we had to hop on for cipher
API compatibility to the one that's officially published by
RustCrypto. This closes#168.
turns out, they're pretty useless. I mean, for debugging actual
running code. it tells us basically what we know already which is
99.99% of the time is spent waiting on inter-thread messaging
in hosted mode.
noticed it's hitting exactly when NTP updates because it also
happens on a 24-hour schedule, staggering a bit to try and keep
everything from hitting exactly at the same time once a day.
The crate we originally grabbed for AES-KWP seems to be non-compliant
(or I'm an idiot and I can't figure out how to call it in a way
that is compliant).
See https://github.com/jedisct1/rust-aes-keywrap/issues/9
Turns out that `aes_kw` *is* compliant, but unfortunately, it didn't
exist back when the PDDB was being written. The PDDB started in October
2021, and `aes_kw` was pushed in Jan 2022. *just* barely missed it.
Anyways, caught this error while trying to implement some CI for
backups. Hurray for testing!
This patch removes the encryption path using the incorrect version
of the algorithm, but leaves the decryption path in place. The
system will try to decrypt initially with the correct version,
but if that fails, it will fall back and try the incorrect version.
If that succeeds, it returns an error code with the correct key and
an updated wrapped key, which the caller can store so that future
calls do not fail. Thus, we have a seamless migration from the
incorrect version to the correct version.
Have verified that the aes_kw version *does* work with the CI
script, insofar as it has been written (we just got to that point
and got stuck). So, full speed ahead. Users shouldn't notice this
change.
Also, fwiw, I did take a look at the "incorrect" implementation,
and it doesn't look obviously insecure. Printing out the various
rounds of the wrapping function looks like the first couple rounds
match, but it diverges somewhere probably due to an off-by-one
in a counter or something like that. I didn't chase it out -- just
wanted to make sure that nothing looked like a huge data compromise.
Good news is this comes right on the heels of v0.9.9 release, so
this will be pushed as a hotfix right away.
this facility is necessary in case a worker accidentally picks
the wrong setting, disabling the feature for the device. When
the factory test completes, a call is made to reset the parameter.
This was a slightly complicated affair because the the don't ask
flag is contained within the rootkeys enclave itself, and we can't
just give access to rootkeys willy-nilly. So the flag is passed
through the PDDB and then to the rootkeys enclave. It's not harmful
to have it reset by an unauthorized actor; the pop-up will not
happen if passwords are, in fact, already set.
Instead of using the ticktimer server directly, use
`std::time::Instant::now()`. This enables tests to be run in hosted
mode.
Signed-off-by: Sean Cross <sean@xobs.io>
Add support for libstd calls to pddb.
The pddb uses `:` as the path separator, and grafts the concept of paths
onto dicts.
Currently implemented features:
* mkdir
* rmdir
* create file
* open file
* read file
* write file
* unlink file
* seek file
Other calls may be added as needed.
Signed-off-by: Sean Cross <sean@xobs.io>
- fix problem where we were generating a size based off of total disk
size, instead of fraction of available space
- enumerate FSCB entries into the page table scan. This is because
the page table cache could hold used entries that aren't reflected
in the low-level disk scan, and there is no way to trigger a
flush at the point of re-allocation.
When the "autobasis" feature is selected, the hardware block
will track all basis that have been opened and unlock them when
it comes time to scan/generate more FSCB space. It also bypasses
all UX normally invoked.
This allows us to create tests that open and close Bases automatically,
but of course, totally destroys the notion of plausible deniability.
So, don't turn the feature on for production.
need to cover the case that no enumeration was done, and just
show a "null" list confirming basically nothing other than
the system basis is disclosed.
1. the large key read algorithm was broken. this is now fixed. we only
saw this triggered when the large key was much larger than 8kiB (and
the original CI tests just use 8kiB keys for "large").
2. the cursor position in read() and write() were being incremented
by the expected read amount, not the actual read amount.
This has worked in testing, but:
- There seems to be a new error unrelated to this with allocating
very large blocks of memory, which needs to be traced down.
- Because this operation will typically fully block the caller,
UX redraws can't happen (unless the caller has a fully multithreaded
redraw routine). This needs some thought because:
1. If you have several basis you're trying to unlock, eventually,
you'll overflow the application incoming queue with redraw requests,
and the system will crash.
2. It's cosmetically disturbing to have the dialog boxes piling up
on top of each other during a critical operation like this.
I *think* the right thing to do is to refactor the GAM a bit
to handle pop-up modals that block a bit more gracefully. But
I think it's more urgent to trace down the problem with large blocks
of memory being allocated, since it is a functional issue and not
a cosmetic one.
I had thought there was an issue with heap size limit increases
and multithreading but I think actually the problem is the
PDDB runs pretty close to the stack size limit. This increases
the stack size limit for the PDDB on boot and also bumps up
the heap limit so caching can work more flawlessly.
Instead of writing `Some(Opcode::*)` repeatedly, change it to simply
match against `Opcode::*`. This is accomplished by defining a new opcode
`Invalid` that will be matched against as a default.
Signed-off-by: Sean Cross <sean@xobs.io>
The new ordering remains the same size, but is no longer
`#[repr(packed)]`. This allows it to be `#[repr(align(4096))]` which
means we can get away with using a stack page for this struct instead of
allocating a new object every time.
Signed-off-by: Sean Cross <sean@xobs.io>
this periodically polls the PDDB and has it check the memory allocation
levels, and if too high, invokes a scrub attempt.
This bit is probably the most poorly tested of all the code, but
also some of the simpler logic. The more important part this code
brings is analytics on actual heap and cache usage, which will help
inform us on how urgent/bad things are getting in terms of PDDB
cache usage.
This macro will be removed, since it is no longer necessary. Remove this
macro across the project and replace it with a standard `main()`
function.
Signed-off-by: Sean Cross <sean@xobs.io>
We're trying to bring the AES version up so we can do CBC
(required for FIDO2) but unfortunately the aes-gcm-siv crate
is locked to an outdated version of AES, and this is blocking
on a type that is missing in the `block-ciphers` crate.
For now, we're using a patched version of aes-gcm-siv from
rozbb's repo, which allows us to move forward, but this needs
to be reverted once a new version of the official crate is patched.
A lot of packages are depending on 0.6 api, which requires a crate
that has OS specific TRNG calls. This patch keeps the 0.5 calls
inside the TRNG crate as direct API calls, and migrates to support
a 0.6 api to a patched version of getrandom that supports Xous.
is_efuse_secured() comes from rootkeys, but we don't want
to expose the full rootkeys API to the world.
PDDB already has an approved capability for accessing rootkeys,
so this single API is passed through via the PDDB object here,
allowing other procesess to query it.
change the notification modal to accept the primary message and an
Option of independent text to display as a qrcode
pub fn show_notification(&self, notification: &str, qrtext:Option<&str>) -> Result<(), xous::Error> {}
Probably nobody is really using secret Bases right now, but it's
probably a good idea to lay out all the infrastructure and start
testing it for when it comes time to do it with data that really
matters.
Previous version was v0.0.1.1
New version is v0.0.2.1 (don't ask me why I skipped v0.0.2.0)
The most significant change is that the key derivation for the basis
is now split into two keys. One key is used for the page table ECB
encoding, and a different key is used for the data AES-GCM-SIV encoding.
This is because someone much smarter than me looked at my scheme
and said it is very dangerous to use the same key for two different
types of ciphers. This person is correct, so it had to be rectified.
For the System Basis, the keys are encrypted using the root keys
wrapper API. In this case, we simply create two random keys, one for
the page table, and one for the data region, and wrap them using
the root keys API.
For user secret Basis, the derived keys are sent through HKDF
with two different "info" strings to derive a pair of keys from
a common password.
In case the cryptographer who looked at this is reading along,
note that the basis name is encoded as a fixed-length, 0-padded
record, so, we don't encode the length of the basis in the hash
used to derive the keys. When I explained the scheme I thought
I was using a variable-length basis name to derive the key, but
when I checked the code, it's always 64 bytes long, and bytes
that are not used are 0-padded, so, I think we don't need the
length encoding field.
This doesn't work because the shutdown call destroys all the servers then terminates
the processes. But, the problem is that the processes are still
running, but servers are disappearing. So what you end up with is a
panic when the first message gets delivered to a server that no longer
exists, and then you end up with the program hanging on that panic
because some servers haven't been destroyed/exited yet and are now
just spinning waiting to be scheduled. so the call works in the tidy
kernel test cases but to get it to work on a full OS you'd need to
have a way to put every server into an eternal blocking state before
making the shutdown call (to prevent new messages from being issued
while servers are going away), which is more work than i'm willing to
put into the problem right now.
An edge case was missed where a page which was allocated out of
the FSCB and then later written to disk was having its journal
number set to 0.
The root cause of the logic issue is that the page table itself
does not store a journal number. It's assumed that the page table
is ground truth, and is unjournaled.
As long as the pages don't exist in the FSCB, that's not a problem,
the journal number doesn't matter.
But, let's say this sequence happens:
1. FSCB allocates some free space
2. process takes that free space allocation and commits it to its own page table
3. FSCB update log gets a note that free space is now allocated
4. system reboots
5. process reads its page table from disk. There are no journals associated
with the page table on disk, so, it assigns a journal number of 0 by default.
6. we now have a conflict between the Process' view of the journal age
of that page table entry, and the FSCB's view.
The fix to this is during the page table scan, to check and see if
any candidate pages happen to already exist inside the FSCB. If it does,
then you adopt the latest journal number for that page in the FSCB.
This keeps the two views consistent.
A second bug was also identified while fixing this where the "Random"
order we were hoping for in visiting the free space cache as an
iterator is less random than we had hoped. So, there is a patch to
extract the free pages and pull one random one from the pool, using
entropy coming from the TRNG. This is less computationally efficient,
but at least it is correct and has more transparent properties
than relying on the indeterminism of the HashSet implementation.
- add a "dangerous debug" command that only exists when the feature
"pddbtest" is enabled. The feature can't work on real hardware, but
still, it enables things like dumping of the database and storing
the keys for inspection, so it's isolated in a feature that prevents
the code from being compiled in the first place when it's not selected.
- fix a bug where keys that have zero length aren't being created
correctly. this is a common use case for reserving some space for
data that isn't yet available.
I did observe some PDDB corruption that happened today that looks
ostensibly linked to a record that didn't sync up with the disk,
so, let's be safer and sync everything. We can work on optimizing
performance after correctness is guaranteed.
The problem with the current is_mounted() call is that it blocks
while the PDDB is requesting a password or actively formatting
(otherwise it doesn't block). It's actually just...partially blocking.
That doesn't cut it for libstd-linked system services that need to run but
before the PDDB comes up. In particular, we need to be able to respond
to time queries *somehow*, even before our timezone and offsets are
loaded from the PDDB.
I considered doing a callback mechanism for mounting, but this
is kind of complex and probably not worth it. Instead, a separate
thread that can return the value of a single boolean revealing
the mount state of the PDDB is created, and you can create an
object that's capable of inspecting that boolean.
This will create a 100MiB file in tools/ppdb-images/hosted.bin
(sorry!!) which is persistent storage for hosted mode.
Delete the file if you want to start over from scratch.
It's...pretty brittle. I expect problems, but maybe it will at
least get things moving.