Commit graph

392 commits

Author SHA1 Message Date
bunnie
be227adde1 removed sized strings from pddb interchange structures
I think this is safe to do because no strings can be directly
serialized into the PDDB's structures. The place to check for
this is inside the backend/ directory. For example, `DictName`
is where the name field is serialized into a structure that
goes onto the disk. It's explicitly length-checked.

What this means is that we could have calls into the pddb crate
with invalid length strings, however, if you check the lib
implementation, all of the APIs also pre-flight the length of
arguments to ensure they fit into the destination string.

So mostly what we're losing is the constraints on the intermediate
IPC representation, but the length is still checked on either side.
2024-09-22 15:44:05 +08:00
bunnie
34e6f46d63 formatting changes that resulted from the regex search
a bunch of lines are shorter or different length now which
triggers a mandatory reformat. this patch cleans that up,
so that going forward the manual rework patches are clearer and
not tied up with formatting patches.
2024-09-22 15:31:14 +08:00
bunnie
f2fbc7fafd regex search/replace instances of xous_ipc::String to String
more regex results
2024-09-22 14:19:32 +08:00
bunnie
0c7be94c22 re-introduce "wait for hardware" hashes for "big hashes" 2024-02-01 00:52:21 +08:00
bunnie
9a5fac408c update pddb to use new sha api
there is one interesting thing that happens with the api
incompatibilities in that importing the KeyInit trait affects
both AES and SHA identically, so this trait now has to be imported
just at the function level instead of crate level because we don't
want it to affect both algorithms so broadly.
2024-01-31 04:11:30 +08:00
bunnie
659796916a chore: cleanup typos in docs, comments 2024-01-15 02:16:43 +08:00
bunnie
b8dc0a981c edge cases in cargo-fmt
so...i think in this case, maybe the build.rs wasn't formatted?
it got picked up the second time when i ran the command.

in the case of pddbkey.rs, I think the changes made by rustfmt
were so dramatic it required two passes for it to become clean.
2024-01-15 00:56:24 +08:00
bunnie
9b211d4cb0 work around rustfmt issue 2024-01-14 23:23:05 +08:00
bunnie
679435c5f4 format sevices/pddb
wow the formatter hated this code.

hope i can recognize it later on when it needs work,
nobody else probably wants to touch it. :-/
2024-01-14 22:35:23 +08:00
bunnie
8ec08c9c76 fix rust v1.75.0 warnings that are probably benign
a lot of re-import warnings have cropped up. While I appreciate
their studiousness on this, the problem is that removing some
of these exports in the API crate could mean that future users
who want access to some API features that aren't used in this
configuration are now unaccessible.

I think it's not a huge problem at the moment, just spam-cleaning
the warnings because there are too many to deal with carefully
at the moment.
2024-01-04 19:15:28 +08:00
bunnie
2a521b5b86 fix stdseek to match std word ordering 2023-12-10 12:53:50 +08:00
bunnie
0a0ac292c3 modernize opcode debug on PDDB 2023-12-07 18:41:51 +08:00
bunnie
7da070f37c fix #460 - prevent PDDB mount dialog from popping up during updates
check the newly-minted EC ready bit prior to running the PDDB mount
handler.
2023-12-07 01:31:06 +08:00
bunnie
5aff0f2413 cleanup doc comment errors 2023-12-01 22:08:34 +08:00
Sean Cross
5dfad296ae xous: mark MemoryRange::as_slice and friends as unsafe
Mark `MemoryRange::as_slice()` and `MemoryRange::as_slice_mut()` as
unsafe.

This is due to the fact that not all representable types can be cast to
an arbitrary value. For example, if Process A sends random values to
Process B, then Process B casts the memory to a slice of `bool`, then
any values that are not 0 or 1 are undefined.

Because these functions can leak UB into the program, they are marked as
unsafe.

This patch adds `unsafe {}` blocks to all instances where this function
is called. Additionally, all instances have been checked to verify that
they do not cast to any invalid types -- most use `u8` or `u32`.

Additionally, the long-deprecated `MemoryRange::from_parts()` call was
removed, since that causes UB without being marked as unsafe.

Signed-off-by: Sean Cross <sean@xobs.io>
2023-09-12 10:07:38 +08:00
bunnie
f0b351c2e0 suggested method to preserve intent of drop to fix error
However, I didn't write this library (and I don't have a test
suite for it), so I'm not quite sure if this preserves the
invariants on the fd_mapping() structure. Needs @xobs' review.
2023-07-16 17:58:40 +08:00
bunnie
9932d971c4 clean up the warnings that are deemed safe to clean up
one warning remains, will address this in a future PR.
2023-07-16 17:55:28 +08:00
bunnie
b2c64eb61a aesthetic cleanup; change default font for all modals
I decided to go back on the idea of not changing the modals
and now all modals use the new larger font as well.

This creates issue #396, which I will deal with later once
I have the time to do some more testing. I'm pretty sure
some other dialog boxes are broken now too.
2023-07-05 15:13:38 +08:00
bunnie
ced2502559 remove hard-fault on restore that I think is not critical
prints to console instead. could monitor with serial port
2023-06-21 03:43:19 +08:00
Jean-Pierre De Jesus DIAZ
2af737b57e
xous-rs: move LANG variable to locales
This change is made with the intention of making the xous-rs crate a bit
more freestanding without depending on the xtask build system.

Signed-off-by: Jean-Pierre De Jesus DIAZ <me@jeandudey.tech>
2023-06-15 18:23:12 +02:00
bunnie
fa436c0845 fix bugs so that the implementation is working
as suspected, the shared lock is killer on performance.
2023-06-05 22:50:59 +08:00
bunnie
4d118e72b8 fix 355
The resolution is to close the previously open basis if one with
the smae name is opened. This is in part because there isn't
a universally sensible/deterministic way of resolving behavior
in this case, and partially because the structure of the basis
map is a hash map keyed by name ... to differentiate basis with
same name/different password would require...leaking some information
about the password? i guess we could go strictly by opening order
by adding an opening parameter but basis management is already
confusing enough.

A message is popped up to indicate when the previous basis is
being unmounted.
2023-04-07 14:51:37 +08:00
bunnie
a59de6d655 cleanup hosted mode warning (new with rust 1.68) 2023-03-17 22:19:04 +08:00
bunnie
fa20b90973 add rand_core trait to functions that require fill_bytes() 2023-02-26 01:01:30 +08:00
bunnie
d1f59a68e9 induce pddb-ci to exit
it exits with a panic, but at least it exits now!
2023-01-31 16:04:10 +08:00
bunnie
2b6e0d2b89 fix return code bug on bulk deletion 2023-01-29 17:32:41 +08:00
bunnie
004599543a add the evicted record and remove a debug line i forgot to trim 2023-01-29 04:26:08 +08:00
bunnie
e500588360 fix bugs in key invalidation/eviction logic
1. track evictions as separate from cleanliness. not clean means
the data is dirty and needs writing back. eviction simply means
the data needs be re-retrieved from disk.

2. actually fill the data when we do a sync -- previously
we only copied the old data into the new page but then forgot
to allocate the data and assign it to the cache (oops)

3. sync the small pools after doing a remove inside the update()
command to make sure we've met invariants for future calls
to cache fills

so many corner cases, i'm going around in circles....
2023-01-29 04:21:31 +08:00
bunnie
99a88ff25a only invalidate the key entry if it is within the current range of interest
previously, we were invalidating at the first instance of
encountering the key; instead we should only invalidate once
we've processed it in our range of interest.
2023-01-29 04:19:24 +08:00
bunnie
70c41ef244 increase threshold for GC on PDDB
for some reason it's being called a bit earlier than i expected.
2023-01-29 04:17:57 +08:00
bunnie
df2dda32be add api call to increase PDDB debug level at runtime 2023-01-29 04:17:42 +08:00
bunnie
145e0469bc fix another edge case in key deletion
invalidated keys that were not dirty were not being removed
also added routine to fill keys that are evicted from the cache
during the update routine
2023-01-28 19:08:01 +08:00
bunnie
7cb98699aa add some consistency check and repair facility to PDDB 2023-01-27 15:26:46 +08:00
bunnie
9c5efebf38 add bulk deletion of keys from a single dictionary 2023-01-27 01:43:24 +08:00
bunnie
c3ab5ca126 fix bug where old key data was not being zeroized on delete
on valid pages were being updated in a dictionary entry as
it is updated; actually, entries that are invalid should also
be zeroized, otheriwse they "Stick around" as valid entries.
2023-01-26 23:32:22 +08:00
bunnie
f6a8d678cc fix bug in small pool re-packing after delete
the new offset address was being used for copying data, instead
of the old offset, which was causing key data to refer to
an incorrect entry after deletion
2023-01-26 22:05:03 +08:00
bunnie
31d958d41e improve debug output on PDDB 2023-01-26 18:01:28 +08:00
bunnie
705427b2c4 Merge branch 'fido21' into main 2023-01-26 04:43:27 +08:00
bunnie
78866199ea fix zero-length key readback error in bulk reads
zero-length keys return a data section of None, instead
of Some(<empty vec>), as the latter representation deserializes
to a null pointer panic.
2023-01-25 23:17:30 +08:00
bunnie
b7d5eca720 fix missing corner case in cache refill 2023-01-25 13:50:06 +08:00
bunnie
c083a9ca85 chore: fix typos 2023-01-25 00:55:32 +08:00
bunnie
71a1dddf15 fix some more edge cases in PDDB key deletion 2023-01-20 18:07:24 +08:00
bunnie
ea913a6fb4 fix initialization order so that CI can run 2023-01-20 18:07:09 +08:00
bunnie
fb648b4e21 Merge branch 'maybe-fix-delete' into main 2023-01-19 23:53:49 +08:00
bunnie
0d65d9a7ca chore: fix another typo 2023-01-19 15:04:00 +08:00
bunnie
71294f7591 add a missing sync after key deletion 2023-01-19 14:46:25 +08:00
Sean Cross
dc4ca79834 services: pddb: use an enum for file types
Rather than hardcoding numbers, use an enum with the constant values.
This will make it easier to reason about what the constants mean.

Signed-off-by: Sean Cross <sean@xobs.io>
2023-01-18 09:06:16 +08:00
bunnie
f71278b422 fix issue #109 - refactor key fill/scan to not crash after pruning
After a prune, key fill/scan can crash because there is an assumption
that data is always resident in the cache for a small key.

This code handles the case where the small key data was evicted during
a prune, and reads the data back in just for that key on demand.

Furthermore, during a `sync` call, it will pull in the key data
that corresponds to the page containing just the small key, allowing
that page to be re-composed and replaced in memory. caching is used
on the decryption of the small key page so the expensive bit
of the call should only happen once per small key bank.

tested with pddb-ci and manually in hosted mode.

A portion of code was re-written to make it clear that only the
length of allocated data is copied to/from disk and memory, instead
of the reserved space as the previous code would indicate. The reserved
space is guaranteed to be zero because the RAM-based page for composing
the small key block is zero-ized prior to operations; left comments
reminding me of this in the future so I don't accidentally optimize
something out and cause data leakage between small keys because I'm
allowing the reserved areas to contain stale data.
2022-12-29 22:04:55 +08:00
bunnie
e25da3a720 add implemenation of the prune debug call
this allows us to trigger the cache prune manually, instead of
in response to memory backpressure only
2022-12-29 22:04:15 +08:00
bunnie
1e56598b10 chore: fix some typos 2022-12-29 00:53:32 +08:00
bunnie
5370c333da add callback for monitoring basis lock/unlock events 2022-12-29 00:10:42 +08:00
bunnie
ad79ebf8ba chore: more typos. no really, i can stop at any time. 2022-12-13 23:02:09 +08:00
bunnie
b1b6d46d27 chore: more typos
probably should not have tried to spell check this, now i'm wasting
my time.
2022-12-13 22:56:37 +08:00
bunnie
52955681d5 reflect unmount status after tryunmount
this fixes a problem where backups fail when autobacklight is
enabled, because autobacklight is constantly querying the PDDB
for the length of the autobacklight timeout, even after the
PDDB is unmounted.
2022-12-02 20:10:10 +08:00
bunnie
c3ba30d55d expand on errors reported 2022-11-09 14:25:26 +08:00
bunnie
0646ec23ba track "attempted" mounts in PDDB
this is to allow a UX process to clear the "waiting for boot"
indicator once an attempt has completed.

right now, it responds *after* the mount process is done,
so the "please waiting" text persists even doing a format.

I think that is...appropriate?
2022-11-07 00:21:52 +08:00
bunnie
3ee36f41ee remove UX delays on PDDB path
retain just one of them going into the erase routine, because
it consumes enough CPU that the dialog box takes a bit too long to
pop up.
2022-11-06 18:45:17 +08:00
bunnie
fa90e74a10
Merge pull request #270 from gsora/feat/pddb-init-notification
feat: show boot process notifications for PDDB
2022-11-06 03:40:17 +08:00
bunnie
7eecf2b1ce remove the initial wait from PDDB; move the pop-up modal location
The initial wait message is now rendered by the first app
that owns the screen (currently shellchat).

The location where the pop-up modal is rendered is also now bracketed
directly around the mount call to avoid interference with other
edge cases that may pop up modals.
2022-11-06 03:07:35 +08:00
Gianguido Sorà
d277ae9551 feat: show boot process notifications for PDDB
This commit shows two notifications:
 1. as soon as the device boots, a notification greets the user and
 tells them everything is fine, the device is booting
 2. as soon as the user enters (or doesn't) enter the password, a
 notification tell the user Xous is loading and mounting PDDB

I spent too much time on this but I believe it's a nice UX improvement.
2022-11-04 17:31:27 +01:00
bunnie
5de44a9502 don't fail if keyattribute not found in current basis when no basis specified
This is not actually an error, we just need to try another basis
down the list. This is a copypasta bug that was only caught now. :-/
2022-11-04 17:05:51 +08:00
bunnie
0d9a39a349 print debug information when a key goes missing in bulk read
this helps debug problems down the road
2022-11-04 17:05:32 +08:00
bunnie
d74673ab29 allow main menu to be used if there are no root keys 2022-11-04 03:55:57 +08:00
bunnie
80a1bd3654 resolve issue #267 2022-11-03 23:18:13 +08:00
bunnie
ff2a7f8240 some more debugging to help trace through dict sync issues 2022-11-03 21:06:04 +08:00
bunnie
b04dcc21ef add smalldb flag, which builds a PDDB with a size of 4MiB
do not interchange this on devices with the full sized
PDDB, you may corrupt the database
2022-11-03 21:04:14 +08:00
bunnie
719020b08b don't allow polls to the unmount to turn true until other callbacks have been allowed 2022-11-03 17:23:11 +08:00
bunnie
7af04b7000 a bit more debugging around dictionary creation 2022-11-03 01:38:18 +08:00
bunnie
d63c452fb6 catch serialization buffer edge case
if the buffer fills up to the point where there isn't enough
space for the 8-bytes of header, a split_at() call will fail before
the routine even gets to decide if the record can fit.

abort the loop if the buffer is that full.
2022-10-27 23:00:10 +08:00
bunnie
c82a79dab9 add performance monitoring in for bulk reads 2022-10-27 16:45:04 +08:00
bunnie
bef734553e clarify some comments and clean up debug messages 2022-10-27 02:53:38 +08:00
bunnie
6992b31104 fix the bufserializer approach
this is much more efficient than the alloc serializer
2022-10-27 02:41:46 +08:00
bunnie
6272a94604 intermediate attempt to extract a length parameter 2022-10-27 01:57:42 +08:00
bunnie
ac01088ec7 Merge branch 'pddb-bulk' into pddb-bulk-bufserializer 2022-10-27 01:30:27 +08:00
bunnie
3d1885e165 add test code
results are slightly better on the first run (3700ms instead of 4900ms)
but there seems to be really excessive allocations using the heap
allocated serializer by rkyv. We really need to use the bufserializer
I think for this to work.
2022-10-27 01:10:17 +08:00
bunnie
721dbc82d7 add an attempt to use bufserializer that didn't work 2022-10-26 23:29:43 +08:00
bunnie
d9bffb3556 add some notes about making this more efficient 2022-10-26 22:06:00 +08:00
bunnie
665e7dce51 fix minor bugs, now passing basic test 2022-10-26 15:16:40 +08:00
bunnie
bffe92cab9 add first draft of bulk serialize/deserialize functions 2022-10-26 05:29:44 +08:00
bunnie
2e768dd19e Merge branch 'main' of github.com:betrusted-io/xous-core into pddb-bulk 2022-10-25 23:15:51 +08:00
bunnie
9e39c432bd Merge branch 'main' into pddb-bulk 2022-10-25 17:41:58 +08:00
bunnie
f5289be6fe experiment with 'std' in rkyv for pddb crate 2022-10-25 17:41:12 +08:00
bunnie
0ec75861e6 cleanup a normalization that escaped due to merge conflict 2022-10-25 17:40:18 +08:00
bunnie
30ea893c74
Merge branch 'main' into xous-rs-os-selection 2022-10-25 17:28:33 +08:00
bunnie
67fc21fbdd harmonize hosted mode selection, but retain an explicit hosted flag
In general, hosted mode is assumed "by default" if the target
is not Xous. This allows packages to be built cleanly on command
lines that don't know to pass a --feature flag to specify a target.

However, an explicit "hosted" flag is still retained so that we
can confirm the build system intent. This saves us from the failure
mode that we thought we were building for target A, but accidentally
ommitted the flag, and instead we've been silently building for
"hosted" mode without any warning. The main consumer of the "hosted"
flag is now just the build.rs script inside of the utralib.
2022-10-25 17:21:02 +08:00
bunnie
97f0d34daa more analytics on PDDB debugging 2022-10-24 02:35:55 +08:00
bunnie
83d31ff8b9 fix non-perf builds for pddb 2022-10-24 00:37:05 +08:00
bunnie
e515092636 add performance profiling options to PDDB 2022-10-23 23:14:57 +08:00
bunnie
fc8f2a25e8 expand on "internal error" for .get() results from PDDB 2022-10-20 17:40:11 +08:00
bunnie
d7392d3704 improve comments and add count to the user-failed mount case 2022-10-17 03:51:36 +08:00
bunnie
de41e5ecff rework password login to have a count and reboot/suspend after N retries
The implementation is a bit clumsy because the original
premise of the login loop is that it should be infalliable, and
because of the isolation of the password mechanism there isn't
a great way to pass number of failures back and forth across
that firewall (and I don't think there should be).

The application calling the password routine tracks failures
but the mounting routine has to kick that up to the main loop.
However, the PDDB main loop can't get stuck on the mount attempt
if you want to suspend, because it has to respond to the suspend
request.

So the failure message has to get kicked all the way back to the
status thread, which is what kicks off the mount, and then handled
there.
2022-10-17 02:40:51 +08:00
bunnie
ec1d1b4444 don't try to unmount anything if nothing was mounted 2022-10-17 00:32:06 +08:00
bunnie
f815ed85b5 fix rust 1.64.0 rkyv warning regressions on unused enum types
rkyv-derived enums throw warnings that they are unused.
Not sure how to get rid of them; I've tried surrounding them in
`#[allow(dead_code)]` directives, and also explicitly creating
each enum variant that I'm aware of through some dummy code and
some log prints of the dummy code calls to ensure the results are
used, and the warnings still persist.

Significantly, the warnings only happen for enums that take
a value. Simple rkyv enums do not throw a warning.

These warnings overwhelm the build logs, so they need to be isolated
in order for us to ensure that the build is clean of other more
important warnings.

The method chosen to isolate them is to take the enums that are
problematic, carve them into their own special file, and add
a file-wide `#![allow(dead_code)]` directive. This only happens
in 7 service crates, and this change commits all of the modifications
in a single big commit that could be undone later if this turns out
to be a compiler regression or something similar.

I also opened an issue with the rkyv maintainer to see what his
opinion is on the situation.

https://github.com/rkyv/rkyv/issues/318
2022-10-09 17:17:20 +08:00
bunnie
2ff9163789 add checksums to PDDB backup
every 1MiB block is checksummed with a sha512/256 truncated to
128 bits. This is used to confirm the media and transport integrity
only. It is not intended to be a cryptographic check, it just confirms
that the backup data was transferred correctly assuming no intentionally
malicious actors in the middle.
2022-10-09 01:10:42 +08:00
bunnie
62af36560b checks for cache coherence issues
Most of the changes introduced in this commit were rolled back,
but, there is now a test facility that's hanging out for the PDDB
to help with future debug efforts.

The patch was motivated by a problem observed in FSCB consistency
when it's updated. Ultimately the root cause was traced to a
cache coherence problem and it's fixed by a patch in the spinor
crate.

However, this patch still retains a check that the data is coherent,
in case the flushing routine isn't complete or some other factor
was missed. It also includes a large amount of commented-out code
that keeps a bit of history used to debug the problem -- eventually,
we can just cull that but it's kept around in case we got the
root cause wrong (or there were multiple causes and coherence is
just one of them).
2022-10-07 18:52:57 +08:00
bunnie
4e40ae6c5d move from target selectors to feature selectors for back-ends
This will allow for e.g. xous on riscv32-imac but not a precursor
target, for example: by using feature selectors we get to have
a bigger range of targets that we support, such as for implementations
that haven't been upstreamed or have official target-triples.

The downsides are:

- patch override libraries don't absorb feature flags. so, for
example, the AES crate that is used to patch other crates won't
be aware of the target features without modifying the referencing
crates.

- features are normally meant to be additive, but target features
are mutually exclusive. The mutex is enforced by a check in
utralib that throws a panic on build if two targets are accidentally
selected at the same time.
2022-10-02 02:56:44 +08:00
bunnie
85279566f0 fix #77
This adds some UX feedback in the edge case that a suspend could not
happen due to a blocking sever, e.g. a sha512 hash pending or
a SPINOR flash write/erase pending.

The UI will tell the user to retry again; usually the ops in
a normally functioning system for hashes or SPI are very short,
so we could almost just retry automatically but I think for now
I want to see if this edge case actually happens.
2022-09-24 02:40:11 +08:00
bunnie
9225cbaba3 resolve issue #206 by adding check for existing basis 2022-09-20 23:06:22 +08:00
bunnie
4ae3308fbe add PIN change to the PDDB 2022-08-12 23:10:18 +08:00
bunnie
2b4ce52a10 fix some regressions on empty key listings 2022-08-02 02:52:04 +08:00