e27b30d818
A particular edge case was finally reproduced & fixed. The edge case occurs when a swap is triggered while handling an interrupt, and an OOM happens during the context switch to/from the interrupt. The background on this is that the hardware drivers for the swapper, as well as much of the logical heavy lifting on e.g. what pages to swap out, live in a userspace process known as xous-swapper. The userspace process has to run free of interrupts - when you're OOM'd or trying to swap, interrupts should not happen. The way to do this in Xous is to run that code inside an interurpt context, as interrupts are not interruptable. Thus, the xous swapper userspace consists of two halves: a regular userspace stub that provides some amount of control/debug over the swapper, and a handler that gets run in a "pseudo-interrupt" context. Like all other interrupts, the handler "borrows" the stack of the currently running thread - interrupt contexts don't have an explicit stack, they just grow on top of the current stack of the current thread in the current process at the time of the interrupt. This is broken because when you have a swap event inside an interrupt context, there are conditions where you might not have a stack. What was in fact happening is that the stack pointer was being orphaned as being the "exception stack" - which is a temporary location, only accessible by the kernel, used for storing all the registers going into and out of exceptions. Since no stack pointer is set up, the moment the userspace starts to run and spill its registers to stack, the system trips a "nested swap" event, which leads to a panic because this is not allowed. The fix to this is to devote two hard-wired pages, mapped in the loader, to be the swapper's private stack area. Instead of behaving like an interrupt and borrowing the caller's stack, the swapper instead goes back to its hard-wired stack to do all the processing. This should also improve the performance of swap handling in some cases because we can now guarantee the stack is always in-memory and doesn't need to be swapped in during swap processing. One complication is that the "reserved memory" that the swapper puts aside in case it *does* need to allocate extra pages during handling can't be freed at the top of the OOM routine. Thus if the swapper ends up using new pages, we should expect yet another crash. However, in practice, this doesn't seem to happen, because the cases in which we hit needing a new page are typically related to cases early on in boot where superpages/new page table entries are being allocated, or if we're growing the heap that passes data back and forth between swap and userspace. Because this call triggers in such a specific condition, I hypothesize that actually in practice it should generally be able to run without allocations. One case where maybe it could hit an allocation is if the swapper hit, for the first time, in an interrupt handler that was swapped out, and never hit before, and the interrupt handler needed a superpage to be allocated, thus triggering an allocation request for a new PTE or two. I think this is probably pretty unlikely, but if we do hit it, the fix would be to, for the purposes of targets with "swap" enabled, to mark unmap_memory and map_memory to be legal syscalls in interrupt contexts. These are normally illegal because interrupts should always return - and these calls could potentially fail or block. However, I think in this case it's a trade-off between "we should be failing if we run out of memory" and that failure being a potentially incorrect behavior that is guarded against by a runtime check to discourage such calls. |
||
|---|---|---|
| .. | ||
| bin | ||
| curve25519-dalek-loader | ||
| ed25519-dalek-loader | ||
| sha2-loader | ||
| src | ||
| test | ||
| .gitignore | ||
| build.rs | ||
| Cargo.toml | ||
| LICENSE | ||
| link.x | ||
| README.md | ||
Xous Stage 1 Loader
This contains the stage1 for Xous. This program is responsible for parsing the initial boot arguments, setting up memory maps, and preparing the kernel and its initial programs to run.
This program runs in Machine mode and has access to the entire system. When it jumps to the kernel, the system will be in Supervisor mode with no way to return to Machine mode.
Building
To build the stage-1 module, you will need a riscv32i-unknown-none-elf
or riscv32imac-unknown-none-elf target for Rust.
- Decide what target you want. For simple, embedded systems this could
be
riscv32i-unknown-none-elf, and for more complex systems with compressed instructions you could useriscv32imac-unknown-none-elf. - Get Rust. Go to https://rustup.rs/ and follow its instructions.
- Install the proper toolchain:
rustup target add ${target_arch} - Build the stage-1 loader:
cargo build --release --target ${target_arch} - Get an executable binary:
riscv64-unknown-elf-gcc -O binary target/${target}/xous-stage1 xous-stage1.bin
If you have made changes to the assembly trampoline which sets up
the loader, you will need to manually rebuild this, by running assemble.sh.
This will replace a set of objects in the bin directory.
Using
This requires additional components in order to function:
- A kernel to run, and optionally additional init processes
- A kernel arguments structure to indicate memory regions
- A stage-0 bootloader to set up the arguments
The stage-0 bootloader needs to load the kernel arguments structure into
memory at some address, then set register $a0 (also known as $x10 or
R10 in RISC-V) to point there. It then needs to jump to that address.
For example, in Renode, this is accomplished with the following commands:
sysbus LoadELF @../stage1/target/riscv32i-unknown-none-elf/debug/xous-stage1
sysbus LoadBinary @../tools/args.bin 0x40800000
# Set $a0 to point at the args binary
cpu SetRegisterUnsafe 10 0x40800000
cpu PC 0x20500000
There is no additional setup required.
Testing
Because xous-stage1 spends much of its time setting up pagetables for
RV32, it depends on a system with 32-bit pointers. If you attempt to
run it on a system with 64-bit pointers, it will panic.
The easy solution is to install a 32-bit toolchain and test with that.
To get a list of possible toolchains, run rustc --print target-list
and pick one that starts with i686-, armv7-, or some other 32-bit
arch.
$ rustup target add i686-unknown-linux-musl
15:26:39 info: downloading component 'rust-std' for 'i686-unknown-linux-musl'
15:26:41 info: installing component 'rust-std' for 'i686-unknown-linux-musl'
$
To run tests, use cargo --test. For example:
❯ cargo test --target i686-pc-windows-msvc
Compiling lazy_static v1.4.0
Compiling xous-stage1 v0.1.0 (D:\\Code\\Xous\\stage1)
Finished test [unoptimized + debuginfo] target(s) in 6.32s
Running target\\i686-pc-windows-msvc\\debug\\deps\\xous_stage1-4e16f995c7c21520.exe
running 6 tests
test test::copy_processes ... ok
test test::full_boot ... ok
test test::allocate_regions ... ok
test test::parse_args_bin ... ok
test test::read_initial_config ... ok
test test::tracker_sane ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
❯
Contribution Guidelines
Please see CONTRIBUTING for details on how to make a contribution.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide its terms.
License
Copyright © 2020
Licensed under the Apache License 2.0 LICENSE