e27b30d818
A particular edge case was finally reproduced & fixed. The edge case occurs when a swap is triggered while handling an interrupt, and an OOM happens during the context switch to/from the interrupt. The background on this is that the hardware drivers for the swapper, as well as much of the logical heavy lifting on e.g. what pages to swap out, live in a userspace process known as xous-swapper. The userspace process has to run free of interrupts - when you're OOM'd or trying to swap, interrupts should not happen. The way to do this in Xous is to run that code inside an interurpt context, as interrupts are not interruptable. Thus, the xous swapper userspace consists of two halves: a regular userspace stub that provides some amount of control/debug over the swapper, and a handler that gets run in a "pseudo-interrupt" context. Like all other interrupts, the handler "borrows" the stack of the currently running thread - interrupt contexts don't have an explicit stack, they just grow on top of the current stack of the current thread in the current process at the time of the interrupt. This is broken because when you have a swap event inside an interrupt context, there are conditions where you might not have a stack. What was in fact happening is that the stack pointer was being orphaned as being the "exception stack" - which is a temporary location, only accessible by the kernel, used for storing all the registers going into and out of exceptions. Since no stack pointer is set up, the moment the userspace starts to run and spill its registers to stack, the system trips a "nested swap" event, which leads to a panic because this is not allowed. The fix to this is to devote two hard-wired pages, mapped in the loader, to be the swapper's private stack area. Instead of behaving like an interrupt and borrowing the caller's stack, the swapper instead goes back to its hard-wired stack to do all the processing. This should also improve the performance of swap handling in some cases because we can now guarantee the stack is always in-memory and doesn't need to be swapped in during swap processing. One complication is that the "reserved memory" that the swapper puts aside in case it *does* need to allocate extra pages during handling can't be freed at the top of the OOM routine. Thus if the swapper ends up using new pages, we should expect yet another crash. However, in practice, this doesn't seem to happen, because the cases in which we hit needing a new page are typically related to cases early on in boot where superpages/new page table entries are being allocated, or if we're growing the heap that passes data back and forth between swap and userspace. Because this call triggers in such a specific condition, I hypothesize that actually in practice it should generally be able to run without allocations. One case where maybe it could hit an allocation is if the swapper hit, for the first time, in an interrupt handler that was swapped out, and never hit before, and the interrupt handler needed a superpage to be allocated, thus triggering an allocation request for a new PTE or two. I think this is probably pretty unlikely, but if we do hit it, the fix would be to, for the purposes of targets with "swap" enabled, to mark unmap_memory and map_memory to be legal syscalls in interrupt contexts. These are normally illegal because interrupts should always return - and these calls could potentially fail or block. However, I think in this case it's a trade-off between "we should be failing if we run out of memory" and that failure being a potentially incorrect behavior that is guarded against by a runtime check to discourage such calls. |
||
|---|---|---|
| .. | ||
| aes | ||
| aes-test | ||
| bao-console | ||
| bao-video | ||
| bao1x-emu | ||
| bao1x-hal-service | ||
| bao1x-mbox1 | ||
| bao1x-mbox2 | ||
| codec | ||
| com | ||
| content-plugin-api | ||
| cram-console | ||
| dns | ||
| doc | ||
| early_settings | ||
| ffi-test | ||
| gam | ||
| graphics-server | ||
| ime-frontend | ||
| ime-plugin-api | ||
| ime-plugin-shell | ||
| ime-plugin-tts | ||
| jtag | ||
| kernel-test | ||
| keyboard | ||
| keystore | ||
| libstd-test | ||
| llio | ||
| log-test-client | ||
| modals | ||
| net | ||
| pddb | ||
| root-keys | ||
| shellchat | ||
| skeleton | ||
| spinor | ||
| status | ||
| swaptest1 | ||
| swaptest2 | ||
| test-spawn | ||
| test-swapper | ||
| ticktimer-test-client | ||
| trng | ||
| tts | ||
| usb-bao1x | ||
| usb-device-xous | ||
| usb-test | ||
| xous-log | ||
| xous-names | ||
| xous-susres | ||
| xous-swapper | ||
| xous-ticktimer | ||
| README.md | ||
Xous Services
Xous is a microkernel OS. The kernel itself is a small program, which runs within just one 4kiB page of RAM. The kernel in and of itself hands out pages from the page table, and routes messages. Everything else is managed by "servers".
Interprocess messages are fast and light-weight in Xous. Scalar messages pass data using registers on the stack, and are the fastest messages. Memory messages work by remapping memory pages between processes. send, lend, and lend_mut of memory pages is supported. send creates a full copy of the page, and is non-blocking; lend remaps the page into the borrowing process with read-only privileges, blocking the lending process until the borrowing process returns the page. lend_mut is like lend but with read/write privileges allowed on the lent memory.
The underlaying hardware is entirely memory-mapped; this includes the notional "disk" which is actually FLASH ROM. Every hardware peripheral has its addresses page-aligned, so that individual peripherals may be "owned" by a given server. The kernel itself "owns" no hardware other than a UART for debugging and a private port into the TRNG.
The kernel starts up a core set of servers, which are broadly divided into three categories:
- "Well known" servers
- "HAL" servers
- "Application" servers
Almost all servers in xous are accessed through a 128-bit, cryptographically random SID. These random IDs change with every boot. Thus, in order to connect to a server, one must first resolve its name into an SID. In order to facilitate this process, three "well known" servers are provided:
xous-namesticktimer-serverlog-server
These three servers can be accessed by any process, regardless of their trust level, as their SID is actually a well-known number. The ticktimer-server is well-known so that servers can delay their boot (if necessary; so far it hasn't been). The log-server is well-known so that errors in connecting to the name server can be debugged.Please refer to the README for xous-names for more details on how name resolution works.
Initially, xous-names functions as a permissive oracle that allows kernel servers to freely look up SIDs using the set of names defined in names.rs. Once the kernel is initialized, a lockdown message is sent to xous-names which prevents any further lookups on servers that are meant to be private to the kernel. (TODO: a coherent, maintainable method for declaring private servers to xous-names)
The distinction between "HAL" servers and "Application" servers is fuzzy; but as a rule of thumb, "HAL" servers are private to the kernel and strive for minimal complexity and maximal clarity in their writing. "Application" servers remain discoverable by user applications after the lockdown event, and are responsible for gate-keeping and vetting requests to the HAL layer.
An example of this is the graphics-server and the gam.
The graphics-server is a HAL; any process that can talk to it is capable of drawing pixels on the screen at any location. The graphics-server implements drawing primitives like Rectangle, Line, Circle, and TextView, as message-passing at the pixel level would not be performant. User processes should not be allowed to talk to it, because it would allow user processes to create a fake UI and thus could fool users into typing passwords into fake dialog boxes.
The gam is a Graphical Abstraction Manager. It manages various Canvas objects that have trust levels associated with them. Primitives such as TextView are bound to Canvas objects, and the gam will handle vetting if the objects are allowed to be drawn, clipping, and so forth. Note that the Xous philosophy is that applications do not get to control the exact look-and-feel of their UI; this is to prevent less trusted applications from masquerading as trusted applications, but also assists with things like accessibility and internationalization. For example, applications cannot pick their font; they can only request a style class such as "small", "regular", and "bold". Thus application developers should think of rendering their UI more as hints and directives to the gam, and less as direct pixel pushing.
Conceptually, a complete Xous OS implementation would have HALs that claim the pages of every hardware device, and application servers that claim all the other remaining memory pages -- this includes unused pages and aliased regions -- except for the free memory reserved for applications themselves. Thus, Xous as a kernel does nothing to prevent anyone on its own right from mapping a page; it's up to the complete set of services frameworks to ensure that all the requisite pages have been mapped. (TODO: an audit tool to check the memory space and verify that 1. all CSRs and memory spaces explicitly declared in soc.svd are mapped, and 2. checks that otherwise remaining free space is mapped to a "gutter" that panics if accessed).
We're trying to track progress on implementing the various servers using github projects.
Server Limit
Xous is currently limited to 128 servers. Each potential server reserves 40 bytes of space in the kernel. Xous 0.8 carries about 30 servers, so 128 servers should be reasonably future-proof. Adjusting the limit requires making a change in two places (note the permalink is to an older version of Xous):
Process count is adjusted by altering https://github.com/betrusted-io/xous-core/blob/master/kernel/src/arch/riscv/process.rs#L15. Xous 0.8 carries about 10 processes, and the current limit is 63. Each additional process reserves about 16 or so additional bytes for tracking information.
Well-Known Servers
Here is the list of well-known services:
ticktimer-server-- allows processes to know elapsed time since start, and request sleeps. Also manages WDT.xous-names-- resolves names intoSIDs.log-server-- implements info and error logging. Has a separate UART from the kernel UART.
Hal Servers
Here are a list of HALs, implemented and projected, and their functions:
com-- manages requests to and from the ECgraphics-server-- manages the frame buffer and basic drawing primitives. Talks to the MEMLCDkeyboard-- key matrix management; debounce; ScanCode conversion. Keyboard layouts (qwerty, dvorak, azerty, qwertz, braille) are interpreted in this servertrng-- manages the TRNG hardware, provides TRNGs for other processesllio-- manages I2C, RTC, GPIO, pin interrupts, soft reboot, and power pins. Also home for info, build IDs, etc.codec-- basic buffering of frames into and out of the audio CODECjtag-- manages the JTAG interface (used for key fusing)keys-- management of cryptographic key storeengine25519-- Engine25519 interfacesha512-- SHA512 interfacespinor-- manages the erasure and programming of the SPINORram-- manages allocation of RAM for applicationsscheduler-- manages thread scheduling and priorities
Application Servers
Here are a list of application servers, implemented and projected, and their functions:
fcc-agent-- use-once -- a special agent meant to run on a minimal Xous for FCC testing. Should never be compiled into production firmware.trng-tester-- use-once -- a special server used to facilitate testing of the TRNG. Pipes TRNG output to memory regions that can be read out usingbt-rngdand fed into Dieharder for analysis.gam-- pre-alpha -- managesCanvasobjects, and provides an abstract framework for applications. Also manages status bar, context menus and pop-up notifications.ime-- liases withkeyboardandgamto handle keyboard inputpddb-- plausibly deniable database. Used in lieu of a conventional filesystem for storing key/value pairs in a plausibly deniable fashion.update-ec-- manages the updating of the ECupdate-soc-- manages remote (non-USB) updates of the FPGA and kernelnet-- manages connections to the Internetwifi-- manages wifi configurationpower-- intermediates requests to the backlight, battery status, charging, RTC, etc.accel-- intermediates requests to the accelerometeraudio-- intermediates requests to the audio hardware. Does stream mixing, etc.usb-- handles USB connectionscredentials-- trusted PIN/password entry mechanism, manages currently activated credentials
Applications
Here are a list of user applications:
shell-- refactoring -- provides a "chat-like" interface for talking to Xouschat-- text and audio secure chat applauncher-- an application launcherpassword-- a password vault