xous-core/docs/processes.md

Processes in Xous

Xous supports multiple processes, each with their own address space, with potentially multiple threads inside a single process. Xous has no concept of priority, or indeed even a runqueue. It only knows how to pass messages, and that it should call the idle process when there are no other tasks to run.

Multitasking is technically co-operative, except that there is a userspace server inside susres which does nothing but configure TIMER0 to perform a pre-emption interrupt once every system tick that causes the currently running process to return control to th parent. Thus, the system has the feel of a pre-emptive multi-tasking operating system, but the pre-emption is implemented entirely in userspace.

In addition, there is a ticktimer-server which is used by processes as a resource to manage the flow of time, and to implement polling loops. Processes which need to periodically check on things start a dedicated thread which uses the sleep_ms routine on the ticktimer_server to efficiently de-schedule the process from the run queue. As a result, the ticktimer-server lays the foundation for a user-space nexus that can implement scheduling priority.

This document covers the Xous kernel-native multi-processing implementation, along with several possible implementations of the idle process.

Kernel Processes

Each process is a self-contained entity. It has its own memory space, and can only interact with the kernel by making syscalls. Processes are heavyweight. Each process takes up at least 16 kB of RAM:

• Root pagetable
• At least one leaf pagetable
• Context data
• RAM

Naturally, if a process uses more than one page for .stack + .data, or if the .text section isn't XIP, or if it has a heap, it will require more memory.

Note that because Xous will auto-expand the stack when it expires, to save memory you can structure your program's memory space such that the .data section occupies the upper portion of a page, with stack growing down from beneath. For example:

| Start | End | Description | ===== | === | =========== | 0xd0000900 | 0xd0000ffc | Data section | .... | 0xd00008fc | Stack (grows down)

In this example, stack will start at address 0xd00008fc and occupy the same page as the data section. If it grows past 0xd0000000, then it will pagefault and the kernel will allocate a new page.

Process Data Structures

/// PIDs are 8 bits to reduce the size of the MMU table.
/// This may be revisited at a later time.  A `XousPid` of
/// 0 is invalid, and a `XousPid` of !0 (i.e. 0xff) is owned by the kernel
type XousPid = u8;

/// Represents one process inside the Xous operating system.
/// The memory manager will have memory tables that use the pid to
/// keep track of where memory is allocated, however that is not
/// part of the process table.
struct XousProcess {

    /// Process ID
    pid: XousPid,

    /// Parent Process ID
    ppid: XousPid,

    /// Various process-specific flags
    flags: XousFlags,

    /// MMU offset -- i.e. contents of the `satp` register
    // for this process.
    mmu_offset: usize,

    /// Last address of the stack pointer.
    sp: usize,

    /// The number of bytes that have been allocated to this
    /// process heap.  This can be changed with `sbrk`.
    heap_size: usize,
}

The kernel keeps track of what memory is owned by a particular process, and handles the actual context switch (i.e. remapping the MMU and saving/restoring registers when moving between processes). It also handles message routing.

Message Data Structures

When a Xous process starts a server, the following structure is allocated in the kernel:

struct XousServer {
    /// The textual name of this server, used when calling `client_connect()`
    name: String,

    /// The process ID of the controlling server
    pid: XousPid,

    /// Stack pointer of the thread blocked by `server_receive()`.
    /// If the process is not blocked, then this is `None`.
    sp: Option<usize>,
}

struct XousMessage {
    /// The process ID of the process that sent this message
    pid: XousPid,

    /// The stack pointer of the process that sent this message
    sp: usize,
}

Message Routing

When a process wants to start a message server, it calls server_register() to register itself with a particular name. Then it can call server_receive() in order to actually start to process messages. To reply to a message, the server must call server_reply().

If a process wants to send a message to a server, it calls message_send(). If another process is waiting on a message (by blocking on server_receive()), then that process is activated immediately and it takes over the remaining quantum of the sending process. If there is no pending server, then the process yields its timeslice and the idle process is resumed.

Context Switches

During a context switch away from a process, all of its registers are pushed to its stack, then the stack pointer is saved into the process' sp entry. Similarly, when switching to a process the memory tables are restored and sp is recovered, and then all registers are restored from the stack.

If the kernel has nothing to do, it activates the parent process. If a process sends a message, or calls yield, or indicates that it has no more work to do, the kernel will resume the parent process. If there is no parent process, the kernel will wait for an interrupt, and await a message to be delivered from userspace.

Implementing Multi Threading

note As of Xous 0.8 all the documentation in this article about multi-threading is deprecated. These sections need a complete re-write.

A process' parent is allowed several specialized syscalls. For example, it has the ability to change the sp of a child process. In order to implement threads in userspace, the parent would keep track of what threads are owned by a particular process and then call resume_at() in order to resume a process at a different program counter.

Resource-constrained "Embedded-style"

It is possible to use Xous to create an "embedded-style" PID 1 that runs all computing inside a single process. In this system, there would only be a single process.

An interrupt handler would fire that calls resume_at() on the single process in order to switch between various threads. However, from the perspective of the kernel, this is all taking place within a single process.

Multi-Threaded "Desktop-style"

This describes one possible task launcher that could be used to provided multiple threads across multiple processes.

Note that in this system, spawn() would be implemented as a message that gets sent to the task manager in order to call sys_process_spawn(). This would take care of setting up threads and running the process.

Process Structures

The following process structures are kept within the task manager, hereafter called task_manager:

/// Runnable priority level -- higher values interrupt lower ones
type UserPriority = u8;

// type UserThreadList = [&UserThread];
type UserThreadList = Vec<UserThread>;

enum UserProcessState {
    /// This process does not exist, and is free to be allocated.
    /// Seen when using statically-allocated process tables.
    Empty,

    /// The process was created, but hasn't been set up yet.
    /// It is still missing things such as a text section.
    Created,

    /// Process is ready for execution
    Ready,

    /// This process is currently running
    Running,

    /// All threads are blocking waiting on a message
    WaitMessage,
}

/// Desktop-style process wrapper in userspace around the kernel structure
struct UserProcess {

    /// Kernel PID of this process
    pid: XousPid,

    /// Priority of this process -- higher values interrupt lower ones
    priority: UserPriority,

    /// Current runnable state of this process
    state: UserProcessState,

    /// All threads of this process that share a memory space.
    /// If this goes to 0, then the process must terminate.
    threads: UserThreadList,
}

Threads

Each process has one or more threads. If all of the threads exit, then a process will crash.

/// TIDs are 16-bits so we can have lots of threads per-process.
type UserTid = u16;

/// Current runnable state of an individual thread.
enum UserThreadState {
    /// This thread doesn't exist, and is free for use.  This
    /// is used when thread tables are statically allocated.
    Empty,

    /// Thread is ready for execution
    Runnable,

    /// Thread is currently running
    Active,

    /// Thread has terminated, and this slot is free for reuse
    Terminated,
}

/// One thread of execution. Every process must have at least one
/// thread.  Note that the current definition here is two words (8 bytes)
/// of overhead per thread, plus the stack space required to do a
/// context switch.
struct UserThread {
    /// Thread ID
    tid: UserTid,

    /// Current runnable state of this thread
    state: UserThreadState,

    /// Priority of this thread -- higher values interrupt lower ones
    priority: UserThreadPriority,

    /// Address of the stack pointer.  This is only valid when
    /// the `state` is `Ready`.
    /// Upon context switch, the entire register set is pushed
    /// to the stack and the resulting `$sp` value is stored here.
    /// When resuming, `$sp` is restored first, and then the register
    /// set is restored from the stack.
    sp: usize,

    /// Proces that this thread is waiting on.  When this thread
    /// is to be resumed, this process will be woken up instead.
    wait_process: Option<XousPid>,
}

Process Creation

A process is created with the process_create() syscall:

pub fn sys_process_create(pages: [XousPage], base_address: usize) -> XousResult<XousPid, XousError>

This will move the memory pages passed in to the new process to the base address specified in base_address, and return the new PID. These pages are removed from the current memory space regardless of whether the call succeeds.

1. The kernel allocates a new XousProcess
   • A new MMU page is allocated to the kernel, which will become mmu_offset. This page is cleared to 0.
   • The heap_size is set to 0
   • $sp is set to $STACK_BASE-$REG_COUNT*usize

At this point, the process is ready to be run. The value of $pc should be correctly set in the initial pages that get passed to process_create().

The initial program loader should take care of extending its own address space if necessary. For processes that don't require additional memory, the entire program can be passed to process_create().