previously was setting all pages as USER even if they shouldn't
be. Trust the flags are set correctly, just mark the valid bit.
Also, detect swaps in an IRQ handler and manage accordingly.
In particular, don't re-disable IRQs (as it overwrites the
saved IRQ mask), don't re-enable IRQs on exit, and don't
modify process Run states.
- no swapping of kernel pages
- mark L1 PT's properly in the memory allocation table
(was accidentally marking them with the VA of the first address
allocated)
the cases are delineated but still getting a fault on return
from the handler.
also optimize out a memory space swap that was incurred due to
inappropriate abstraction boundaries.
- invoking the swapper context is different in an IRQ because
we don't want to modify the runnability states of the original
process
- didn't want to make a custom version of swap resume but
with the IRQ issue it made sense. In the process of doing this,
we can also optimize out a memory space transition, so it's
probably worth it even if we're not entirely going down the
right path to fix the bug of interest.
probably should have done this in smaller commit chunks but it
was hard to say if this was going in the right direction until now.
at the moment, system can boot and evict quite a bit of memory before
hitting a panic. The current state is that we panic on returning
from an interrupt handler that hit a swapped page.
This is a pretty decent point to check point everything. hopefully
this next arc will be a bit cleaner.
some thought needs to go into when and how we call oom_doom
but let's test more core functionality of the swapper before we
worry about that. I can think of several places where we could
use that call, the main tricky bit is that oom_doom diverges
and so wherever we patch it has to be able to resume. If we put
it into e.g. somewhere in the main memory alloc loop, the trouble
is we would have to track all of the multiple places that we
could have arrived there...ideally we can find a single spot
to patch this, with a clean entry and exit path.
instead of sending scalar messages on every page alloc/free, which
is a lot of traffic, introduce an API where the swapper can request
a snapshot of the alloc status by mapping the kernel's alloc
table as a read-only structure into the swapper's memory space,
and then invoking the swapper in its interrupt context to read that out.
The structure is unmapped on return from the context.
additionally, a running tally of free space is now maintained
and this can be queried with a fast syscall.
Thus, the normal workflow is the swapper would have a timer that
periodically queries the kernel's free space level, and when it
falls below a critical amount, it will invoke the expensive snapshot.
this can also be extended so that the kernel can call into
the swapper's interrupt context with a "help, OOM imminent"
message down the road.
I think that structurally we might never exceed a nesting depth
of 2, but there are some specific cases where an alloc can call
an alloc in the kernel (e.g., having to create a new superpage
in the midst of a routine alloc), therefore, we have to handle
some nesting.
There's also chance that maybe nesting could be due to race
conditions or self-circular behavior in the swapper, but ideally
we avoid that entirely because we can't handle non-structural,
unbounded nesting of re-entry to the swapper routine.
We want to use the finish_callback_and_resume() function because
it's well-tested and sets up everything just right, but it expects
that we're coming from another process. Unfortunately, in the
middle of all of this, we need to map a page back into the
destination process' memory space, so we end up starting from
the swapper's memory space, changing into the destination process'
memory space, then mapping the page, then changing back into
the swapper's memory space, then using the finish_callback_and_resume()
call.
I think the impact might not be *that* bad, mainly it's a TLB
miss and the PTW isn't super bad, and this comes after a disastrously
expensive decrypt-from-RAM operation -- so probably not worth it
to angonize over optimizing this bit at this point, and stick
with well-tested primitives.
There's two ways we can enter the swapper:
1. via an interrupt due to a page fault
2. via a syscall that generated a memory event (e.g. unmap)
This patch handles case (2) better and clarifies variable names
around that to make the code read better.
It also has logic to handle divergent looping through a large
unmap request.
The unmap syscall is a little bit tricky, because it can de-allocate
several pages at once, which can trigger an advisory de-allocate
message to the swapper in the middle of the loop. The entire structure
has to be re-written to handle this case, so we patch this right at
the top level syscall point and handle it entirely with custom
logic in the swapper.
@xobs made the `state` element of a process publicly read-only visible
through a function call. I don't think this is a security issue but
please lmk if that's a problem.
for now, we just count how mayn pages we missed an advisory on,
those all effectively are "wired". If we're fast enough on registering,
it turns out we actually miss nothing.
DUe to https://github.com/rust-lang/rust/issues/114447 direct access to
static mut variables causes a warning which will be turned into an
error in the 2024 edition.
The workaround is to use `addr_of!()` and `addr_of_mut!()` macros and
then dereference that.
Signed-off-by: Sean Cross <sean@xobs.io>
