In general, hosted mode is assumed "by default" if the target
is not Xous. This allows packages to be built cleanly on command
lines that don't know to pass a --feature flag to specify a target.
However, an explicit "hosted" flag is still retained so that we
can confirm the build system intent. This saves us from the failure
mode that we thought we were building for target A, but accidentally
ommitted the flag, and instead we've been silently building for
"hosted" mode without any warning. The main consumer of the "hosted"
flag is now just the build.rs script inside of the utralib.
rkyv-derived enums throw warnings that they are unused.
Not sure how to get rid of them; I've tried surrounding them in
`#[allow(dead_code)]` directives, and also explicitly creating
each enum variant that I'm aware of through some dummy code and
some log prints of the dummy code calls to ensure the results are
used, and the warnings still persist.
Significantly, the warnings only happen for enums that take
a value. Simple rkyv enums do not throw a warning.
These warnings overwhelm the build logs, so they need to be isolated
in order for us to ensure that the build is clean of other more
important warnings.
The method chosen to isolate them is to take the enums that are
problematic, carve them into their own special file, and add
a file-wide `#![allow(dead_code)]` directive. This only happens
in 7 service crates, and this change commits all of the modifications
in a single big commit that could be undone later if this turns out
to be a compiler regression or something similar.
I also opened an issue with the rkyv maintainer to see what his
opinion is on the situation.
https://github.com/rkyv/rkyv/issues/318
This will allow for e.g. xous on riscv32-imac but not a precursor
target, for example: by using feature selectors we get to have
a bigger range of targets that we support, such as for implementations
that haven't been upstreamed or have official target-triples.
The downsides are:
- patch override libraries don't absorb feature flags. so, for
example, the AES crate that is used to patch other crates won't
be aware of the target features without modifying the referencing
crates.
- features are normally meant to be additive, but target features
are mutually exclusive. The mutex is enforced by a check in
utralib that throws a panic on build if two targets are accidentally
selected at the same time.
fully integrates wifi on and off commands. measured standby power
when wifi is off is "immeasurable" (0 mA), so probably < 1-2mA.
fix ui transitions so that the off request is respected, through
sleep/resume cycles as well.
instead of sharing the wifi buffer loopback has its own buffer.
this eliminates a race condition that happens whenever an Rx
happens to happen on the wifi interface while there's loopback
traffic in progress.
also, i see a lot of opportunities to optimize the net stack now
that i have tweaked this path; it might be interesting to break
the receive interrupt out directly into the COM crate instead
of passing it through LLIO then NET then COM and having COM
drain the RX buffer immediately into a local "infinite" sized
queue.
The llio->net->com loop exists because the interrupt handler
for all the i/o is shared within one hardware block, but maybe
i can create an additional interrupt responder for the com
interface just to optimize things a bit in a future SOC update.
Alternatively, I could move all the interrupts into COM and not
have them resident in LLIO. This would optimize for COM latency
at the expence of I2C latency, but that's probably OK because
I2C is much less latency sensitive (i2c and com interrupt, as
mentioned above, share the same hardware block for external
pin-based interrupts).
- Peek now has a handler for reads that can't be met immediately
- add a count to the number of times a close socket has been polled,
if it exceeds a threshold close the socket anyways. This is a compromise
between trying to figure out how to implement a fully non-blocking TcpClose,
and having an interlocked one that cleans up state but ends up blocking
the case where you have, in the same thread, a sender and receiver
(which is ironically somehow legal and does not cause deadlock,
because the TcpClose in Unix is not blocking on internal state
being cleared up).
0 is a "valid" port and it means OS please give me a port.
This implements a method where a random port is chosen until
you find one that's not occupied.
If a Stream is opened and a process isn't constantly polling it,
it'll never close if the remote side goes away -- you'll just
end up hanging in a FIN-WAIT-2 condition.
Add some logic to detect this and close out streams automatically
where the remote has hung up.
I'm not actually sure this is what you're *supposed* to do, but,
lacking a formal specification, all I can do is try to get
this thing to pass the test bench. And pass it does.
TcpClose should not immediately remove the socket. It
has to wait until the FIN comes. Add a fin waiter state.
Also rewrite to use "retain". Woo. I'm starting to get
the hang of Rust closures. Only took a couple years.
RxWait gets stuck in CLOSE_WAIT state. It's a state where
you "could" transmit, but you can't receive anymore.
In this case respond with a 0-length receive? That seems
to be what leads to the right behavior. I would have thought
you'd want to respond with a timeout error but that doesn't
match what's in the test bench.
- add 127.0.0.1 to the list of IPs listened to by smoltcp
- add some static Atomic types to store the IP and MAC address
so we can process packets in other areas to detect the loopback
state
- add a routine to extract the loopback packets when they happen
- bunch of debug visibility tweaks
loopback packets are intercepted on the way to the COM and
re-injected directly into the Rx queue, along with a message
sent back to the main loop to pump the Rx buffer to extract the
loopback packet.
the CM can't always detect if the EC changes state; this is
especially true after an update or a forced reset where
the state caches are not reliably updated. If no packets are
seen for a while, poll the actual state and force a sync-up.
Also try to make sure that the interrupts are enabled correctly.
This macro will be removed, since it is no longer necessary. Remove this
macro across the project and replace it with a standard `main()`
function.
Signed-off-by: Sean Cross <sean@xobs.io>
smoltcp is written to allow multiple adapters, so 0.0.0.0 as
the binding for UDP is illegal. However, in Rust, it seems that
0.0.0.0 is "totally fine" and most OSes will resolve this to
a "default" adapter.
this patch introduces the behavior of taking 0.0.0.0 and mapping
it to a default adapter. In fact, we only allow binding to the
default adapter. The reasoning behind this is that if our IP
address changes, we would expect that 0.0.0.0 resolves to that.
Where this scheme fails is if someone binds to 127.0.0.1 expecting
it to not respond to packets coming in from the outside, e.g.
they are using UDP as a loopback interface. I think this isn't
a common use case for Xous, but if it does become one I suppose
we could add a code path to also check for the 127.0.0.1 case
and try to handle it...
smoltcp requires some polling to be done in the future for it
to work with low latency. It returns a time in the future for
the next poll.
This refactors the polling algorithm to:
1. use a fixed thread pool. This burns a little extra memory in
exchange for not having to allocate/de-allocate threads every
time a follow-up is needed.
2. use a short/long scheduler, where short delays (shorter than
the scheduler quantum) are grouped together, and only a single
follow-up is fired off within that time range if needed. Longer
delays have a larger window (roughly +/- 100ms) of grouping,
so fewer such events are also required in the distant future.
Empirically the delays seem to fall into two groups, either "short"
(less than 20ms) or "long" (around 700ms).
This improved scheduler uses fewer threads and generates less
messages, so instead of 16 threads and encountering exhaustion
we have 9 and have yet to see more than 7 used even under the
most strenuous tests (multiple while-true curl loops + fountain
test + ping simultaneously).
Some helpful debug is also printed now when scheduler runs out
of slots, so we can see the distribution of pending delays.
we no longer create a new interface on net reset, which means that
the socket handles don't blow up if we change IP addresses.
However, the ARP cache is still stale and there seems to be on way
to force it to flush....but I guess in theory, eventually it
should fix itself, if smoltcp is working correctly?
the refactor changes the way sockets are handled, so now, when
the interface is rebound we are certain to get panics.
working on resolving this detail now, but at least the interface
is working now and it seems more stable than before.
total threads in the net crate are around 10 right now, 32 max, so
increasing the sleeping thread limit to 16, since every now and then
we get exhaustion notes.
- allow for zero-length tx/rx returns
- implement "shutdown" function, which is different from close:
shutdown requires any pending connections to return Ok(0) immediately,
but the socket is not dropped.
now that the suspender is in its own thread, we can do this safely.
It's nice to have a bit of a delay because we want other system recovery
things to take precedence and settle out before we try pinging the COM
to see how the wifi changed while we were asleep.
the connection manager actually wants a ping on resume,
and when we suspend we suspend -- "it is what it is". However,
the connection manager is a hairball that can block for a while,
so, to prevent failures to suspend, we put the suspend listener
in its own thread, and on resume the suspend listener will
automagically kick out an event to tell the connection manager
to deal with stuff.
Due to overly liberal use of `.take()`, file handles were getting closed
when doing ordinary operations. This stops that.
Signed-off-by: Sean Cross <sean@xobs.io>
This now adds a `timeout` parameter to the connect() API call.
Additionally, connections will not be returned until they are determined
to be open, or until an error occurs (such as a timeout).
Signed-off-by: Sean Cross <sean@xobs.io>
It's crappy because it uses a busy wait loop to poll the udp
rx socket in order to accurately emulate the Xous async callback
feature (which is _not_ emulated for Tcp, for example).
It's more accurate because this emulation happens. But it's a lot
more complicated to implement it because zero-cost async UDP callbacks
aren't really a host-native feature (but they are a Xous-native feature!),
and I didn't want to .. pull in tokio or mio code surface into
the Xous hosted mode, seemed a bit like crushing a fly by dropping
a house on it. Maybe that's a fine solution in Oz, but Xous doesn't live in Oz.
Total bodge -- we lop off the TCP stack at the Xous API level, and
substitute it with a native host TCP stack.
Caveat hacker: this means that a lot of OS-specific net properties
will carry through to the hosted mode emulation. This is adequate
for "smoke testing" a build but do not be shocked, *shocked* if
there are more bugs to be found because of some weird host integration
issue.
It's not a good time to bug the EC to do network stuff when it's updating
itself.
...might even want to send the network into a disconnected state,
but, let's start with just turning off the connection manager
for now.
Spent a couple hours chasing down a "bug" which is acutally just
something about how http works that I don't understand. TCP was
working just fine but there is ... idk some sort of keep alive
thing that occupies a socket so the "web server" blocks waiting
for a read. Anyways, I guess the "right" way to solve that would
be to make a multi-threaded web server
OH GOD STOP
right. it's just a demo to prove that TCP works. done.
Open an issue if you find a problem with TCP kthx!
Good enough so that if you type `net server` and you hit the IP
address of the device, it'll respond with something like
Hello from Precursor!
I have been up for 86087 ms.
There's still all kinds of bugs, I'm sure. But ... I'm also not
sure /why/ you would want a web server on Precursor. This is more
just to make sure that the TCP stack isn't totally broken.
The "web server" as it may be is quite broken and doesn't parse
any headers and probably sends a bogus amount of data because
I can't figure out if the content length includes some new lines
or not. But really, this just isn't in scope. Works with Chrome,
Firefox shows the status page and then says "connection reset" and hides
all the information. Whatever, browser compatibility is well beyond
my pay grade!
I wrote in a commit yesterday how there was no ROI on doing this
yet somehow I kept on doing it. I blame poor impulse control.
Turns out there was a series of race conditions and bugs that
caused this thing to "mostly work" on the first go at it but
really hard to debug and get very right.
1. Litex UART has a weird undocumented API. You advance the Rx
FIFO by writing `1` to the PENDING bit, and no other way. However,
you determine if there are any more characters to receive by
reading the EMPTY bit and seeing if it is 1. The previous method
wrote `1` to the PENDING bit and then used the PENDING bit to
see if a character was available, under the theory that the
PENDING bit would be set again if there was still a character.
This is somewhat true, except that the PENDING bit is set only
some amount of time later which is about comparable to a few
CPU instructions, so if you had a fast loop path the PENDING
bit would show as cleared and you'd exit, which would cause
no further interrupts to fire because the interrupt was
effecitvely not acknowledged.
This isn't written up anywhere but I figured this out by
just reading the source code for Litex UART and BIOS:
e8be391504/litex/soc/cores/uart.py (L264-L279)ccd3ab17be/litex/soc/software/libbase/uart.c (L38-L44)
2. I had the Key Injector IP block in the `sys` domain, which
would have its clock cut when the CPU went to sleep, which would
cause input characters to disappear when the CPU went into power
saving. moving this to "always on" domain fixes that.
3. I had the UART PHY in the always on block (so you could
send or receive exactly one character), but not the
separate UART block (which contained the FIFOs).
The UART block would also go into power
savings, and this would cause FIFO entries to go missing.
4. Escape sequences get sent as a burst of characters. The Key Injector
originally did not have a FIFO of its own, so it would only
remember the *last* key of the sequence that was written, causing
some keys to be dropped occassionally. Adding a 16-deep FIFO
to the injector fixes this.
5. The code I had to read out the injector's results was designed with
the "strobe" bit (to indicate that the FIFO had characters) in the
same regsiter as the character itself. Thus the loop that 1. checked
if the fifo was empty then 2. read and forwarded the character
would actually perform two reads from the FIFO. However, this "mostly worked"
I think because the FIFO retains the last value that was written
even when it was empty, so if only one character was read at a time
everything was fine. It only didn't work when you stuffed two characters
in there, as you'd drop the first one. Reading the FIFO once and
then interpreting the code with a bitmask fixes this.
6. The escape sequences, in their full glory with no dropped characters,
required a refactored parsing routine, which I have now added.
7. backspace->delete mappings, the bane of computer engineers since 1960.
OK. very bald yak. Here I lay down the shears next to several
mooshi pillows stuffed with the finest Yak hair.
Always reset the EC on boot. If we have a cold-boot situation,
it's hard to know what state the EC is in versus our stack;
a hard reset on the EC guarantees we at least always start from
the same point.
Note that on suspend/resume, there is no reset - the EC continues
trundling onwards, but the system was designed for this.
This is not the fix I was hoping for for connection stability,
but unfortunately this is where we are at.
The problem is that at some point the WF200 throws a "host interface
error"; but the error codes show no specific error. After throwing
this error, it can respond to some basic commands (such as the one
asking what the current status and error codes are), but higher level
APIs seem to time out. There's no command to bring the WF200 stack
to a sane state (like a warm reset or buffer flush) as far as I
can see, so, the solution is to detect the issue and send a reset
to the WF200. This keeps the link up in the long term, but
you will experience drop-outs about once every 2-3 hours for
about 20 seconds while the system detects and fixes the problem.
I think for this one, more data will be needed from customer units
to see if this could be just a local network issue with my config
or a more pervasive problem.
Each Cargo.toml now includes a comment which remnids maintainers
of the dependency policy for Xous (that is, nail down every dependency
to the minor number).
Versions are pinned down to the minor number. Yes, this means that
we don't automatically pull in features and bug updates from crates.io,
but it also means we don't pull in new attack surfaces and code that
hasn't been tested or reviewed. I'd like this process to be very deliberate
and not automatic for Xous.
The `rand` crate is almost pinned down to a single version, except
for an odd dependency that's pulled in by hosted mode that depends
on rand 0.8.4 via a subordinate dependency via the `tempfile` crate.
Pondering if it's worth it to try and eliminate that, because the
0.8.4 version doesn't get compiled into any version that touches
real hardware. It just makes it harder to analyze the Cargo.lock
file, mostly. I think I'll leave it as-is.
an activity_interval Arc::AtomicU32 is shared between the
connection manager thread and the main net thread. Every
time there is a packet received, the activity interval is
bumped to 0. If two interval polls pass, then a more
expensive wlan_status() call is issued to the COM to see
if we're really still connected.
On a typcial network, the ARP traffic alone is sufficient
to keep the status timeout from happening. The status interval
is currently set at slightly over 10 seconds, so if you fall
off the network entirely, about 20 seconds will pass before
an automatic rejoin is issued. This can be adjusted by tuning
the POLL_INTERVAL_MS variable inside the net service's
connection_manager.rs module.