the CM can't always detect if the EC changes state; this is
especially true after an update or a forced reset where
the state caches are not reliably updated. If no packets are
seen for a while, poll the actual state and force a sync-up.
Also try to make sure that the interrupts are enabled correctly.
This macro will be removed, since it is no longer necessary. Remove this
macro across the project and replace it with a standard `main()`
function.
Signed-off-by: Sean Cross <sean@xobs.io>
smoltcp is written to allow multiple adapters, so 0.0.0.0 as
the binding for UDP is illegal. However, in Rust, it seems that
0.0.0.0 is "totally fine" and most OSes will resolve this to
a "default" adapter.
this patch introduces the behavior of taking 0.0.0.0 and mapping
it to a default adapter. In fact, we only allow binding to the
default adapter. The reasoning behind this is that if our IP
address changes, we would expect that 0.0.0.0 resolves to that.
Where this scheme fails is if someone binds to 127.0.0.1 expecting
it to not respond to packets coming in from the outside, e.g.
they are using UDP as a loopback interface. I think this isn't
a common use case for Xous, but if it does become one I suppose
we could add a code path to also check for the 127.0.0.1 case
and try to handle it...
smoltcp requires some polling to be done in the future for it
to work with low latency. It returns a time in the future for
the next poll.
This refactors the polling algorithm to:
1. use a fixed thread pool. This burns a little extra memory in
exchange for not having to allocate/de-allocate threads every
time a follow-up is needed.
2. use a short/long scheduler, where short delays (shorter than
the scheduler quantum) are grouped together, and only a single
follow-up is fired off within that time range if needed. Longer
delays have a larger window (roughly +/- 100ms) of grouping,
so fewer such events are also required in the distant future.
Empirically the delays seem to fall into two groups, either "short"
(less than 20ms) or "long" (around 700ms).
This improved scheduler uses fewer threads and generates less
messages, so instead of 16 threads and encountering exhaustion
we have 9 and have yet to see more than 7 used even under the
most strenuous tests (multiple while-true curl loops + fountain
test + ping simultaneously).
Some helpful debug is also printed now when scheduler runs out
of slots, so we can see the distribution of pending delays.
we no longer create a new interface on net reset, which means that
the socket handles don't blow up if we change IP addresses.
However, the ARP cache is still stale and there seems to be on way
to force it to flush....but I guess in theory, eventually it
should fix itself, if smoltcp is working correctly?
the refactor changes the way sockets are handled, so now, when
the interface is rebound we are certain to get panics.
working on resolving this detail now, but at least the interface
is working now and it seems more stable than before.
total threads in the net crate are around 10 right now, 32 max, so
increasing the sleeping thread limit to 16, since every now and then
we get exhaustion notes.
- allow for zero-length tx/rx returns
- implement "shutdown" function, which is different from close:
shutdown requires any pending connections to return Ok(0) immediately,
but the socket is not dropped.
now that the suspender is in its own thread, we can do this safely.
It's nice to have a bit of a delay because we want other system recovery
things to take precedence and settle out before we try pinging the COM
to see how the wifi changed while we were asleep.
the connection manager actually wants a ping on resume,
and when we suspend we suspend -- "it is what it is". However,
the connection manager is a hairball that can block for a while,
so, to prevent failures to suspend, we put the suspend listener
in its own thread, and on resume the suspend listener will
automagically kick out an event to tell the connection manager
to deal with stuff.
Due to overly liberal use of `.take()`, file handles were getting closed
when doing ordinary operations. This stops that.
Signed-off-by: Sean Cross <sean@xobs.io>
This now adds a `timeout` parameter to the connect() API call.
Additionally, connections will not be returned until they are determined
to be open, or until an error occurs (such as a timeout).
Signed-off-by: Sean Cross <sean@xobs.io>
It's crappy because it uses a busy wait loop to poll the udp
rx socket in order to accurately emulate the Xous async callback
feature (which is _not_ emulated for Tcp, for example).
It's more accurate because this emulation happens. But it's a lot
more complicated to implement it because zero-cost async UDP callbacks
aren't really a host-native feature (but they are a Xous-native feature!),
and I didn't want to .. pull in tokio or mio code surface into
the Xous hosted mode, seemed a bit like crushing a fly by dropping
a house on it. Maybe that's a fine solution in Oz, but Xous doesn't live in Oz.
Total bodge -- we lop off the TCP stack at the Xous API level, and
substitute it with a native host TCP stack.
Caveat hacker: this means that a lot of OS-specific net properties
will carry through to the hosted mode emulation. This is adequate
for "smoke testing" a build but do not be shocked, *shocked* if
there are more bugs to be found because of some weird host integration
issue.
It's not a good time to bug the EC to do network stuff when it's updating
itself.
...might even want to send the network into a disconnected state,
but, let's start with just turning off the connection manager
for now.
Spent a couple hours chasing down a "bug" which is acutally just
something about how http works that I don't understand. TCP was
working just fine but there is ... idk some sort of keep alive
thing that occupies a socket so the "web server" blocks waiting
for a read. Anyways, I guess the "right" way to solve that would
be to make a multi-threaded web server
OH GOD STOP
right. it's just a demo to prove that TCP works. done.
Open an issue if you find a problem with TCP kthx!
Good enough so that if you type `net server` and you hit the IP
address of the device, it'll respond with something like
Hello from Precursor!
I have been up for 86087 ms.
There's still all kinds of bugs, I'm sure. But ... I'm also not
sure /why/ you would want a web server on Precursor. This is more
just to make sure that the TCP stack isn't totally broken.
The "web server" as it may be is quite broken and doesn't parse
any headers and probably sends a bogus amount of data because
I can't figure out if the content length includes some new lines
or not. But really, this just isn't in scope. Works with Chrome,
Firefox shows the status page and then says "connection reset" and hides
all the information. Whatever, browser compatibility is well beyond
my pay grade!
I wrote in a commit yesterday how there was no ROI on doing this
yet somehow I kept on doing it. I blame poor impulse control.
Turns out there was a series of race conditions and bugs that
caused this thing to "mostly work" on the first go at it but
really hard to debug and get very right.
1. Litex UART has a weird undocumented API. You advance the Rx
FIFO by writing `1` to the PENDING bit, and no other way. However,
you determine if there are any more characters to receive by
reading the EMPTY bit and seeing if it is 1. The previous method
wrote `1` to the PENDING bit and then used the PENDING bit to
see if a character was available, under the theory that the
PENDING bit would be set again if there was still a character.
This is somewhat true, except that the PENDING bit is set only
some amount of time later which is about comparable to a few
CPU instructions, so if you had a fast loop path the PENDING
bit would show as cleared and you'd exit, which would cause
no further interrupts to fire because the interrupt was
effecitvely not acknowledged.
This isn't written up anywhere but I figured this out by
just reading the source code for Litex UART and BIOS:
e8be391504/litex/soc/cores/uart.py (L264-L279)ccd3ab17be/litex/soc/software/libbase/uart.c (L38-L44)
2. I had the Key Injector IP block in the `sys` domain, which
would have its clock cut when the CPU went to sleep, which would
cause input characters to disappear when the CPU went into power
saving. moving this to "always on" domain fixes that.
3. I had the UART PHY in the always on block (so you could
send or receive exactly one character), but not the
separate UART block (which contained the FIFOs).
The UART block would also go into power
savings, and this would cause FIFO entries to go missing.
4. Escape sequences get sent as a burst of characters. The Key Injector
originally did not have a FIFO of its own, so it would only
remember the *last* key of the sequence that was written, causing
some keys to be dropped occassionally. Adding a 16-deep FIFO
to the injector fixes this.
5. The code I had to read out the injector's results was designed with
the "strobe" bit (to indicate that the FIFO had characters) in the
same regsiter as the character itself. Thus the loop that 1. checked
if the fifo was empty then 2. read and forwarded the character
would actually perform two reads from the FIFO. However, this "mostly worked"
I think because the FIFO retains the last value that was written
even when it was empty, so if only one character was read at a time
everything was fine. It only didn't work when you stuffed two characters
in there, as you'd drop the first one. Reading the FIFO once and
then interpreting the code with a bitmask fixes this.
6. The escape sequences, in their full glory with no dropped characters,
required a refactored parsing routine, which I have now added.
7. backspace->delete mappings, the bane of computer engineers since 1960.
OK. very bald yak. Here I lay down the shears next to several
mooshi pillows stuffed with the finest Yak hair.
Always reset the EC on boot. If we have a cold-boot situation,
it's hard to know what state the EC is in versus our stack;
a hard reset on the EC guarantees we at least always start from
the same point.
Note that on suspend/resume, there is no reset - the EC continues
trundling onwards, but the system was designed for this.
This is not the fix I was hoping for for connection stability,
but unfortunately this is where we are at.
The problem is that at some point the WF200 throws a "host interface
error"; but the error codes show no specific error. After throwing
this error, it can respond to some basic commands (such as the one
asking what the current status and error codes are), but higher level
APIs seem to time out. There's no command to bring the WF200 stack
to a sane state (like a warm reset or buffer flush) as far as I
can see, so, the solution is to detect the issue and send a reset
to the WF200. This keeps the link up in the long term, but
you will experience drop-outs about once every 2-3 hours for
about 20 seconds while the system detects and fixes the problem.
I think for this one, more data will be needed from customer units
to see if this could be just a local network issue with my config
or a more pervasive problem.
Each Cargo.toml now includes a comment which remnids maintainers
of the dependency policy for Xous (that is, nail down every dependency
to the minor number).
Versions are pinned down to the minor number. Yes, this means that
we don't automatically pull in features and bug updates from crates.io,
but it also means we don't pull in new attack surfaces and code that
hasn't been tested or reviewed. I'd like this process to be very deliberate
and not automatic for Xous.
The `rand` crate is almost pinned down to a single version, except
for an odd dependency that's pulled in by hosted mode that depends
on rand 0.8.4 via a subordinate dependency via the `tempfile` crate.
Pondering if it's worth it to try and eliminate that, because the
0.8.4 version doesn't get compiled into any version that touches
real hardware. It just makes it harder to analyze the Cargo.lock
file, mostly. I think I'll leave it as-is.
an activity_interval Arc::AtomicU32 is shared between the
connection manager thread and the main net thread. Every
time there is a packet received, the activity interval is
bumped to 0. If two interval polls pass, then a more
expensive wlan_status() call is issued to the COM to see
if we're really still connected.
On a typcial network, the ARP traffic alone is sufficient
to keep the status timeout from happening. The status interval
is currently set at slightly over 10 seconds, so if you fall
off the network entirely, about 20 seconds will pass before
an automatic rejoin is issued. This can be adjusted by tuning
the POLL_INTERVAL_MS variable inside the net service's
connection_manager.rs module.
much better, but still eventually the stack can fall over eventually.
also, the stack doesn't behave well when you try to initiate new
commands in the middle of a flood, because the back-pressure mechanisms
are primarily in-stack at the moment.
I think i have an idea on how to fix that but this will have to be later.
- moved most ping state into the Net crate proper, since the smoltcp
implementation assumes a tightly bound set of objects and lifetimes
in its icmp abstraction
- created various convenience abstractions for handling single and
multiple pings, blocking and non-blocking.
- refactored the "net" command to use the new form
Initial testing indicates there are still some bugs (seems to be
with the way we are handling timestamping and doing pong retirement)
but at least we're not getting tons of async callback errors like
we were before, and we're not shuttling obscene amounts of state
using Arc-Mutex between the input and output of the ping operations.
Now that I understand how smoltcp does ping better I have regrets.
The demo example in smoltcp uses a funky macro that was a bit
misleading. I have gotten rid of that but I probably would have made
different decisions in implementing this had I understood the API better.
Anyways. It works, but has some trouble handling the cases of timeouts
sometimes.
not fully working yet, but the architecture is there and it builds.
smoltcp's ping implementation (or at least its demo code) was not
very amenable to splitting into modules, had to recode most of it.
the ipv6 is still a hack.
Multicast/Broadcast is not supported by smoltcp it seems, so those
calls are just unimplemented. But we are able to clone UdpSocket,
for the sick puppies who like to do that sort of thing.