Back

While I wait for the front/rear panel cables (expected to ship tomorrow) I've been working on some other parts of the firmware.

In particular, rather than using embedded Linux like most people probably would have, I wanted to keep this bare metal. So I found myself implementing things like a SSH server for bare metal STM32 with no OS (and no dynamic memory allocation).

Currently I'm working on a SNTP client that syncs the STM32 RTC to a network time server. Almost done with the NTP side of things, just have to write the RTC driver.

Other than that, pending TODOs:
* IPv6 support in the TCP/IP stack
* libscopehal driver support for the SCPI commands (already implemented in firmware) to set input threshold and output drive voltage
* Finish the CDR trigger mode. Right now I have 8b/10b and 64b/66b decoding in the FPGA bitstream, but need to add a pattern matching engine and SCPI commands to configure it
* Add commands for deep BER integration (continuous sampling reporting number of PRBS errors since last clear or something)
* SFTP-based OTA firmware update for (at least) the main processor and FPGA. Will probably try and get the front panel processor as well, but neither the supervisor nor the IBC has enough flash for A/B images. I'll probably switch to different STM32L0's with more flash in future projects to enable OTA flashing of the entire system.

I think OTA support is probably the next priority after I finish NTP.

Once I get working OTA support and all of the front/rear panel cables are installed, I'll be able to close the thing up and free up a ton of bench space.

Although I might not rack it right away, since having access to JTAG will still be handy for active firmware development.

NTP client is now pretty-printing a human readable yyyy-mm-dd hh:mm:ss.uuuuuu timestamp with hard-coded UTC-to-local offset.

Next step: make that offset configurable, get the RTC driver written up so I can store the timestamp to the RTC, and add a #define to logtools to make it print the timestamp from the RTC instead of since boot.

NTP on the STM32 is working!

First line: RTC timestamp prior to sync
Second line: timestamp received from time server, after correcting for network latency
Third line: RTC time read back after synchronizing it to match the NTP timestamp

Only thing left is logging library integration.

Let the NTP firmware run over a few hours while having dinner and doing family stuff.

It looks like the local oscillator on my MCU is just a tad fast relative to GPS time (the stratum 1 NTP source on my lab LAN).

The polls are approximately 1024 seconds apart, and I'm running about 2.8 - 2.9 ms fast every time I re-sync the clock to NTP.

Let's call this 2.85 ms of error into 1024 seconds, which comes out to about 2.78 ppm.

Not bad considering i'm using a non-temperature-compensated quartz oscillator with a datasheet tolerance of +/- 25 ppm.

And I think that's all the CLI and library plumbing needed for full NTP integration including the logging library.

Only thing missing is adding configurable UTC offset and DST support at some point, although that's not an immediate priority since I'm the sole user of the system and the next DST transition is quite a ways off...

Results of an overnight test run. Not bad.

The reason for the large negative step errors on first sync, if anyone is curious, is that the RTC is clocked by the HSE oscillator (25 MHz source) which does not keep running across resets or power-down.

Or, more importantly, when flashing a new firmware via JTAG.

So any time I firmware update the board the RTC stops for a few seconds and it lags behind actual time until the next NTP sync.

(I wasn't originally planning on using the RTC on this board so I never put a low-speed crystal on it)

Ok, got some housekeeping stuff out of the way. Confirmed some of the display corruption I was seeing on the front panel was due to overflowing the SPI FIFO when data came from the FPGA too fast, but I had plenty of RAM so I just made the FIFO a bit bigger and the problem went away.

Also made the SPI class have compile-time variable FIFO sizes which will simplify things a bunch.

Still probably going to have to make some tweaks to some of these peripheral drivers to make things nicely integrated without adding too much overhead. It's going to be a process.

Anyway, the elephant in the room is OTA firmware update via SFTP. This is going to take some time to get right.

Before I work on that I have a few other more small yaks to shave, like https://github.com/ngscopeclient/scopehal/issues/866.

So I think that ticket will be the next focus, it should be straightforward.

M2.5 screws for attaching the front/rear panel SMA cables (expected Monday) to the chassis came in.

The fit is snugger than I'd like, I have to actually screw them in rather than just pushing.

But I think I can make it work without going down to M2. We'll find out once I have the cables and try actually screwing a few of them in place.

At this point, I think the baseline firmware/gateware feature set is complete: I can use it for everything I originally had planned when I started the project.

Now it's just a question of how many more useful capabilities I can cram in.

And building OTA update since this project was kinda intended to be the crash test dummy for that subsystem.

Semirigid cable came in! Had a production issue on the rear panel cables I need to reach out to the vendor about, but the front panel ones look awesome. Now I just need to bend them all...

Front panel is taking shape nicely!

The cables are on the long side since they're phase matched pairs and I needed enough space for the 90 degree bends to come off the PCB (these are not right angle connectors, as those had worse performance).

Rear panel will have to wait since those cables were assembled with the wrong connector. Gonna be a fun call with my sales rep tomorrow but I'm sure they'll make it right, this is the first time I've had an issue with an order from them.

I think the front side of the unit is finished, display and all.

So, that's all I can do on hardware until I get the rear side cables redone.

Still need to work on OTA update but I'm not in the mood to start that tonight so I think I'll try to memory map the QSPI first.

Welp. Memory mapping anything but actual memory via OCTOSPI seems to be full of dragons, I have a support case opened with ST but am not hopeful.

Tl;dr there is a 32-byte prefetch cache that doesn't seem possible to turn off. So any kind of read-with-side-effects or status register doesn't seem practical to implement.

Unless there's a chicken bit to turn it off, which is always possible (I opened a support ticket to ask).

Anyway, for now it's time to fall back to the code I had before doing indirect access - slower but gets the job done.

There's definitely potential for more optimizations on the FPGA-MCU communication, having the MCU spend more time in sleep, etc.

But I think at this point it's probably worth starting to work on OTA.

I think I'll work on OTA of the front panel MCU first.

The front MCU is a STM32L431 with 256 kB of flash and 64 kB of RAM.

Flash is organized as 128 2 kB pages, so (unlike the main processor) I have plenty of granularity for exactly where I want to put the bootloader vs the application.

The front panel does not have any persistent configuration storage, all of its settings are pushed each boot from the main processor.

The front panel firmware is small enough (38K stripped ELF) that it should easily fit almost anywhere. Hmm...

Ok so I think this is going to be the plan for the front panel:

* Main MCU accepts SFTP command, initializes SFTP server routine

* Main MCU sees a "write file" command for the front panel MCU's firmware file

* Main MCU sends SPI command to front panel MCU to reboot in DFU mode

* Main MCU parses incoming ELF as it comes in, finds data that needs to go to flash, and pushes it over SPI to front panel

* Final CRC verification, if this fails front panel remains in DFU mode

* Main MCU sends SPI command to front panel MCU to reboot in normal mode with new firmware

All of this has to be done "fire-and-forget" right now, since the SPI SO pin on the front panel MCU is unusable due to an errata (if I enable it, JTAG stops working and the chip soft-bricks).

I'm not sure if there's any way around this, perhaps by clever use of open drain signaling somewhere to signify "ready"? Otherwise I may have no choice but to run open-loop and just hard code conservative timeouts on the main MCU side.

Thinking a bit more, I don't think I actually need to have a hard "never use SO" rule.

What I can do instead is, default SO to tristated / JTAG mode.

And in a few specific commands like "query bootloader flash status" enable SPI mode on that pin (resetting jtag in the process due to the errata) and then immediately return to normal mode.

It means that single-stepping through that part of the code won't work, but I'll still be able to reset or power cycle the chip and have JTAG functional again for flashing or debugging of anything but the bootloader.

Starting work on the bootloader for the front panel MCU.

First obstacle: I never implemented support for the STM32L4 flash controller in my peripheral library. I should fix that.

Well, after a few bug fixes, here we are.

It's now a bootloader with no firmware update capability, so not super useful.

But it does boot time CRC32 verification (would be easy to swap out with a SHA, HMAC, curve25519 signature, etc if I wanted cryptographic checks, but that's not necessary for this application) to detect flash corruption, automatically updates the saved CRC if a new firmware version is loaded over JTAG, and then boots the app.

What's missing is:
* Enter firmware update flow, rather than infinite looping, if the CRC check fails
* Enter firmware update flow if application crashes (hard fault, WDT failure, etc) repeatedly
* Enter firmware update flow when requested from the main application (i.e. specific SPI command saying "enter DFU mode")
* Make said firmware update flow actually do something

Bootloader progress: all of the crash exception vectors in the application now set a status code in backup RAM which is passed to the bootloader, so the bootloader can tell if it was invoked as a result of an application crash, a warm reset at the application's request, a power cycle, or a request by the application to enter DFU mode.

If the application crashes, the bootloader defaults to entering DFU mode immediately (I might add a timeout counter in the future where a few crashes are allowed, but the goal is to avoid a crash loop). After a new, hopefully less buggy firmware is flashed the bootloader clears the fault and boots it. Since there's no backup battery, a full power cycle of the system will also allow restarting the crashy firmware build should you so desire.

I think I'm at a pretty good stopping point for the bootloader itself now. Next step will be adding SFTP server support to my SSH stack on the main MCU so I can actually get a binary to push to said bootloader.

Well, I ended up finding a yak and implementing support for the "exec" service request first.

But I also did add the subsystem request for SFTP.

Now to actually build the SFTP protocol layer so it can do something...

Lack of access to RFCs during the fiber cut (aside from the one I had already opened) slowed me down, but now I have SFTP working in basic write-only mode.

It doesn't actually do anything with the inbound data, so the next step will be bolting an ELF parser onto this so I can actually figure out what data has to get flashed where.

I guess this is my evening... Heavy metal seems an appropriate soundtrack for bare metal firmware development.

Progress! I can SFTP a file to the board, it parses the outer ELF header, finds the program header table, and identifies the PT_LOAD sections.

Now to make it actually write that data to flash...

And finally! Here it is. I don't have the actual bootloader interface logic done yet (the SFTP class needs to take the write data block and push it over SPI to the front panel, and the bootloader needs to actually write it to flash).

But everything is done on the SFTP protocol and ELF parser to the point that it knows what data to write to what flash addresses.

I should probably sleep now.

The plot thickens.

I need to be able to get data out of the front panel MCU (e.g. to know if it's in DFU or application mode, or if a flash block was successfully erased/written).

But PB4 is NJTRST / SPI1_MISO is my only way to do that, and due to an errata if I ever configure it in any alt mode other than 0 (NJTRST) the JTAG TAP resets.

Current workaround is to configure it in JTAG mode except when actively executing a readback operation, at which point I change it back to SPI mode.

This isn't working and I'm not yet sure why. Hopefully implementation bug somewhere in the chain of hardware.

Two-way SPI communication between the bootloader/application firmware on the front panel, and the application firmware on the main MCU, is now working.

Next step, actually writing the contents of the loadable sections to flash... and we should have a fully working SFTP OTA boot flow (for one of the five programmable chips on the board, but hey it's a start!)