Back

While I wait for the front/rear panel cables (expected to ship tomorrow) I've been working on some other parts of the firmware.

In particular, rather than using embedded Linux like most people probably would have, I wanted to keep this bare metal. So I found myself implementing things like a SSH server for bare metal STM32 with no OS (and no dynamic memory allocation).

Currently I'm working on a SNTP client that syncs the STM32 RTC to a network time server. Almost done with the NTP side of things, just have to write the RTC driver.

Other than that, pending TODOs:
* IPv6 support in the TCP/IP stack
* libscopehal driver support for the SCPI commands (already implemented in firmware) to set input threshold and output drive voltage
* Finish the CDR trigger mode. Right now I have 8b/10b and 64b/66b decoding in the FPGA bitstream, but need to add a pattern matching engine and SCPI commands to configure it
* Add commands for deep BER integration (continuous sampling reporting number of PRBS errors since last clear or something)
* SFTP-based OTA firmware update for (at least) the main processor and FPGA. Will probably try and get the front panel processor as well, but neither the supervisor nor the IBC has enough flash for A/B images. I'll probably switch to different STM32L0's with more flash in future projects to enable OTA flashing of the entire system.

I think OTA support is probably the next priority after I finish NTP.

Once I get working OTA support and all of the front/rear panel cables are installed, I'll be able to close the thing up and free up a ton of bench space.

Although I might not rack it right away, since having access to JTAG will still be handy for active firmware development.

NTP client is now pretty-printing a human readable yyyy-mm-dd hh:mm:ss.uuuuuu timestamp with hard-coded UTC-to-local offset.

Next step: make that offset configurable, get the RTC driver written up so I can store the timestamp to the RTC, and add a #define to logtools to make it print the timestamp from the RTC instead of since boot.

NTP on the STM32 is working!

First line: RTC timestamp prior to sync
Second line: timestamp received from time server, after correcting for network latency
Third line: RTC time read back after synchronizing it to match the NTP timestamp

Only thing left is logging library integration.

Let the NTP firmware run over a few hours while having dinner and doing family stuff.

It looks like the local oscillator on my MCU is just a tad fast relative to GPS time (the stratum 1 NTP source on my lab LAN).

The polls are approximately 1024 seconds apart, and I'm running about 2.8 - 2.9 ms fast every time I re-sync the clock to NTP.

Let's call this 2.85 ms of error into 1024 seconds, which comes out to about 2.78 ppm.

Not bad considering i'm using a non-temperature-compensated quartz oscillator with a datasheet tolerance of +/- 25 ppm.

And I think that's all the CLI and library plumbing needed for full NTP integration including the logging library.

Only thing missing is adding configurable UTC offset and DST support at some point, although that's not an immediate priority since I'm the sole user of the system and the next DST transition is quite a ways off...

The reason for the large negative step errors on first sync, if anyone is curious, is that the RTC is clocked by the HSE oscillator (25 MHz source) which does not keep running across resets or power-down.

Or, more importantly, when flashing a new firmware via JTAG.

So any time I firmware update the board the RTC stops for a few seconds and it lags behind actual time until the next NTP sync.

(I wasn't originally planning on using the RTC on this board so I never put a low-speed crystal on it)

Ok, got some housekeeping stuff out of the way. Confirmed some of the display corruption I was seeing on the front panel was due to overflowing the SPI FIFO when data came from the FPGA too fast, but I had plenty of RAM so I just made the FIFO a bit bigger and the problem went away.

Also made the SPI class have compile-time variable FIFO sizes which will simplify things a bunch.

Still probably going to have to make some tweaks to some of these peripheral drivers to make things nicely integrated without adding too much overhead. It's going to be a process.

Anyway, the elephant in the room is OTA firmware update via SFTP. This is going to take some time to get right.

Before I work on that I have a few other more small yaks to shave, like https://github.com/ngscopeclient/scopehal/issues/866.

So I think that ticket will be the next focus, it should be straightforward.

M2.5 screws for attaching the front/rear panel SMA cables (expected Monday) to the chassis came in.

The fit is snugger than I'd like, I have to actually screw them in rather than just pushing.

But I think I can make it work without going down to M2. We'll find out once I have the cables and try actually screwing a few of them in place.

Good progress on the PC software side, plus some firmware work to support that.

The ngscopeclient driver now supports changing input thresholds and output drive levels, as well as assigning nicknames to channels.

And the threshold/drive/nickname settings are persisted on device in the KVS, so that the next time you reconnect they're still there even if it's rebooted.

The intent of preserving these settings device-side is that there's probably only going to be one deployment of the thing (i.e. i'm not going to be suddenly changing the input threshold of my PicoScope's external trigger input, or which port the WaveRunner's trigger output goes to) and I don't want to have to reconfigure that every time I load a new firmware or reboot the thing.

Other settings, like which output is driven by which input, are more volatile and are expected to change often so they won't be persisted on device (however if you save a config to a .scopesession they will)

At this point, I think the baseline firmware/gateware feature set is complete: I can use it for everything I originally had planned when I started the project.

Now it's just a question of how many more useful capabilities I can cram in.

And building OTA update since this project was kinda intended to be the crash test dummy for that subsystem.

Semirigid cable came in! Had a production issue on the rear panel cables I need to reach out to the vendor about, but the front panel ones look awesome. Now I just need to bend them all...

Front panel is taking shape nicely!

The cables are on the long side since they're phase matched pairs and I needed enough space for the 90 degree bends to come off the PCB (these are not right angle connectors, as those had worse performance).

Rear panel will have to wait since those cables were assembled with the wrong connector. Gonna be a fun call with my sales rep tomorrow but I'm sure they'll make it right, this is the first time I've had an issue with an order from them.

I think the front side of the unit is finished, display and all.

So, that's all I can do on hardware until I get the rear side cables redone.

Still need to work on OTA update but I'm not in the mood to start that tonight so I think I'll try to memory map the QSPI first.

Welp. Memory mapping anything but actual memory via OCTOSPI seems to be full of dragons, I have a support case opened with ST but am not hopeful.

Tl;dr there is a 32-byte prefetch cache that doesn't seem possible to turn off. So any kind of read-with-side-effects or status register doesn't seem practical to implement.

Unless there's a chicken bit to turn it off, which is always possible (I opened a support ticket to ask).

Anyway, for now it's time to fall back to the code I had before doing indirect access - slower but gets the job done.

There's definitely potential for more optimizations on the FPGA-MCU communication, having the MCU spend more time in sleep, etc.

But I think at this point it's probably worth starting to work on OTA.

I think I'll work on OTA of the front panel MCU first.

The front MCU is a STM32L431 with 256 kB of flash and 64 kB of RAM.

Flash is organized as 128 2 kB pages, so (unlike the main processor) I have plenty of granularity for exactly where I want to put the bootloader vs the application.

The front panel does not have any persistent configuration storage, all of its settings are pushed each boot from the main processor.

The front panel firmware is small enough (38K stripped ELF) that it should easily fit almost anywhere. Hmm...

Ok so I think this is going to be the plan for the front panel:

* Main MCU accepts SFTP command, initializes SFTP server routine

* Main MCU sees a "write file" command for the front panel MCU's firmware file

* Main MCU sends SPI command to front panel MCU to reboot in DFU mode

* Main MCU parses incoming ELF as it comes in, finds data that needs to go to flash, and pushes it over SPI to front panel

* Final CRC verification, if this fails front panel remains in DFU mode

* Main MCU sends SPI command to front panel MCU to reboot in normal mode with new firmware

All of this has to be done "fire-and-forget" right now, since the SPI SO pin on the front panel MCU is unusable due to an errata (if I enable it, JTAG stops working and the chip soft-bricks).

I'm not sure if there's any way around this, perhaps by clever use of open drain signaling somewhere to signify "ready"? Otherwise I may have no choice but to run open-loop and just hard code conservative timeouts on the main MCU side.

Thinking a bit more, I don't think I actually need to have a hard "never use SO" rule.

What I can do instead is, default SO to tristated / JTAG mode.

And in a few specific commands like "query bootloader flash status" enable SPI mode on that pin (resetting jtag in the process due to the errata) and then immediately return to normal mode.

It means that single-stepping through that part of the code won't work, but I'll still be able to reset or power cycle the chip and have JTAG functional again for flashing or debugging of anything but the bootloader.

Starting work on the bootloader for the front panel MCU.

First obstacle: I never implemented support for the STM32L4 flash controller in my peripheral library. I should fix that.

Well, after a few bug fixes, here we are.

It's now a bootloader with no firmware update capability, so not super useful.

But it does boot time CRC32 verification (would be easy to swap out with a SHA, HMAC, curve25519 signature, etc if I wanted cryptographic checks, but that's not necessary for this application) to detect flash corruption, automatically updates the saved CRC if a new firmware version is loaded over JTAG, and then boots the app.

What's missing is:
* Enter firmware update flow, rather than infinite looping, if the CRC check fails
* Enter firmware update flow if application crashes (hard fault, WDT failure, etc) repeatedly
* Enter firmware update flow when requested from the main application (i.e. specific SPI command saying "enter DFU mode")
* Make said firmware update flow actually do something

Bootloader progress: all of the crash exception vectors in the application now set a status code in backup RAM which is passed to the bootloader, so the bootloader can tell if it was invoked as a result of an application crash, a warm reset at the application's request, a power cycle, or a request by the application to enter DFU mode.

If the application crashes, the bootloader defaults to entering DFU mode immediately (I might add a timeout counter in the future where a few crashes are allowed, but the goal is to avoid a crash loop). After a new, hopefully less buggy firmware is flashed the bootloader clears the fault and boots it. Since there's no backup battery, a full power cycle of the system will also allow restarting the crashy firmware build should you so desire.

I think I'm at a pretty good stopping point for the bootloader itself now. Next step will be adding SFTP server support to my SSH stack on the main MCU so I can actually get a binary to push to said bootloader.

Well, I ended up finding a yak and implementing support for the "exec" service request first.

But I also did add the subsystem request for SFTP.

Now to actually build the SFTP protocol layer so it can do something...

Lack of access to RFCs during the fiber cut (aside from the one I had already opened) slowed me down, but now I have SFTP working in basic write-only mode.

It doesn't actually do anything with the inbound data, so the next step will be bolting an ELF parser onto this so I can actually figure out what data has to get flashed where.

I guess this is my evening... Heavy metal seems an appropriate soundtrack for bare metal firmware development.

@0h00000000 It's a bit nore complex than that. I have a L431 and H735 and FPGA that all need to be updateable via SFTP running on the H735, plus (eventually) signed updates and (in the near term) simple crc verification. The actual flashing is the easy part.

Also fallback code and linker script modifications so i can have a/b images or at least fallback to bootloader if things don't work out.

@0h00000000 And if this goes well, in future systems there will be two additional L0 series parts that need updating too (for this project I'm probably treating those two as SWD-update-only since they don't have the flash for a fallback image, future designs will use alternate parts with more space)

@0h00000000 Yeah my FPGA is a Kintex-7, I'm not fitting that image in the STM32 flash :)

My flash is single bank on the H735, eight 128K sectors. There's an option bit to select which sector to boot from.

My current setup for the H735 is to keep the default boot from 0x08000000 and put the bootloader in the first sector. The last two sectors are used by the microkvs key-value store that all of my configuration data lives in.

Rather than doing A/B images, my plan is to have the bootloader/updater live in flash sector 0 (should be possible to squeeze into 128 kB easily enough with -Os) while the main application lives in sectors 1-5.

The system will come up in bootloader mode initially then check some health state in backup SRAM to see if the last boot of the main image was successful or if there was a failure (hard fault, watchdog reset, etc).

If boot / image verification fails, or if the user explicitly requested a return to bootloader mode to flash the main processor, the bootloader will remain active. In this state it will be running out of the first flash sector, using the same SSH keys and IP config as the full firmware image, and wait for you to SFTP over an image which will then be written to sectors 1-5.

Once the new image is written and verified, the bootloader will then trigger a reset and jump to it.

@0h00000000 For the FPGA, it's easy because the FPGA has multi-boot support natively and I have enough SPI flash for multiple full bitstreams so I should have no problem doing full A/B there.

The L431 on the front panel also has plenty of flash space (and small sectors) so bootloader + full A/B should be viable there as well.

@azonenberg Oh yeah, I saw this processor/DAC/ADC and thought you might be interested in checking it out, 2x1gsps ADC, 4x500msps ADC, 2x500msps DAC, 6x VSPA vector accel, 6x e200 cores, PCIe. Interesting beast.
https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/layerscape-processors/layerscape-access-la12xx-programmable-baseband-processor:LA12xx

@space_head Sounds like a DLL search path issue. We're working on retooling the Windows build process and packaging to make it a bit cleaner

Are you running from the mingw shell you did the compile in, or just double clicking the EXE, or what?

@space_head @david_rysk you've been doing a bunch of packaging work, is this a known problem?