Back

That's the last power rail alive.

I'll do another sweep later on and do full ripple characterization under load of each rail (I spot checked for obvious problems but that was with no firmware on any of the chips except the supervisor)

On the DAC front, U21's output is still flatlined (IN0-IN7).

But I'm seeing the expected 500 mV on the outputs of U22 (IN8-IN11 plus the CDR trigger input).

So with a few bitstream changes, I should be good to go test the trigger inputs on channels 8-11.

For anyone not keeping up with the thread, here's where things stand. I'm still in bringup mode with lots of hard coded configs etc, so nowhere near full application functionality on either FPGA or MCU side.

* Fan control: not tested as the board runs quite cool. I don't expect to need fans in the final system (just passive cooling) but put the connectors on so I could validate fan control logic, plus in case the board ran unexpectedly warm.

* Trigger outputs: Working great, need to measure rise times and such but the waveforms all look good on the scope

* Trigger inputs: Channels 0-7 blocked by U21 DAC not working, channels 8-11 should be working (testing in progress pending firmware build)

* Relays: Working fine

* Comms with IBC: Working fine, although IBC firmware sometimes hangs when supervisor resets midway through an I2C transaction (need to add a timeout interrupt or something)

* Power supply: Functional but not ideal. 3V0_N rail is super spiky and needs further investigation. All rails need full ripple characterization done. GTX_1V8 is a bit noisier than I'd like. PGOOD for 1V8 is nonfunctional and ignored by current supervisor firmware, presumed solder defect under DC-DC module that I've decided to just live with as no other functionality is impacted.

* Front panel interface: Nothing done yet, this is just a header with a SPI bus in case I want to put some LEDs or something on the front of the chassis.

* Front panel SERDES interfaces: SYNC output validated with good eye performance. TX/RX lanes 0/1, and CDR trigger input, not yet tested. Expecting TX/RX to work fine as they're hardwired direct to GTX, CDR trigger might need more work.

* FPGA boot from external flash: not yet tested

* 1000base-T management: working fine

* SFP+ management: links up, have not yet tested passing packets but no cause for concern if the link is stable

... Well this isn't good.

The MAX40026 comparator I'm using on the inputs specifies a common mode voltage range of 1.5 to Vcc.

And my frontend has a 2:1 attenuation.

Experimentally, if the input voltage is a 0-2V squarewave (0-1V after attenuation), it works fine with a threshold of 850 mV but with a 750 mV threshold it stops working (constant output even if input is still toggling above/below the input).

What are the odds of being able to find a suitable pin-and-supply-compatible comparator with a wider common mode range that fits the same footprint? If not I'm gonna have a lot of rework to do.

The only other part I see that passes my initial filters has a minimum common mode of Vee + 1.5V, and the board is wired with Vee = 0V so that would still be a minimum 1.5V common mode range.

Let's see... If I'm willing to do some channel-specific rework for specific instruments, I think this might be salvageable.

The LeCroy scopes are 1V into high-Z output. So if I remove the 50 ohm terminations and replace the attenuators with passthroughs, I'll have un-attenuated high-Z directly to the comparator, and I already know 1V with an 850 mV threshold will work.

The PicoVNA and Siglent SSG pulse trigger outputs, and I think the PicoScope, are 3.3V into high-Z. I should be able to do the same thing there (the comparator can take a 3.3V input).

Then for the Siglent SSG (and future function generator) which have 5V TTL outputs, keep the attenuating stage, but the 5V signal will have enough swing to trip the comparator.

So in short, for all of the input channels that do not need 5V tolerance, remove R53, R77, R89, and replace R65 with a 0R, at which point I have roughly 1-3.3V usable input range high-Z inputs.

And for the inputs that *do* need 5V tolerance, accept that ~1.6V is the minimum trigger level that will work reliably.

Ok I mapped out the instruments I have / am planning to acquire in the near to mid term.

It looks like if I bodge 9 of the 12 inputs to make them hi-Z 3.3V max, leaving the remaining three as 50 ohm 5V tolerant, I should be able to salvage this without messing with the comparators themselves.

Welp my bench is a horrible mess but the rework is done and I have it triggering well from a 0-1V high-Z input.

I think this is the last bodging to do, other than possibly adding some extra bypass caps to the CDR trigger input.

Will do some flux cleanup and tidying on Saturday when my ESD scrub brushes come in.

Somewhere in this pile are my single pair Ethernet boards including one gigabit one in need of rework, plus the new test fixture and parts to populate it.

Did some code refactoring and additional testing.

Brought up a second GTX interface (TX1) sending a PRBS-15 pattern.

For whatever reason, I didn't include AC coupling caps on the PCB for the transceivers. Not a big deal, I can use an external SMA DC block on the front panel ports, but still odd. Not sure why I did that.

SSH console is now alive. Still has all of the limitations of my TCP/IP stack (e.g. incomplete retransmits) but now I have a dev platform to work on that.

Trigger out waveforms look mostly good (here's a LVCMOS25 output) but there does seem to be a small reflection (also visible on the GTX output at lower data rates). Maybe my SMPM launch isn't quite as good as I thought it was. Not remotely enough to impair functionality, but definitely something to fix before I spin more boards on this stackup.

The 10GbE interface isn't doing anything yet (links up but no traffic is passed since there's nothing attached to the MAC).

Plan is to make the FPGA mux between the two, using the SFP preferentially if both are up and otherwise using whichever one has a link.

I still have to test the CDR trigger input port, but otherwise I think I'm basically done with bringup and rework. Everything is fixed and although I had to do a few passive swaps and make a pin-swapped cable, there were no actual PCB edits needed which I'm happy about (it beats LATENTPINK in that regard).

At some point soon I want to start thinking about the front panel. I have a connector earmarked for the purpose that hooks up 3V3_SB, 3V3, a SPI bus, and the soft power/reset buttons.

I'm thinking I probably want to put on push buttons at minimum and maybe also some indicator LEDs and/or a small LCD.

Ok so.... let's see about doing final-ish power rail validation with semi-complete firmware (nowhere near feature complete but most major subsystems at least exist and are drawing power).

Goal here is to either sign off each rail as being OK, or to identify a specific cause for concern and make a plan for rework to correct it.

12V0: 11.952V with 168 mV p-p / 3.6 mV RMS ripple. Short spikes, likely SMPS transients.

Some broadband noise around 300 MHz then narrowband peaks at 500, 1250, 2500 MHz.

The ripple might be a minor EMC concern but I'm not worried from a device functionality perspective. Let's call this good.

5V0. This is used as an analog supply for the DACs and input to the power opamps producing VCCIO for each output channel. Regulated with an LDO off of 12V0.

5.034V, 13.29 mV p-p, 0.458 mV RMS ripple. No concerns here.

3V3_SB. This is the supply for the supervisor MCU as well as the sensors and monitoring MCU on the IBC.

Regulated with an LDO off of 12V, but on the IBC (so fairly well isolated from noise on the main logic board).

3.2955V, 5.028 mV p-p, 0.463 mV RMS ripple. This one is fine too.

3V3 is decidedly less pretty. This is the main 3.3V rail for the board.

It runs lots of stuff:
* Input to 3V0_N buck-boost
* Input to integrated buck on main MCU
* Main digital IO supply including QSPI from FPGA to MCU, and RGMII to baseT PHY
* Input for SFP+ filtering network

3.295V, 55.95 mV p-p / 2.21 mV RMS ripple.

I think this is still well within spec for everything but it's an EMC concern. I also just don't like rails looking this noisy.

I suspect those spikes are caused by the 3V0_N converter and will hopefully go away (or at least shrink) if I put some more input capacitance on it.

But we'll find out more once I probe 3V0_N and 3V3 simultaneously (planned for later).

VDDA for the main MCU. This is an analog rail filtered off of 3V3 but is vastly cleaner: 7.62 mV p-p, 0.93 mV RMS ripple.

I'm quite happy with this as-is.

A3V3 for the baseT PHY. Also an analog rail filtered off of 3V3 (last 3.3V power domain on the board, I promise! at least other than the ones for the SFP+ which don't have test points)

This one is sagging a bit due to resistive losses in the filter network, down to 3.284V. Still well within tolerance for the PHY.

Ripple is 6.941 mV p-p, 0.934 mV RMS. I'm happy with that.

VSMPS for the STM32H7 (main MCU). This is the nominally 1.8V output of the MCU's integrated DC-DC converter which is then LDO'd down to run the various internal power domains with dynamic voltage scaling.

1.801V so right where it should be. Ripple is 12.00 mV p-p, 0.70 mV RMS, so no cause for concern. This is clean enough I don't think it's the source of the junk on 3V3.

Even far more dirty power would be fine here seeing as this rail is LDO'd before use. So no need to revisit it.

1V8. This is VCCAUX for the FPGA plus VCCIO for the first 4 output channels, as well as running the FPGA input buffers for all of the LVDS trigger inputs.

Very good: 1.793V average. Ripple is 5.49 mV p-p / 0.64 mV RMS. Well within spec.

Finally the last 1.8V power domain: GTX_1V8. This is the VCCAUX supply for the SERDES quad on the FPGA.

Average is right where I want, 1.796V.

Using a MYMGK for this with 4A rated output was probably overkill and the very light load is probably the reason the ripple is slightly out of spec (14.81 mV p-p / 1.63 mV RMS, datasheet wants no more than 10 mV p-p).

That said, I might still be OK since this is the ripple measured on the plane close to the DC-DC and I actually have a 4.7 uF cap across the via going to the FPGA ball that will attenuate some of the sharper spikes.

At some point I might try to take a differential measurement across that cap but for now I think I'll call this "not great, but probably tolerable". The main impact of a bit more ripple here would be degraded jitter performance on the GTX (since this is the PLL supply) and I'm seeing eyes that look perfectly fine.

Moving on down the list to lower voltage domains we get MCU Vcore, the output of the on die LDO for the STM32H7.

Not a whole lot going on here. I might poke more when I have a slightly busier firmware running but I'm not worried.

I think around the 7.5us mark we might be seeing the CPU activity increase when An Ethernet frame arrives? There looks like an activity spike in the spectrogram on the 256 MHz AHB bus frequency, but it's really hard to see if you don't know what to look for.

Average voltage is 1.364V which is right where it should be. Ripple is 5.92 mV p-p / 0.68 mV RMS, totally fine.

Next, 1V2. This is the 1.2V domain that runs the Ethernet PHY, as well as the MGTAVTT rail for the FPGA transceivers.

We can see some 250 MHz activity from the PHY as well as spiky broadband noise around 500-600 MHz from the switching spikes.

This is where it should be voltage wise (1.1996V) but noisier than I want: 31.63 mV p-p / 0.96 mV RMS. Spec for the GTX is, again, 10 mV p-p.

But actually, looking more closely at the datasheet requirement, the ripple requirement is specified from 10 kHz to 80 MHz. And this switching noise is much higher frequency (presumably within the range that substrate bypass caps would notch most of it out).

With a FIR LPF applied to the measured waveform to cut off everything past 80 MHz, I'm only seeing about 2 mV of ripple.

And with a 200 MHz frontend bandwidth limiter on the scope I'm seeing 4.80 mV p-p / 0.26 mV RMS ripple. So this rail actually *is* in spec.

Backing up to GTX_1V8, it's also (barely) in spec with the bandwidth limit: 8.97 mV p-p / 1.62 mV RMS.

I don't like the high frequency spikes but it does meet the datasheet noise limits.

Next, D1V2. This is the digital core of the KSZ9031 PHY, filtered off of 1V2 to prevent noise from the Ethernet subsystem leaking into the transceivers.

1.180V, ripple is 14.20 p-p and 3.57 mV RMS. Noise is dominated by 125 MHz and harmonics thereof. Gee, I wonder why an Ethernet PHY would be doing that...

Next, A1V2 on the KSZ9031, the 1.2V filtered analog rail. Decidedly quieter, 9.043 mV p-p / 1.44 mV RMS.

Some spikes at expected harmonics of 125 MHz and also one at 1.125 GHz which is interesting. Maybe that's what the internal PLL runs at?

Anyway, looks fine to me.

This one is interesting. A1V2_PLL on the KSZ9031. Higher than the other filtered rails at 1.191V, probably because lower I*R loss across the ferrite (the PLL doesn't pull much current). Ripple is almost nonexistent and dominated by low frequency from the power supply: 6.98 mV p-p / 0.84 mV RMS.

But the FFT picks up a lot of stuff that's too weak to see in the time domain view.

Big spike at the 125 MHz core clock, then a nice comb pretty much every 25 MHz from there on out.

Super quiet overall and well within acceptable limits.

For being the digital core of the FPGA this rail is shockingly quiet.

I mean, I don't have THAT much stuff going on in the FPGA but I was ping flooding the thing and it was buffering a bunch of input clocks to various outputs etc.

996.3 mV (nominal 1.0), ripple is 6.13 mV p-p / 0.42 mV RMS. One of the quietest rails on the board despite being digital.

Almost done, I promise. This is power domain 16 for anyone keeping count.

GTX_1V0, the digital core of the SERDES block. 999.901 mV average on a nominal 1.0V, absolutely nailed it! Ripple is 7.22 mV p-p / 0.85 mV RMS without any bandwidth limit. This is far enough inside the spec that I didn't even bother measuring within the specified 10 kHz - 80 MHz band.

And finally, power domain 16. The one I was saving for last because it was both the last numerically, and because I knew it would be horrible from previous testing.

3V0_N is good on average at -2.984V.

But the ripple is absolutely atrocious, ranging from -2.75 to -3.35V (596 mV p-p, 7.7 mV RMS).

Also of note is that the 3V0_N rail has negative going spikes exactly when the 3V3 rail has positive spikes and vice versa. Perfectly phase aligned.

So it definitely looks like the 3V0_N supply is the source of all the junk on the 3V3 rail. Fix that and it should clean up both rails at once.

Also, I now have 2.2 GB of waveform data saved (almost, but not exclusively, from the rail validation work). Will be useful to be able to go back and look at saved waveforms and do A/B testing showing the impact of rework.

So now the question is, where do I go from here?

The SMPS is fairly straightforward, basically reference schematic for the LMR70503.

Layout wise, I see one potential issue: while the bulk caps (at least the nearest one) are connected to the chip on the top layer on the 3V3 side, the ground is not (only by vias to layer 2). So that adds a bit of loop inductance.

Not much, but some.

For initial rework, I'm thinking of shoving an 0.47 uF 0402 X7R across the C2/C3 GND/3V3 vias. Unless anybody else has better ideas?

Actually an 0402 wouldn't fit on the top side. I could do one on the back side, which would add via inductance, but anything I put on the top would have to be an 0201.

I don't work with 0201s all that much.

My options in inventory look to be CL03A104KQ3NNNH (0.1 uF) or GRM033R60J474KE90D (0.47 uF).

Both are 6.3V rated and would be operating at 3.3V.

The Samsung 0.1 uF loses 50% of its capacitance at 3.3V, thus making it 50 nF.

The Murata 0.47 uF loses 62% at 3.3V (due to the higher capacitance and thinner dielectric leading to higher field strength) but starts with much more capacitance - it's still 178 nF, about 4x the Samsung, so I'll go with that.

Here's the area in question under the microscope.

I need to bodge this capacitor between the two vias at the south side of the WLCSP (U26) at the center of the image, in the gap between it and C154 (a massive 0805).

It's an annoying angle but I have plenty of space. I might almost be able to fit an 0402 in there if I tried hard but let's see what the 0201 does first.

Soldermask removed, test fitting the part.

An 0402 might fit but it'd be annoying to solder because it would be pretty much LGA style mounting with no free pad to get the iron onto.

I think I'll put the 0201 here and if I want more capacitance put an 0402 on the back side of the same via.

And rework complete, let's see how it turned out...

I'm gonna say *that* did something.

602 -> 380 mV p-p ripple reduction from that one cap.

I can't easily fit much more on the top so let's see what happens if I add an 0402 on the bottom straddling the same vias. Hopefully the extra ESL won't matter TOO much.

And after the second cap the 3V3 spikes are massively reduced and the output ripple is down to 286 mV p-p.

I think that's it for tonight as it's getting late. Tomorrow I'll see about adding some back side caps and see if I can cut these spikes down even further.

Clearly on the right track, though.

Added ground strap on the top layer from the main ground via to the north end of C154. This actually *increased* the p-p ripple to 335 mV. Maybe because more high freq switching noise is getting onto the ground plane due to the lower ESL?

I think I'll keep it though, and just add more capacitance.

Second 0201 on the input doesn't seem to have helped. Maybe I need to add a bulk cap on the bottom now?

Ok I think I've hit the limit of what I can do with input capacitance.

Going from nothing, 0201 on output, 0201 on input, ground strap on top, second 0201 on top, 0402 on bottom:

VOUT p-p ripple 596, 380, 286, 333, 328, 319 mV.

VIN p-p ripple 62.6. 66.4, 43.7, 42.6, 39.7, 34.9.

So let's see what happens if I bodge another cap on the output side next.

Top side 0201 on the output didn't change output ripple at all (319 mV) but dropped input to the lowest it's ever been (32.6). Huh.

Let's try an 0402 0.47 on the top right near the output cap.

That helped a lot. Output ripple now 236 mV. Let's see if I can find space for another.

The capacitive tower of babel is growing. I'm about to add my third cap to the second story.

And with that, ripple is down to 171 mV p-p.

Still not *great* but I'm tempted to stop soon. It seems like I'm hitting diminishing returns of what I can do without a re-layout of the supply.

Maybe one or two more, though. After lunch.

Nope, two more 0402s on the output piggybacked above the bulk cap did zilch. Probably too much inductance for them to help.

I guess I'll have to live with a 3.4x reduction from the original ripple. Improved, if not as far as I'd like.

This is the last planned rework.

So I'm gonna pull the board and give it a nice scrub in some SWAS to deflux, then set it back up on the bench and focus on firmware dev from here on out.

Have a few chores to do but when I'm done it's cleanup time.

This is the best flux remover I've found for no-clean fluxes, but it's not particularly good for you (10-30% tetrahydrofurfuyl alcohol).

The stuff is stinky enough that after rereading the SDS I've decided to stop using it outside the fume hood.

Board is cleaned up and back on the bench.

Tried out the CDR trigger input for the first time and I'm getting flatlined output. Not sure what happened there.

But it's a pretty low priority for debug because I have so many other things to do and I already have a differential CDR trigger capability in hardware (no firmware to implement it, but it's there when I have time). The single ended was really just a bonus.

I'm not having good luck with comparators on this project though...

Aaaand I'm not getting signals on several of the input channels and the DAC Vref looks bad.

Thinking back, I don't think I ever solved the problem of DAC0 not giving valid output.

So cleaning the board might have been a bit premature, the DAC might still need rework...

In the meantime, let me at least validate all of the other stuff I can before decabling *again*.

FPGA and MCU firmware seems to be at MVP level (that's "minimum viable prototype", as in all core functionality needed for operation has been demonstrated but not necessarily fully debugged).

In particular, I can plug the device into a 10/100/1000baseT LAN (the SFP+ side still needs more work but it's functional without that) and it comes up on a hard coded IP address because I haven't got around to adding UART console commands to change that. It's reachable via SSH using a hard coded username and password and gives you a shell equivalent to what you get from the UART.

There's a SCPI server on port 5025 that provides commands to change direction of bidirectional ports, set threshold of inputs, and set swing of outputs.

Oh, and the crossbar itself is done and working and you can set mux selectors via SCPI.

The CDR trigger needs more hardware debug. The SFP+ links up and should be able to receive frames, but the arbitration logic for deciding whether to send frames to the SFP+ or the RGMII MAC isn't done yet so right now I hard code sending to RGMII.

The BERT/pattern generator ports are working and spitting out hardcoded PRBS15, but I don't have any firmware/gateware to control swing, pattern, inversion, FFE taps, or the receiver yet.

As far as hardware goes, since this board seems to have had a lot of solder issues, let's just go through and validate every single one of the IO ports.

IN0...IN7 are nonfunctional due to the DAC being flatlined; all testing on them will have to be tabled until I rework the DAC that I somehow forgot to do earlier. I swear I had got to it... this is what happens when you're doing bringup in between parenting a toddler I guess lol. I should keep better notes.

All of the unbuffered outputs (OUT0-3) work.

All of the buffered unidirectional outputs (OUT4-7) work.

All of the bidir IOs (IO8-11) work as both inputs and outputs.

I strongly suspect the inputs on 0...7 will work fine once I fix the DAC. Guessing just bad soldering since the SPI bus to it looks fine. Just mad I didn't do that already as I thought I had finished the rework...

There's some strange hangs in the MCU firmware where sometimes it takes longer than expected to process a SCPI command, and it seems to take a while to start responding to ping when first powered up. So still debug to do on that front too.

Reflowed the DAC and now it's definitely improved.

IN2/3 is flatlined despite having valid input and Vref looking correct.

Probably a solder defect, these are ones that I reworked early on and it looked good visually but I didn't do a functionality check yet as I didn't have the necessary firmware done.

Also definitely need to debug why the firmware suddenly stops responding when I'm sending it SCPI commands, then is fine when I reconnect.

So much for thinking I was done with rework, lol.

But I think this actually *will* be the last rework considering I've now validated pretty much everything hardware wise.

Famous last words.

Anyway it's getting late enough I don't want to do any fine soldering but I am going to start work on getting the BERT functionality up.

I'm somewhat impressed. These two random 36" KF047 cables I had lying around are phase matched tightly enough that the ML4039-BTP BERT can send a 10.3125 Gbps PRBS15, with totally guesstimated FFE settings, to the Kintex-7 GTX with a very low BER (zero errors in the minute or so that I felt like waiting).

I guess now I have to implement all of the glue I need to support proper BERT operation now. Like runtime configurable rates (ideally DRP to dynamically tweak PLL configuration but we'll see how that goes), eye scan, inversion, etc.