The day before, the VEML7700 light sensor had worked one hundred percent on the desk.
Then I soldered it into the weather station's outdoor hub, at the end of a four-metre FTP cable, and most reads ended with no answer or read failed.
The hub carries three sensors. The SHT45 measures temperature and humidity, the VEML7700 light and an INMP441 microphone noise. On the other end of the cable sit an ESP32-WROOM-32 and a BMP280. The sensors use I2C, the microphone I2S, and everything runs through a single cable with four twisted pairs:
| pair | wires |
|---|---|
| 1 | microphone SD + GND |
| 2 | SDA + 3V3 |
| 3 | SCL + GND |
| 4 | microphone SCK + WS |
When designing it I made sure the microphone clocks never ran in the same pair as SDA or SCL. I thought that took care of it.
Most transfers gone to hell
The diagnostic sketch prints a line with the state of all sensors every two seconds. A bus scan found all three addresses (0x10 0x44 0x76), so the wiring was right.
But at 100 kHz roughly one read in twelve went through cleanly. After lowering to 20 kHz it was 9 of 19, and three of those nine returned nonsense, for example white 65391.
Meanwhile the SHT45 and BMP280 on the same bus did not report a single error.
A faulty module or a cold solder joint in the hub was the obvious suspect. But that module had worked on the desk the day before. The difference between the desk and the hub was mainly the cable: the cable was four metres long and the microphone clocks ran in the neighbouring pair.
So I stopped the microphone clocks. With the clocks stopped, 19 of 19 reads went through, at 20 kHz in 40 seconds.
The principle of the interference
Each sample takes two 32-bit slots (left and right channel), so at a 16 kHz sample rate SCK runs at 16,000 × 64 = 1.024 MHz. That is fifty times faster than I2C at 20 kHz, with sharp edges and no pause.
A twisted pair protects against interference only when both the signal and its return current flow in it. But pair 4 carries two clocks and no ground. The return current of both SCK and WS therefore looks for a path through the ground wires in pairs 1 and 3, and SCL lies in pair 3. I had no oscilloscope for this, so that is my explanation, not a measurement. All I know for sure is that with the clocks stopped, the errors disappeared.
The BMP280 sits next to the ESP32 on short wires, so its clean reads did not surprise me. Why the SHT45 at the end of the same cable did not report errors, though, I do not know.
With the VEML7700 the fault is all the more treacherous because it has no CRC. The SHT4x sends a checksum along with the data, so the firmware can recognize a corrupted read. A value like als 65307 from the VEML7700 leaves as a valid number.
The fix: the clocks stand still while reading
I did not feel like pulling a new cable, so the fix ended up in the firmware. Before every sensor read the microphone task is paused and i2s_channel_disable() stops both the SCK and WS clocks. After the read i2s_channel_enable() starts them again. In the diagnostic sketch it looks like this:
// I2C only with the mic clocks stopped
if (i2sReady) i2s_channel_disable(i2s.rxChan());
reportBmp();
reportSht();
reportVeml();
if (i2sReady) {
i2s_channel_enable(i2s.rxChan());
delay(WARMUP_MS);
i2s.readBytes((char*)frames, sizeof(frames)); // drop what the DMA held and the muted start
}
It does not come free. After the clocks start, the INMP441 stays silent for 2^18 SCK cycles, which at 1.024 MHz is 256 ms, and the firmware discards a bit more to be safe. Together with the read itself at 20 kHz, every sensor read costs roughly half a second of noise measurement. The station reads twenty times per ten-minute window, so about ten seconds of noise are missing in every window. I can live with that.
At the same time I set the lowest drive strength (GPIO_DRIVE_CAP_0) on SCK and WS so the edges are gentler. I deployed both at once, so I do not know whether the weaker driver alone would have been enough.
And the first version had a hole. bmp.readPressure() stayed outside the section with the stopped clocks.
The BMP280 does sit next to the ESP32, but SDA and SCL are still the same wires that run four metres through the cable. It only dawned on me a commit later.
Diagnostics after the fix: ok=29 fail=0 bad=0 in 60 seconds.
The weak spot stayed in the cable
Stopping the clocks is just a crutch. For the next revision of the hub I have written down that every signal above roughly 100 kHz gets its own ground in the pair. It is best to run I2S in a separate cable, SCK, WS and SD each in a pair with a ground and the shield grounded on the ESP32 side. If I2C still misbehaved after that, the PCA9615 differential extender is next. The cleanest option, though, would be to process the microphone signal with a dedicated microcontroller right in the hub, or to move the microphone next to the ESP32 and not drag megahertz clocks across four metres at all.
One rule applies right away, though. Every new I2C sensor that is to hang on the shared cable, I test straight away on the whole cable and with the microphone clocks running. A desk with short wires tells me nothing about interference.