July 2023 – Our Journey to Scotland

ESP32 with PlatformIO, C++, Unity and ESPIDF

As a follow up to my post about ESP32 with PlatformIO and Arduino, in this post I present how to use PlatformIO with the Espressif IDF (or ESPIDF, for short) in conjunction with C++ and Unity as a UnitTesting framework.

It proved to be much more difficult to get this running than with the Arduino framework.

Here are our requirements for development:

Support (hardware and framework independent) unit tests to be run on the local dev machine (aka env:native).
Support hardware and framework specific unit tests to be run on the actual microcontroller.

Here are some similarities and differences between ESPIDF and Arduino:

Again, the native environment will be compiled via SYS2/Mingw64 whereas the microcontroller environments are compiled by the compilers provided by the PlatformIO toolchain.
ESPIDF uses app_main() instead of the setup()/loop() construct in Arduino.
ESPIDF by default creates a main.c instead of a main.cpp file. We therefore have to use extern "C" { } to unmangle the symbols in our code.
For whatever reason the use of #ifdef __cpluscplus always evaluated to false and was therefore not usable. Thus, I used extern "C" unconditionally in the code.
To detect the ESPIDF framework, I used the ESP_PLATFORM symbol (instead of the ARDUINO symbol).
All framework dependent cpp and h lib files are guarded with #if defined(ESP_PLATFORM).
All test code (test_embedded and test_native) has to be surrounded with extern "C" as well (only the code and certinaly not the #includes).
Classes and code in lib_dir should not be surrounded with extern "C".
Also, I pretty much moved all the code to lib_dir, so the main.cpp is essentially only a stub.
We have to manually enable exceptions to support throw etc via build_flags: -fexceptions.
(not unit test related) Reading out GPIO to get the state of an LED always returns 0.

The final result can be found here.

Summary

Again, it is quite quirky to setup the development environment. Plus, I could not find a single example out in the wild (PlatformIO in conjunction with C++, Unity with embedded and native testing, ESPIDF).

In the end, I now have a working environment where I can hopefully do what I want to do: sending and receiving CAN messages via the TWAI interface. We will find out …

Reverse engineering the BYD Battery-Box Premium LVS CAN Protocol for Victron Venus OS

On my goal, to build a battery with a Venus OS compatible CAN interface I decided to have a look at he BYD CAN protocol – for several reasons:

It is supported with Victron and Venus OS.
I happen to have a BYD Battery-Box Premium LVS dual battery system.
I heard mixed information about the Pylontech CAN protocol implementation.

So, I got myself a Kvaeser Memorator Light, in order to be able to sniff the CAN traffic between a Venus OS and the BYD BMS. For whatever reason, I did not get it to work, so I ended up with candump – which proved to be more that sufficient for what I needed.

Note: of course, before I started reverse engineering the protocol, I made some effort to find resources and someone who might have already done that – but no luck. However, there were some fragments regarding HVS systems. But they did not seem to be compatible with the LVS implementation.

If you are interested in the result, you can head right here. Otherwise, stay with me and I explain my approach to correlate the identifiers and data pieces.

First, I just started candump to check the general message flow and to see some recurring patterns (working/normal operation, UseCase A).
I then verified that the communication (which runs at 500kb/s) only consists of 11bit identifiers and no FD frames.
I then identified the several message ids based on sender (TX, Venus OS) and receiver (RX, BYD).
I then monitored the message flow, when I disconnected the BYB BMS temporarily (UseCase B).
And then I monitored the message flow, when there was no BYD BMS present at start and then powered it on (UseCase C).
Have alarms and warnings being sent by the BYD BMS (UseCase D). (see note below)

Things to consider:

What are the units of the data being sent (e.g. temperature came in Kelvin/K)?
What is the byte ordering (e.g for WORDs expect the low byte first and then the high byte)?
Is there a scaling on the data being sent (e.g. 1/10mV)?
Is information distributed over different messages? Or does one message have a special meaning in correlation to another message? (e.g. cell voltage and temperature)

For most of the parts, I *knew* what data to expect or to look for. I just looked at the BMS device inside the Venus OS and looked for data that matched the information shown on the GUI.

In the end, I identified most of the messages. For alarms and events, I will verify them once I have a working prototype on my ESP32 by simulating and sending them to Venus OS. [Edit: Alarms and Warnings are now identified and described. Events seem to be not supported. With 17 frames/messages I can now setup a complete BYD battery simulation towards a Venus OS.]

Here is a Summary: BYD Battery-Box Premium LVS CAN Protocol. There you find also some image of Venus OS with the correspnding information as shown on the GUI.

Hope you find this useful.

ESP32 with PlatformIO, C++, Unity and Arduino

After a quick adventure with the .NET nanoFramework on microcontrollers, I sort of came back to my senses and continued with something that seemed to have a brighter future (read: more supported boards, more documentation, bigger community, …). So, after a quick look around, I tried: PlatformIO:

Loads of supported microcontrollers, and boards
Arduino and Espressif IDF supported
Programming language is C or C++
Unit Testing possible
Remote debugging
IDE is a VS Code extension
Runs on Windows, Linux and Mac

Sounds just too promising. At least promising enough to reactivate my C++03 knowledge and bring it up to at least C++17 (spoiler: after all there _was_ a reason why I switched to C#).

On the way I found out, that we now support auto, bool and lambda expressions but kept the splitting-of-declaration-and-implementation-nightmare – yikes …

Installation

Installation of PlatformIO was really straightforward. After the installation of VSCode, I installed the Python, C/C++ and the C++ extension beforehand. That automatically brought me CMake as well. And after that I just had to add the PlatformIO extension.

From there I could start and create my first project. And depending on the framework chosen (Arduino in my case) the main.cpp comes with either setup() and loop() or just app_main() (EspIdf).

Note: if the main file (under the src folder) is a main.c we have to rename it to main.cpp to use C++ features – I totally forgot about that …

Unit Testing

Building and flashing the controller “just worked”. So, I started to port my C# HelloWorld morse code generator to C++. Certainly, I wanted to write some unit tests along that way. And there the “trouble” started …

There is documentation, but I totally missed the way how unit testing are to be done with PlatformIO (at least when it is one with Unity:

First, unit tests are either “local” or “native” tests (on your dev machine) or “embedded” tests. We have to set up a separate *environment* for each.
The microcontroller framework (Arduino, in my case) is not supported on the native environment (not even an #include <Arduino.h> is allowed). So, we have to make sure, we use only code that is totally hardware independeent.
PlatformIO does not install a toolchain for the native environment (i.e. we have to install a C/C++ compiler ourselves). And on Windows, it is recommended to use MSYS2 with Mingw64. That effectively means, we have different compilers depending on the environment. Something, that just feels weird to me. And something that could cause problems, as I later should find out.
Every test is compiled as a separate executable with a main() function. Something I am not used to in a .NET environment. And here is again, it matters which framework (Arduino or EspIdf) we are using, as we have to repeat setup()/loop() or app_main() again and again.
The main code in the src folder is compiled as well, which leaves us with duplicate main() function. Preprocessor with #if defined() to the rescue – quite clumsy …
And then the main thing: we essentially have to move all the application code to the lib folder, as -by default- the src code is not included when unit testing. That is not only strange to me, but leaves the src folder being an empty stub, as all the code now lives in the the (private) lib folder.
Documentation or the Calculator example were only partially helpful. I ended up with the weirdest compilation and linker errors I never dared to imagine.
Unity requires we need to specify all tests manually if we want to run them.

But in the end, I got it working. Here is what I did:

Guard your src/main.cpp with #if defined(ARDUINO) / #endif.
Move everything else to lib (hardware / framework dependent and indepedent code).
Create test_embedded and test_native folders under test.
Create a env:native enviroment and exclude test_embedded. Exlcude test_native on the environment of your real board(s).
Code that relies on the (Arduino) framework is guarded with #if defined(ARDUINO) / #endif as well.
Create a test file with setup()/loop(), app_main() and main(). And have a central function runUnityTests that runs all the tests.

Summary

My first impression is … mixed. On the one hand, PlatformIO makes it relatively easy to develop for different hardware/boards. Due to VSCode the “developer experience” is much better than with the Arduino IDE.

But … setting up Unit Testing and how it is implemented is rather awkward. Needless to say, that error messages are not for the faint of the heart.

I cannot say, that I miss my C++ days. On the other hand, not something I could not get used to and around with it.

Hello, world! morse code generator on an ESP32

C# .Net on a Raspberry Pi 400 running Venus OS

Today, I tried to run a C# console application on a Venus OS – and it pretty much worked right away. But why would I want to do that?

The answer is simple: a couple of weeks ago I started to add some “drivers” to Venus OS to support additional features like using a MultiPlus-II as a charger for top balancing cells. With Venus OS, most of the examples I found were written in Python (except for some C++ extensions). And it is no secret that I am not too fond of that. So, why not using my favourite programming language on Venus OS as well?

My first thought was, I would have to install the .Net framework on Venus OS. But, with the advent of self-contained (and thus framework-independent) executables this is not needed.

First, I installed .NET on a Raspberry Pi 400 with Raspbian (just for the fun of it). I basically followed Deploy .NET apps on ARM single-board computers:

curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin --channel STS

echo 'export DOTNET_ROOT=$HOME/.dotnet' >> ~/.bashrc
echo 'export PATH=$PATH:$HOME/.dotnet' >> ~/.bashrc
source ~/.bashrc

dotnet --version

… and there it is!

And then it was time for another infamous Hello, world!:

dotnet new console -o HelloWorld
cd HelloWorld

And now for the compilation.

dotnet publish --sc -r linux-arm -c Release -p:PublishTrimmed=true

linux-arm was needed, as Venus OS is a 32-bit operating system (regardless of the 64bit architecture of the Pi 400). I chose PublishTrimmed to save some space. And of cource, --sc for self-contained.

I then gzipped the publish folder and copied it to the Venus OS (via WinSCP). After uncompressing the files (with permissions left intact), I ran the program and got this error:

Process terminated. Couldn't find a valid ICU package installed on the system. Please install libicu (or icu-libs) using your package manager and try again. Alternatively you can set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support. Please see https://aka.ms/dotnet-missing-libicu for more information.

Enabling invariant mode seemed to be the easier choice. After all, my future drivers would hopefully not need globalisation support anyway. So, I recompiled after adjusting the .csproj file:

<PropertyGroup>
    <InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>

… and it worked:

root@raspberrypi4:~# publish/HelloWorld
Hello, World!

I executed this on a Raspberry Pi 400 running Venus OS v3.00 and .Net 7.

From there, I wanted to connect to D-Bus which proved to be more difficult. Following the Connecting .NET Core to D-Bus I had to find out that Tmds.DBus.Tool was not compatible with .Net 7. I will have to look into that separately.

Note about IL trimming: the size difference is really noticable. In my example the untrimmed compilation was around 65MB and the trimmed version around 13MB. However, it seemed to me that the trimmed version took slightly longer to load and execute. So, I am not sure if I will keep this switch on.

So, what would be the perceived advantages of using .Net on Venus OS for me?

Known developing environment
Better type safety
Reusability of a lot of basic code
Easier testing and mocking

But this is only my personal opinion and preference. Yours might differ.

Initial setup of a Venus OS on a Raspberry Pi without a wired network connection

When setting up a Raspberry Pi to run Venus OS, the GUI is not available on the local HDMI port – it is running headless by default. However, in order to connect to a Wireless network, we need to access that UI.

As mentioned in the above link, there is a workaround to it: renaming (or removing) the /etc/venus/headless file. This can be done by connecting via the serial port to the PI using the Adafruit USB to TTL serial cable. There is a very thorough article on how to connect to the port.

In short, pin 8 (GPIO14, TX) is white; pin 10 (GPIO15, RX) is green and pin 6 can be used as ground (black) and DO NOT USE the red wire. See here for the actual Pin layout. We can then use Putty to make a connection to the Pi (at 115200bps).

So far, so good. However, when trying to rename the headless file the following error message appears: mv: can't rename 'headless': Read-only file system

As pointed out in Cannot change headless – Read only filesystem on Rasp4 for Venus OS large two options exist:

Make the file system read/write via /opt/victronenergy/swupdate-scripts/remount-rw.sh
Enable superuser access (unfortunately, this requires GUI access – chicken-egg-problem here)

But instead of making the filesystem read-write until the next firmware update, I would rather only temporarily remount via mount -o remount,rw /.

And after that, renaming/removing the headless file succeeds.

Following the next reboot, the file system is then mounted read-only again and the GUI appears on the local HDMI port.

Now we can configure WLAN settings and everything else (such as superuser access) without the need for a wired network.

And in case you are in the need of a very small keyboard / display combination, you can use

a Rii X1 mini wireless keyboard
and a Atomos Connect 4K or one of its successors and your favourite camera application to see the Pi’s local screen (works great on Android as well)

Connecting to Wi-Fi with via Bluetooth and Victron Connect

In case you only want to connect to Wi-Fi and do not happen to have a serial cable, but you want to use the Raspberry Pi’s bluetooth connection, you can use Victron Connect to configure wireless network settings.

For this you start up Victron Connect on an Android (or Apple i device, Windows will nork work for that) and discover the Raspberry you want to connect. When pairing with the Pi use 000000 as the pin code.

After that you will find the gear icon in the upper right corner. From there you can select Network settings and connect to your WLAN.

Below you find some screenshots.

Bluetooth connection to Venus OS via Victron Connect

Enabling WiFi on a Raspberry Pi 400 with Venus OS

When we run Venus OS without any modifications on a Raspberry Pi 400 no WiFi is detected – though the Pi 400 certainly has WiFi onboard.

As it seems, I am not the first one to notice that. bipedalprimate presented a solution by copying a bunch of Raspbian /lib/firmware files to the Venus OS. But as it turns out, things can be achieved much simpler.

It seems, that the driver on the 400 is differs from the chipset of a _regular_ Pi 4: it is the brcmfmac43456.

When looking at the /lib/firmware/brcm folder of a Venus OS these drivers are missing:

Venus OS v3.00 contents of /lib/firmware/brcm

On a Raspberry Pi 400 things look different:

Raspberry Pi 400 Raspbian 6.1.21 contents of /lib/firmware/brcm

As it seems, only a few files are required for a Raspberry Pi 400 and only a few belong to the brcmfmac43456. Most of the files are in fact links to other files (and some are in the cypress directory).

So, I did the following: I copied the brcm and cypress directories to a USB stick and inserted it into the Pi 400. From there I copied the driver files to the respective directories inside /lib/firmware, added some links and adjusted the permissions. Below you see the commands I used.

Note1: I am a novice when it comes to Linux, so pls do not expect any sophisticated shell scripting.

Note2: by default the root file system is _read-only_. Therefore I re-mounted it as read-write (so, maybe our changes will not survive a firmware update).

Note3: my USB stick was mounted as /run/media/sda1. Yours might be different.

	mount -o remount,rw /

	cd /lib/firmware/cypress
	cp /run/media/sda1/cypress/cyfmac4356* .
	chmod 644 cyfmac4356*

	cd /lib/firmware/brcm
	cp /run/media/sda1/brcm/brcmfmac4356-pcie.gpd-win-pocket.txt .
	chmod 644 brcmfmac4356-pcie.gpd-win-pocket.txt

	ln ../cypress/cyfmac4356-pcie.bin brcmfmac4356-pcie.bin
	chmod 777 brcmfmac4356-pcie.bin

	ln ../cypress/cyfmac4356-pcie.clm_blob brcmfmac4356-pcie.clm_blob
	chmod 777 brcmfmac4356-pcie.clm_blob

	ln ../cypress/cyfmac4356-sdio.bin brcmfmac4356-sdio.bin
	chmod 777 brcmfmac4356-sdio.bin

	ln ../cypress/cyfmac4356-sdio.clm_blob brcmfmac4356-sdio.clm_blob
	chmod 777 brcmfmac4356-sdio.clm_blob

	ln brcmfmac4356-sdio.AP6356S.txt brcmfmac4356-sdio.khadas,vim2.txt
	chmod 777 brcmfmac4356-sdio.khadas,vim2.txt

	ln brcmfmac4356-sdio.AP6356S.txt brcmfmac4356-sdio.vamrs,rock960.txt
	chmod 777 brcmfmac4356-sdio.vamrs,rock960.txt

	cp /run/media/sda1/brcm/brcmfmac43456-sdio.bin .
	chmod 644 brcmfmac43456-sdio.bin

	cp /run/media/sda1/brcm/brcmfmac43456-sdio.clm_blob .
	chmod 644 brcmfmac43456-sdio.clm_blob

	cp /run/media/sda1/brcm/brcmfmac43456-sdio.txt .
	chmod 644 brcmfmac43456-sdio.txt

	ln brcmfmac43456-sdio.bin brcmfmac43456-sdio.raspberrypi,400.bin
	chmod 777 brcmfmac43456-sdio.raspberrypi,400.bin

	ln brcmfmac43456-sdio.clm_blob brcmfmac43456-sdio.raspberrypi,400.clm_blob
	chmod 777 brcmfmac43456-sdio.raspberrypi,400.clm_blob

	ln brcmfmac43456-sdio.txt brcmfmac43456-sdio.raspberrypi,400.txt
	chmod 777 brcmfmac43456-sdio.raspberrypi,400.txt

	mount -o remount,ro /

view raw RaspberryPi400WlanSupportOnVenusOS.sh hosted with ❤ by GitHub

Commands necessary to enable WLAN support on Raspberry Pi 400 for Venus OS v3.00

Two brcm-links are giving errors, but this can be ignored. The links on the Raspbian are not working either.

After copying the files both directories looked like this:

Venus OS v3.00 firmware brcm and cypress folder after adding the driver files

After a reboot I could browse and connect to my SSID via Settings, Wi-Fi:

Established connection to a Wi-Fi network

And from the serial console, ifconfig also showed our new interface:

Venus OS recognising the Raspberry Pi 400 WiFi interface

Now, the Raspberry Pi 400 can be used like any other Pi with Venus OS.

Thanks again to bipedalprimate for pointing me in the right direction!