Building the custom Edge Impulse Thingy91 Firmware and Connecting to MQTT
PART 3 of the 4 part article series on using the Nordic Thingy91 with Edge Impulse
Part 1: From Zero to Hero with the Nordic Thingy91 and Edge Impulse
Part 2: Getting started with Edge Impulse and the Thingy91
This section covers the process of building the Edge Impulse firmware locally with your trained model included. The perquisite to this section is to have the nRF Connect Toolchain setup locally and to have a copy of the firmware.
The process outlined below assuming you are running Windows 10 and your local carrier supports NB-IoT. Since we will be using the command line to build the steps are portable across platforms provided you have the toolchain setup properly locally. You won’t need to use an IDE such as Segger or VS Code for this.
You are able to build this firmware yourself locally by downloading the firmware repository for the Thingy91 from Edge Impulse’s GitHub and configuring the MQTT topic and setup the connectivity properly. For this guide I am describing the process using NB-IoT.
First things first go back to nRF Connect and install the Toolchain Manager by clicking the Install button.
After installing this click “Open” to go into the Toolchain manager and then install nRF Connect SDK 1.7.0 by clicking “Install”.
If installation completed successfully then you will see the option to use VS Code and Segger Studio and drop-down menu.
Clicking on the drop-down menu reveals options to open a Bash or Command Prompt. Select “Open command prompt” and this will open a Windows command prompt that is setup with the environment variables and paths setup to build the for the nRF Connect SDK. If you open a normal command prompt using the start menu these settings won’t be in place and you won’t be able to build without setting everything up manually.
The nRF SDK uses CMake and Ninja behind the scenes to build Zephyr OS based projects. Zephyr has created their own build tool called west that we will use. To see if everything is setup properly run “west –version” from the Command Prompt you opened via the nRF Connect Toolchain manager.
If you see something like the above, you should be ready to proceed.
Change directly any other directly of your choice and download the official Edge Impulse Thingy91 firmware source using
“git clone https://github.com/edgeimpulse/firmware-nordic-thingy91.git”
Navigate to the firmware folder by typing “cd firmware-nordic-thingy91”.
You are now ready to build the firmware locally and you can test that the build works perfectly by running the command:
west build -b thingy91_nrf9160_ns
Depending on your workstation it should take 5–10 minutes to complete. The command is telling west to build the target application for the nRF9160 itself and not the connectivity bridge as we don’t need that if you have already flashed it earlier. For the rest of this tutorial, we are only building the main application firmware. Like all modern toolchains the nRF SDK supports incremental builds so unless you make significant changes to your code subsequent builds will be quicker. The build should complete, and you should see something like below.
The firmware you just built is the default Edge Impulse firmware that doesn’t contain your model. It does in fact contain and indoor/outdoor detection model you can flash to the Thingy91 and test using the “edge-impulse-run-impulse” command.
To Flash power down the Thingy91 and then power up while holding the main push button (SW3). Then follow the steps in the previous section on flashing using the nRF Connect Programming software. You will need to click “Clear files” and then “Add Hex file”.
The custom-built version of your firmware file is located relative to the source you downloaded in the generated “build/zephyr” folder. The file name is “app_signed.hex” be sure to select this one as it also includes the MCUBoot bootloader. In case you are wondering the “firmware.hex” you downloaded from the prebuilt firmware in the studio is essentially this same file renamed. With that said click “Write” to flash the Thiny91 with the custom-built firmware.
To include your own custom model first need to build the C++ Library from the Edge Impulse Studio. The C++ Library is the portable library that includes your full Impulse (the DSP code for feature extraction and the model itself).
When you create the Library in the Edge Impulse Studio is creates a portable C++ source code archive that downloads as a zip file which contains the following folders:
All you need to do overwrite the folders with the same names in your source code folder with these new versions. In Windows 10 you can opt to replace the existing files. You now have your model embedded into the local copy of the firmware source on your workstation.
If you are using NB-IoT you need to also edit the “prj.conf” file and add the following line using a text editor.
This should let your firmware connect via NB-IoT when you run the AT+CONNECT command from the console.
Once that’s done run the following command again to rebuild.
west build -b thingy91_nrf9160_ns
The project will rebuild most of the files and it will take a short while again, once its done simply flash the “app_signed.hex” file in the “build/zephyr” subdirectory.
You have now replaced the firmware on the device with one with your own custom model built in having now done in a more manual manner what the Studio does when you build for a Thingy91 target from the Web interface. Go ahead and open a serial terminal as shown with Putty earlier and run-in sequence
AT+CONNECT
AT+RUNIMPULSE
Since the firmware publishes to the public HiveMQ broker at “broker.hivemq.com” you can use any MQTT client to connect to that broker and subscribe to the default topic “ei/demo”
The easiest way to do this is using the WebSocket web client at http://www.hivemq.com/demos/websocket-client/ as seen below.
Connect to broker.hivemq.com and the WebSocket port is 8000.
Once the connection is established and inference is running you will see your labels appear in the client. Every time inference completes the class has the highest value is published as this is most likely the correct class being detected. You can observe this in the “Messages” field.
You might be asking yourself why go through all the extra effort to do this manually. For one you are now able to make changes to the configuration and even the source code yourself. Since the Zephyr build system makes use of a tool called KConfig that allows you to make changes to the build configuration before rebuilding.
The Edge Impulse firmware gives you the option to change the MQTT broker as well as the Topic
To access the build menu, you run
west build -t menuconfig
This presents you with the high-level build configuration menu:
The first 3 options apply to the Edge Impulse firmware’s specific configuration option and the last one Zephyr Kernel is where you can make changes to the Zephyr RTOS. It is advisable not make any changes to that option unless know what you are doing with the Zephyr as you can break your project.
You can press Enter on an option to get a modification input field and confirm by pressing Enter again. To commit changes your and either press Q, S or escape repeatedly until you are given the option to save. Rebuilding after making changes to the configuration takes longer because the entire source often needs to be rebuilt.
Let’s look at the Edge Impulse options. First, the Inference thread stack size refers to the memory allocated to the inference thread and the thread priority is also telling the RTOS how important the Inference thread is. Edge Impulse have set these up for you and you need to keep in mind that behind the scenes there is parallel code executing on the nRF9160. You may need to adjust these if your model is pushing the hardware too its resource limit, but these defaults should be good for most applications.
Going into the MQTT Edge Impulse Demo sub menu you can make the changes to the MQTT client to allow you to customize where you want to send the data.
The options are:
MQTT publish topic: This is the topic on the broker that you will publish to, and the receiver will subscribe too
MQTT broker port: The TCP port that the broker expects connections on, consult your broker documentation or configuration. The default of 1883 is commonly used but can be changed.
Seconds to delay before attempting to reconnect to the broker: If the client loses connection this is the timeout before trying to connect again. This could be because an initial connection failed
Seconds to delay before retrying LTE connection: Before the MQTT broker can connect the LTE, connection must be established. This setting controls how long the Thingy91 will take to reconnect to your carrier’s cellular connection
Menuconfig works on the command prompt so will work across platforms including where there is no GUI available. If you have a GUI such as on Windows, you can also run the following command to get a GUI interface that does the same thing as Menuconfig, but you can more easily edit and see the hierarchy of settings thanks it being GUI based
west build -t guiconfig
The Guiconfig menu also provides some hand context information at the bottom area of the display as well.
Before building you need to save your changes. With menuconfig you can keep pressing escape till prompted to save settings and then press “Y” or just press “S” and then ENTER to save to the default file location (very important do not change the file).
To build you would need to exit to return to the command line. Simarly with Guiconfig you need to click the Save button on top and then you wont be able to access the command prompt in the same console you started it from until you fully exit by closing the Guiconfig window.
Try experimenting with the settings and rebuilding and flashing before running AT+CONNECT and AT+RUNIMPULSE from the serial console.
This enables you to explore the full potential of the Thingy91 by publishing data with from your own models. The ability to change the topic particularly on the public broker means you can establish privacy on your data by creating your own private topic.
Next up is the last part of this series of articles where you will be introduced to a version of the Edge Impulse firmware that allows you to add interactivity to your Thingy91 and use it in a more standalone manner.
Next is the final Part 4: Building Interactive standalone Edge Impulse Models with MQTT Connectivity on the Nordic Thingy91
