PART 2 of the 4 part article series on using the Nordic Thingy91 with Edge Impulse
Part 1: From Zero to Hero with the Nordic Thingy91 and Edge Impulse
Part 3: Building the custom Edge Impulse Thingy91 Firmware and Connecting to MQTT
Edge Impulse officially supports the Thingy91 by providing firmware you can flash to the board that in turn allows you to connect to the Edge Impulse Studio to collect data. There is also the option of generating firmware from your own impulses directly from the web interface to deploy to the Thingy91 to run your ML models directly on the device.
Before going further please ensure that you have nRF Connect 3.7.1 installed on your PC. This guide assumes you are using Windows 10. Besides some of the steps for setting up of the Nordic environment many of the Nordic and all of Edge Impulse Web Studio steps are generic across Windows, MacOS and Linux.
Note that at the time of writing only nRF Connect 3.7.1 can be used on Windows and Linux, version 3.7.2 if you are running MacOS.
If you have a later version already installed, you need to do the following to downgrade to nRF Connect 3.7(if you don’t you will encounter issues such as No Trait found errors when attempting to flash firmware):
- Download the most up to date sub version of 3.7 for your platform from this link (https://www.nordicsemi.com/Products/Development-tools/nRF-Connect-for-desktop/Download?lang=en#infotabs). This will either be 3.7.1 (Windows/Linux) or 3.7.2 (Mac).
- Install this download and start it. You will be prompted to upgrade and click No.
- Select Settings from within nRF Connect and select “Add source” and then paste the following URL http://developer.nordicsemi.com/.pc-tools/nrfconnect-apps/3.7-apps/apps.json
- You should see “3.7 Compatible apps, v x.x.x” appear on all the tools together with the latest “official” versions meaning you will see more than one version of each tool. If you are only using the Thingy91, you will be using only the 3.7 compatible versions so an optional step to make your life easier is to click Filter as shown below and filter out the Official/latest versions by deselecting “Official”.
This guide also focuses on using the USB port to flash firmware however if you have Segger J-Link debugger or a board with one built in you can also use the debug port as well to directly flash both IC’s (nRF9160 and nRF52840) on the device.
Next you must install at least the Programmer Add on in nRF Connect by clicking the install button that will then change to “Open” as per below once the programmer has been installed.
Take note that it will say 3.7 compatible apps if you have followed the steps to setup the older version correctly.
When you click Open on the Programmer App it will open the window like below and you are now all good to go.
These steps are documented on Edge Impulse’s site and parts on Nordic’s site. Some of the steps you need for follow in this guide are ones you will be repeating often and having them here in one place will prevent you from having to jump back and forth while reading. The steps here are intended to give you a feel for the full end to end process flow for using the Thingy91 together with Edge Impulse. Following this guide will also help you familiarize yourself with the process of building on the Thingy91 itself. So, let’s get to it then.
The first step is to is flash the device with the Edge Impulse Firmware so that you can connect to the device to the Edge Impulse studio to start collecting data to train your mode. The Edge Impulse firmware is maintained by Edge impulse and available to download prebuilt here:
When you download it and extract the archive you will find the following 3 files:
The specific files you are need are the .hex files which are the compiled firmware ready to flash to the chips on the Thingy91. The “connectivity-bridge.hex” file is for the nRF52840 which acts as the serial to USB interface on the board hence the name connectivity bridge. This firmware doesn’t play much of a role in your final application, and you will only need to flash this once if your hardware version is lower than 1.4.0. The Hardware version is indicated on the label as shown below. If in doubt flash anyway as it only needs to be done once.
The “firmware.hex” file is the main application firmware that runs the inference and manages the sensors and connections to the web-based Edge Impulse Studio. This file is loaded to the nRF9160’s built in application processor when you follow the steps below. Besides allowing you to connect to the Edge Impulse Studio, this firmware also manages model inference, Command Line control, the sensors and of course the LTE Modem. There is also an MQTT client enabled as well for publishing data remotely. The first version you download will not contain your model and is only used for the initial step of connecting to the Studio and gather data. Later you will reflash the nRF9160 with an updated firmware containing your trained model with everything setup and ready to go, this will overwrite the version you initially downloaded You will learn how to do this, step by step.
Since the process of flashing requires you to power cycle your Thingy91 using the power sliding switch SW1 as seen below I highly recommend completely removing the orange cover while doing development as it makes it easier to do.
Starting with flashing the Connectivity Bridge firmware to the nRF52840 you need to plug the device into your PC via USB and then turn the power off.
Locate SW4 which is the nRF52840 recovery button and is a very small button located adjacent to the nRF9160 (big shiny IC) on the PCB.
Be careful as this is a very small switch and its best to use something blunt and made of plastic to press this such as a back of a marker or pen. You can even use the edge of IBasis Sim holder provided. This button is near components including the nRF52840 so be sure not to let whatever you use to press this switch slip off and short out any exposed pads or dislodge any components from the board.
With the nRF Connect Programmer Software open simply power up the Thingy91 by sliding SW1 on while holding SW4 down. This is best done using both hands.
In the Programmer software the device will appear as a single Serial port. The number will vary depending on your system as it’s a virtual com port created by Windows in this case. You should see the grey dot turn green to show its successfully connected to the Thingy91. Select the “Enable MCUboot” checkbox in red.
Go to “Add Hex files” and browse to the “connectivity-bridge.hex” firmware and click “Write” highlighted in green above. The MCU Boot DFU Window will then appear. Click “Write” again on this popup and flashing will begin.
Once complete you should be done with the Connectivity Bridge and won’t need to use SW4 again. MCUBoot is the secure Bootloader that is built into your hex files that allows you to be able to write software via the USB virtual com port, otherwise you would need a Debugger such as the Segger J-Link which in turn means extra hardware you need to source.
Thanks to MCUBoot all you need is a USB Cable so ensure that you always select that option. The Edge Impulse firmware has the MCUBoot loader baked in, so you don’t have to worry about that. This applies to both the nRF9160 and nRF52840 (firmware.hex and connectivity-bridge.hex).
If you encounter issues at this point or errors, then go back and make sure you have v3.7.x installed as described above. Also, if MCUBoot isn’t working you may need a Segger J-Link debugger to directly flash the MCUBoot firmware. If the device is brand new this shouldn’t be a problem, but it could be an issue if someone else has used the device and decided to not load a bootloader.
Next up is the actual firmware file to the main nRF9160 application processor and for that you follow the same process as above, but you need to first click “Clear files” highlighted in blue and then browse for the “firmware.hex”. Like the firmware you flashed to the nRF52840 for the connectivity bridge, the main firmware also contains MCUBoot so that you can flash your application via USB. At this point the following steps to flash the main application to the nRF9160 itself are what you may be using over and again as you update the firmware.
This time turn the Thingy91 off and turn it on while holding down SW3 which is the bigger main push button that is much easier to activate by hand. This can be done with one hand once you get used to it.
This time the device will appear as two serial ports which means everything is good to go.
You can flash the firmware by ensuring MCUBoot is checked and clicking Write. You will get a similar MCUBoot DFU window as with the nRF52840 earlier that will popup, just click Write again.
That’s it’s you are now ready to start using the Thingy91 with Edge Impulse and you can start sampling data. When you reflash firmware later you will be using the last part of this process each time (power cycle while holding the main button SW3).
Edge Impulse Workflow
The general process you will follow with the Thingy91 will be to flash the connectivity bridge once, then flash the main firmware to be able to connect to studio and collect data. As referred to earlier, once you have built and trained and tweaked your model and you are ready to run it on the Thingy91, the Edge Impulse Studio allows you to build a fresh firmware containing your trained model by selecting the Thingy91 build target from the Deployment menu.
Once you click build and its finish building, the Studio will provide you with a new firmware download just like the initial firmware you which will also have the “connectivity-bridge.hex” but this time it contains an updated version of “firmware.hex” and that has your deployed model baked in. All you need to do is reflash the “firmware.hex” using the process above to the Thingy91 and then that you can run it directly on the device either from the command line or a serial console. Note that you don’t need to flash the connectivity bridge firmware again, but it’s included in the download in case you are using it for the first time. Hopefully the process is clear at this point, but practice makes perfect, so I encourage you to complete the steps till you get used to the device.
An alternative to building the firmware directly in the studio other approach you can follow is to build and export the C++ version of our model from the Edge Impulse Studio Deployment page. This is a portable C++ library that contains everything to need to run your model (the feature extraction DSP code as well as the model itself). This can then be dropped into the firmware source and rebuilt from the command line or from an IDE. If you follow this approach, you can then make changes to the firmware, and you won’t downloading prebuilt hex files. More on that in the next section for now we will use the firmware build function in Studio.
Let’s go through these steps to build a basic motion classification model that will form the basis of the rest of this project. At this point of you are familiar with Edge Impulse the process of creating models feel free to skip on to the section on building firmware locally.
The first step is to Log into the Edge Impulse Studio at www.edgeimpulse.com and click the “Create new project” button:
Start by creating an empty Accelerometer project:
Select “Let’s get Started” to open the Studio Dashboard.
At this point you have two ways to connect the Thingy91, one is to use the Command Line Interface (CLI) command “edge-impulse-daemon” or thanks to the WebUSB you can also connect it directly via the Studio Interface in your browser. WebUSB is a newer feature that uses HTML5 APIs to connect the browser directly to local serial ports. It’s the simplest and quickest way to get connected.
Just like in the Nordic Programming Software 2 serial ports appear and they should be numbered the same as you saw in Nordic Programming software unless you are doing this from a different workstation. As long as you see the two ports select the lowest numbered one and click “Connect”.
After a few moments your device is ready for sampling and the ‘Record new data” block will show the Device ID which for the Thingy91 is derived from the LTE modem’s IMEI which is unique for each nRF9160.
You can sample data from the different sensors individually or from a combination of the sensors as seen above. Keep in mind that the Environment Sensor covers the Temperature, Humidity, Pressure and Gas sensors as one option and will sample 4 parameters. Likewise, the Light sensor includes the RGB sensing and the light intensity lux sensing functions and will also return 4 parameters as seen below. The Accelerometer behaves like other boards and returns the 3 axis. When sampling the LED will flash blue except for when sampling light the LED is turned off otherwise it will interfere with the light sensing.
Below you can see examples of the sampling of environment and light separately.
Experiment with this and observe the different sampling length limitations as well as sampling frequencies that were selected by the experts at Edge Impulse to ensure that you get the optimum sample set for each specific type of sensors
For this tutorial we will do basic motion classification example to get you up to speed on Edge Impulse while also getting familiar with the Thingy91.
To build a classifier you will collect data and assign collected data to distinct classes or labels before training. In the case of a motion classifier, you will collect data by moving the device in distinct ways and assigning data collected to distinct and appropriately named labels. Later when you run the model you will move the device in different ways and see how it classifies into one of the trained labels.
What I did was select the Accelerometer only option under Sensor and create samples with the following labels, “motion_up”, “motion_foward”, “motion_side”, “stationery”. Note in this example “motion_up” means up and down and so forth, the names care kept short for brevity, but you can feel free to use which ever labels you want so long as you keep track of what they mean to you in your application.
To sample from the Thingy91, type in the label name in the “Label” field and then click “Start Sampling”.
The example above will sample for 10 seconds collecting 5 samples per second. You can adjust the length and sampling frequency but for this example 5Hz works fine and I would suggest increasing the length, so you don’t have to click “Start sampling” repeatedly.
For the “motion_forward” label I moved the Thingy91 forward and backwards as shown by the blue arrows while sampling was running. I chose the Y Axis as the forward/backward direction. You can see the axis labelled on the PCB above the Sim card holder.
Do the same for up and down motion and label it “motion_up” and then for side-to-side movement and label that “motion_side”. Lastly leave the Thingy91 stationery and sample data after labelling it “stationary”
Once done you will have collected data representing the 3 translational degrees of freedom and the stationary condition. I collected a total of just under 5minutes or just over 1 minute per label.
Before moving on to building your machine learning model, you first need to split your data between training and test sets. If you know what you are doing and have a dataset that is known to work for you could skip this step and train on the training set only. It is however highly advisable when building models. The options on the top of the “Data Acquisition” screen show you how your data is split. The Data Collected block shows you how much data you have collected and visually shows you how its split between classes as different colors.
The same applies for the split between training and test data. You can manually move data between the two sets by clicking the 3 dots on each sample and selecting to move the data and you can view each set by means of the “Training data” and “Test data” options on top. You can also hover over the pie charts to see popups showing you which label it is and the number of samples which is very handy.
A good split is 80% for Training and 20% for test and you can do this automatically from the Dashboard by clicking the “Balance dataset” button which tries to match this split as best as possible based on your data set size.
Once that is done click “Create Impulse” and start by adding a feature processing block by simply clicking on the “Add a processing block”.
For motion classification choose the” Spectral Analysis” processing block which extracts frequency domain features. If you are not sure what means don’t worry about it too much, know that Edge Impulse has provided this block as the optimal processing option for motion classification.
When you deploy your model to the Thingy91 later, Edge Impulse will insert Digital Signal Processing (DSP)processing code into your model to perform this spectral analysis in real time on the device before it feeds the processed features into the neural network model on your device in real time. You never need to worry about it other than knowing it exists. If you were building an embedded ML model from scratch or using other tools you would need to be competent in DSP and know how to do things like Fast Fourier Transforms which involve complex Math and signal processing theory as taught in Engineering classes and then figure out how to implement them. This could greatly delay you in getting your model onto the device. With Edge Impulse it took just a few clicks.
Before finishing you need to click “Add learning block” to insert the ML model itself into the processing pipeline. Select the Classification model. Once again there is no need to worry about the model architecture however you can access the Keras TF code to make changes or implement your own architectures if you come from a ML background. This model uses a very simple but effective architecture.
By now you should see below and all you need to do is click “Save Impulse”.
At this point you have built an impulse that runs from left to right in terms of processing. The menu on the left will now show sub menus under “Impulse design” that will allow you to access the configuration of the blocks added.
In case you are wondering when we are going train the model we are almost there. Before we can do that, we must first generate the features on our training data that we sampled earlier. This important step is required to train the model which actually takes the features as its input and not the raw data, but also to help us check our data if it will result in a model that can classify well. To generate the features, click on “Spectral features” and leave all the options as default by clicking “Save parameters”. Then Click “Generate features” on the Window that appears.
After a few moments a 3D visualization similar to below is shown. In my case I collected the data by ensuring I only moved along one axis at a time and when I saw the features below, I could immediately see the model is going to perform well because the features have separated well.
You should see a similar display and you can rotate it in 3D and click the dots which represent each sample. As long as they are clustered together by label (colour coded according to legend on the left) you should be ok. If you didn’t move yourThingy91 in more or less straight lines and say in circles you would see samples floating away from others or overlapping the wrong class.
The power of machine learning is that you don’t need to move in precise straight lines for an example like this, just more or less. If you moved in exact straight lines the dots would overlap each other and this actually wouldn’t be good because the model would not learn to deal with the kinds of variations from ideal exist in real life. If you needed to detect precise straight lines in this way you wouldn’t need machine learning and could code this based on accelerometer data only. The only catch would be that you would not be able to detect different users’ interpretations of straight-line motion making the system very rigid.
Interestingly look at the red dots which represent stationary, this is one class where you don’t want any movement, so you want the samples to be as close as possible and the dots therefore as closely packed as possible. Try to think about why that is the case.
The last thing left to do was train the model so this was done by clicking on “NN Classifier”, leaving the defaults and clicking “Start training”. Training runs on Edge Impulse’s Cloud backend using their resources, so it happens relatively quickly.
If you have performed a split between test and training sets you can manually test your model using the Model. Below you can see the results from testing for this example and they show good performance on the test set.
Since I have used a small dataset that isn’t ideal, I have only run the test on small set of data by using the “Classify all” feature against my test set. The bigger the data sets the better the model will generalize over unseen conditions. Keep that in mind for when you build production models.
When the training is done you then simply go to Deployment in the Navigation menu in the left and select Nordic Thingy91 as the target and click Build.
Once the firmware is built you simply open the zip file that downloads automatically, and you extract and flash the “firmware.hex” file to the nRF9160 as described above.
Running your impulse on your device
To run your impulse, you have two options the first is to use the CLI command, “edge-impulse-run-impulse” which connects your device over serial (USB to serial for the Thingy91) and allows you to run your impulse and view the results real time. You will need to have the Edge Impulse CLI installed to use run this command.
Select the lowest port number press Enter and your model will start executing on the Thingy91 showing you the results of the inference.
The inference will keep running repeating every 2 seconds and the number next to each label your trained shows you the confidence that the model has that the Thingy91 is in that state. The range is from 0–1 and can be converted to percentage by multiplying by 100. The above example shows that the device is 98% likely to be stationary which is the case. Note that the forward and side categories are registering slightly. This is due to the small dataset.
The second option is to use a serial console and start the Impulse using the Edge Impulse AT Command interface-built Edge Impulse firmware. At this point don’t confuse this with the AT command interface on the nRF9160’s modem.
Putty is a good open-source console/terminal application that allows you to connect to serial ports. In the Session. Use the lowest port number detected and setup Putty to connect over serial at 115200 baud.
You should see the below and then you can you start your impulse by typing “AT+RUNIMPULSE” all in upper caps. This starts running your model continuously.
Your model will execute on the device and will show the same information that you saw when running “edge-impulse-run-impulse”. Not you can use any serial console application not just Putty.
Cloud Connectivity
The Thingy91 is somewhat unique in that it offers full TCP/IP internet connectivity thanks to the LTE modem. To make it simple to use a MQTT client is implemented directly in the firmware so that you don’t have to worry about implementing your own protocol. MQTT has become the defacto standard for IoT due to its simplicity and reliability. MQTT is a publish/subscribe brokered protocol in that it requires a central broker that clients connect to.
Data is sent or rather published to topics which are free form and can be created on the fly by the publishing client and data is read by other clients who subscribe to the topic. Topics have a structure similar to file system paths. For example, the Edge Impulse firmware publishes to the public HiveMQ broker (broker.hivemq.com) on the topic “ei/demo”. The Public broker doesn’t require authentication and any client can connect to and subscribe to that topic. If someone else is experimenting with their Thingy91 at the same time using the default topic their data will be published as well.
To connect to the public broker, you must issue the AT+CONNECT command using the console and this command is blocking in that you can’t do anything till it completes. You may have to wait a couple of minutes. If you see the bellow error, then connectivity didn’t work using the Studio built firmware.
The Thingy91 supports both LTE-M and NB-IoT and at you need to make sure you check with your locally carrier which is supported. If you only have NB-IoT available, you may have issues connecting with the firmware you build in the cloud and be unable to connect. You will most likely need to build the Edge Impulse Firmware locally.
The Thingy91 you receive should have the latest modem firmware loaded but you may discover later that your Modem firmware needs to be updated this is quite straight forward to do. If in doubt I would suggest updating it once, and like the connectivity bridge this is also a once off activity.
The modem firmware is also hosted on the nRF9160 just in a specific memory region associated with the Modem’s private processor, remember that your nRF9160 has an application processor and Modem included in it. The Modem firmware implements an AT command set that allows your host application to control the Modem such as performing initialization and transmitting data etc.
The reason you want to update the Modem firmware is if the board doesn’t connect but that would be the first step in troubleshooting connections. The firmware is located here: https://www.nordicsemi.com/Products/Development-hardware/Nordic-Thingy-91/Download#infotabs
Be sure to select the latest version and at the time of writing mfw_nRF9160_1.3.1.zip was the latest working version. The file will start with mfw_nRF9160.
To update the modem firmware, you follow the same process to put the nRF into MCUBoot mode by holding down the main button SW3 while powering up, this time you select the Update modem button under Cellular Modem.
Instead of browsing to a Hex file you browse the Zip archive you downloaded, and the Programmer software will handle the rest for you.
It’s a two-step process and takes a bit of time and generally would only need to be done once.
Now that you are familiar with flashing firmware and and working with Edge Impulse the next step is to learn how to build the firmware locally, to see how to do this continue on to Part 3: Building the custom Edge Impulse Thingy91 Firmware and Connecting to MQTT
