PART 1 of the 4 part article series on using the Nordic Thingy91 with Edge Impulse
Part 2: Getting started with Edge Impulse and the Thingy91
Part 3: Building the custom Edge Impulse Thingy91 Firmware and Connecting to MQTT
The Thingy91 is a cellular IoT prototyping platform kit from Nordic Semiconductor that not only showcases the capabilities of the nRF9160 SiP but also has an interesting combination of sensors making it ideal for not only testing but also deploying tinyML based IoT Applications.
Unlike many other sensor development kits that connect via Bluetooth to a phone the Thiny91 stands out because it can stand on its own thanks to the LTE modem which allows it to connect directly to the Internet.
This is made possible by the fact that the nRF9160 has a very capable powerful ARM Cortex M33 Application Processor ideal for running your tinyML model complemented by an integrated LTE Modem with an eSim slot supporting NB-IoT and LTE-M together with a GPS Module and a host of useful peripherals.
Together with the main push button input are variety of sensors specifically, Environmental (temperature, pressure, humidity, and gas), Light (Lux and RGB color sensing) and Motion sensing. There are even some useful outputs in the form of dual RGB LEDs, Transistors and even a Buzzer all of which allow the Thingy91 to not only sense but to also interact with the environment.
The unit ships in a plastic housing with a protective rubber cover and includes a rechargeable LiPo battery for deploying your prototype easily. It can also be powered via USB and there is no need for a debugger or other tool as you can flash it via USB as well. Nordic have even added a Sim Card from IBasis with 10Meg loaded to get you going as quickly as possible.
This series of posts will cover the following
1. Overview of Nordic Thingy91
2. Getting started with Edge Impulse on the Thingy91
3. Custom Thingy91 Firmware for Edge Impulse Experimentation
By the end of this series of articles you should have a better understanding of what the Thingy91 is and how you can use it to build tinyML applications using Edge Impulse and hopefully have some fun along the way by building applications on it. So, lets dive straight in starting with a broad overview of the Thing91 hardware.
Thingy91 Hardware Overview
This section contains a more detailed description and overview of the hardware and its features you can however skip on to the section on getting started (Getting started with Edge Impulse on the Thingy91) if you are in a hurry to use it with Edge Impulse. Understanding the hardware features and capabilities is advisable as it will help you with understand the capabilities of the Thingy 91 and may even give you new ideas and use cases for it.
The Thingy91 is a Prototyping platform that has been designed to be both versatile and compact catering for as a wide variety of Cellular IoT use cases.
All the functionality is implemented on a double-sided PCB that can be removed from the supplied case and dropped into your own application if need be. For more advanced applications there are test points and power connectors for external power supplies directly accessible on the PCB. For most applications the USB port and battery will be adequate.
The Thingy91 is designed to be a self-contained all in one unit that gives you everything you need to both sense and interact with the environment, location awareness and Internet connectivity. You don’t need any external components other than a USB cable to start working with the Thingy91. You will need a sim card if you want to make use of Low Power Cellular Internet Connectivity, but it is just as capable when used offline as well.
Nordic has a development kit the nRF9160 DK built specifically around the nRF9160 which you can also use a development and deployment platform, the Thingy91 provides all the key features in one ready to deploy product to make it easier to transition from a bench prototype to something you can put into the field.
The simplified block diagram gives a simplified overview of the architecture of the Thingy91 platform:
At its core the Thingy91 is centered around the nRF9160 which is Nordics flagship cellular device which is constructed as a System in a Package (SiP) and contains both an Application Processor and a Modem with an integrated GNSS receiver. Included are power and battery charge management and RF front ends for the Modem and GPS. There is an eSim holder that takes a Nano (4FF) sim so that you can connect to your local cellular carrier. The diagram above shows the key components of the System Architecture you will interact with when working with the Thingy91.
The application processor is an ARM Cortex M33 clocked at 64MHz with 1MB of flash memory and 256K of RAM together with the standard peripherals found on all MCU’s. The M33 has been designed specifically for IoT applications and balances power efficiency with performance and security features for backend connectivity as well as DSP functionality for signal processing. The M33 is also ideal for running your tinyML models as you will see later and its super easy to use with Edge Impulse.
The Cellular Modem is a separate functional unit from the application processor housing the GPS integrated and its own processor running the modem AT Command firmware. You won’t directly develop for this processor but instead interact with the AT command processor in the Modem by writing commands and reading responses. This greatly simplifies the process of using the Modem and means you don’t need to spend a lot of time getting familiar with the low-level details of the Modem. Nordic even wraps the AT commands in a nice high-level API to reduce the burden on the developer.
There is more to the Thingy91 than just the nRF9160 though because they have cleverly included a nRF52840 as the USB connectivity bridge. What this means is that instead of using one of the popular USB to Serial interface ICs used on many other devkits Nordic instead opted to use their own Bluetooth Low Energy (BLE)SoC for this purpose. The nRF52840 adds Bluetooth and NFC functionality and additional spare I/O’s accessible via a header. Using the functionality of the nRF52840 will require some additional development and synchronization between the two processors that communicate with each other. Nordic have included a Bluetooth and NFC antenna just in case you wish to use these features without any further need for hardware changes. It’s worth noting that the NFC tag is a passive tag meaning that the nRF5480 is unable to generate the RF field to read other tags but instead acts as a dynamically programmable passive tag that can read by a NFC capable phone for example.
So, if you plan to use the NFC feature think of it as your own customizable NFC tag that you can change based on your tinyML model’s predictions by sending data back to the nRF52840 for example. The main application processor you will use in this guide and for most applications will be the nRF9160’s application processor as all the of sensors and output peripherals connect directly to the nRF9160.
While the MCUBoot Bootloader and the Connectivity bridge ensures you can flash custom firmware with nothing more than just a USB cable and the nRF Connect software, you may need to do some advanced debugging. That is why there is a physical debug port located next to the USB port. The Debug port is shared by both nRF9160 and nRF5280 and this the pins effectively connect to either means by a sliding switch, so you in essence have two development kits in one. Behind the scenes there is some multiplexing circuitry that allows this to be possible. Through this port you can debug and step through your code to troubleshoot issues live.
A 16Kbit EEPROM is also present on the board with 512 bytes reserved and the rest of the storage space available for use in your custom application. This could be used to store historical measurements or GPS satellite data for example or data for batch upload when there is no signal or the modem is powered down to save power. The EEPROM adds more capability to platform for IoT applications.
The latest boards above V1.4.0 ship with a 1350maH Lipo battery to allow the device to run untethered. With the default Asset tracking application supplied by Nordic this lasts around 1 full day. If you employ careful power management in your application this can be greatly extended. It’s also possible to replace this with a larger external battery. Note that the battery only charges via USB when the main power switch is on.
Sensors
One of the features that really differentiates the Thingy91 is its environmental sensor that is based on the Bosch Sensortec’s BME680 which is a compact high performance low power environmental sensor. This device packs 4 different sensors in one small footprint housing as seen below.
Not only does it measure relative humidity and temperature from -40 to 85 degrees Celsius it also measures pressure ranging from 300 -1100hPa and of course gas in MOhms. The gas sensor is designed to measure volatile organic compounds (VOC’s) which can help detect various kinds of gases. Below are some examples of the types of gases detected in human breath as noted in the BME680 datasheet.
To ensure there is airflow to the sensor on the PCB the Thingy91 housing has an environmental access port that will allow air to pass through a grille on the housing.
Naturally the rubber cover also has holes over this port so make sure when you reattach the rubber cover that you don’t block these holes. This is easy to do if you align the USB port plug correctly. Upon closer inspection the housing also is also ducted to direct moving air down onto the PCB and a cutout around the BME680 so that there is adequate ventilation of the sensor for improved sensing even from within the housing.
Environmental sensing is where the power tinyML and of Edge Impulse comes to the rescue because you would normally need to understand what you are looking for and code the solution accordingly. For example, if you wanted to measure air quality which is an ideal use case for this sensor you would need to know what gasses to look for and their outputs together with the other ambient parameters and then experiment and tweak you application accordingly. You will most likely not cater for all use cases easily without a lot of effort and empirical measurements.
Using Edge Impulse and embedded machine learning you can cut down the development time significantly while improving performance by taking measurements in real or simulated environments that represent the different air quality scenarios and then allow the training algorithm to figure out the solution for you. The machine learning approach will also be able to handle a greater degree of variation from your trained environment than a hard coded solution thereby allowing it to operate with unseen data. That is provided you build your model correctly.
If you are new to tinyML and even machine learning in general, Edge Impulse will make sure you can do that by hand holding you through the process of building out and tweaking your data set and finally by ensuring that your trained neural network is comfortably squeezed onto the chip with room to spare for your custom application.
Next are the Motion sensors. The Thingy91 has onboard accelerometers and provides a low power accelerometer that can operate in lower power modes and a high G accelerometer that uses more power but is better able to detect shocks from impact. Once again thanks to tinyML you could train the low power accelerometer to detect shocks under certain conditions and maintain the lower power performance for Asset Tracking applications where battery life and longevity are important.
Lastly there is the optical sensor that functions as a RGB/Infrared sensor and a general light sensor. The RGB sensing can measure colour whereas the light sensing capabilities of the sensor measures overall light intensity and colour temperature. The infrared capability allows for measurement of infrared intensity. These functions have different use cases where the colour sensor is used to measure specific colors and could be used for product detection where the light intensity can be used for tamper detection as well as controlling indoor and outdoor lighting for example. The infrared sensing is useful for flame detection applications especially when combined with the environmental sensor.
When using the light sensor be careful to note that using the LED’s can cause unexpected interference. Light is directed to the light sensor by means of a light pipe which is a plastic channel designed into the housing focusing light onto the sensor from the rear of Thingy91. Be sure point the light pipe at your target when taking measurements which means facing the rear towards your target. One little feature that was added to help with colour sensing on surfaces is the addition of an auxiliary LED to illuminate the surface so that the reflective light can be measured. With colour sensing on surfaces you will need this illumination so that the reflected wavelengths can be measured to determine the surface color. There is an additional light pipe to direct this LED’ light to the surface and this will only work in close proximity but provides more accurate color measurement than depending on ambient or natural illumination.
Outputs
If the sensors weren’t enough Nordic has included some handy outputs that you can use to interact with the physical world. As mentioned in the previous paragraph there are LED’s, more specifically dual RGB LED’s, which illuminate a Light well in the casing which acts a diffuser and allows you to see the light from most viewing angles. Being RGB there is the possibility to show different colors by adjusting the primary colors.
For audible output there is an onboard buzzer that is driven by PWM. This buzzer has a wide frequency output however its resonant frequency is 2.7Khz which means it gives the loudest audible output (85dB at 10Cm) at 2.7Khz. Its best driven at 50% duty cycle and for most applications its audible output level is adequate to be noticed in an adjacent room.
The outputs culminate in 4 NMOS transistor outputs which you can use to activate physical outputs and control things in the real world. You do need to do some soldering as these outputs are accessed by the header shown below.
Connectivity
The LTE Modem integrated into the nRF9160 only supports NB-IoT and LTE-M targeting lower power IoT applications where traditional Cellular Technologies tend to be more power hungry.
For example, in NB-IoT Power Saving Mode the specified minimum current of the nRF9160 is only 2.7uA. The lower power Cellular IoT technologies are ideal for use cases where you hit limitations on data throughput with traditional LPWAN technologies but still require a lower power footprint than traditional Cellular technologies (2/3/4G).
You could almost think of the modem as being an external device connected to a serial port and you send it AT commands from your application devkit much like many external modems work. Only in this case because it’s a System in a Package they are all in the same package of the nRF9160 and wiring has been done for you. Included in the Modem is the GPS receiver and the modems own RAM and flash storage.
The modem runs independently from the application processor and implement an AT command set that your application then uses to control the modem and GPS receiver. The AT commands control both the data Modem and the GPS. Both the GPS and LTE radio share the same antenna on the board. This imposes some constraints to be kept in mind and you generally need to turn the LTE modem off when you access the GPS.
Ideally if you are using NB-IoT and your network supports Power Saving Mode (PSM) or extended Discontinuous Reception (eDRX) where the network forces the modem to turn off communication and transmit using power saving time windows then you can use these windows to collect GPS data more frequently while keeping battery drain low.
Should you wish to use an external GPS antenna there is a U. FL connector where you can attach an external antenna. You will however need to modify your project configuration to use the external antenna this is because the onboard antenna has some circuitry to help boost the signal, specifically a Low Noise Amplifier which needs to be disabled or bypassed. This amplifier is required to boost the signal from the on-board antenna which is not as sensitive as a bigger external antenna. GPS works ideally in an outdoor setting and the onboard antenna provided does work reasonably well however you may want to augment the accuracy by using Assisted GPS (A-GPS) which is supported but requires internet connectivity. If you embed the Thingy91 in a vehicle as a tracker for example, then this would be where you may need an external antenna.
Finally, there is a coax SWF connector labelled “LTE” which is a test point for measuring LTE power output and for an external LTE antenna. Take note that for older versions of the Thingy91 both the GPS and LTE connectors are the Coax SWF type connectors. For the versions 1.0.0 and above your GPS should have changed to U.FL.
The IBasis sim card provided does not work in all regions at the time of writing. The easiest way to see if the sim works is to follow the process of activating the sim using the nRF cloud with the default Asset Tracker firmware. If you follow the instructions here and don’t see your device appear in your nRF cloud Dashboard, then its most likely the sim card doesn’t work. Your only option is to source a SIM card from your locally carrier after confirming that either NB-IoT or LTE-M is available in your region. You may also have to explicitly set your carrier’s APN using the LTE Link Manager that is part of the nRF Connect Suite when testing this. Fortunately, you don’t need connectivity to use the device with Edge Impulse
That rounds out the overview of the Thingy91 hardware and its features. By now you can see that it packs a lot of punch in small little package!
The Thingy91 at first looks like another Wireless sensor node for experimenting with IoT but it’s a capable device that gives a lot of bang for buck and has everything you need to not only develop IoT and tracking solutions but to also build tinyML based projects thanks to the right balance of hardware chosen by the designers. Nordic refer to it as prototyping platform, rightly so since you could use this as a basis for advanced prototypes and possibly even embed it into final product depending on the application. Hardware still requires software to do what you intend so this overview would not be complete without introducing the options available to developers.
Software Development
To use the Thingy91 and develop applications the official supported environment is the nRF Connect SDK from Nordic. This is the SDK that has drivers and APIs for handling connectivity and I/O as well as peripherals. Some features such as the Buzzer will however require you to develop implement your own code to provide you with the required output. The nRF9160 is completely supported in that there are high level APIs to access not only the Modem but the GPS and all the MCU peripherals.
Up until now no mention has been made of Zephyr which is the Real Time Operating System (RTOS) that Nordic has built the nRF Connect SDK on top of. Zephyr is an embedded RTOS created in the spirit of Linux and maintained by the Linux foundation to cater for embedded target that can’t run Linux. It is a very power OS that provides a very high level of abstraction over the hardware which may seem overkill but is there to allow code to be portable across many target platforms.
Nordic ships the Thingy91 with an Asset Tracker Application that talks to the nRF cloud backend all built on top of the nRF Connect SDK which in turn is built on top of Zephyr. The source code for both the original (v1) and current (v2) Asset Trackers are available. The Asset Tracker App connects to Nordic’s nRF Cloud is a IoT visualization platform that has widgets to show the GPS location on a map and real time displays of the sensor information, events, and trends. A good starting point when testing the Thingy91 is to load the Asset Tracker app as it also helps verify all is working with the hardware. It is a professionally developed application and nRF Cloud perfectly complements it so you can get going with this use case out of the box if you wish.
For development Nordic provides you with a free license for Segger Embedded to use with Nordic products and you can now also use nRF Connect SDK with Visual Studio Code. There is also the option of simply using a text editor and the command line thanks to Zephyr’s West meta tool.
Edge Impulse also provides firmware built on top of Zephyr and nRF Connect to allow you to collect data and run models. Alternatively, you can also incorporate your Edge Impulse models directly into nRF Connect thanks to support in place for Edge Impulse
In the last part of this article, you will be introduced to customizable firmware that allows you to build some interaction and extend the features using a version of the official Edge Impulse without any code changes required. We will be using just the command line and west to build this later.
Before that lets first get started with Edge Impulse on the Thingy91. If you have a Thingy91, now is the time to get it out and if not, you can order one from most parts of the world here.
Be sure to check out the competition below at the time of writing (January 2022) submissions for project ideas are open till 9 February 2022 to be eligible for a free Thingy91.
There are some really cool cash prizes up for grabs and a special bonus prize for the best project that makes use of Machine Learning with Edge Impulse. Be sure to check it out.
The next part will teach you how to use Edge Impulse with the Nordic Thingy91, what you learn using Edge Impulse Studio applies to any other device. If you don’t have a Thingy91 and plan to enter the competition go ahead and use your mobile phone to start prototyping ideas you have.
Now that you have learned about the Nordic Thing91 you are ready to build your first TinyML model with the Thingy91 this is covered in Part 2: Getting started with Edge Impulse and the Thingy91
