TS MK II – Mesh Network Pt.2

As part of my Temperature Sensor upgrade, I’ve started looking into Mesh Networking support.

Documenting my progress

Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.

The beginning

I recommend you read the first post before picking up here.

31th July

I carried on through the developer study guy and implemented the basic Light. The messaging flow is now making sense to me. I knew the Microbit’s 16KB of RAM wasn’t going to be enough for any real project, so I decided to bite the bulled and pick up a Adafruit nRF52832 device.

Ordered from www.coolcomponents.co.uk, they arrived pretty quickly. I plugged it in, downloaded the Bluefruit app from the app store and was able to discover and connect to the device pretty quickly. So far, so good.

Unfortunately, I couldn’t flash it my Zephyr project. More digging around, using the Nordic Tools and I discovered that the West tool couldn’t detect the device. Strange, as I could flash it with simple Arduino sketches. I trawled the documentation. Tried changing drivers. Updated the bootloader. Nothing had any effect.

During the umpteenth reading of the Zephyr docs, I finally realised why:

Flashing Zephyr onto the nrf52_adafruit_feather board requires an external J-Link programmer. The programmer is attached to the X1 SWD header.

https://docs.zephyrproject.org/latest/boards/arm/nrf52_adafruit_feather/doc/index.html

I have emphasized external in the quote above. I totally missed that key piece of information. More googling and I learn that Segger J-Link is a special protocol and that a small unit is required to interface your computer with the board! Everything fell into the place. The different drivers and the talk of the SWD header.

Unfortunately, my Adafruit boards don’t have a SWD header. Fortunately, the board has space and solder pads for one to be added.

The SWD header goes here

Adafruit sell the headers, but nobody in the UK had them available. I searched around and found something on RS that looked right.

I wasn’t prepared for the tininess of the header

I found a J-LINK device on The PiHut which I ordered, along with a short cable. I’ll wait for that to arrive before I attempt to solder this header onto the board. I’m going to need some steady hands!!

3rd August

J-Link Device arrived and connected up. Managed to solder on the header without any issue.

Added the RS 10 pin SWD header to my Adafruit nRF52

I think hooked up my newly purchased J-LINK device. Side node here – I ordered a serial cable with this as PiHut said it was recommended. One came in the box, so the extra one I ordered was redundant! I’ll be writing to them about that.

Hooked up (I should point out that this shows the correct cable orientation – see below)

Running the nordic command shows it has found the J-LINK connector!

Sadly, nothing is easy…

Big cryptic error trying to flash the device

My attempt to flash the hello_world sample failed. Ah man – with some help from this https://embeddedcomputing.weebly.com/segger-j-link-with-adafruit-feather-nrf52.html I realised I had the cable connected the wrong way around. I could find no information on the orientation of the header or cable, but I reversed it based on that pic and boom…..

SUCCESS!

Zephyr hello_world sample successfully written to device

It appeared that I had successfully flashed the device with the sample code.

Terminal output shows hello_world is working

Bloomin’ heck. It did actually work!

With the flash process *finally* working, I create a new “light” project, using what I’d learned from the BBC Microbit. I was then able to reuse the BBC Microbit switch.

I’d consider that a success! I think I’ve enough of a basic grasp of BLE Mesh networking to park this part of the project and move on!

Advertisements

TS Mk II – Mesh Network Pt.1

As part of my Temperature Sensor upgrade, I’ve started looking into Mesh Networking support.

I was aware that the Expressif ESP32 supports the new Bluetooth LE Mesh protocol, and as Bluetooth Low Energy would help stretch battery life, it seemed like the natural place to start my investigations.

I was initially going to try the WiFi mesh, but it didn’t really seem to suit devices like temperature sensor as they are actually asleep most of the time. If nodes in the mesh keep turning on and off, the other devices in the mesh would be constantly reorganizing themselves to plug the gaps. I do want to learn about ESP’s WiFi mesh, but for now, I’ll focus on Bluetooth

Documenting my progress

Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.

Getting started

I’ve already got the ESP-IDF environment setup and have two different dev boards. I’m going to try and approach this in stages

  • Copy one of the sample projects and get it running
  • Add another node
  • Send data from one node to another
  • Add MQTT and WiFi to get data out of the mesh

9th July 2019

I pulled down the ESP-IDF code from their mesh branch and spun up on the samples. After faffing around with the menuconfig settings, I got the Node example compiled and deployed onto the device. It also appeared appeared in the Nordic iOS Mesh App.

I found a good post on the provisioning process, https://www.novelbits.io/bluetooth-mesh-tutorial-part-3/ and this helped me make sense of what I was seeing.

Once I tapped on the node, I was able to tap “Identify”, which pulled back more information. I think this is what novelbits called “Invitation”. I then tapped “Provision” and after a few seconds, and a lot of logging from the device, I had the first Node in my mesh!

11th July

Now that I had my first mesh node, I quickly realised why the Expressif code was only at version 0.6 – none of the provisioning data was being saved. This meant that each time I flashed the device, I had to provision it again. This became annoying after the fifth time, so I started digging into the actual provisioning process, to see if I could put in something temporary.

I was digging around on google and I came across the Zephyr RTOS. In their examples, they had a simple 2 node mesh demo. The readme page states that no provisioning is required, and whilst it’s insecure, it’s sufficient for the demo. In their code, I could see a few lines of code where they manually provision the device with their own network and device keys.

static const u8_t net_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};
static const u8_t dev_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};
static const u8_t app_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};

err = bt_mesh_provision(net_key, net_idx, flags, iv_index, addr, dev_key);

This seemed ideal for my tinkering. I also found reference to the Zephyr code in Expressif’s news release about their BLE Mesh, where they state they are basing their implementation on top of the Zephyr . This meant I could probably find simple code in the esp-idf repo.

Running this code on the device resulted in it saying that provisioning was complete.

19th July

After more digging around the ESP-IDF examples, I came across some testing code, which appeared to provision using the esp code.

bt_mesh_device_auto_enter_network()

Try as I might, I couldn’t get that to work. Just kept reporting an error of -22.

UPDATE 13th August: During some experimenting with how to bridge the mesh network with MQTT, I managed to get this code working eventually, meaning:

static const u16_t net_idx;
static const u16_t app_idx;
static const u32_t iv_index;

struct bt_mesh_device_network_info info = {
        .net_key = { 
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    },
        .net_idx = net_idx,
        .flags = flags,
        .iv_index = iv_index,
        .unicast_addr = 0x0002,
        .dev_key = {
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
        },
        .app_key = {
		0x01,
		0x23,
		0x45,
		0x67,
		0x89,
		0xab,
		0xcd,
		0xef,
		0x01,
		0x23,
		0x45,
		0x67,
		0x89,
		0xab,
		0xcd,
		0xef,
},
        .app_idx = app_idx,
        .group_addr = 0xc000
    };

    err = bt_mesh_device_auto_enter_network(&info);

20th July

After more attempts to provision the device directly, I’ve decided to kinda give up. As I mentioned, the Zephyr code is easier for me to understand at this stage, so I’m going to try and get a build of the Zephyr samples running on my ESP32 boards.

23rd July

After hours of playing around trying to get the Zephyr SDK compiling against the ESP32 toolchain, I decided to stop. I found an excellent resource at over on the Bluetooth SIG website, called “An Introduction to Bluetooth Mesh Networking“. The samples in their course use the Zephyr SDK and the BBC Microbit. I had a glance though it and most of the main concepts are covered, so I think I’ll order some BBC Microbits and run through it.

I also came across an interest post at https://hutscape.com/, which is worth checking out. The author, Sayanee, uses an Adafruit board to build a UV sensor. Pretty cool. At £26 (and no Friend Node support in the Nordic SDK), I decided against buying any for the time being.

24th July

I got a BBC Microbit and tried the Zephyr mesh_demo code. After more mucking around, I managed to flash it with their code.

Building the mesh_demo firmware
The output from the microbit shows it working

As suggested by the Bluetooth study guide, I turned on the logging and then

Enable the logging and I use up too much RAM

The damn firmware, with logging, was too big for the BBC Microbit’s 16KB of ram. Bloody hell. Nothing is every easy. Some googling revealed that the small amount of SRAM makes it almost impossible to run the mesh. The Apollo Computer ran on 4KB of RAM ūüôā

Hardware development is tough! That said, these microbits are perfect for me to dabble in the fundamentals of Bluetooth Mesh and to get to grips with the models. Hopefully there is enough space for me to try my own code. Perhaps it’s possible to compress the Zephyr RTOS runtime down a little??

25th July

Been reading about how to figure out the size of the runtime. You just run the command ninja rom_report

The mesh_demo code takes up just over 3% of the total size
The bluetooth code takes up 66%

I don’t know what those numbers actually mean. I had assumed bytes, but that would mean the bluetooth library, if measured in bytes, was around 50KB and I know that’s not right.

28th July

I experimented with the Zephyr mesh_demo sample, but couldn’t make it work. Kept getting error codes when trying to send messages. I tried to tweak it here and there, but I was just strumbling around in the dark. I returned, instead, to the Bluetooth developer guide and gave that go. The guide starts at the Switch module.

The sample code wasn’t up-to-date with the latest Zephyr code, so I had to tweak it somewhat, but I got it to compile and, amazingly, it worked.

One Microbit is the light, the other the switch
Turning the light on and off over BLE Mesh!

You can continue following my adventures here

Temperature Sensor – Mk II

My first go at creating a simple battery powered temperature sensor was really interesting and fun to do, but, like any project, there are always way to make it better!

For my Mark II sensor, there were a few things I wanted to improve:

Mesh Networking

I’m interesting in exploring the concept of mesh networking. Within the Expressif family of devices, there are two flavors that I’m aware of; WiFi mesh and Bluetooth Low Energy Mesh. The ESP32 platform supports both of these. My current Temperature Sensor is built upon the ESP8266, but moving to the ESP32 should be straight forward.

In Progress – .

More Data

Mk I of my sensor was a simple temperature probe. I wanted to expand on this after I found the Bosch BME280 sensor. This little sensor can capture temperature, pressure and humidity. The boards I ordered work on the I2C protocol, so I’ll have to figure that out.

Not started.

MQTT Discovery

I use the Hass.io platform to manage my automation at home. It supports a simple MQTT interface for receiving data and sending commands. Typically, you would configure the devices within a config file, but it does support a discovery protocol, which allows devices to make themselves known to Hass and to provide all the information required for Hass to use them.

I would like my temperature sensors to support this mode, so that when they fire themselves up for the first time, they would register themselves automatically.

Not started.

Battery Life

Not started.

Housing

I’d like to enclose my sensor in a nice case, so that I can mount them on the wall in way that doesn’t look completely ugly!

Not started.

Low battery notifications

Not started.

Adventures in deep sleep with ESP8266 and a temperature sensor

One of the small ESP8266 projects I did was the creation of a simple temperature sensor.

I build a simple circuit, following the guidance from the following articles:

I opted to use a probe on a long wire rather than using the little component. I ordered some of these from Amazon – https://amzn.to/2IiYEQs. I thought these would give me more flexibility when it came to positioning the probe.

The plan is to build a couple of these sensors and put then around my house. Mostly out of curiosity. I want to see how the house warms and cools during the coarse of the day and what impact the heating & occupancy has (more sensors!)

I had come across this article – https://openhomeautomation.net/esp8266-battery – which describes how an ESP8266 can run for years on a large battery. Seemed very straight forward and I figured that I could use a 9v battery to power one of these units for a few months at the very least.

I built the circuit and wrote the code using VSCode and the amazing PlatformIO plug-in. Essentially, my code connects to my WiFi, connects to an MQTT server (running on Hassio) and sends the temperature value. It will then go into a deep sleep for five minutes and repeat the process. This felt straight forward.

I put the circuit together on a breadboard to ensure I got the temperature reading working. I then moved the assembly onto a small circuit board and soldered it all in.

The 8266 comes requires 3.3v to operate, but comes with an in-built power regulator allowing you to power it with anything between 3.3v and 12v.

This is what I connected my 9v battery to. Everything seemed to work perfectly and my Hassio instance was recording the temperature coming from the sensor.

Temperature logged every 30 seconds until about 6am

Leaving it over night, I checked it in the morning. The temperature was logged every 30 seconds (I initially went for this to test its behaviour) until about 6am, when it went dead.

Some quick investigation showed the 9v battery had drained too much and the ESP8266 was no longer able to connect to the WiFi. That wasn’t what I expected. I knew it would use power when it was connected to the WiFi etc. but to drain a battery in a day? I decided to try and work it out.

First, let’s assume it’s running all the time. According to my multi meter, it’s drawing ~80mA.

This didn’t seem like a whole lot, until I checked this page – http://www.techlib.com/reference/batteries.html – and discovered that there are only 500mA hours in one of those batteries. So, at 80mA draw, the battery would supply the ESP8266 for 500/80 = 6.

Wow. Six hours wasn’t what I had in mind!

I’m not reading the temperature all the time, using a deep sleep as I mentioned before, to reduce its power usage between temperature reads. So if we assume it draws next to nothing when its asleep, let’s work out the power consumption over the course of an hour.

It takes about ten seconds to connect to the WiFi, connect to MQTT and send the temperature reading. This happens every thirty seconds, to in total, it’s essentially forty seconds between each read Read (10) | Sleep (30) | Read (10) etc.

So in an hour, we’ll take 90 readings.

This means we run at ~80mA for 90 x 10s or 15 minutes, per hour. This means our 500mA battery should run for about 24 hours. Except, it didn’t.

Using the multi meter again, I discovered the 8266 was drawing ~11mA when it was in deep sleep.

This was much, much higher than the power consumption outlined by the articles I had read online. Some further researching showed other people have the same problem.

It means the 8266 draws 11mA for 45 minutes of every hour. Out 500mA battery could only run for about forty five hours. That’s not even two days. It should be able to run for over a year on a battery when it’s asleep.

More reading on t’internet about the NodeMCU and one of the comments in this post (https://openhomeautomation.net/esp8266-battery) mentions the voltage regulator on the board draws ~11mA. Something called
Quiescent Current. I was supplying the board with 9v input to the GND/VIN pins, which would require the voltage regulator to kick in. Could this be the problem?

Using my bench power supply, I connected it directly to the 3.3v input and set the voltage to a constant 3.3v. I also removed the temperature sensor hardware and replaced it with a fixed software measurement to ensure no current was drawn through the board.

Zero Effect!

Still drawing 11mA in deep sleep. Note discrepancy between bench and multi-meter

Using the bench power supply did make me aware of one thing. The current readings were different across the two devices.

I had received delivery of some Wemos D1 modules, which other people had reported some success with. I wired up one of these and loaded on the same program (with temperature reading code completely removed).

According to my multi meter, it was using over 150mA, but wasn’t actually working (no message being sent via MQTT). This threw me completely!

In the spirit of clueless experimentation, I moved the input on the multi-meter from mA to 10A and, lo and behold, the current dropped and my MQTT subscriber started getting data. I got some help from @themainframe via Twitter and he told me this down to impedance and quite normal. I made a note to research this topic a little further.

Sending the temperature uses some power

The deep sleep also dropped the current being drawn to a level so low that it didn’t register on my bench supply. Result!!

Nothing being consumed in Deep Sleep

The next step was to re-enable the code and circuity that actually took the temperature reading. In my original design, I was feeding power to the probe from a 3.3v output on the ESP dev board. Following the circuits others designed, this time around I would power the circuit from the power supply directly.

Temperature probe connected to directly to power supply

This worked like a treat! Measuring the current, I could see the draw whilst it was measuring and transmitting the temperature was very low.

Really small power draw

The next step was moving from bench power to battery power. I recalled another blog post where the author had used a very large lithium ion battery to power his IoT device. I found the same battery and ordered a pair and also got some holders for them. It’s an 18650 type, which is 3.7v with a large 2000mAh capacity.

I also needed a way to drop from 3.7v to 3.3v. I had used a voltage regulator on my smart lamp project (blog post pending) so that seemed like what I wanted. I came across the MCP1700-3302E regulator, which seemed perfect. They seemed like they would deliver the 3.3v I needed from the 3.7v battery. They were also able to handle 250mA, which was more than enough for my setup.

Putting it all together and I had a working circuit!

Battery powered temperature with deep sleep (finally working!)

Next step was reducing this down to fit in the little box. I bought some veroboard and set to work.

Using some vero board to create a complete circuit.
I used pin holders rather than soldering the D1 directly to the board.
With battery and sensor attached
The board and battery back into the original box.

Leaving the unit over night and logging to the same Hass.io queue, I was delighted to discover when I came down in the morning that the device was still transmitting. It was interesting to see my kitchen falling to around 15 degrees.

The raw data being written
A smoother graph in Hass, which looks nicer!

With it all packaged up, the last step is to charge the LiPo battery fully before I “deploy” the device.

Once it’s been running for a week or so, I’ll have a much better idea of how reliable it is and then I will look at the next steps. I want to design a PCB for the sensor and get a better box to house it all. The software could also be improved, for example, only sending an updated temperature if it has actually changed. I would also like a way to dynamically alter the sleep period, for example, checking very 30 minutes between midnight and 5am.

It has been interesting to get back into some electronics. I forgot how much I really enjoy experimenting and learning with physical stuff, not just software!

Project: Water Softener Salt Level Detector

In my house, I have a water softener. This requires topping up with salt at varying intervals, depending on the usage. In the year I’ve been living here, I’ve forgotten twice. Whilst it’s not the end of the world, it does mean that we get some limescale build up on the¬†taps and, more annoyingly, the shower head. I’ve tried¬†a recurring calendar entry to remind me, but the reminds would happen pop up when I was at work, or out and about etc. By the time I got home, I’d forgotten about it.¬†This sort of small annoyance was something I felt could be fixed with a overly complicated, time consuming, technological solution!

The problem

My water softener has a large compartment at the front, which you fill with salt tablets. Overtime, these get used up as the software regenerates itself.

I wanted a way to detect when the level of the salt had dropped to a low level.

How to measure the level?

My first pass at this problem involved using ultra sonic ranging device. I purchased a simple unit that was good for distances up to three metres. More than enough for my needs. To power this unit and make the necessary calculations, I chose a Raspberry Pi Zero, the latest entry to the Raspberry Pi Family. At about £9 it seemed like a good option to get me started on my first IoT project. To be honest, I bought one on a whim and needed to put it to work!

I connected it all up and, in testing, it worked very well. I went as far as to build a simple iOS app to display the current level (I even build a simple web page with SignalR!!!)  Unfortunately, it was useless when it came to the salt. Ultrasonic works best with flat surfaces and the top of the a pile of salt tables is anything but level! I thought about placing a piece of word or plastic on top of the pile, but this seemed like a hack. Okay, the whole thing is a hack, but I have to maintain some standards!

I went back to the web and started researching how people measure levels like this in real life, coal bunkers and whatnot. They use lasers. I didn’t want to spend any more money than absolutely necessary, so I would need a different approach.

The idea when struck me that I could use a reed switch (one that turns on in the presence of a magnetic field) and a wooden rod with a small magnet on it. The rod would rest of the pile of salt and move downwards, inching the magnet closer to the switch. On paper, this seemed like it would work, but in practice, I didn’t have enough space above the softener for the rod to protrude.

 

img_5492
The initial circuit board with the reed switch

I then had another brainwave. Replace the rod with a piece of string and a weight! The effect was the same, but it didn’t require the space! Genius.

img_5493
Waterproofing the *complex* electronics

img_5494
A lid with the mechanism attached

img_5495
Side view showing the string coming through

I put all the electronics into a water proof box and fed in the USB power cable. I added some LEDs to give me some idea it was working (green indicated my software was running and red indicated low salt). I moved the reed switch to the end of a piece of cable, so I could better position it. Some gorilla tape and a straw (to ensure the string moved unimpeded) and my contraption was complete!

I mounted the contraption onto my softener, replacing the lid.

img_5497

The software is designed to send me an alert once the salt runs low. As tempted as I was to write my own app and host a back end server, sanity prevailed and I just used IFTTT to raise an alert.

img_5498
An IFTTT alert telling me to top up the salt

Summary

The contraption has been in place now for a few months (it’s taken me that long to write this bloody blog post!) and it’s not been a mixed bag. I’ve gotten one alert, which arrived at 3 in the morning. Another time, I just checked the softener on a whim to discover that the salt was low, but the string had snagged and got stuck. I freed it and a second later got the IFTTT alert. At least it proved my python script was reliable.

I’ve put some plastic on top of the wood to ensure the string never snags again and I’m expecting an alert in the next few weeks.

The future?

With version one in place and working, I’ve started thinking about version 2. These are the improvements I plan on making:

  1. Switch to an MCU (Micro Control Unit). The Pi is great, but it’s overkill (running a full OS) and uses much more power than necessary. The NodeMCU board, for example, can be put into a deep sleep, which users very little power.
  2. Replace the breadboard with a PCB to reduce space.
  3. Trigger the measurement at fixed intervals and only notify me when I’m *actually* home. The last part I’m not sure about, but I suspect IFTTT can *know* where you are.
  4. Potentially use an IR sensor for distance measurement. Whilst this would eliminate any moving parts, it would require a hole into the salt holder and I’m worried that water might splash out or onto it.

Running my .Net Core mDNS service on Windows IoT

As part of my ongoing experiments with C# and mDNS (to build my own Homekit accessory), I got to the point where I wanted to run my code on a Raspberry Pi.

Microsoft have released a version of Windows 10 that will run on a Raspberry Pi in a headless mode. The main advantage of this is that it contains the .Net Core runtime!

To get started, download Microsoft IoT manage and fire it up. Under the menu, you’ll see an option to Set up a new device. This lets you flash an SD card with the code necessary.

iot flashing UI.PNG

iot flashing.PNG

Once the formatting and flashing is complete, you’ll have an SD card containing Windows IoT. Pop this into your Raspberry PI, and after a minute or five it will appear under the My devices view.

To get me started, I launched Remote Powershell. When it started, I got prompted for credentials.

iot powershell ip.PNG

This includes the IP Address and the account Administrator (the password for which, you set earlier). I found that this just didn’t work. I replaced the IP address with the machine name (minwinpc) and it worked!

iot powershell ip domain.PNG

I just realised this is a real Windows XP dialog!

After Powershell started, I opened a file share and created a directory called Climenole. To get my little Console app ready, I needed to compile it for ARM. Thankfully, this was straightforward. From the command prompted, I ran this command

dotnet publish -r win-arm

This compiles the app for the win-arm architecture. From the project directory, you can access

\bin\Debug\netcoreapp2.0\win-arm

to see all the published files. I took all these files and copied them across to the Climenol directory I created on the Raspberry Pi.

Once the copy completed, I could run my app using Powershell.

iot wrong ip address.PNG

It didn’t work the first time as the hardcoded IP address wasn’t right, but after a tweak,

iot responding to packets.PNG

It started answering responses and even appeared in Homekit.

img_48941.png

A good first step and proved that my mDNS was portable.

Next challenge is getting all the crypto code working, so that I can actually add and control the accessory from Homekit. It was a pain to get it working on the .Net Framework, so I suspect it will be no picnic on Core!

Update (07 Jan, 2019) I’ve had a couple of people ask to see the mDNS code. It’s up on Github –¬†https://github.com/tomasmcguinness/dotnet-mdns