With the very sad news that Den Automation is having financial troubles, I decided it was time to revisit my investigation into the protocol that Den uses.
I had read on their website that they used both a cloud based and local network approach. In fact, this was one of the things that appealed to me most. Offering local control is an important aspect in any smart home as it insulates you from internet outages.
I had looked into their protocol a few weeks after setting up my Den hub, but it turned up a dead end for me.
Based on what I’d learned my own adventures in IoT, I started working under the assumption that the local protocol would be HTTP based and use probably use mDNS. Using Charles, the populate iOS proxy, I was able to capture some of the traffic from my phone whilst using the Den app.
Unfortunately, all the requests were targeted at Den’s cloud. I was able to see the details of my deployment, the various switches and sockets, but I wasn’t too interested in taking this approach, since I had Google Home and Alexa integrations already setup.
After more digging around, I eventually capture the Publish command used to control the state of the individual devices.
This enabled me to update the state and thus turn the switches on and off.
I’ve created a simple web app – https://denclient.azurewebsites.net/ – which can be used to control the connected hardware. You just need to use the same process as I did to sniff the packets.
At the time of writing this, the Den app was still working for me, which enabled me to perform some basic packet sniffing. Since I performed this investigation, the Den App is no longer working. I presume that AWS/Azure have killed the Den backend due to unpaid bills.
I carried on through the developer study guy and implemented the basic Light. The messaging flow is now making sense to me. I knew the Microbit’s 16KB of RAM wasn’t going to be enough for any real project, so I decided to bite the bulled and pick up a Adafruit nRF52832 device.
Ordered from www.coolcomponents.co.uk, they arrived pretty quickly. I plugged it in, downloaded the Bluefruit app from the app store and was able to discover and connect to the device pretty quickly. So far, so good.
Unfortunately, I couldn’t flash it my Zephyr project. More digging around, using the Nordic Tools and I discovered that the West tool couldn’t detect the device. Strange, as I could flash it with simple Arduino sketches. I trawled the documentation. Tried changing drivers. Updated the bootloader. Nothing had any effect.
During the umpteenth reading of the Zephyr docs, I finally realised why:
Flashing Zephyr onto the nrf52_adafruit_feather board requires an external J-Link programmer. The programmer is attached to the X1 SWD header.
I have emphasized external in the quote above. I totally missed that key piece of information. More googling and I learn that Segger J-Link is a special protocol and that a small unit is required to interface your computer with the board! Everything fell into the place. The different drivers and the talk of the SWD header.
Unfortunately, my Adafruit boards don’t have a SWD header. Fortunately, the board has space and solder pads for one to be added.
I found a J-LINK device on The PiHut which I ordered, along with a short cable. I’ll wait for that to arrive before I attempt to solder this header onto the board. I’m going to need some steady hands!!
J-Link Device arrived and connected up. Managed to solder on the header without any issue.
I think hooked up my newly purchased J-LINK device. Side node here – I ordered a serial cable with this as PiHut said it was recommended. One came in the box, so the extra one I ordered was redundant! I’ll be writing to them about that.
Running the nordic command shows it has found the J-LINK connector!
I was aware that the Expressif ESP32 supports the new Bluetooth LE Mesh protocol, and as Bluetooth Low Energy would help stretch battery life, it seemed like the natural place to start my investigations.
I was initially going to try the WiFi mesh, but it didn’t really seem to suit devices like temperature sensor as they are actually asleep most of the time. If nodes in the mesh keep turning on and off, the other devices in the mesh would be constantly reorganizing themselves to plug the gaps. I do want to learn about ESP’s WiFi mesh, but for now, I’ll focus on Bluetooth
Documenting my progress
Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.
I’ve already got the ESP-IDF environment setup and have two different dev boards. I’m going to try and approach this in stages
Copy one of the sample projects and get it running
Add another node
Send data from one node to another
Add MQTT and WiFi to get data out of the mesh
9th July 2019
I pulled down the ESP-IDF code from their mesh branch and spun up on the samples. After faffing around with the menuconfig settings, I got the Node example compiled and deployed onto the device. It also appeared appeared in the Nordic iOS Mesh App.
Once I tapped on the node, I was able to tap “Identify”, which pulled back more information. I think this is what novelbits called “Invitation”. I then tapped “Provision” and after a few seconds, and a lot of logging from the device, I had the first Node in my mesh!
Now that I had my first mesh node, I quickly realised why the Expressif code was only at version 0.6 – none of the provisioning data was being saved. This meant that each time I flashed the device, I had to provision it again. This became annoying after the fifth time, so I started digging into the actual provisioning process, to see if I could put in something temporary.
I was digging around on google and I came across the Zephyr RTOS. In their examples, they had a simple 2 node mesh demo. The readme page states that no provisioning is required, and whilst it’s insecure, it’s sufficient for the demo. In their code, I could see a few lines of code where they manually provision the device with their own network and device keys.
This seemed ideal for my tinkering. I also found reference to the Zephyr code in Expressif’s news release about their BLE Mesh, where they state they are basing their implementation on top of the Zephyr . This meant I could probably find simple code in the esp-idf repo.
Running this code on the device resulted in it saying that provisioning was complete.
After more digging around the ESP-IDF examples, I came across some testing code, which appeared to provision using the esp code.
Try as I might, I couldn’t get that to work. Just kept reporting an error of -22.
UPDATE 13th August: During some experimenting with how to bridge the mesh network with MQTT, I managed to get this code working eventually, meaning:
After more attempts to provision the device directly, I’ve decided to kinda give up. As I mentioned, the Zephyr code is easier for me to understand at this stage, so I’m going to try and get a build of the Zephyr samples running on my ESP32 boards.
After hours of playing around trying to get the Zephyr SDK compiling against the ESP32 toolchain, I decided to stop. I found an excellent resource at over on the Bluetooth SIG website, called “An Introduction to Bluetooth Mesh Networking“. The samples in their course use the Zephyr SDK and the BBC Microbit. I had a glance though it and most of the main concepts are covered, so I think I’ll order some BBC Microbits and run through it.
I also came across an interest post at https://hutscape.com/, which is worth checking out. The author, Sayanee, uses an Adafruit board to build a UV sensor. Pretty cool. At £26 (and no Friend Node support in the Nordic SDK), I decided against buying any for the time being.
I got a BBC Microbit and tried the Zephyr mesh_demo code. After more mucking around, I managed to flash it with their code.
As suggested by the Bluetooth study guide, I turned on the logging and then
Hardware development is tough! That said, these microbits are perfect for me to dabble in the fundamentals of Bluetooth Mesh and to get to grips with the models. Hopefully there is enough space for me to try my own code. Perhaps it’s possible to compress the Zephyr RTOS runtime down a little??
Been reading about how to figure out the size of the runtime. You just run the command ninja rom_report
I don’t know what those numbers actually mean. I had assumed bytes, but that would mean the bluetooth library, if measured in bytes, was around 50KB and I know that’s not right.
I experimented with the Zephyr mesh_demo sample, but couldn’t make it work. Kept getting error codes when trying to send messages. I tried to tweak it here and there, but I was just strumbling around in the dark. I returned, instead, to the Bluetooth developer guide and gave that go. The guide starts at the Switch module.
The sample code wasn’t up-to-date with the latest Zephyr code, so I had to tweak it somewhat, but I got it to compile and, amazingly, it worked.
I’m interesting in exploring the concept of mesh networking. Within the Expressif family of devices, there are two flavors that I’m aware of; WiFi mesh and Bluetooth Low Energy Mesh. The ESP32 platform supports both of these. My current Temperature Sensor is built upon the ESP8266, but moving to the ESP32 should be straight forward.
In Progress – .
Mk I of my sensor was a simple temperature probe. I wanted to expand on this after I found the Bosch BME280 sensor. This little sensor can capture temperature, pressure and humidity. The boards I ordered work on the I2C protocol, so I’ll have to figure that out.
I use the Hass.io platform to manage my automation at home. It supports a simple MQTT interface for receiving data and sending commands. Typically, you would configure the devices within a config file, but it does support a discovery protocol, which allows devices to make themselves known to Hass and to provide all the information required for Hass to use them.
I would like my temperature sensors to support this mode, so that when they fire themselves up for the first time, they would register themselves automatically.
I’d like to enclose my sensor in a nice case, so that I can mount them on the wall in way that doesn’t look completely ugly!
A few months back, after we discovered mould growing on our bathroom ceiling, I replaced the wall mounted extractor fan with an in-line model, mounted in the loft. I never grasped how important the position of the van was in relation to the source of steam and the source of fresh air. Without going into too much detail, the new vent, directly over the shower, is doing its job.
My extractor fan, like most people’s, is wired to the lighting of the bathroom, so turning on the light also turns on the fan. I have two problems with this:
You need to have the light on, even if it’s the middle of the day.
The fan runs in the middle of the night after a trip to loo.
The first point is my preference. I like natural daylight, so it does annoy me having to turn on the lights. Opening the window just isn’t as effective as the fan. And whilst the in-line unit I purchased is very quiet, in the dead of night, you can hear it roaring away .
It would be nice to only have the fan on when needed e.g. having a shower. Some fans use humidity sensors, but inline fans don’t have that option. Besides, they require the humidity level to rise enough to trigger them, which kinda defeats the point. My wife suggested we add an extra switch to control it, but it’s not practical as I’m not able to get a wire down into the socket. I’ve started looked at some sort of smart alternative.
I’ve been thinking about a solution and I’ve got an idea that I think will work. There are three parts to it.
#2 Disconnect the fan from the bathroom light and control it using an ESP8266 and a relay. This will allow it to be turned off and on via an MQTT instruction. I will also reduce the fans overrun to zero and control that via software instead.
#3 Create an automation to engage the extractor fan when the temperature sensor reaches a certain temp and to turn it off when the temperature drops back down.
In order to have a fast response response to changes in temperature, the unit is going to have to be powered constantly. Fortunately, there is an easy access between the shower pipes and the cupboard under the stairs. I’ll be able to run a cable up there with ease.
The Fan Control
The fan that I’m using in my bathroom is a bog standard inline extractor. It takes three inputs. Mains power, live and neutral and a switch live, which is tied to the bathroom lights.
My plan was to use the existing mains power to power my control circuit and to put the switched live under the control of a simple relay, rather than the light switch. I will also use Hass to manage the fan’s overrun, rather than using the inbuilt circuit. This gives me more control and flexibility.
I have a few Hi Link transformers, which take mains voltage and spit out 5v. The Wemos D1 boards I have will take a 5v input and the small relays I have are also powered using 5v.
The operation of the control would be pretty basic. Using MQTT, it would listen for a run command, the body of which would hold a count in minutes. When received, it would engage a relay to the switched live input of the fan. Once the specified number of minutes elapsed, it would disengage the relay and the fan would stop. If a run command was received whilst the fan was already running, the time would just be reset. If a stop command was received, the relay would automatically disengage.
The controller would also publish its current state when it was changed, something that isn’t necessary, but useful for keeping Hassio up-to-date with changes it didn’t cause!
I assembled a small prototype using some veroboard and a relay I had. It is a low trigger relay, which means it’s engaged when the the indicator input has a low voltage. This is a little annoying (it’s using power to engage the contact). I ordered a high trigger relay, but pressed on with the prototype just the same.
Sadly, this protoype didn’t work! Everything powered up, but the program on the ESP8266 didn’t appear to boot up. Experience taught me this was because of a lack of power, but I expected no more than 300mA to be drawn by both the relay and D1 board.
I attached my bench supply to get a better view of the power consumption.
So it’s wasn’t a power issue. I poked around a little more and after reviewing the MQTT logs, I was happy that the code on the D1 was actually running correctly. This was puzzling. All information pointed to the relay being a problem.
I reconnected to my PC via USB and ran the code again. Everything worked perfectly.
As I’m using a low trigger relay, this means that the relay engages when the output PIN is set to ground. Using my multi meter, I measure the voltage across the PIN and got a value of 0.095V
Reconnecting my rectifier, I tested the voltage again and got a different value!
The penny then dropped. What if the cut off was 0.1v? I quickly popped a resister in series and bingo, the relay engaged!
As I already knew, the low trigger relay didn’t align with my requirements, so aside from a fun diversion, this wasn’t a problem I needed to solve.
With the fan control hardware prototype up and running, I turned my attention to Hass control. As I was using the MQTT protocol, this was very straight forward.
After some experimenting, I had to bin my approach of passing the running time as part of the body because of how MQTT works in HASS. Whilst HASS will issue MQTT commands for a particular device, it also listens for status changes from the device. The body of these two messages must match up. For example, if I issued START5, my board would respond with START5 and HASS would turn the fan yellow to indicate its running. If I send the MQTT command outside of HASS (MQTT directly, for example) with a body of START10, HASS won’t identify the fan has started and its state would be wrong. I don’t want to force myself into relying only on HASS to start the fan. Commands like START and STOP are sufficient enough for my needs.
It’s not the end of the world. The fan already has a fixed overun time, so I won’t be any worse off. In the future, I could look at another MQTT command that could dynamically change the overrun time.
More to do
The idea was sound, but I have much more work to do.
Replace the replay with a high-trigger one
Connect the control unit to the extractor fan
Connect the temperature sensor to the hot water pipe
I opted to use a probe on a long wire rather than using the little component. I ordered some of these from Amazon – https://amzn.to/2IiYEQs. I thought these would give me more flexibility when it came to positioning the probe.
The plan is to build a couple of these sensors and put then around my house. Mostly out of curiosity. I want to see how the house warms and cools during the coarse of the day and what impact the heating & occupancy has (more sensors!)
I had come across this article – https://openhomeautomation.net/esp8266-battery – which describes how an ESP8266 can run for years on a large battery. Seemed very straight forward and I figured that I could use a 9v battery to power one of these units for a few months at the very least.
I built the circuit and wrote the code using VSCode and the amazing PlatformIO plug-in. Essentially, my code connects to my WiFi, connects to an MQTT server (running on Hassio) and sends the temperature value. It will then go into a deep sleep for five minutes and repeat the process. This felt straight forward.
I put the circuit together on a breadboard to ensure I got the temperature reading working. I then moved the assembly onto a small circuit board and soldered it all in.
The 8266 comes requires 3.3v to operate, but comes with an in-built power regulator allowing you to power it with anything between 3.3v and 12v.
This is what I connected my 9v battery to. Everything seemed to work perfectly and my Hassio instance was recording the temperature coming from the sensor.
Leaving it over night, I checked it in the morning. The temperature was logged every 30 seconds (I initially went for this to test its behaviour) until about 6am, when it went dead.
Some quick investigation showed the 9v battery had drained too much and the ESP8266 was no longer able to connect to the WiFi. That wasn’t what I expected. I knew it would use power when it was connected to the WiFi etc. but to drain a battery in a day? I decided to try and work it out.
First, let’s assume it’s running all the time. According to my multi meter, it’s drawing ~80mA.
This didn’t seem like a whole lot, until I checked this page – http://www.techlib.com/reference/batteries.html – and discovered that there are only 500mA hours in one of those batteries. So, at 80mA draw, the battery would supply the ESP8266 for 500/80 = 6.
Wow. Six hours wasn’t what I had in mind!
I’m not reading the temperature all the time, using a deep sleep as I mentioned before, to reduce its power usage between temperature reads. So if we assume it draws next to nothing when its asleep, let’s work out the power consumption over the course of an hour.
It takes about ten seconds to connect to the WiFi, connect to MQTT and send the temperature reading. This happens every thirty seconds, to in total, it’s essentially forty seconds between each read Read (10) | Sleep (30) | Read (10) etc.
So in an hour, we’ll take 90 readings.
This means we run at ~80mA for 90 x 10s or 15 minutes, per hour. This means our 500mA battery should run for about 24 hours. Except, it didn’t.
Using the multi meter again, I discovered the 8266 was drawing ~11mA when it was in deep sleep.
This was much, much higher than the power consumption outlined by the articles I had read online. Some further researching showed other people have the same problem.
It means the 8266 draws 11mA for 45 minutes of every hour. Out 500mA battery could only run for about forty five hours. That’s not even two days. It should be able to run for over a year on a battery when it’s asleep.
More reading on t’internet about the NodeMCU and one of the comments in this post (https://openhomeautomation.net/esp8266-battery) mentions the voltage regulator on the board draws ~11mA. Something called Quiescent Current. I was supplying the board with 9v input to the GND/VIN pins, which would require the voltage regulator to kick in. Could this be the problem?
Using my bench power supply, I connected it directly to the 3.3v input and set the voltage to a constant 3.3v. I also removed the temperature sensor hardware and replaced it with a fixed software measurement to ensure no current was drawn through the board.
Using the bench power supply did make me aware of one thing. The current readings were different across the two devices.
I had received delivery of some Wemos D1 modules, which other people had reported some success with. I wired up one of these and loaded on the same program (with temperature reading code completely removed).
According to my multi meter, it was using over 150mA, but wasn’t actually working (no message being sent via MQTT). This threw me completely!
In the spirit of clueless experimentation, I moved the input on the multi-meter from mA to 10A and, lo and behold, the current dropped and my MQTT subscriber started getting data. I got some help from @themainframe via Twitter and he told me this down to impedance and quite normal. I made a note to research this topic a little further.
The deep sleep also dropped the current being drawn to a level so low that it didn’t register on my bench supply. Result!!
The next step was to re-enable the code and circuity that actually took the temperature reading. In my original design, I was feeding power to the probe from a 3.3v output on the ESP dev board. Following the circuits others designed, this time around I would power the circuit from the power supply directly.
This worked like a treat! Measuring the current, I could see the draw whilst it was measuring and transmitting the temperature was very low.
The next step was moving from bench power to battery power. I recalled another blog post where the author had used a very large lithium ion battery to power his IoT device. I found the same battery and ordered a pair and also got some holders for them. It’s an 18650 type, which is 3.7v with a large 2000mAh capacity.
I also needed a way to drop from 3.7v to 3.3v. I had used a voltage regulator on my smart lamp project (blog post pending) so that seemed like what I wanted. I came across the MCP1700-3302E regulator, which seemed perfect. They seemed like they would deliver the 3.3v I needed from the 3.7v battery. They were also able to handle 250mA, which was more than enough for my setup.
Putting it all together and I had a working circuit!
Next step was reducing this down to fit in the little box. I bought some veroboard and set to work.
Leaving the unit over night and logging to the same Hass.io queue, I was delighted to discover when I came down in the morning that the device was still transmitting. It was interesting to see my kitchen falling to around 15 degrees.
With it all packaged up, the last step is to charge the LiPo battery fully before I “deploy” the device.
Once it’s been running for a week or so, I’ll have a much better idea of how reliable it is and then I will look at the next steps. I want to design a PCB for the sensor and get a better box to house it all. The software could also be improved, for example, only sending an updated temperature if it has actually changed. I would also like a way to dynamically alter the sleep period, for example, checking very 30 minutes between midnight and 5am.
It has been interesting to get back into some electronics. I forgot how much I really enjoy experimenting and learning with physical stuff, not just software!