A few months ago, Den Automation Ltd appeared to drop off the face of the earth. My hub disconnected from the internet and the Den app would only work whilst I was at home.
I should point out that I invested in Den via Seedrs.
Continued Operation is possible
I took time to try and understand the protocol and eventually managed to write some code that allowed me to continuing using them. I created a Flow in my Home Automation installation using NodeRed. This essentially bridged messages from my Hass over to my Den Hub.
This has been reasonable successful, but there are some issues.
#1 The Den Hub needs to be restarted every week or so. I don’t know what’s happening, but I suspect the MQTT server they are using has a memory leak. After a few days of successful operation, it disconnects from NodeRed and refuses new connections. A restart clears this.
#2 Speed is inconsistent. Sometimes operating a light or socket from within HA will result in speedy action from the Den unit. Sometimes it takes several seconds and sometimes nothing happens.
#3 Disconnections from the switches and sockets. This has happened my bathroom switch. It just no longer operates or sends its state. I guess it has lost contact with the Hub. I might try and reset it, but I think that’s probably a crap shoot since I have no App to guide the process.
Den Automation Ltd. are officially in liquidation.
As somebody who invested in both shares and product, I’ve been hit twice.
I’m also in the process of trying to recover some of the money I spend on pre-orders, as some of the items were never delivered. I’ve submitted the paperwork, but I have yet to get a response.
End of the line
With Den officially in Liquidation, there is no longer any hope for the company.
I’ve also decided to start the process of removing all their products from my home. First to go was the double socket connected to my Kitchen’s TV. I’ve replaced this with a Sonoff switch (flashed with Tasmota).
I’ve had a great experience with the Shelly 1 relay product, so my plan is to remove all the Den light switches upstairs and restore the original switches, with Shelly relays going into the lights.
The loss of Den’s cloud crippled the product for most people who had purchased it. I was lucky that I was had the knowledge and tools to determine the local protocol and leverage it from my own ends. Sadly, I wasn’t able to help anyone else.
Den’s local control ability was the primary thing that attracted me to their product line. The relied on the Cloud for user account’s and when that went, their app became useless, actually logging the users out. Not as “local” as I had hoped.
Sonos’s recent announcement about their hardware has made me come to the conclusion that any hardware that needs a Cloud is bad. It’s hardware you own, but don’t own. I’ve got £250 of Den products that I can’t really use because a company was poorly managed, despite the product itself being fully operational.
With news that Den Automation were likely going into administration (due to massive debt) and the discontinuation of their Cloud services, my dreams of adding more Den switches were well and truly dashed!
I’d added a few Aqara temp sensors and contact sensors to my Home Automation setup and I discovered that they had a property Wall Switch, one that claimed to work without a neutral wire. I’d ran into this limitation before with smart switches. I found a vendor on Aliexpress and ordered some. After some issues with packages being returned and duty being paid, they finally arrived.
I ordered a single gang and a pair of two-gang switches. Aside from everything being in Chinese (I think) they looked pretty solid.
On inspection of the individual units, I realised immediately there would be some issues.
First, there was the depth of the backing box. It was around 27mm, which I knew would be too deep for the boxes I my own house. I had the same problem with the Den switches, but they provided a spacer to compensate. These switches had no spacer.
The second problem was the two gang switches weren’t double pole, meaning the two way switches I had hoped to replace were out of bounds. I should have guessed this really. I had some other two gang switches in my house, which might be good candidates to replace.
WARNING! Please don’t attempt anything related to mains electricity unless your confident in your ability. Always remember to isolate circuits from the fuse board before doing anything!!I am not responsible if anything happens.
So, back to basics. Did they work? I turned to my trusty test rig!
Confident the switch actually work, I chose the downstairs toilet switch to replace. It was a single gang switch, which mirrored my test rig.
Amazingly, the switch did fit. Mostly down to the thickness of the wall plaster and the empty backing box.
I popped the cover back on and clicked it a few times to confirm it worked.
Adding to my Home Assistant
This part proved more difficult.
The scan for a new switch didn’t find anything. I tried it several times, resetting the Aqara switch over and over. It was only after I put my laptop down in frustration that I spotted lots of new entries under the “Lights” section!
Turns out it was treating them as Lights, but calling them Smart Plugs.
Not a big deal, but something to watch out for.
I synced Home Assistant and it popped up as a Plug.
I choose the toilet for another reason. My son is using the toilet now and, due to his height, or lack thereof, he has to shout for somebody to turn on the light.
I installed one of the Aqara Push switches at my toddler height
I then took the opportunity to install Node-RED and to configure a simple flow. When the event from the button is detected, it will turn on the switch above for three minutes.
It’s a nice bit of hardware. I’m a little mystified about how it *actually* it works. I’ve read that it only works with dimmable bulbs, so I guess it doesn’t actually every disconnect the circuit. It’s reliable and switches pretty quickly. It doesn’t suffer the same limitation as the Den Automation switches, which can only be switched once every minute.
Due to the limited design of the two gang switches, I can’t replace any of the two way switches in the house. There are two other two gang switches in my house. One of the boxes is packed to the gills, so I won’t be touching that.
The outdoor lights might be an option, but I haven’t opened up the switch to take a look.
All in all, it seems like a pretty solid offering. The depth of the switch is certainly an issue and the lack of a supplied spacer might result in you having to mix and match with some off the shelf ones, which isn’t ideal.
Aqara also do a battery version in the same style, so you can mix and match. I’ve picked a few of them up too and I’m hoping to use them in conjunction with some Shelly 1 relays to swap out the two way switches.
With the very sad news that Den Automation is having financial troubles, I decided it was time to revisit my investigation into the protocol that Den uses.
I had read on their website that they used both a cloud based and local network approach. In fact, this was one of the things that appealed to me most. Offering local control is an important aspect in any smart home as it insulates you from internet outages.
I had looked into their protocol a few weeks after setting up my Den hub, but it turned up a dead end for me.
Based on what I’d learned my own adventures in IoT, I started working under the assumption that the local protocol would be HTTP based and use probably use mDNS. Using Charles, the populate iOS proxy, I was able to capture some of the traffic from my phone whilst using the Den app.
Unfortunately, all the requests were targeted at Den’s cloud. I was able to see the details of my deployment, the various switches and sockets, but I wasn’t too interested in taking this approach, since I had Google Home and Alexa integrations already setup.
After more digging around, I eventually capture the Publish command used to control the state of the individual devices.
This enabled me to update the state and thus turn the switches on and off.
I’ve created a simple web app – https://denclient.azurewebsites.net/ – which can be used to control the connected hardware. You just need to use the same process as I did to sniff the packets.
At the time of writing this, the Den app was still working for me, which enabled me to perform some basic packet sniffing. Since I performed this investigation, the Den App is no longer working. I presume that AWS/Azure have killed the Den backend due to unpaid bills.
I carried on through the developer study guy and implemented the basic Light. The messaging flow is now making sense to me. I knew the Microbit’s 16KB of RAM wasn’t going to be enough for any real project, so I decided to bite the bulled and pick up a Adafruit nRF52832 device.
Ordered from www.coolcomponents.co.uk, they arrived pretty quickly. I plugged it in, downloaded the Bluefruit app from the app store and was able to discover and connect to the device pretty quickly. So far, so good.
Unfortunately, I couldn’t flash it my Zephyr project. More digging around, using the Nordic Tools and I discovered that the West tool couldn’t detect the device. Strange, as I could flash it with simple Arduino sketches. I trawled the documentation. Tried changing drivers. Updated the bootloader. Nothing had any effect.
During the umpteenth reading of the Zephyr docs, I finally realised why:
Flashing Zephyr onto the nrf52_adafruit_feather board requires an external J-Link programmer. The programmer is attached to the X1 SWD header.
I have emphasized external in the quote above. I totally missed that key piece of information. More googling and I learn that Segger J-Link is a special protocol and that a small unit is required to interface your computer with the board! Everything fell into the place. The different drivers and the talk of the SWD header.
Unfortunately, my Adafruit boards don’t have a SWD header. Fortunately, the board has space and solder pads for one to be added.
I found a J-LINK device on The PiHut which I ordered, along with a short cable. I’ll wait for that to arrive before I attempt to solder this header onto the board. I’m going to need some steady hands!!
J-Link Device arrived and connected up. Managed to solder on the header without any issue.
I think hooked up my newly purchased J-LINK device. Side node here – I ordered a serial cable with this as PiHut said it was recommended. One came in the box, so the extra one I ordered was redundant! I’ll be writing to them about that.
Running the nordic command shows it has found the J-LINK connector!
I was aware that the Expressif ESP32 supports the new Bluetooth LE Mesh protocol, and as Bluetooth Low Energy would help stretch battery life, it seemed like the natural place to start my investigations.
I was initially going to try the WiFi mesh, but it didn’t really seem to suit devices like temperature sensor as they are actually asleep most of the time. If nodes in the mesh keep turning on and off, the other devices in the mesh would be constantly reorganizing themselves to plug the gaps. I do want to learn about ESP’s WiFi mesh, but for now, I’ll focus on Bluetooth
Documenting my progress
Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.
I’ve already got the ESP-IDF environment setup and have two different dev boards. I’m going to try and approach this in stages
Copy one of the sample projects and get it running
Add another node
Send data from one node to another
Add MQTT and WiFi to get data out of the mesh
9th July 2019
I pulled down the ESP-IDF code from their mesh branch and spun up on the samples. After faffing around with the menuconfig settings, I got the Node example compiled and deployed onto the device. It also appeared appeared in the Nordic iOS Mesh App.
Once I tapped on the node, I was able to tap “Identify”, which pulled back more information. I think this is what novelbits called “Invitation”. I then tapped “Provision” and after a few seconds, and a lot of logging from the device, I had the first Node in my mesh!
Now that I had my first mesh node, I quickly realised why the Expressif code was only at version 0.6 – none of the provisioning data was being saved. This meant that each time I flashed the device, I had to provision it again. This became annoying after the fifth time, so I started digging into the actual provisioning process, to see if I could put in something temporary.
I was digging around on google and I came across the Zephyr RTOS. In their examples, they had a simple 2 node mesh demo. The readme page states that no provisioning is required, and whilst it’s insecure, it’s sufficient for the demo. In their code, I could see a few lines of code where they manually provision the device with their own network and device keys.
This seemed ideal for my tinkering. I also found reference to the Zephyr code in Expressif’s news release about their BLE Mesh, where they state they are basing their implementation on top of the Zephyr . This meant I could probably find simple code in the esp-idf repo.
Running this code on the device resulted in it saying that provisioning was complete.
After more digging around the ESP-IDF examples, I came across some testing code, which appeared to provision using the esp code.
Try as I might, I couldn’t get that to work. Just kept reporting an error of -22.
UPDATE 13th August: During some experimenting with how to bridge the mesh network with MQTT, I managed to get this code working eventually, meaning:
After more attempts to provision the device directly, I’ve decided to kinda give up. As I mentioned, the Zephyr code is easier for me to understand at this stage, so I’m going to try and get a build of the Zephyr samples running on my ESP32 boards.
After hours of playing around trying to get the Zephyr SDK compiling against the ESP32 toolchain, I decided to stop. I found an excellent resource at over on the Bluetooth SIG website, called “An Introduction to Bluetooth Mesh Networking“. The samples in their course use the Zephyr SDK and the BBC Microbit. I had a glance though it and most of the main concepts are covered, so I think I’ll order some BBC Microbits and run through it.
I also came across an interest post at https://hutscape.com/, which is worth checking out. The author, Sayanee, uses an Adafruit board to build a UV sensor. Pretty cool. At £26 (and no Friend Node support in the Nordic SDK), I decided against buying any for the time being.
I got a BBC Microbit and tried the Zephyr mesh_demo code. After more mucking around, I managed to flash it with their code.
As suggested by the Bluetooth study guide, I turned on the logging and then
Hardware development is tough! That said, these microbits are perfect for me to dabble in the fundamentals of Bluetooth Mesh and to get to grips with the models. Hopefully there is enough space for me to try my own code. Perhaps it’s possible to compress the Zephyr RTOS runtime down a little??
Been reading about how to figure out the size of the runtime. You just run the command ninja rom_report
I don’t know what those numbers actually mean. I had assumed bytes, but that would mean the bluetooth library, if measured in bytes, was around 50KB and I know that’s not right.
I experimented with the Zephyr mesh_demo sample, but couldn’t make it work. Kept getting error codes when trying to send messages. I tried to tweak it here and there, but I was just strumbling around in the dark. I returned, instead, to the Bluetooth developer guide and gave that go. The guide starts at the Switch module.
The sample code wasn’t up-to-date with the latest Zephyr code, so I had to tweak it somewhat, but I got it to compile and, amazingly, it worked.