Blast from the past!

I came across this screenshot from my first iPhone app, Caffeine Club. Not the most visually pleasing app ever created, but I had fun building it!

2015/01/img_2836-1.jpg

pkpassvalidator.com update

I have just published a small update to https://pkpassvalidator.com/

The signature now undergoes additional checks, to ensure that the WWDR certificate is used. This is one of the most problematic things for people using my https://github.com/tomasmcguinness/dotnet-passbook library.

Hopefully this small change will help.

Den Automation – API

With the very sad news that Den Automation is having financial troubles, I decided it was time to revisit my investigation into the protocol that Den uses.

I had read on their website that they used both a cloud based and local network approach. In fact, this was one of the things that appealed to me most. Offering local control is an important aspect in any smart home as it insulates you from internet outages.

I had looked into their protocol a few weeks after setting up my Den hub, but it turned up a dead end for me.

Based on what I’d learned my own adventures in IoT, I started working under the assumption that the local protocol would be HTTP based and use probably use mDNS. Using Charles, the populate iOS proxy, I was able to capture some of the traffic from my phone whilst using the Den app.

Unfortunately, all the requests were targeted at Den’s cloud. I was able to see the details of my deployment, the various switches and sockets, but I wasn’t too interested in taking this approach, since I had Google Home and Alexa integrations already setup.

I fired up WireShark and used Apple’s guide on how to intercept all traffic from my iPhone – https://developer.apple.com/documentation/network/recording_a_packet_trace

I immediately spotted a HTTP request, which was a protocol switch request. More digging and some Google-fu and I leaned that this was just MQTT over websockets on port 1884

I loaded up MQTT Explorer and fed in the details (IP address of my hub and Port 1884) but got a 401 (Authentication failed). This was progress as I knew some thing was listening.

Fast forward a few hours and I’d extracted this packet. Thanks to this protocol document ( http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/MQTT_V3.1_Protocol_Specific.pdf ) I was able to determine this was a connect request (0001), which contained a username and password.

The request to login, with a client name of MQTT and the username and password

Armed with some credentials, I fired up MQTT Explorer and was connected.

After a few minutes, I started to get information over the connection! MQTT Explorer includes the # and $SYS/# subscriptions, so it basically gets everything published by the hub.

I knew that the device was one of the double sockets (the TV was on) and it was sending measurements!

{“43de2329-8808-57bc-af5b-ea01bd6a6d11”:{“online”:true},”c8de59a4-29e5-52ad-84b5-2574f005544b”:{“software”:[{“type”:”USER_1″,”version”:”1.3.0″}],”battery_level”:29,”online”:false,”status”:{“0”:{“last_state_changed”:”2019-08-09T15:46:24Z”},”1″:{“last_state_changed”:”2019-08-09T15:46:24Z”}}},”7039ee52-d25a-59c5-b957-844ff067233f”:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:false,”status”:{“0”:{“state”:0,”shutter”:1,”nfc_tag”:{“uid”:”04:cd:be:d2:35:5e:80″,”atqa”:”0x44″,”sak”:”0x00″,”version”:0,”name”:””,”type”:”APPLIANCE”,”keep_on”:false,”timeout”:{“enabled”:false,”duration”:0}},”last_state_changed”:”2019-10-09T11:18:17Z”},”1″:{“state”:0,”shutter”:1,”nfc_tag”:{“uid”:”04:f2:97:ca:35:5e:80″,”atqa”:”0x44″,”sak”:”0x00″,”version”:0,”name”:””,”type”:”APPLIANCE”,”keep_on”:false,”timeout”:{“enabled”:false,”duration”:0}},”last_state_changed”:”2019-10-09T10:59:42Z”}}},”1cef1fcd-3aa7-5a49-80b4-ea98b85c5827″:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:true,”status”:{“0”:{“state”:1,”shutter”:1,”nfc_tag”:{“uid”:”04:61:8a:7a:33:5e:80″,”atqa”:”0x44″,”sak”:”0x00″,”version”:0,”name”:””,”type”:”APPLIANCE”,”keep_on”:false,”timeout”:{“enabled”:false,”duration”:0}},”last_state_changed”:”2019-10-09T18:04:06Z”},”1″:{“state”:1,”shutter”:1,”nfc_tag”:{“uid”:”04:5a:b2:d2:35:5e:81″,”atqa”:”0x44″,”sak”:”0x00″,”version”:0,”name”:””,”type”:”APPLIANCE”,”keep_on”:false,”timeout”:{“enabled”:false,”duration”:0}},”last_state_changed”:”2019-10-09T18:04:08Z”}}},”c0713ae1-1913-5ab7-944b-d94ea4b5477d”:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:false,”status”:{“0”:{“state”:0,”is_ready”:true,”last_state_changed”:”2019-10-09T17:30:34Z”}}},”4f832cb2-824f-58dc-87ad-5a89e121b653″:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:true,”status”:{“0”:{“state”:0,”is_ready”:true,”last_state_changed”:”2019-10-07T17:23:55Z”}}},”59607912-dda1-5e97-9e3d-6b065b9288db”:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:false,”status”:{“0”:{“state”:0,”is_ready”:true,”last_state_changed”:”2019-10-08T18:59:02Z”}}},”5a008677-5c2e-5405-a6b8-357a7e64642a”:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:true,”status”:{“0”:{“state”:0,”is_ready”:true,”last_state_changed”:”2019-10-08T21:19:37Z”}}},”8016c79f-8502-5b5c-b480-bb6361af1398″:{“software”:[{“type”:”USER_1″,”version”:”1.4.0″}],”online”:true,”status”:{“0”:{“state”:0,”shutter”:0,”nfc_tag”:null,”last_state_changed”:”2019-10-07T11:13:41Z”},”1″:{“state”:0,”shutter”:0,”nfc_tag”:null,”last_state_changed”:”2019-09-29T12:54:36Z”}}}}

Switching stuff on and off

After more digging around, I eventually capture the Publish command used to control the state of the individual devices.

A 142 bytes WebSocket message sent when I turned on a switch
The content is an MQTT publish (0011)

This enabled me to update the state and thus turn the switches on and off.

I’ve created a simple web app – https://denclient.azurewebsites.net/ – which can be used to control the connected hardware. You just need to use the same process as I did to sniff the packets.

At the time of writing this, the Den app was still working for me, which enabled me to perform some basic packet sniffing. Since I performed this investigation, the Den App is no longer working. I presume that AWS/Azure have killed the Den backend due to unpaid bills.

TS MK II – Mesh Network Pt.2

As part of my Temperature Sensor upgrade, I’ve started looking into Mesh Networking support.

Documenting my progress

Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.

The beginning

I recommend you read the first post before picking up here.

31th July

I carried on through the developer study guy and implemented the basic Light. The messaging flow is now making sense to me. I knew the Microbit’s 16KB of RAM wasn’t going to be enough for any real project, so I decided to bite the bulled and pick up a Adafruit nRF52832 device.

Ordered from www.coolcomponents.co.uk, they arrived pretty quickly. I plugged it in, downloaded the Bluefruit app from the app store and was able to discover and connect to the device pretty quickly. So far, so good.

Unfortunately, I couldn’t flash it my Zephyr project. More digging around, using the Nordic Tools and I discovered that the West tool couldn’t detect the device. Strange, as I could flash it with simple Arduino sketches. I trawled the documentation. Tried changing drivers. Updated the bootloader. Nothing had any effect.

During the umpteenth reading of the Zephyr docs, I finally realised why:

Flashing Zephyr onto the nrf52_adafruit_feather board requires an external J-Link programmer. The programmer is attached to the X1 SWD header.

https://docs.zephyrproject.org/latest/boards/arm/nrf52_adafruit_feather/doc/index.html

I have emphasized external in the quote above. I totally missed that key piece of information. More googling and I learn that Segger J-Link is a special protocol and that a small unit is required to interface your computer with the board! Everything fell into the place. The different drivers and the talk of the SWD header.

Unfortunately, my Adafruit boards don’t have a SWD header. Fortunately, the board has space and solder pads for one to be added.

The SWD header goes here

Adafruit sell the headers, but nobody in the UK had them available. I searched around and found something on RS that looked right.

I wasn’t prepared for the tininess of the header

I found a J-LINK device on The PiHut which I ordered, along with a short cable. I’ll wait for that to arrive before I attempt to solder this header onto the board. I’m going to need some steady hands!!

3rd August

J-Link Device arrived and connected up. Managed to solder on the header without any issue.

Added the RS 10 pin SWD header to my Adafruit nRF52

I think hooked up my newly purchased J-LINK device. Side node here – I ordered a serial cable with this as PiHut said it was recommended. One came in the box, so the extra one I ordered was redundant! I’ll be writing to them about that.

Hooked up (I should point out that this shows the correct cable orientation – see below)

Running the nordic command shows it has found the J-LINK connector!

Sadly, nothing is easy…

Big cryptic error trying to flash the device

My attempt to flash the hello_world sample failed. Ah man – with some help from this https://embeddedcomputing.weebly.com/segger-j-link-with-adafruit-feather-nrf52.html I realised I had the cable connected the wrong way around. I could find no information on the orientation of the header or cable, but I reversed it based on that pic and boom…..

SUCCESS!

Zephyr hello_world sample successfully written to device

It appeared that I had successfully flashed the device with the sample code.

Terminal output shows hello_world is working

Bloomin’ heck. It did actually work!

With the flash process *finally* working, I create a new “light” project, using what I’d learned from the BBC Microbit. I was then able to reuse the BBC Microbit switch.

I’d consider that a success! I think I’ve enough of a basic grasp of BLE Mesh networking to park this part of the project and move on!

TS Mk II – Mesh Network Pt.1

As part of my Temperature Sensor upgrade, I’ve started looking into Mesh Networking support.

I was aware that the Expressif ESP32 supports the new Bluetooth LE Mesh protocol, and as Bluetooth Low Energy would help stretch battery life, it seemed like the natural place to start my investigations.

I was initially going to try the WiFi mesh, but it didn’t really seem to suit devices like temperature sensor as they are actually asleep most of the time. If nodes in the mesh keep turning on and off, the other devices in the mesh would be constantly reorganizing themselves to plug the gaps. I do want to learn about ESP’s WiFi mesh, but for now, I’ll focus on Bluetooth

Documenting my progress

Rather than try and put a post together at the end, I’m going to try and document my progress as I go, with additions to this post each time I do something significant.

Getting started

I’ve already got the ESP-IDF environment setup and have two different dev boards. I’m going to try and approach this in stages

  • Copy one of the sample projects and get it running
  • Add another node
  • Send data from one node to another
  • Add MQTT and WiFi to get data out of the mesh

9th July 2019

I pulled down the ESP-IDF code from their mesh branch and spun up on the samples. After faffing around with the menuconfig settings, I got the Node example compiled and deployed onto the device. It also appeared appeared in the Nordic iOS Mesh App.

I found a good post on the provisioning process, https://www.novelbits.io/bluetooth-mesh-tutorial-part-3/ and this helped me make sense of what I was seeing.

Once I tapped on the node, I was able to tap “Identify”, which pulled back more information. I think this is what novelbits called “Invitation”. I then tapped “Provision” and after a few seconds, and a lot of logging from the device, I had the first Node in my mesh!

11th July

Now that I had my first mesh node, I quickly realised why the Expressif code was only at version 0.6 – none of the provisioning data was being saved. This meant that each time I flashed the device, I had to provision it again. This became annoying after the fifth time, so I started digging into the actual provisioning process, to see if I could put in something temporary.

I was digging around on google and I came across the Zephyr RTOS. In their examples, they had a simple 2 node mesh demo. The readme page states that no provisioning is required, and whilst it’s insecure, it’s sufficient for the demo. In their code, I could see a few lines of code where they manually provision the device with their own network and device keys.

static const u8_t net_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};
static const u8_t dev_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};
static const u8_t app_key[16] = {
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
	0x01, 0x23, 0x45, 0x67, 0x89, 0xab, 0xcd, 0xef,
};

err = bt_mesh_provision(net_key, net_idx, flags, iv_index, addr, dev_key);

This seemed ideal for my tinkering. I also found reference to the Zephyr code in Expressif’s news release about their BLE Mesh, where they state they are basing their implementation on top of the Zephyr . This meant I could probably find simple code in the esp-idf repo.

Running this code on the device resulted in it saying that provisioning was complete.

19th July

After more digging around the ESP-IDF examples, I came across some testing code, which appeared to provision using the esp code.

bt_mesh_device_auto_enter_network()

Try as I might, I couldn’t get that to work. Just kept reporting an error of -22.

UPDATE 13th August: During some experimenting with how to bridge the mesh network with MQTT, I managed to get this code working eventually, meaning:

static const u16_t net_idx;
static const u16_t app_idx;
static const u32_t iv_index;

struct bt_mesh_device_network_info info = {
        .net_key = { 
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    },
        .net_idx = net_idx,
        .flags = flags,
        .iv_index = iv_index,
        .unicast_addr = 0x0002,
        .dev_key = {
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
                    0x01,
                    0x23,
                    0x45,
                    0x67,
                    0x89,
                    0xab,
                    0xcd,
                    0xef,
        },
        .app_key = {
		0x01,
		0x23,
		0x45,
		0x67,
		0x89,
		0xab,
		0xcd,
		0xef,
		0x01,
		0x23,
		0x45,
		0x67,
		0x89,
		0xab,
		0xcd,
		0xef,
},
        .app_idx = app_idx,
        .group_addr = 0xc000
    };

    err = bt_mesh_device_auto_enter_network(&info);

20th July

After more attempts to provision the device directly, I’ve decided to kinda give up. As I mentioned, the Zephyr code is easier for me to understand at this stage, so I’m going to try and get a build of the Zephyr samples running on my ESP32 boards.

23rd July

After hours of playing around trying to get the Zephyr SDK compiling against the ESP32 toolchain, I decided to stop. I found an excellent resource at over on the Bluetooth SIG website, called “An Introduction to Bluetooth Mesh Networking“. The samples in their course use the Zephyr SDK and the BBC Microbit. I had a glance though it and most of the main concepts are covered, so I think I’ll order some BBC Microbits and run through it.

I also came across an interest post at https://hutscape.com/, which is worth checking out. The author, Sayanee, uses an Adafruit board to build a UV sensor. Pretty cool. At £26 (and no Friend Node support in the Nordic SDK), I decided against buying any for the time being.

24th July

I got a BBC Microbit and tried the Zephyr mesh_demo code. After more mucking around, I managed to flash it with their code.

Building the mesh_demo firmware
The output from the microbit shows it working

As suggested by the Bluetooth study guide, I turned on the logging and then

Enable the logging and I use up too much RAM

The damn firmware, with logging, was too big for the BBC Microbit’s 16KB of ram. Bloody hell. Nothing is every easy. Some googling revealed that the small amount of SRAM makes it almost impossible to run the mesh. The Apollo Computer ran on 4KB of RAM 🙂

Hardware development is tough! That said, these microbits are perfect for me to dabble in the fundamentals of Bluetooth Mesh and to get to grips with the models. Hopefully there is enough space for me to try my own code. Perhaps it’s possible to compress the Zephyr RTOS runtime down a little??

25th July

Been reading about how to figure out the size of the runtime. You just run the command ninja rom_report

The mesh_demo code takes up just over 3% of the total size
The bluetooth code takes up 66%

I don’t know what those numbers actually mean. I had assumed bytes, but that would mean the bluetooth library, if measured in bytes, was around 50KB and I know that’s not right.

28th July

I experimented with the Zephyr mesh_demo sample, but couldn’t make it work. Kept getting error codes when trying to send messages. I tried to tweak it here and there, but I was just strumbling around in the dark. I returned, instead, to the Bluetooth developer guide and gave that go. The guide starts at the Switch module.

The sample code wasn’t up-to-date with the latest Zephyr code, so I had to tweak it somewhat, but I got it to compile and, amazingly, it worked.

One Microbit is the light, the other the switch
Turning the light on and off over BLE Mesh!

You can continue following my adventures here

Temperature Sensor – Mk II

My first go at creating a simple battery powered temperature sensor was really interesting and fun to do, but, like any project, there are always way to make it better!

For my Mark II sensor, there were a few things I wanted to improve:

Mesh Networking

I’m interesting in exploring the concept of mesh networking. Within the Expressif family of devices, there are two flavors that I’m aware of; WiFi mesh and Bluetooth Low Energy Mesh. The ESP32 platform supports both of these. My current Temperature Sensor is built upon the ESP8266, but moving to the ESP32 should be straight forward.

In Progress – .

More Data

Mk I of my sensor was a simple temperature probe. I wanted to expand on this after I found the Bosch BME280 sensor. This little sensor can capture temperature, pressure and humidity. The boards I ordered work on the I2C protocol, so I’ll have to figure that out.

Not started.

MQTT Discovery

I use the Hass.io platform to manage my automation at home. It supports a simple MQTT interface for receiving data and sending commands. Typically, you would configure the devices within a config file, but it does support a discovery protocol, which allows devices to make themselves known to Hass and to provide all the information required for Hass to use them.

I would like my temperature sensors to support this mode, so that when they fire themselves up for the first time, they would register themselves automatically.

Not started.

Battery Life

Not started.

Housing

I’d like to enclose my sensor in a nice case, so that I can mount them on the wall in way that doesn’t look completely ugly!

Not started.

Low battery notifications

Not started.

Set extractor fan to automatic – Part 1

A few months back, after we discovered mould growing on our bathroom ceiling, I replaced the wall mounted extractor fan with an in-line model, mounted in the loft. I never grasped how important the position of the van was in relation to the source of steam and the source of fresh air. Without going into too much detail, the new vent, directly over the shower, is doing its job.

My extractor fan, like most people’s, is wired to the lighting of the bathroom, so turning on the light also turns on the fan. I have two problems with this:

  • You need to have the light on, even if it’s the middle of the day.
  • The fan runs in the middle of the night after a trip to loo.

The first point is my preference. I like natural daylight, so it does annoy me having to turn on the lights. Opening the window just isn’t as effective as the fan. And whilst the in-line unit I purchased is very quiet, in the dead of night, you can hear it roaring away .

It would be nice to only have the fan on when needed e.g. having a shower. Some fans use humidity sensors, but inline fans don’t have that option. Besides, they require the humidity level to rise enough to trigger them, which kinda defeats the point. My wife suggested we add an extra switch to control it, but it’s not practical as I’m not able to get a wire down into the socket. I’ve started looked at some sort of smart alternative.

The plan

I’ve been thinking about a solution and I’ve got an idea that I think will work. There are three parts to it.

#1 Add a temperature sensor to the hot water pipe that feeds the shower. I’ve built one for my son’s room, battery powered, so we can see how hot/cold his room gets during the night. The temperature sensor could tell when the shower is running as the pipe would get up to 50 or 60 degrees Celsius.

#2 Disconnect the fan from the bathroom light and control it using an ESP8266 and a relay. This will allow it to be turned off and on via an MQTT instruction. I will also reduce the fans overrun to zero and control that via software instead.

#3 Create an automation to engage the extractor fan when the temperature sensor reaches a certain temp and to turn it off when the temperature drops back down.

The Temperature Sensor

For this project, I’m planning on using a variation on my own home made temperature sensor.

In order to have a fast response response to changes in temperature, the unit is going to have to be powered constantly. Fortunately, there is an easy access between the shower pipes and the cupboard under the stairs. I’ll be able to run a cable up there with ease.

The Fan Control

The fan that I’m using in my bathroom is a bog standard inline extractor. It takes three inputs. Mains power, live and neutral and a switch live, which is tied to the bathroom lights.

My plan was to use the existing mains power to power my control circuit and to put the switched live under the control of a simple relay, rather than the light switch. I will also use Hass to manage the fan’s overrun, rather than using the inbuilt circuit. This gives me more control and flexibility.

I have a few Hi Link transformers, which take mains voltage and spit out 5v. The Wemos D1 boards I have will take a 5v input and the small relays I have are also powered using 5v.

Converts mains voltage to a steady 5V DC, perfect for Wemos

The operation of the control would be pretty basic. Using MQTT, it would listen for a run command, the body of which would hold a count in minutes. When received, it would engage a relay to the switched live input of the fan. Once the specified number of minutes elapsed, it would disengage the relay and the fan would stop. If a run command was received whilst the fan was already running, the time would just be reset. If a stop command was received, the relay would automatically disengage.

The controller would also publish its current state when it was changed, something that isn’t necessary, but useful for keeping Hassio up-to-date with changes it didn’t cause!

I assembled a small prototype using some veroboard and a relay I had. It is a low trigger relay, which means it’s engaged when the the indicator input has a low voltage. This is a little annoying (it’s using power to engage the contact). I ordered a high trigger relay, but pressed on with the prototype just the same.

The initial prototype

Sadly, this protoype didn’t work! Everything powered up, but the program on the ESP8266 didn’t appear to boot up. Experience taught me this was because of a lack of power, but I expected no more than 300mA to be drawn by both the relay and D1 board.

I attached my bench supply to get a better view of the power consumption.

W1 and the relay (engaged) draw 347mA, well below the 0.6A rating

So it’s wasn’t a power issue. I poked around a little more and after reviewing the MQTT logs, I was happy that the code on the D1 was actually running correctly. This was puzzling. All information pointed to the relay being a problem.

I reconnected to my PC via USB and ran the code again. Everything worked perfectly.

As I’m using a low trigger relay, this means that the relay engages when the output PIN is set to ground. Using my multi meter, I measure the voltage across the PIN and got a value of 0.095V

USB power gives 0.095v when the output pin is set to LOW

Reconnecting my rectifier, I tested the voltage again and got a different value!


Bench power gives 0.11v when the output pin is set to LOW

The penny then dropped. What if the cut off was 0.1v? I quickly popped a resister in series and bingo, the relay engaged!

As I already knew, the low trigger relay didn’t align with my requirements, so aside from a fun diversion, this wasn’t a problem I needed to solve.

HASS control

With the fan control hardware prototype up and running, I turned my attention to Hass control. As I was using the MQTT protocol, this was very straight forward.

After some experimenting, I had to bin my approach of passing the running time as part of the body because of how MQTT works in HASS. Whilst HASS will issue MQTT commands for a particular device, it also listens for status changes from the device. The body of these two messages must match up. For example, if I issued START5, my board would respond with START5 and HASS would turn the fan yellow to indicate its running. If I send the MQTT command outside of HASS (MQTT directly, for example) with a body of START10, HASS won’t identify the fan has started and its state would be wrong. I don’t want to force myself into relying only on HASS to start the fan. Commands like START and STOP are sufficient enough for my needs.

It’s not the end of the world. The fan already has a fixed overun time, so I won’t be any worse off. In the future, I could look at another MQTT command that could dynamically change the overrun time.

Tapping this runs the fan

More to do

The idea was sound, but I have much more work to do.

  • Replace the replay with a high-trigger one
  • Connect the control unit to the extractor fan
  • Connect the temperature sensor to the hot water pipe

I will cover these in future posts on the subect!