Saturday, January 24, 2015

ESP8266 based Serial to UDP Bridge

I finally got around to building a very simple Serial to UDP bridge based on the ESP8266 board.  I tracked down the reboot issues to the fact that the LWIP stack needs to be fully initialized before wifi connection can be made.  There is a system_init_done_cb() method that allows a callback function to be registered, and putting the wifi connection and network initialization code in there fixed the reboot problem.  This is a problem that could've been solved with halfway decent documentation...  but I also appreciate the fact that I can put something together based on a board that cost $3 including shipping!

The code simply connects to the Wifi station specified in user_config.h file and sends anything received on the UART to the server specified in fixed UDP packets.  It's a completely one way data flow that suits the Battery Monitor use case just fine!  The UDP path isn't as well documented as the TCP one, but it wasn't too difficult to get it up and running once I was able to get out of the reboot loops.  What I think is more valuable, though, is just getting through the initialization and connection steps to get to a reasonable starting point for future projects.

On the ESP8266 Developer environment setup as logged before, it should just require some simple changes to the Makefile to point to the new locations for the compiler toolchain and the SDK.  I can simply run "make" afterwards to get it all built and use the "" script to flash it onto the device (remember to ground GPIO0 pin for programming mode!)

Next weekend, I'm hoping to carve out a little time to try to get it all wired into the house - hopefully I won't run into any other issues and will finally be able to start collecting some data!  I'm also toying with the idea of trying to get Lua running on the ESP8266 boards... but we'll see if I can find a block of time to work on this...

Saturday, December 27, 2014

Energy Monitors

Whenever I have some time to spare, I've been tinkering with the ESP8266 board, slowly getting it to act as a simple serial to Wifi gateway.  As the simplest path forward, I've setup the serial input to be read into a buffer, which then gets flushed as an UDP packet to the server when filled up.  The code seems simple enough, but I'm running spontaneous reboots, which I suspect is caused by the watchdog timer.  What I wouldn't give for a proper debugger!  I'm hoping I'll be able to get it up and running soon, though.

Meanwhile, I've been keeping a close eye on power consumption over time as best as I could by manually checking the battery monitor at different points in time.  One of the most important part about the battery monitor reading is that it literally monitors just the batteries, not any other parts of the electrical system.  You can infer a lot of things from getting the voltage and current information, but they are no direct measurements.  The diagram below shows where the battery monitor measurement shunt is located.
Power System High Level Layout
If the battery monitor reading shows positive current (in other words, the batteries are being charged), it could mean that it's nice and sunny outside and the solar panels are generating power, or the diesel generator has kicked in and is running, and the total power being generated is more than the power being consumed by the house, the water pump, and all the electrical loads.  Reverse is also true - the Solar Panel could be generating power, but if microwave and hair dryer is running at the same time, you'd see a negative value on the battery monitor.

One of the things that I noticed right away was that there was a spike in power consumption whenever water was being used - using both cold and hot water would result in a brief spike in power consumption of up to ~500W for the duration of the water usage, which was a surprise to me.  As it turns out, the water pressure in the house - both hot and cold water as well as the sprinkler system - is maintained at 50 psi by an electric water pump which kicks in automatically when the water pressure drops below that point.  It would be nice if there's a way to store the excess power during the day in the form of pressurized water or water with higher potential energy, but the terrain at the house does not have enough elevation gain to have decent water pressure and apparently the existing system is about as energy efficient as water systems get these days.

The current set of solar panels, under full Sun close to Winter Solstice, gets about 45A, or little over 2kW.  The diesel generator seems to be rated for around 3kW, peaking at 60A.  The house is fairly efficient, but in the middle of the night with everything turned off, there's base consumption of ~7A or 350W which seems kind of high.  I walked around the house with a Kill-A-Watt meter to see if anything was amiss, and noticed that a camper that was plugged in was using 40W, but 200VA with power factor of 0.20.  I wasn't able to figure out the actual effect of unplugging the trailer (it was during the middle of sunny, gorgeous day with solar panels going full tilt!) but I'm hoping getting a timer and having the camper powered on only during the day will make a sizable dent in the power consumption.

Going back to the diagram, even if I were to get the Battery Monitor information, there's a lot of information missing from the battery monitor readings - it'd be nice to get a finer breakdown of where the energy is going, and how much is actually being generated and being used up.  In trying to answer that question, I got introduced a whole world of Home Energy Monitors, from fancy industrial looking units to cute user friendlier ones with names like TED.  They all work in similar manner - a set of current transformer on relevant circuits feeding data into some sort of control unit which then sends the data over to a display unit.  Many of them are designed to be wired into the circuit panel, and uses proprietary wireline and wireless protocols to send the data to a receiving unit.  Being able to collect data to a computer is unfortunately considered a "professional" use case that requires more expensive units to be purchased.  In the house, having energy monitor wired to the circuit panel would mean that I'd get good visibility within the house itself, but I wouldn't be able to easily figure out the amount of energy used by the water pump for example.

Then I found a product called mFi/mPort from a company called Ubiquiti.  I've actually already been looking at their wireless last-mile product line in trying to get non-Satellite internet up at the house, and it was a pleasant surprise to see that they have nicely designed, cheap(-er) line of IoT sensor products.  As far as I can tell, you purchase a mPort, which can support 2 different sensors (or your own sensors over RS232/422/485!).  Ubiquiti sells a variety of sensors, though the ones that I'm interested in are the current sensing modules.  mPort itself is wireless / ethernet based and as far as I can tell, it should be fairly easy to get the data off of the device itself.

For about $100, I should be able to get two current readings (AC only, as the sensors are current transformer based, not hall effect), so that should allow for Water Pump and Rest of the House energy consumption.  This ends up killing two birds with one stone, as it gives me a proxy way to measure the water consumption in the house as well.

I'm slowly getting all the pieces in place to get the house to be a little bit smarter, but everything's taking much longer than originally expected...

Sunday, November 30, 2014

Best laid plans...

I know I was being overly optimistic, but sometimes everything just works out the way you had hoped.  Sadly, this was not one of those times.

I got to the house ready to get to work - my NUC with all the bluetooth doohickey configured, a simple shell script to pipe the serial output to a rotated text file while I figure out how to store the data for the long term, the HC-06 serial bluetooth dongle, and the electronics project box that contains all the little miscellaneous wires and parts.

Getting the NUC up and running was the easy part, luring me into a false sense of achievement even though there's nothing special about it.

Wee!  Computer is up and running
Then pretty much everything started going wrong.  There were two major issues that I was worried about.  First was the range of the bluetooth wireless network.  The HC-06 bluetooth module is a Bluetooth Class 2 device, with typical range of about 10 meters.  I was hoping to put the server near where the satellite internet modem, wireless router, and the mobile microcell was located, and putting them all on an uninterruptible power supply to protect them from invariable power surges and brown-outs common to off-grid homes.  This would place the bluetooth devices just a little over 11 meters apart with a wall in-between.

Second possible issue was the power consumption of the HC-06 bluetooth module.  The datasheet shows average pairing current of about 25mA, with spikes up to 40mA.  However, once paired, the current consumption should stay at around 8mA.  The Trimetric 2025 down-converts the 48V available at the battery down to 5V using linear regulators, so the current available at the 5V rail is fairly limited.  Bogart engineering folks said there should be at least 5mA available at the 5V rail, and that the display uses another 15~20mA or so.  I was hoping that by turning the display off and by putting a little capacitor to take the load, I'd be able to get the HC-06 module to at least pair after which I'd be able to turn the display back on and use the Battery Monitor while recording the data.

For both the issues, unfortunately, I was proven overly optimistic.

Range testing the Bluetooth Module
On the range side, the effective range I was able to get out of the module and the NUC's bluetooth antenna was about 8 meters.  But no big deal, NUC is connected using Wifi, so I can always put it somewhere closer...

Unhappy Trimetric
But on the power side, I wasn't even close!  Even with the display turned off, plugging in the HC-06 module into the Trimetric collapsed the 5V rail, causing the display to glitch wildly and resetting the microcontroller.  Even when I tried to get the HC-06 paired first, I was not able to maintain the Bluetooth connection with the Trimetric's display turned on.


The wall where the battery monitor is mounted is behind the counters, and there's a heavy one-piece granite countertop making it very difficult to access the area.  There's no other connection in the box other than the wires coming from the battery box, no easy way to get the data out or power in.  And I really didn't want to look for yet another 48V to 5V converter which may also skew the voltage readings in the battery monitor.

I went down to the subfloor to see if there was any hope from underneath.

Sealed tight
Everything seems to be sealed pretty tight.  And I didn't realize it until my friend Phil pointed out, but in the top center, above the black pipes, there's a giant spider just hanging out.  Nope nope nope.  I was starting to lose hope when Phil chimed in again - there in fact is pre-existing wiring going into the battery monitor!  The wiring from the battery box to the battery monitor some sort of shielded 6 conductor cable, of which 4 conductors are used.  That means there's 2 wires available for me!

I see the light!
The battery monitor only transmits at 2400bps, but at 5V TTL signaling instead of RS-232.  If it was RS-232, we should be good to 60m, but given that I don't know how long the cabling is, I need to figure out if it makes more sense to try to get power in rather than to get the serial signal out.  I also need to figure out where the end-point of this cabling is...  but I'd imagine it should be somewhere within an easy reach of a wall-wart!

This means that I probably need to go back to ESP8266 for sending the data back to the computer.  Wifi should have more than enough range to, and now that power's no concern, Wifi's back on the table!  I'm leaning towards having the ESP8266 simply sending out UDP packets with the serial data to a hardcoded address, but we'll see what's easiest to implement.  Hmm, I wonder if there's enough memory to fit a Lua VM in the ESP8266!

Monday, November 17, 2014

Rewiring the Battery Box

As mentioned before, the lead acid batteries that power the house has seen better days.  The batteries being used are 24 6V batteries, wired in 3 parallel strings of 8 batteries with system voltage of 48V.  I was not able to read any direct model numbers off the batteries, but they probably have rated capacity of 300~350Ahr.  The actual capacity at time point is probably significantly lower, though.

Lead Acid Batteries

Below is the actual wiring diagram of how the system was originally wired up.  The three parallel strings are made up of batteries 5~12, 17~24, and 1~4,13~16.
Battery Pack Wiring
Due to the physical sizing of the battery box and the fact that these are 6V batteries, something had to give, and in this case, it was how the three parallel strings are wired up.  Best practices for battery sizing seems to guide people to minimize parallel strings to reduce possibility of imbalance.  In fact, the ideal battery configuration seems to be 2V x 24 (or however you may need for target system voltage) or just one series string of batteries.  All I've read seems to heavily discourage 2+ parallel strings, and if you end up with a 3+ parallel batteries, not to tie them to each other, but to run similar length wires individually to each parallel string and tie them together at a common point.

As you can see above in the wiring diagram, this was not how the system was wired.  The 1~4,13~16 battery string is wired using a wire that's more than 6 feet long meaning it sees significantly higher resistance than the other two strings.  I tried to top up the batteries using a generator and made voltage measurements at rest and under load, which came up as follows:

At Rest:
6.18V 6.18V 6.07V 6.16V 6.37V 6.36V 6.40V 6.33V 6.38V 6.33V 6.35V 4.61V
6.20V 6.16V 6.16V 5.97V 5.83V 6.18V 6.25V 6.19V 6.21V 6.20V 6.07V 6.11V

Under Load (~C/20)
6.04V 6.13V 5.88V 6.01V 6.25V 6.25V 6.27V 6.20V 6.23V 6.24V 6.25V 2.40V
6.14V 6.13V 6.08V 3.50V 3.80V 6.07V 6.11V 6.06V 6.06V 6.09V 5.93V 5.99V

Red colored readings mean that the battery's basically dead, and the orange means it's compromised.  If I overlay the battery information on the wiring chart, I get the following:

Compromised Batteries

You can see that the dead or compromised batteries are the ones closest to parallel wiring, where it was likely seeing significant amounts of overcharging and overdrawing during the charge / discharge cycles.  The long term solution would be to replace the battery pack with properly designed one, but that has to happen after we had more solar generation capacity - currently, the solar panels are not quite large enough to fully charge up the battery pack even on a sunny summer day.

For the time being, I simply rewired battery pack to get us through the winter - I pushed out the batteries 3, 12, 16, 17, 23, 24 to the end and removed the 1~4,13~16 string.  The battery pack now has two parallel strings, with the parallel wires being doubled up for extra current carrying capacity.  The two parallel strings now go positions 3~10 and 15~22.  At +100lb, this was surprisingly good workout. :-)

The system voltage is now in the correct range instead of swinging between 45V~47V.  Unfortunately I forgot to bring the Bluetooth serial module that I was going to use to read out and store the battery monitor information.  Hopefully I'll remember next time I head down!

Monday, November 10, 2014

Time Series Database

I've been thinking about what would be the best way to store the various sensor data that'll be generated in a way that's scalable, easy to visualize, and easy to analyze.  In reality, the amount of data that'll be generated from a handful of sensors scattered around the house will be pretty small and whatever I'll end up building or using will be a complete overkill...  but it's always fun thinking about big infrastructure to solve minor problems. :-)

There are some very unique attributes to sensor data.  First, each piece of data has time associated with it, which needs to be captured and stored.  In a sensor data stream, each reading is also completely independent of each other.  This means that in terms of consistency, even if you lose a data point or ten, it's still OK.  Another interesting attribute is that once the data point has been generated, it's completely immutable.  There's no need to ever go back to a previous data point to update some value.  And for analysis, you'll never be looking at an individual data points, but rather a series of data points over time.

A tool that I instinctively reach for is a relational database - MySQL, SQLite, whatever - it's flexible, easy to use, and most importantly, I know how to use it.  But in this case, it just seemed ill-suited to me.  The sensor data has enough unique and weird attributes that even just trying to map the data to a database schema in an efficient way ran into roadblocks.  A typical analysis might be to find minimum or maximum value in a sensor stream - so do I need to create an index of the data as well as ones for the timestamp and sensor ID?

The other way to move forward was just storing it in a flat file of some sort, which will work just fine.  However, one of the things I had wanted to do is to create a way to replicate and sync across different stores of data - so that the the system can be decentralized and resilient in the face of network disruptions (like the Satellite internet link going down in a storm).  The fact that the data set itself is independent and immutable makes this easy, but with a flat file, scenarios like a data point in the past appearing after newer data has been written and flushed to the file becomes annoying to deal with (unless, of course, I just drop that old data point).

If pretty graphs are what's needed, something like RRDtool might be exactly what's needed here - I've spent enough time working online to have seen plenty of MRTG graphs, and with RRDtool, you have fixed data set size regardless of how long you capture the data for.  But this means some of the data is being thrown away, and in this age of people having a terabyte harddrive full of cat pictures, it seems... wasteful to throw any valuable data away!

Googling around to see what others have done, I stumbled across the Time Series Database wikipedia page!  Now that I think about it, things like stock ticks and interest rates over time behaves in a very similar way to sensor data.  There's a name to the problem that I'm trying to solve, though these people are trying to solve it in a much bigger scale. :-)

Wikipedia-surfing from Time Series Database page, I ended up with two open source packages that tries to solve the IoT data storage problem - nimbits and OpenTSDB.  Both seems to be a Java service that expose a REST API front-end for devices to report to, somewhat like what the Sparkfun folks are doing at with Phant.

After half a day of clicking around and reading documentations and papers, it feels like I'm just as far as where I had started off from...  But at least I know the name of what it is that I'm looking for now! :-)

Monday, November 3, 2014

HC-06 Bluetooth Serial Module

I've been messing around with the ESP8266 module for the last little while - starting with the SDK demos, it's fairly easy to hack the code to connect automatically to a given router and starting sending data over the network to a given target.  Even though only UART and GPIO is routed out in the PCB, the part has support for i2c and SPI and would make a great starting point for a networked device.

Unfortunately, for the specific task I had in mind - getting data out from battery monitor with power budget measured in tens of mA - it doesn't quite fit the bill.  The power consumption is about a factor of 10 too high, and bandwidth requirements (2400 baud uni-directional) and range requirements are also a little off.  So, time to look seriously down the Bluetooth path.  After being amazed with the Chinese ingenuity in ESP8266, I figured I'd look there before grabbing a $35 bluetooth serial module and I came across the world of HC-0# series of serial port profile bluetooth modules available from eBay for low price of about $5.  I ended up ordering the HC-06 which is a consumer level bluetooth serial port slave device, which acts as a transparent serial to bluetooth bridge once paired.

HC-06 Wired Up
There seems to be several different manufacturers of the HC-06 part, none of which provides usable documentation. :p  Fortunately, there are sites that document their experiences with the parts that I've found useful.  In terms of wiring it up, it seems like the module can accept 5V power, though the serial rx/tx lines are limited to 3.3V logic.  My HC-06 module came pre-configured as 9600 8n1 and I created a simple voltage divider to connect 5V serial output to the 3.3V input on HC-06.

HC-0# family of devices also use AT command set, but the HC-06 I found has an interesting quirk - the page above touch on it for a bit, but instead of using \r\n as end-of-command, the HC-06 requires you to enter the entire command within a certain timeframe for it to be processed correctly.  This in essence means that in the serial console, you need to copy and paste the commands (AT+VERSION, for example) sans \r\n to allow commands to be accepted.  Since there's no echo, you basically then sit around waiting for the response to appear (OK, etc).  It's a little awkward, but the AT command based setup just has to be done once, I suppose.  All I had to do is to run AT+BAUD2 to set serial port to 2400 baud to talk to the battery monitor.

I've gotten a NUC to serve as a smart home hub - this was the first time manually pairing a bluetooth device to Linux computer all from command line, and it was surprisingly annoying - there seems to be multiple entry points into bluez - having run through them, I'm surprised that bluetooth works as well as it does on Android!

First, I had to see if I could see the bluetooth device and get the address:
daniel@smarthub:~$ hcitool scan
Scanning ...
20:14:04:29:34:92 HC-06

I then had to edit the rfcomm configuration file to add the entry:
daniel@smarthub:~$ cat /etc/bluetooth/rfcomm.conf
# RFCOMM configuration file.
rfcomm0 {
        # Automatically bind the device at startup
        bind yes;
        # Bluetooth address of the device
        device 20:14:04:29:34:92;
        # RFCOMM channel for the connection
        channel 1;
        # Description of the connection
        comment "HC-06 Serial Dongle";
HC-06 ships with default pairing password of "1234", and I had to run the following command to get the pairing prompt:
bluez-simple-agent hci0 20:14:04:29:34:92
Then start rfcomm:
daniel@smarthub:~$ sudo rfcomm bind rfcomm0
daniel@smarthub:~$ ls -al /dev/rfcomm0
crw-rw---- 1 root dialout 216, 0 Nov  2 23:40 /dev/rfcomm0
Add my user to dialout group so that /dev/rfcomm0 can be accessed without sudo:
daniel@smarthub:~$ cat /etc/group | grep dialout
Then finally connect!
daniel@smarthub:~$ screen /dev/rfcomm0 2400

I still haven't quite figured out how to collect and store all the data, but it seems like at least the pieces are (finally) coming together!  And as for the Broadcom WICED SENSE module - I might simply end up using it as a fancy and somewhat limited weather station - it's got temperature and pressure covered, but maybe I can glue it on a wind wheel and use the accelerometer and gyro to figure out the wind speed. :-)

Monday, October 13, 2014

Getting started with development on ESP8266

Enterprising tinkerers on the esp8266 message boards have added the Xtensa Call0 ABI support to gcc, allowing a fairly clean way to compile the code and run it on the device.

I'm going try to try to put the steps together here, mainly for my own benefit since I'll probably forgot otherwise. :-)

To get the toolchain up and running, follow the instructions given here:
$ git clone -b lx106 git://
$ cd crosstool-NG
$ ./bootstrap && ./configure --prefix=`pwd` && make && make install
$ ./ct-ng xtensa-lx106-elf
$ ./ct-ng build
This will compile the toolchain targeted at xtensa lx106 into the builds/ directory and will take a while.  You may be required to install some missing dependencies.  Meanwhile, we need to download the SDK and few other odds and ends.

From this post, download the file which contains the "esptool" package which will parse the elf generated by gcc compiler above, extract the relevant sections, and repack it into a binary format ready to be flashed.  While you're there, also download the Makefiles which we'll modify to work with the toolchain above.

From this post, download the esp8266 sdk package.  Extract the SDK package, extract the inside the SDK (as the Makefile looks for SDK_DIR/esptool/esptool executable) and build the esptool executable
$ unzip
$ cp esp8266_sdk_v0.9.1
$ cd esp8266_sdk_v0.9.1
$ unzip
$ cd esptool
$ vim Makefile  # edit target from WINDOWS to LINUX
$ make
To build the "AT" firmware, I had to copy the new updated makefile from Makefile_AT_0.9.1 into the SDK_DIR/examples/at directory and make couple of changes to the Makefile by setting the SDK_BASE, setting EXTRA_INCDIR to add include/ and /usr/include/ fields, and pointing to the toolchain built above for CC/AR/LD calls.
$ cp esp8266_sdk_v0.9.1/examples/at
$ cd esp8266_sdk_v0.9.1/examples/at
$ unzip
$ vim Makefile # Update SDK_BASE, EXTRA_INCDIR, CC, AR, LD
$ make
It would've been lovely if it all had just worked...  but some more tweaks and missing dependencies are required.  First, we need to put the libc.a and libhal.a from extra_libs.tgz from this post and put it in the SDK_DIR/lib directory.  I also had to comment out int32_t in SDK_DIR/include/c_types.h.  After that, though, simple "$ make" and we get the two firmware files, 0x00000.bin and 0x40000.bin!

To download the firmware to the ESP8266, you need to boot the board with GPIO0 tied to ground to have it come up in programming mode.  Once in programming mode, I was able to use the download script to program the new code
python --port COM3 write_flash 0x00000 0x00000.bin
python --port COM3 write_flash 0x40000 0x40000.bin
I got some error messages after erasing and programming had completed, but everything seems to work OK even with the error message.

ESP8266 Communicating
The module uses AT command set, and I was able to test the basic functionality by doing the following:
COM3> AT+CIFSR # Get IP address
# Server mode enabled, test with nc IP_ADDRESS 5000

# Client mode, test with nc -l 6000
COM3> AT+CIPSEND=<length>
What I'm interested in doing doing, though, is mode where the device wakes up by itself, automatically connects to a known Wifi station, and either automatically starts sending data or starts up a server so that data can be pulled out.  This seems closer to the SDK's "IoT_demo" code base.  I'm able to build and flash the resulting image, but I'm not quite sure what it is that the demo does yet.  I suppose if I can't figure it out, I can just write something from scratch instead. :-)