Sunday, November 30, 2014

Best laid plans...

I know I was being overly optimistic, but sometimes everything just works out the way you had hoped.  Sadly, this was not one of those times.

I got to the house ready to get to work - my NUC with all the bluetooth doohickey configured, a simple shell script to pipe the serial output to a rotated text file while I figure out how to store the data for the long term, the HC-06 serial bluetooth dongle, and the electronics project box that contains all the little miscellaneous wires and parts.

Getting the NUC up and running was the easy part, luring me into a false sense of achievement even though there's nothing special about it.

Wee!  Computer is up and running
Then pretty much everything started going wrong.  There were two major issues that I was worried about.  First was the range of the bluetooth wireless network.  The HC-06 bluetooth module is a Bluetooth Class 2 device, with typical range of about 10 meters.  I was hoping to put the server near where the satellite internet modem, wireless router, and the mobile microcell was located, and putting them all on an uninterruptible power supply to protect them from invariable power surges and brown-outs common to off-grid homes.  This would place the bluetooth devices just a little over 11 meters apart with a wall in-between.

Second possible issue was the power consumption of the HC-06 bluetooth module.  The datasheet shows average pairing current of about 25mA, with spikes up to 40mA.  However, once paired, the current consumption should stay at around 8mA.  The Trimetric 2025 down-converts the 48V available at the battery down to 5V using linear regulators, so the current available at the 5V rail is fairly limited.  Bogart engineering folks said there should be at least 5mA available at the 5V rail, and that the display uses another 15~20mA or so.  I was hoping that by turning the display off and by putting a little capacitor to take the load, I'd be able to get the HC-06 module to at least pair after which I'd be able to turn the display back on and use the Battery Monitor while recording the data.

For both the issues, unfortunately, I was proven overly optimistic.

Range testing the Bluetooth Module
On the range side, the effective range I was able to get out of the module and the NUC's bluetooth antenna was about 8 meters.  But no big deal, NUC is connected using Wifi, so I can always put it somewhere closer...

Unhappy Trimetric
But on the power side, I wasn't even close!  Even with the display turned off, plugging in the HC-06 module into the Trimetric collapsed the 5V rail, causing the display to glitch wildly and resetting the microcontroller.  Even when I tried to get the HC-06 paired first, I was not able to maintain the Bluetooth connection with the Trimetric's display turned on.


The wall where the battery monitor is mounted is behind the counters, and there's a heavy one-piece granite countertop making it very difficult to access the area.  There's no other connection in the box other than the wires coming from the battery box, no easy way to get the data out or power in.  And I really didn't want to look for yet another 48V to 5V converter which may also skew the voltage readings in the battery monitor.

I went down to the subfloor to see if there was any hope from underneath.

Sealed tight
Everything seems to be sealed pretty tight.  And I didn't realize it until my friend Phil pointed out, but in the top center, above the black pipes, there's a giant spider just hanging out.  Nope nope nope.  I was starting to lose hope when Phil chimed in again - there in fact is pre-existing wiring going into the battery monitor!  The wiring from the battery box to the battery monitor some sort of shielded 6 conductor cable, of which 4 conductors are used.  That means there's 2 wires available for me!

I see the light!
The battery monitor only transmits at 2400bps, but at 5V TTL signaling instead of RS-232.  If it was RS-232, we should be good to 60m, but given that I don't know how long the cabling is, I need to figure out if it makes more sense to try to get power in rather than to get the serial signal out.  I also need to figure out where the end-point of this cabling is...  but I'd imagine it should be somewhere within an easy reach of a wall-wart!

This means that I probably need to go back to ESP8266 for sending the data back to the computer.  Wifi should have more than enough range to, and now that power's no concern, Wifi's back on the table!  I'm leaning towards having the ESP8266 simply sending out UDP packets with the serial data to a hardcoded address, but we'll see what's easiest to implement.  Hmm, I wonder if there's enough memory to fit a Lua VM in the ESP8266!

Monday, November 17, 2014

Rewiring the Battery Box

As mentioned before, the lead acid batteries that power the house has seen better days.  The batteries being used are 24 6V batteries, wired in 3 parallel strings of 8 batteries with system voltage of 48V.  I was not able to read any direct model numbers off the batteries, but they probably have rated capacity of 300~350Ahr.  The actual capacity at time point is probably significantly lower, though.

Lead Acid Batteries

Below is the actual wiring diagram of how the system was originally wired up.  The three parallel strings are made up of batteries 5~12, 17~24, and 1~4,13~16.
Battery Pack Wiring
Due to the physical sizing of the battery box and the fact that these are 6V batteries, something had to give, and in this case, it was how the three parallel strings are wired up.  Best practices for battery sizing seems to guide people to minimize parallel strings to reduce possibility of imbalance.  In fact, the ideal battery configuration seems to be 2V x 24 (or however you may need for target system voltage) or just one series string of batteries.  All I've read seems to heavily discourage 2+ parallel strings, and if you end up with a 3+ parallel batteries, not to tie them to each other, but to run similar length wires individually to each parallel string and tie them together at a common point.

As you can see above in the wiring diagram, this was not how the system was wired.  The 1~4,13~16 battery string is wired using a wire that's more than 6 feet long meaning it sees significantly higher resistance than the other two strings.  I tried to top up the batteries using a generator and made voltage measurements at rest and under load, which came up as follows:

At Rest:
6.18V 6.18V 6.07V 6.16V 6.37V 6.36V 6.40V 6.33V 6.38V 6.33V 6.35V 4.61V
6.20V 6.16V 6.16V 5.97V 5.83V 6.18V 6.25V 6.19V 6.21V 6.20V 6.07V 6.11V

Under Load (~C/20)
6.04V 6.13V 5.88V 6.01V 6.25V 6.25V 6.27V 6.20V 6.23V 6.24V 6.25V 2.40V
6.14V 6.13V 6.08V 3.50V 3.80V 6.07V 6.11V 6.06V 6.06V 6.09V 5.93V 5.99V

Red colored readings mean that the battery's basically dead, and the orange means it's compromised.  If I overlay the battery information on the wiring chart, I get the following:

Compromised Batteries

You can see that the dead or compromised batteries are the ones closest to parallel wiring, where it was likely seeing significant amounts of overcharging and overdrawing during the charge / discharge cycles.  The long term solution would be to replace the battery pack with properly designed one, but that has to happen after we had more solar generation capacity - currently, the solar panels are not quite large enough to fully charge up the battery pack even on a sunny summer day.

For the time being, I simply rewired battery pack to get us through the winter - I pushed out the batteries 3, 12, 16, 17, 23, 24 to the end and removed the 1~4,13~16 string.  The battery pack now has two parallel strings, with the parallel wires being doubled up for extra current carrying capacity.  The two parallel strings now go positions 3~10 and 15~22.  At +100lb, this was surprisingly good workout. :-)

The system voltage is now in the correct range instead of swinging between 45V~47V.  Unfortunately I forgot to bring the Bluetooth serial module that I was going to use to read out and store the battery monitor information.  Hopefully I'll remember next time I head down!

Monday, November 10, 2014

Time Series Database

I've been thinking about what would be the best way to store the various sensor data that'll be generated in a way that's scalable, easy to visualize, and easy to analyze.  In reality, the amount of data that'll be generated from a handful of sensors scattered around the house will be pretty small and whatever I'll end up building or using will be a complete overkill...  but it's always fun thinking about big infrastructure to solve minor problems. :-)

There are some very unique attributes to sensor data.  First, each piece of data has time associated with it, which needs to be captured and stored.  In a sensor data stream, each reading is also completely independent of each other.  This means that in terms of consistency, even if you lose a data point or ten, it's still OK.  Another interesting attribute is that once the data point has been generated, it's completely immutable.  There's no need to ever go back to a previous data point to update some value.  And for analysis, you'll never be looking at an individual data points, but rather a series of data points over time.

A tool that I instinctively reach for is a relational database - MySQL, SQLite, whatever - it's flexible, easy to use, and most importantly, I know how to use it.  But in this case, it just seemed ill-suited to me.  The sensor data has enough unique and weird attributes that even just trying to map the data to a database schema in an efficient way ran into roadblocks.  A typical analysis might be to find minimum or maximum value in a sensor stream - so do I need to create an index of the data as well as ones for the timestamp and sensor ID?

The other way to move forward was just storing it in a flat file of some sort, which will work just fine.  However, one of the things I had wanted to do is to create a way to replicate and sync across different stores of data - so that the the system can be decentralized and resilient in the face of network disruptions (like the Satellite internet link going down in a storm).  The fact that the data set itself is independent and immutable makes this easy, but with a flat file, scenarios like a data point in the past appearing after newer data has been written and flushed to the file becomes annoying to deal with (unless, of course, I just drop that old data point).

If pretty graphs are what's needed, something like RRDtool might be exactly what's needed here - I've spent enough time working online to have seen plenty of MRTG graphs, and with RRDtool, you have fixed data set size regardless of how long you capture the data for.  But this means some of the data is being thrown away, and in this age of people having a terabyte harddrive full of cat pictures, it seems... wasteful to throw any valuable data away!

Googling around to see what others have done, I stumbled across the Time Series Database wikipedia page!  Now that I think about it, things like stock ticks and interest rates over time behaves in a very similar way to sensor data.  There's a name to the problem that I'm trying to solve, though these people are trying to solve it in a much bigger scale. :-)

Wikipedia-surfing from Time Series Database page, I ended up with two open source packages that tries to solve the IoT data storage problem - nimbits and OpenTSDB.  Both seems to be a Java service that expose a REST API front-end for devices to report to, somewhat like what the Sparkfun folks are doing at with Phant.

After half a day of clicking around and reading documentations and papers, it feels like I'm just as far as where I had started off from...  But at least I know the name of what it is that I'm looking for now! :-)

Monday, November 3, 2014

HC-06 Bluetooth Serial Module

I've been messing around with the ESP8266 module for the last little while - starting with the SDK demos, it's fairly easy to hack the code to connect automatically to a given router and starting sending data over the network to a given target.  Even though only UART and GPIO is routed out in the PCB, the part has support for i2c and SPI and would make a great starting point for a networked device.

Unfortunately, for the specific task I had in mind - getting data out from battery monitor with power budget measured in tens of mA - it doesn't quite fit the bill.  The power consumption is about a factor of 10 too high, and bandwidth requirements (2400 baud uni-directional) and range requirements are also a little off.  So, time to look seriously down the Bluetooth path.  After being amazed with the Chinese ingenuity in ESP8266, I figured I'd look there before grabbing a $35 bluetooth serial module and I came across the world of HC-0# series of serial port profile bluetooth modules available from eBay for low price of about $5.  I ended up ordering the HC-06 which is a consumer level bluetooth serial port slave device, which acts as a transparent serial to bluetooth bridge once paired.

HC-06 Wired Up
There seems to be several different manufacturers of the HC-06 part, none of which provides usable documentation. :p  Fortunately, there are sites that document their experiences with the parts that I've found useful.  In terms of wiring it up, it seems like the module can accept 5V power, though the serial rx/tx lines are limited to 3.3V logic.  My HC-06 module came pre-configured as 9600 8n1 and I created a simple voltage divider to connect 5V serial output to the 3.3V input on HC-06.

HC-0# family of devices also use AT command set, but the HC-06 I found has an interesting quirk - the page above touch on it for a bit, but instead of using \r\n as end-of-command, the HC-06 requires you to enter the entire command within a certain timeframe for it to be processed correctly.  This in essence means that in the serial console, you need to copy and paste the commands (AT+VERSION, for example) sans \r\n to allow commands to be accepted.  Since there's no echo, you basically then sit around waiting for the response to appear (OK, etc).  It's a little awkward, but the AT command based setup just has to be done once, I suppose.  All I had to do is to run AT+BAUD2 to set serial port to 2400 baud to talk to the battery monitor.

I've gotten a NUC to serve as a smart home hub - this was the first time manually pairing a bluetooth device to Linux computer all from command line, and it was surprisingly annoying - there seems to be multiple entry points into bluez - having run through them, I'm surprised that bluetooth works as well as it does on Android!

First, I had to see if I could see the bluetooth device and get the address:
daniel@smarthub:~$ hcitool scan
Scanning ...
20:14:04:29:34:92 HC-06

I then had to edit the rfcomm configuration file to add the entry:
daniel@smarthub:~$ cat /etc/bluetooth/rfcomm.conf
# RFCOMM configuration file.
rfcomm0 {
        # Automatically bind the device at startup
        bind yes;
        # Bluetooth address of the device
        device 20:14:04:29:34:92;
        # RFCOMM channel for the connection
        channel 1;
        # Description of the connection
        comment "HC-06 Serial Dongle";
HC-06 ships with default pairing password of "1234", and I had to run the following command to get the pairing prompt:
bluez-simple-agent hci0 20:14:04:29:34:92
Then start rfcomm:
daniel@smarthub:~$ sudo rfcomm bind rfcomm0
daniel@smarthub:~$ ls -al /dev/rfcomm0
crw-rw---- 1 root dialout 216, 0 Nov  2 23:40 /dev/rfcomm0
Add my user to dialout group so that /dev/rfcomm0 can be accessed without sudo:
daniel@smarthub:~$ cat /etc/group | grep dialout
Then finally connect!
daniel@smarthub:~$ screen /dev/rfcomm0 2400

I still haven't quite figured out how to collect and store all the data, but it seems like at least the pieces are (finally) coming together!  And as for the Broadcom WICED SENSE module - I might simply end up using it as a fancy and somewhat limited weather station - it's got temperature and pressure covered, but maybe I can glue it on a wind wheel and use the accelerometer and gyro to figure out the wind speed. :-)