Saturday, December 27, 2014

Energy Monitors

Whenever I have some time to spare, I've been tinkering with the ESP8266 board, slowly getting it to act as a simple serial to Wifi gateway.  As the simplest path forward, I've setup the serial input to be read into a buffer, which then gets flushed as an UDP packet to the server when filled up.  The code seems simple enough, but I'm running spontaneous reboots, which I suspect is caused by the watchdog timer.  What I wouldn't give for a proper debugger!  I'm hoping I'll be able to get it up and running soon, though.

Meanwhile, I've been keeping a close eye on power consumption over time as best as I could by manually checking the battery monitor at different points in time.  One of the most important part about the battery monitor reading is that it literally monitors just the batteries, not any other parts of the electrical system.  You can infer a lot of things from getting the voltage and current information, but they are no direct measurements.  The diagram below shows where the battery monitor measurement shunt is located.
Power System High Level Layout
If the battery monitor reading shows positive current (in other words, the batteries are being charged), it could mean that it's nice and sunny outside and the solar panels are generating power, or the diesel generator has kicked in and is running, and the total power being generated is more than the power being consumed by the house, the water pump, and all the electrical loads.  Reverse is also true - the Solar Panel could be generating power, but if microwave and hair dryer is running at the same time, you'd see a negative value on the battery monitor.

One of the things that I noticed right away was that there was a spike in power consumption whenever water was being used - using both cold and hot water would result in a brief spike in power consumption of up to ~500W for the duration of the water usage, which was a surprise to me.  As it turns out, the water pressure in the house - both hot and cold water as well as the sprinkler system - is maintained at 50 psi by an electric water pump which kicks in automatically when the water pressure drops below that point.  It would be nice if there's a way to store the excess power during the day in the form of pressurized water or water with higher potential energy, but the terrain at the house does not have enough elevation gain to have decent water pressure and apparently the existing system is about as energy efficient as water systems get these days.

The current set of solar panels, under full Sun close to Winter Solstice, gets about 45A, or little over 2kW.  The diesel generator seems to be rated for around 3kW, peaking at 60A.  The house is fairly efficient, but in the middle of the night with everything turned off, there's base consumption of ~7A or 350W which seems kind of high.  I walked around the house with a Kill-A-Watt meter to see if anything was amiss, and noticed that a camper that was plugged in was using 40W, but 200VA with power factor of 0.20.  I wasn't able to figure out the actual effect of unplugging the trailer (it was during the middle of sunny, gorgeous day with solar panels going full tilt!) but I'm hoping getting a timer and having the camper powered on only during the day will make a sizable dent in the power consumption.

Going back to the diagram, even if I were to get the Battery Monitor information, there's a lot of information missing from the battery monitor readings - it'd be nice to get a finer breakdown of where the energy is going, and how much is actually being generated and being used up.  In trying to answer that question, I got introduced a whole world of Home Energy Monitors, from fancy industrial looking units to cute user friendlier ones with names like TED.  They all work in similar manner - a set of current transformer on relevant circuits feeding data into some sort of control unit which then sends the data over to a display unit.  Many of them are designed to be wired into the circuit panel, and uses proprietary wireline and wireless protocols to send the data to a receiving unit.  Being able to collect data to a computer is unfortunately considered a "professional" use case that requires more expensive units to be purchased.  In the house, having energy monitor wired to the circuit panel would mean that I'd get good visibility within the house itself, but I wouldn't be able to easily figure out the amount of energy used by the water pump for example.

Then I found a product called mFi/mPort from a company called Ubiquiti.  I've actually already been looking at their wireless last-mile product line in trying to get non-Satellite internet up at the house, and it was a pleasant surprise to see that they have nicely designed, cheap(-er) line of IoT sensor products.  As far as I can tell, you purchase a mPort, which can support 2 different sensors (or your own sensors over RS232/422/485!).  Ubiquiti sells a variety of sensors, though the ones that I'm interested in are the current sensing modules.  mPort itself is wireless / ethernet based and as far as I can tell, it should be fairly easy to get the data off of the device itself.

For about $100, I should be able to get two current readings (AC only, as the sensors are current transformer based, not hall effect), so that should allow for Water Pump and Rest of the House energy consumption.  This ends up killing two birds with one stone, as it gives me a proxy way to measure the water consumption in the house as well.

I'm slowly getting all the pieces in place to get the house to be a little bit smarter, but everything's taking much longer than originally expected...

Sunday, November 30, 2014

Best laid plans...

I know I was being overly optimistic, but sometimes everything just works out the way you had hoped.  Sadly, this was not one of those times.

I got to the house ready to get to work - my NUC with all the bluetooth doohickey configured, a simple shell script to pipe the serial output to a rotated text file while I figure out how to store the data for the long term, the HC-06 serial bluetooth dongle, and the electronics project box that contains all the little miscellaneous wires and parts.

Getting the NUC up and running was the easy part, luring me into a false sense of achievement even though there's nothing special about it.

Wee!  Computer is up and running
Then pretty much everything started going wrong.  There were two major issues that I was worried about.  First was the range of the bluetooth wireless network.  The HC-06 bluetooth module is a Bluetooth Class 2 device, with typical range of about 10 meters.  I was hoping to put the server near where the satellite internet modem, wireless router, and the mobile microcell was located, and putting them all on an uninterruptible power supply to protect them from invariable power surges and brown-outs common to off-grid homes.  This would place the bluetooth devices just a little over 11 meters apart with a wall in-between.

Second possible issue was the power consumption of the HC-06 bluetooth module.  The datasheet shows average pairing current of about 25mA, with spikes up to 40mA.  However, once paired, the current consumption should stay at around 8mA.  The Trimetric 2025 down-converts the 48V available at the battery down to 5V using linear regulators, so the current available at the 5V rail is fairly limited.  Bogart engineering folks said there should be at least 5mA available at the 5V rail, and that the display uses another 15~20mA or so.  I was hoping that by turning the display off and by putting a little capacitor to take the load, I'd be able to get the HC-06 module to at least pair after which I'd be able to turn the display back on and use the Battery Monitor while recording the data.

For both the issues, unfortunately, I was proven overly optimistic.

Range testing the Bluetooth Module
On the range side, the effective range I was able to get out of the module and the NUC's bluetooth antenna was about 8 meters.  But no big deal, NUC is connected using Wifi, so I can always put it somewhere closer...

Unhappy Trimetric
But on the power side, I wasn't even close!  Even with the display turned off, plugging in the HC-06 module into the Trimetric collapsed the 5V rail, causing the display to glitch wildly and resetting the microcontroller.  Even when I tried to get the HC-06 paired first, I was not able to maintain the Bluetooth connection with the Trimetric's display turned on.


The wall where the battery monitor is mounted is behind the counters, and there's a heavy one-piece granite countertop making it very difficult to access the area.  There's no other connection in the box other than the wires coming from the battery box, no easy way to get the data out or power in.  And I really didn't want to look for yet another 48V to 5V converter which may also skew the voltage readings in the battery monitor.

I went down to the subfloor to see if there was any hope from underneath.

Sealed tight
Everything seems to be sealed pretty tight.  And I didn't realize it until my friend Phil pointed out, but in the top center, above the black pipes, there's a giant spider just hanging out.  Nope nope nope.  I was starting to lose hope when Phil chimed in again - there in fact is pre-existing wiring going into the battery monitor!  The wiring from the battery box to the battery monitor some sort of shielded 6 conductor cable, of which 4 conductors are used.  That means there's 2 wires available for me!

I see the light!
The battery monitor only transmits at 2400bps, but at 5V TTL signaling instead of RS-232.  If it was RS-232, we should be good to 60m, but given that I don't know how long the cabling is, I need to figure out if it makes more sense to try to get power in rather than to get the serial signal out.  I also need to figure out where the end-point of this cabling is...  but I'd imagine it should be somewhere within an easy reach of a wall-wart!

This means that I probably need to go back to ESP8266 for sending the data back to the computer.  Wifi should have more than enough range to, and now that power's no concern, Wifi's back on the table!  I'm leaning towards having the ESP8266 simply sending out UDP packets with the serial data to a hardcoded address, but we'll see what's easiest to implement.  Hmm, I wonder if there's enough memory to fit a Lua VM in the ESP8266!

Monday, November 17, 2014

Rewiring the Battery Box

As mentioned before, the lead acid batteries that power the house has seen better days.  The batteries being used are 24 6V batteries, wired in 3 parallel strings of 8 batteries with system voltage of 48V.  I was not able to read any direct model numbers off the batteries, but they probably have rated capacity of 300~350Ahr.  The actual capacity at time point is probably significantly lower, though.

Lead Acid Batteries

Below is the actual wiring diagram of how the system was originally wired up.  The three parallel strings are made up of batteries 5~12, 17~24, and 1~4,13~16.
Battery Pack Wiring
Due to the physical sizing of the battery box and the fact that these are 6V batteries, something had to give, and in this case, it was how the three parallel strings are wired up.  Best practices for battery sizing seems to guide people to minimize parallel strings to reduce possibility of imbalance.  In fact, the ideal battery configuration seems to be 2V x 24 (or however you may need for target system voltage) or just one series string of batteries.  All I've read seems to heavily discourage 2+ parallel strings, and if you end up with a 3+ parallel batteries, not to tie them to each other, but to run similar length wires individually to each parallel string and tie them together at a common point.

As you can see above in the wiring diagram, this was not how the system was wired.  The 1~4,13~16 battery string is wired using a wire that's more than 6 feet long meaning it sees significantly higher resistance than the other two strings.  I tried to top up the batteries using a generator and made voltage measurements at rest and under load, which came up as follows:

At Rest:
6.18V 6.18V 6.07V 6.16V 6.37V 6.36V 6.40V 6.33V 6.38V 6.33V 6.35V 4.61V
6.20V 6.16V 6.16V 5.97V 5.83V 6.18V 6.25V 6.19V 6.21V 6.20V 6.07V 6.11V

Under Load (~C/20)
6.04V 6.13V 5.88V 6.01V 6.25V 6.25V 6.27V 6.20V 6.23V 6.24V 6.25V 2.40V
6.14V 6.13V 6.08V 3.50V 3.80V 6.07V 6.11V 6.06V 6.06V 6.09V 5.93V 5.99V

Red colored readings mean that the battery's basically dead, and the orange means it's compromised.  If I overlay the battery information on the wiring chart, I get the following:

Compromised Batteries

You can see that the dead or compromised batteries are the ones closest to parallel wiring, where it was likely seeing significant amounts of overcharging and overdrawing during the charge / discharge cycles.  The long term solution would be to replace the battery pack with properly designed one, but that has to happen after we had more solar generation capacity - currently, the solar panels are not quite large enough to fully charge up the battery pack even on a sunny summer day.

For the time being, I simply rewired battery pack to get us through the winter - I pushed out the batteries 3, 12, 16, 17, 23, 24 to the end and removed the 1~4,13~16 string.  The battery pack now has two parallel strings, with the parallel wires being doubled up for extra current carrying capacity.  The two parallel strings now go positions 3~10 and 15~22.  At +100lb, this was surprisingly good workout. :-)

The system voltage is now in the correct range instead of swinging between 45V~47V.  Unfortunately I forgot to bring the Bluetooth serial module that I was going to use to read out and store the battery monitor information.  Hopefully I'll remember next time I head down!

Monday, November 10, 2014

Time Series Database

I've been thinking about what would be the best way to store the various sensor data that'll be generated in a way that's scalable, easy to visualize, and easy to analyze.  In reality, the amount of data that'll be generated from a handful of sensors scattered around the house will be pretty small and whatever I'll end up building or using will be a complete overkill...  but it's always fun thinking about big infrastructure to solve minor problems. :-)

There are some very unique attributes to sensor data.  First, each piece of data has time associated with it, which needs to be captured and stored.  In a sensor data stream, each reading is also completely independent of each other.  This means that in terms of consistency, even if you lose a data point or ten, it's still OK.  Another interesting attribute is that once the data point has been generated, it's completely immutable.  There's no need to ever go back to a previous data point to update some value.  And for analysis, you'll never be looking at an individual data points, but rather a series of data points over time.

A tool that I instinctively reach for is a relational database - MySQL, SQLite, whatever - it's flexible, easy to use, and most importantly, I know how to use it.  But in this case, it just seemed ill-suited to me.  The sensor data has enough unique and weird attributes that even just trying to map the data to a database schema in an efficient way ran into roadblocks.  A typical analysis might be to find minimum or maximum value in a sensor stream - so do I need to create an index of the data as well as ones for the timestamp and sensor ID?

The other way to move forward was just storing it in a flat file of some sort, which will work just fine.  However, one of the things I had wanted to do is to create a way to replicate and sync across different stores of data - so that the the system can be decentralized and resilient in the face of network disruptions (like the Satellite internet link going down in a storm).  The fact that the data set itself is independent and immutable makes this easy, but with a flat file, scenarios like a data point in the past appearing after newer data has been written and flushed to the file becomes annoying to deal with (unless, of course, I just drop that old data point).

If pretty graphs are what's needed, something like RRDtool might be exactly what's needed here - I've spent enough time working online to have seen plenty of MRTG graphs, and with RRDtool, you have fixed data set size regardless of how long you capture the data for.  But this means some of the data is being thrown away, and in this age of people having a terabyte harddrive full of cat pictures, it seems... wasteful to throw any valuable data away!

Googling around to see what others have done, I stumbled across the Time Series Database wikipedia page!  Now that I think about it, things like stock ticks and interest rates over time behaves in a very similar way to sensor data.  There's a name to the problem that I'm trying to solve, though these people are trying to solve it in a much bigger scale. :-)

Wikipedia-surfing from Time Series Database page, I ended up with two open source packages that tries to solve the IoT data storage problem - nimbits and OpenTSDB.  Both seems to be a Java service that expose a REST API front-end for devices to report to, somewhat like what the Sparkfun folks are doing at with Phant.

After half a day of clicking around and reading documentations and papers, it feels like I'm just as far as where I had started off from...  But at least I know the name of what it is that I'm looking for now! :-)

Monday, November 3, 2014

HC-06 Bluetooth Serial Module

I've been messing around with the ESP8266 module for the last little while - starting with the SDK demos, it's fairly easy to hack the code to connect automatically to a given router and starting sending data over the network to a given target.  Even though only UART and GPIO is routed out in the PCB, the part has support for i2c and SPI and would make a great starting point for a networked device.

Unfortunately, for the specific task I had in mind - getting data out from battery monitor with power budget measured in tens of mA - it doesn't quite fit the bill.  The power consumption is about a factor of 10 too high, and bandwidth requirements (2400 baud uni-directional) and range requirements are also a little off.  So, time to look seriously down the Bluetooth path.  After being amazed with the Chinese ingenuity in ESP8266, I figured I'd look there before grabbing a $35 bluetooth serial module and I came across the world of HC-0# series of serial port profile bluetooth modules available from eBay for low price of about $5.  I ended up ordering the HC-06 which is a consumer level bluetooth serial port slave device, which acts as a transparent serial to bluetooth bridge once paired.

HC-06 Wired Up
There seems to be several different manufacturers of the HC-06 part, none of which provides usable documentation. :p  Fortunately, there are sites that document their experiences with the parts that I've found useful.  In terms of wiring it up, it seems like the module can accept 5V power, though the serial rx/tx lines are limited to 3.3V logic.  My HC-06 module came pre-configured as 9600 8n1 and I created a simple voltage divider to connect 5V serial output to the 3.3V input on HC-06.

HC-0# family of devices also use AT command set, but the HC-06 I found has an interesting quirk - the page above touch on it for a bit, but instead of using \r\n as end-of-command, the HC-06 requires you to enter the entire command within a certain timeframe for it to be processed correctly.  This in essence means that in the serial console, you need to copy and paste the commands (AT+VERSION, for example) sans \r\n to allow commands to be accepted.  Since there's no echo, you basically then sit around waiting for the response to appear (OK, etc).  It's a little awkward, but the AT command based setup just has to be done once, I suppose.  All I had to do is to run AT+BAUD2 to set serial port to 2400 baud to talk to the battery monitor.

I've gotten a NUC to serve as a smart home hub - this was the first time manually pairing a bluetooth device to Linux computer all from command line, and it was surprisingly annoying - there seems to be multiple entry points into bluez - having run through them, I'm surprised that bluetooth works as well as it does on Android!

First, I had to see if I could see the bluetooth device and get the address:
daniel@smarthub:~$ hcitool scan
Scanning ...
20:14:04:29:34:92 HC-06

I then had to edit the rfcomm configuration file to add the entry:
daniel@smarthub:~$ cat /etc/bluetooth/rfcomm.conf
# RFCOMM configuration file.
rfcomm0 {
        # Automatically bind the device at startup
        bind yes;
        # Bluetooth address of the device
        device 20:14:04:29:34:92;
        # RFCOMM channel for the connection
        channel 1;
        # Description of the connection
        comment "HC-06 Serial Dongle";
HC-06 ships with default pairing password of "1234", and I had to run the following command to get the pairing prompt:
bluez-simple-agent hci0 20:14:04:29:34:92
Then start rfcomm:
daniel@smarthub:~$ sudo rfcomm bind rfcomm0
daniel@smarthub:~$ ls -al /dev/rfcomm0
crw-rw---- 1 root dialout 216, 0 Nov  2 23:40 /dev/rfcomm0
Add my user to dialout group so that /dev/rfcomm0 can be accessed without sudo:
daniel@smarthub:~$ cat /etc/group | grep dialout
Then finally connect!
daniel@smarthub:~$ screen /dev/rfcomm0 2400

I still haven't quite figured out how to collect and store all the data, but it seems like at least the pieces are (finally) coming together!  And as for the Broadcom WICED SENSE module - I might simply end up using it as a fancy and somewhat limited weather station - it's got temperature and pressure covered, but maybe I can glue it on a wind wheel and use the accelerometer and gyro to figure out the wind speed. :-)

Monday, October 13, 2014

Getting started with development on ESP8266

Enterprising tinkerers on the esp8266 message boards have added the Xtensa Call0 ABI support to gcc, allowing a fairly clean way to compile the code and run it on the device.

I'm going try to try to put the steps together here, mainly for my own benefit since I'll probably forgot otherwise. :-)

To get the toolchain up and running, follow the instructions given here:
$ git clone -b lx106 git://
$ cd crosstool-NG
$ ./bootstrap && ./configure --prefix=`pwd` && make && make install
$ ./ct-ng xtensa-lx106-elf
$ ./ct-ng build
This will compile the toolchain targeted at xtensa lx106 into the builds/ directory and will take a while.  You may be required to install some missing dependencies.  Meanwhile, we need to download the SDK and few other odds and ends.

From this post, download the file which contains the "esptool" package which will parse the elf generated by gcc compiler above, extract the relevant sections, and repack it into a binary format ready to be flashed.  While you're there, also download the Makefiles which we'll modify to work with the toolchain above.

From this post, download the esp8266 sdk package.  Extract the SDK package, extract the inside the SDK (as the Makefile looks for SDK_DIR/esptool/esptool executable) and build the esptool executable
$ unzip
$ cp esp8266_sdk_v0.9.1
$ cd esp8266_sdk_v0.9.1
$ unzip
$ cd esptool
$ vim Makefile  # edit target from WINDOWS to LINUX
$ make
To build the "AT" firmware, I had to copy the new updated makefile from Makefile_AT_0.9.1 into the SDK_DIR/examples/at directory and make couple of changes to the Makefile by setting the SDK_BASE, setting EXTRA_INCDIR to add include/ and /usr/include/ fields, and pointing to the toolchain built above for CC/AR/LD calls.
$ cp esp8266_sdk_v0.9.1/examples/at
$ cd esp8266_sdk_v0.9.1/examples/at
$ unzip
$ vim Makefile # Update SDK_BASE, EXTRA_INCDIR, CC, AR, LD
$ make
It would've been lovely if it all had just worked...  but some more tweaks and missing dependencies are required.  First, we need to put the libc.a and libhal.a from extra_libs.tgz from this post and put it in the SDK_DIR/lib directory.  I also had to comment out int32_t in SDK_DIR/include/c_types.h.  After that, though, simple "$ make" and we get the two firmware files, 0x00000.bin and 0x40000.bin!

To download the firmware to the ESP8266, you need to boot the board with GPIO0 tied to ground to have it come up in programming mode.  Once in programming mode, I was able to use the download script to program the new code
python --port COM3 write_flash 0x00000 0x00000.bin
python --port COM3 write_flash 0x40000 0x40000.bin
I got some error messages after erasing and programming had completed, but everything seems to work OK even with the error message.

ESP8266 Communicating
The module uses AT command set, and I was able to test the basic functionality by doing the following:
COM3> AT+CIFSR # Get IP address
# Server mode enabled, test with nc IP_ADDRESS 5000

# Client mode, test with nc -l 6000
COM3> AT+CIPSEND=<length>
What I'm interested in doing doing, though, is mode where the device wakes up by itself, automatically connects to a known Wifi station, and either automatically starts sending data or starts up a server so that data can be pulled out.  This seems closer to the SDK's "IoT_demo" code base.  I'm able to build and flash the resulting image, but I'm not quite sure what it is that the demo does yet.  I suppose if I can't figure it out, I can just write something from scratch instead. :-)

Saturday, October 4, 2014

Wireless Power Consumption and ESP8266

I've been in touch with the folks at Bogart Engineering, who makes the "Trimetric" battery monitor used to monitor the 48V lead acid battery pack at the house.  I've been able to successfully read the streaming data out of the device, and have been trying to figure out an easy way to gather and store that data.

Trimetric Battery Monitor
The battery monitor is powered directly by the 48V and is down-converted to 5V.  The conversion is done with a linear regulator, so the main limit is the heat dissipation.  Unfortunately, this means available current at the 5V rail is low tens of mA, which basically rules out running anything that communicates over Wifi, including the ESP8266 module.  I knew there was about a magnitude difference in power consumption between Bluetooth and Wifi, but it really didn't hit me until I was reading through the different datasheet to figure out what would actually work in this case.

Given the power budget, I'm back to looking at Bluetooth serial dongles, and probably get a low end NUC running Linux which will give me more flexibility in case I wanted to add something like a weather station or run a server in the house.

The ESP8266 board finally arrived, and I've turned it on to make sure that it works - and it does seem to really work! :-)  I've also found a fairly active forum with people trying to do exactly what I'm trying to accomplish.  I've been lurking due to lack of time, but it seems like the main issue is that the IoT development kit's precompiled bits are compiled in Tensilica's proprietary compiler in an ABI that's quite different from the one supported in gcc.  Apparently there's a VirtualBox machine image floating around with all the required bits installed, but a lot of people's focus seems to be on trying to figure out how to compile and run code on fully open development stack.  At least on the firmware download side, it doesn't seem like flash chip twiddling is required - someone reverse engineered the flashing tool and created a documented open version of it!

ESP8266 Board
Some random notes about the ESP8266 board that I have - it's running firmware version 00160910 and communicates at 115200 baud instead of 57600 baud reported elsewhere.  The board layout also requires the chip enable to be held high on top of connecting the tx/rx pins and power and ground.

Even though my intended use for this part is gone due to power consumption (seems to be 200~300mA), I'm hoping I can at least help out in the effort to get open stack toolchain working.  Even with the different ABI, it might be possible to create a glue layer that would allow gcc complied code to talk to the precompiled binaries.  Should be an interesting puzzle to try to figure out.

Saturday, September 27, 2014

Powering a House

I did some preliminary assessment on the not-yet-smart-ish home's electrical system to get a better understanding of exactly what I'm working with.  One of the things that's nice about having worked on variety of projects over the years is that you run into familiar sights from time to time.  I should've known that DC power systems will look similar regardless of whether it's for a solar powered car or power generating kites, but it still was a pleasant surprise to peel back the layers and be presented with some vaguely familiar.

Main Inverter and Charge Controllers for the house
As the house is completely off-grid, it's basically a closed system consisting of batteries for energy storage, bunch of solar panels and a backup diesel generator for power generation, and an inverter to supply 110V AC for the house.  Even some of the names were familiar - Trace Engineering, which built the inverter used here, is owned by Xantrex whose chargers we used extensively back in the solar car days.

Lead Acid Battery Pack
The battery pack has nominal voltage of 48V, consisting of 3 parallel strings of 8 x 6V batteries.  I couldn't read any battery pack model numbers besides the maker (Deka) but the size of it looks to be about 300~350 AHr each.  So, across the 3 parallel strings, the rated capacity is probably just around 1000 AHr, giving it ~48kWhr for the entire battery pack (or enough to run a decently sized hair dryer for 24 hours!)

Measuring the voltages of the individual batteries showed some dead cells - since these are pretty old, beat-up, abused batteries, it's not surprising.  Some of the dead cells are on the top batteries from the two parallel strings closest to the battery connector - I'm guessing the relatively lower resistance due to shorter cabling probably resulted in imblanaced current draw.  I'll eventually have to replace them (with something like these - reading through some articles, the general recommendation seems to be to shoot for a single string of 2V batteries sized for the right capacity) but for the time being, I think just wiring them in a way to remove the dead cells and changing this to a 2 parallel strings while we try to get better power consumption numbers might be a way to go.  My rough guestimate is that we currently have ~24kWhr or usable storage capacity in these batteries.

It looks like there's enough solar panels for approximately 2kW of generation capacity.  Using the Solar Irradiance figures (put in California / San Jose / South South West / 53 degrees) shows a figure of 3.48 in December (least sun) and 6.63 in June (most sun), so we're generating about 7kWhr / day in an average sunny day in December, and 13kWhr / day in an average sunny day in June.  Assuming 24kWhr battery capacity, this means that it would take 3.5 days of full sun in December and 2 days of full sun in June to fully charge the battery pack even with degraded batteries and using only 2 strings in parallel.

To me, this seems like the solar panels are quite undersized for the batteries we have - it's interesting to see the trade-offs and engineering decisions made in a world of completely different economics.  When the system was put in place - probably 10+ years ago - solar panels were quite expensive and were likely to be the most expensive part of the system.  In comparison, batteries were cheaper and diesel at near $1 / gallon was probably a viable backup option in case of multiple days of cloudy weather.  In those days, it probably made sense to size the solar panels for about the average daily energy consumption around equinox.  This puts more cycle in the lead acid batteries, but that probably was an acceptable tradeoff to lower the cost of the solar panels.  And if it rained for a week or you get less sun during the winter, you'd just run the generator a bit to make up the difference

In 2014, you can buy ready to be installed solar panels for well under than $1 / watt while battery price has gone up and diesel price has quadrupled to $4+ / gallon.  These days, I think it makes much more sense to oversize the solar panels so that you put minimum number of cycles into (relatively) expensive batteries, and size the batteries to cover several days worth of energy consumption even with cloudy weather to minimize diesel generator having to kick in.

It's quite amazing just how far the solar module prices has come down, and how that completely changed the sizing of various components in the off-grid system.  And given this, for this house, the priority is probably to add more panels for generating capacity - probably at least triple the current capacity.

Saturday, September 13, 2014

Wifi Wifi Everywhere

Intel announced the Edison Module at the Intel Developer Forum this week.  To me, what's really exciting about the Edison is that this really is a full fledged computer (similar to Beagle Board families) with first class built-in support for wireless infrastructure in a tiny form-factor.  There's a huge difference in programming in an embedded quasi-OS environment like the Spark Core and having the full Linux environment available - it really opens up the world of software ecosystem that you can reuse.  Edison has two Silvermont cores and real storage (4GB eMMC) to go along with the connectivity - it really is amazing to see the technology being open and available at such small form factor and low price.

As far as I can tell, the cheapest and least fussy way to get the Edison is through Newegg - Sparkfun seems to planning a series of boards that could be plugged into enable various sensors and IO interfaces but for learning about the system and experimenting, it'd probably be hard to beat the Arduino interface board with 3.3V/5V tolerant I/Os.

At the other extremes, I came across the ESP8266 Wifi Module.  This seems to be something recently discovered by hacker community at large with minimum AT command documentation in Chinese and an information free website, but it's basically a very cheap, very simple Wifi modules available from AliExpress for well under $4 each including shipping.  The firmware that ships with the module exposes AT command set that can be accessed through the UART to use the module as a simple network device from a microcontroller.

What would be ideal, though, is to be able to run my application directly on the ESP8266 part instead of having another microcontroller interface it - it would allow cheap single board solution that can fit in wide variety of IoT applications.  There is a IoT SDK available, though it seems to provide secret bits in precompiled libraries.  From the makefile, I can tell that it's using Tensilica Xtensa core, which is similar to Nios 2, but on synthesizable for ASICs.

I'm trying to order some of the modules (payment verification is giving me some trouble) and once they're here, I'm hoping to figure out how to run my own code on the devices - this will be the perfect way to interface with the battery monitoring system!  The pictures of the module show a "Winbond" chip which I imagine is the serial flash that holds the code, so my plan is to try to tap into that directly and get the currently loaded firmware and to program updates directly to it.  But if I fail, I can always just stick a MSP430 or something to start gathering data from the battery monitor.

Picture from AliExpress - Serial Flash part
Between $4 ESP8266 and $50 Edison, my excitement with Spark Core has basically vanished.  Spark Core's "cloud software" doesn't interest me much (since you get so much more with full Linux running on Linux).  At $39, I can burn through 10 ESP8266 for the price of one Spark Core board.  What is clear, though, is that it's a pretty interesting time to be in this space!

Sunday, September 7, 2014

Wireless Connectivity for Smart-ish Home

There is a practical reason for my interest in Internet of Things, a personal itch that I'd like to scratch.  In the next year or so, I'll be moving to a completely off-grid house - not a single wire or pipe connects the house to rest of civilization.  Electricity comes from Solar with battery system, heating with firewood and propane, and water from a spring.  The house is in a dead zone as far as cellular service goes, so the only link out is through a Satellite internet connection.  It's little amazing that you could be less than 25 miles from the heart of Silicon Valley and be so isolated, but isolated it is.

There's a lot of little things to check to ensure that everything's running smoothly - when power or water goes out, you have no one to blame but yourself.  Right now, all the checks are done manually, but my plan is to automate as much of the monitoring as possible before the move-in date to make it easier to live there.

There's a lot of different random things to check spread over a decent area, and most of the places do not have power source, so ideal sensor network will be very low power, workable over longer distances (100+ meters) or at least mesh capable.  I still haven't figured out how to measure certain things either - it may require a camera pointing at a gauge in the worst case scenario, so even the required data rates aren't clear to me yet.

There is one system, however, that I think will be crucial to automate as soon as possible, and will be easy to do so.  The electrical system runs off of a bank of lead acid batteries at 48V, and there is a battery monitor installed within the house.  The monitor has a serial output at 5V signalling that can be used to get the various real time data including things like pack voltage, current / power being used, and battery state of charge.  As far as I can tell, the battery monitor is not independently powered, and runs from the 48V supplied directly from the batteries.  Since this battery monitor is located within the house, it should be within the range of the wifi router or a computer with a Bluetooth enabled.  Basically, what I need is the ability to wirelessly transmit the serial port contents to a computer somewhere.

On the bluetooth side, something like this will do the job, though it'll require a host computer to pull the data.  With Bluetooth Low Energy, it might be possible to get to much lower power levels, but I don't see an accessible and affordable development platform yet.  Broadcam has announced a low cost development kit that includes whole bunch of sensors that seems very interesting, but the documentation isn't quite there and I'm not certain if it's possible to crack one open to bring out the serial connections out.  But at $19.99, I figured I'd take a gamble. :-)  Once the part is here, I'll be able to open it up and see how extensible the fob is.  Even if it's not useful for this purpose, it might be interesting to have a battery powered Bluetooth Low Energy device with lots of sensors attached.

Going the wireless route seems to make more sense - having some sort of processing and computing ability at the battery monitor itself will help in dealing with things like buffering data in case of network outages and eventually being able to make decisions there.  Looking around for a wifi enabled development options, it seems like Spark Core would fit the need the best.  There's enough processing capacity and memory to do something interesting, and data can be sent directly to a server out on the internet to be stored and analyzed.  Unfortunately, it seems like the Spark Core's uart pins are not 5V tolerant, so I'll have to do something there.

But before I spend more time, money, and effort down the path, next time I head down to the house, I'll take the battery monitor apart to see exactly how the things are wired and use a computer to talk to the serial port on the battery monitor.  From the documentation, it's not clear if 5V rail is available on the header on the battery monitor, and whether there's enough current to drive a Wifi board on top.  If not, I'll have to find a DC-DC converter to bring the 48V pack voltage down to 5V.  Hrm, and that reminds me, I also need to figure out how the grounding is done in relation to the battery pack...  So many little details to figure out!

Monday, September 1, 2014

Software Defined Radio

I figured I'd spend more time reverse engineering and trying to build a custom SPI boot image, but then I came across the Quark SoC X1000 Board Support Package Build Guide which basically documents exactly what I was trying to do.  Good news is that the firmware update image (document calls it the capsule file) format was pretty much what I had found.  Well, next time, I'll try to RTFM harder. :-)

Lately, I've been tinkering with Software Defined Radio (SDR) based on Realtek RTL2832u part.  The chip seems to have an interesting backstory - originally designed for digital TV, someone found a debug raw output mode and reverse engineered it to make it a very low cost SDR platform.  I bought mine from Amazon for less than 9 dollars including shipping though it did take a while to get here from China.

Galileo board (with standoffs!) and the RTL2832u USB Stick
So far, I've been messing with individual tools from the rtl-sdr package and feeding airplane ADS-B data to FlightRadar24.  Airplanes equipped with the proper transponders will broadcast various data including the altitude, heading, ground speed, and GPS coordinates on 1090MHz.  It's been interesting to see the data captured by the radio as I see the planes fly by to land at SFO.

My short term goal is to get the ADS-B feed fully running on the Galileo - the rtl-sdr package relies on libusb package, so there shouldn't be any Linux kernel modules or changes required.  Ideally, I'd be able to basically take the 386 binary and run it on the Galileo, but we'll see how that works out. :-)  It also seems like the antenna that ships with the module is pretty crappy, so that'll eventually have to be resolved somehow.  Tuning to a local radio stations gives mostly muffled sound, though that might also be because I'm messing something up in the software.

RF and radios are something that I have very little experience with, and it's pretty exciting to jump into a field where everything is new and amazing!

Saturday, August 23, 2014

Windows on Intel Galileo

Apparently there's a way to run Windows on Intel Galileo board with the development and debugging happening through Visual Studio!  According to the instructions, you basically need a (decently sized, minimum 16GB) microSD card which you boot off.

Given how the boot process works, it doesn't surprise me that this would work since all you'd need is an EFI executable bootloader on the SD card.  I've always wondered how decoupled Windows kernel is from all the GUI aspect of it, and I'm guessing this is one of the more stripped down version of Windows with minimum required to run Arduino like workloads.

Looking at the Troubleshooting and Advanced Usage pages, the Galileo on Windows shows up as another Windows PC with open mounts and ability to connect kernel debugger to it.

Unfortunately, Windows is only supported on the original Galileo board, not on Gen 2 boards... though given I have no experience with Windows development at all, it's not that big a loss. :-)

Sunday, August 17, 2014

Opening the Galileo Firmware Update Image

In the Galileo tool installation directory, under hardware/tools/x86/bin, there's a file named sysimage-galileo-1.0.2.cap, which looks awfully like the firmware update image.  The file size is little under 8MB, which is the size of the SPI flash part, and the filename seems to hint at this being the system image of some sort.  There's also script, which refers to sysimage files.

When I first opened the file in vim and used the handy xxd command to see the hexdump (:%!xxd, which also works magically on Windows), it felt a little disorienting.  However, there are certain things that I knew would be in the firmware update image.  Since the Galileo board is booting from the device, there has to be some sort of bootloader in there (grub legacy, as seen before) as well as the bootloader configuration information (grub.conf).  I also know that the Linux kernel image and the initramfs image must also be included.  In the bootlog, as Linux kernel boots, it helpfully prints the following which gives me the kernel and initramfs sizes.
[Linux-EFI SPI, setup=0x107f, size=0x1e0da0]
[Initrd SPI, addr=0xdb5f000, size=0x298a13]
That's more than enough to start some spelunking.  Since it's easiest to find strings, and I know grub.conf is a simple text file, I started by looking for the title of the boot targets like 'Clanton' and 'SVP' - Voila!  At 0x06a3990, you see the contents of grub.conf that corresponds to the boot menu that we see in the bootlog.
Claton SVP kernel!

Linux kernel image also has some unique features - I know that it'll start with "MZ" as its magic number and will have some blurb about needing a bootloader.  By looking for that string, I found what looks very much like the executable kernel image at 0x04be990.
Start of the Linux Kernel

As I was browsing around, I noticed a certain pattern appearing close to start of the two sections above.  The entire update image itself starts with header string with the string "HSC_", and this header appears 9 times throughout the file aligned at 16 byte boundaries.  More importantly, this appears exactly 1024 bytes before the two sections that I've found above.  Using this big hint, I looked at the different sections and tried to figure out what they might be - initramfs was easy, as it's the only region big enough to fit the file size above.  Grub EFI executable was also easy to determine, but the first sets of regions are a mystery for now.
0x0000000 (size 0x0021500) - ??? looks like some sort of table
0x0021500 (size 0x0062090) - ??? looks like an executable
0x0083590 (size 0x0040000) - ??? looks like an executable
0x00c3590 (size 0x0040000) - ??? looks like an executable
0x0103590 (size 0x0121000) - ??? looks like an executable
0x0224590 (size 0x029A000) - initramfs
0x04be590 (size 0x01E5000) - Kernel Image
0x06a3590 (size 0x0001000) - grub.conf
0x06a4590 (size 0x0051000) - grub.efi executable
Now that we have the regions, we can extract the kernel and initramfs filesystem with a simple dd commands
$ dd if=sysimage-galileo-1.0.2.cap of=initramfs.lzma bs=1 skip=2247056 count=2722323
$ dd if=sysimage-galileo-1.0.2.cap of=vmlinux bs=1 skip=4974992 count=1986560
 Extracting the rootfs from initramfs is pretty straight forward from this point
$ mkdir rootfs
$ cp initramfs.lzma rootfs/
$ cd rootfs
$ unxz initramfs.lzma
$ cpio -ivd < initramfs
And now you have the exact filesystem that you get within Galileo after Linux boots up. We can even see the firmware flashing code in opt/cln/galileo/!  And just for fun, let's try booting the Kernel and initramfs on QEMU :-)
$ qemu-system-i386 -kernel vmlinux -initrd initramfs.lzma -append ="root=/dev/ram0 console=uart earlyprintk=serial apic=debug rw"
After that, you need to switch QEMU to serial0 by pressing Ctrl + Alt + 3, but once done, you see the kernel booting on a completely different QEMU system!

It's booting up!
There are some issues - the final console never comes up, probably because init routines point the consoles elsewhere upon boot...  but it does shows that Galileo is basically a stripped down PC.  How amazing is it when you can take a random kernel and stick it on a completely different system and have it boot up?

Sunday, August 10, 2014

Getting down to Linux

The standoffs that I've ordered are somewhere between Shanghai and US, but the USB to UART bridge is here, allowing me to see what's going on when the Galileo board boots up.  All I had to do is install the CP2102 drivers and connect using PuTTY using 115200 and the COM port assigned by the drivers.

Boot Console up and running!

I put the Galileo Gen 2 boot log into a github gist since it's pretty long, but there are some very interesting tidbits in there.  First, it's using Grub Legacy as the bootloader!  Another place where the PC background shows up.  I've usually seen U-Boot being used as bootloader in embedded systems, but for booting Linux on x86 platform, Grub is as good as any, I suppose.  The other thing that I found interesting is that there doesn't seem to be a typical PC "BIOS" as we know it.  In an embedded system where you don't need to discover hardware, it makes sense to drop some of the legacy features that are not likely to be used.  Looking at the Galileo board schematics and the Quark X1000 datasheet, you can see that there's 8MB serial SPI flash (part W25Q64FV) being used as the boot memory.  In Quark Datasheet Table 50, it shows reset vector being mapped into "Legacy Bridge" which has support for one SPI serial flash to store boot firmware (section 21.1).

The references to the SPI flash shows up throughout the boot log.  Grub bootloader is located there, as well as the Linux kernel image.  There are mentions of different elements being signed, though since Galileo supports booting from SD card, I probably wouldn't mess too much with the SPI flash anyway to reduce the risk of possibly bricking the board necessitating finding a JTAG programmer somewhere.

When booting without a SD card, the root filesystem seems to be initramfs loaded into memory, which explains lack of persistence between boots.  When booting from the SD card, the ext3 filesystem is loaded allowing changes to persist between boots, including previous sketches automatically being reloaded.

Since Arduino tool allows firmware to be upgraded, I rummaged around the Arduino installation directory and found "sysimage-galileo-1.0.2.cap" file in hardware\tools\x86\bin directory, which is about the right size to fit in 8MB SPI flash.  Opening up the file also has some right strings showing up, so that might be a quick way to figure out what exactly is in the different SPI segments.

There's plenty more to poke around, but to show what you can do by having direct access to the underlying Linux, here's a simple shell script version of the Arduino blink example on Galileo Gen 2 board.

It seems like there are faster ways to access GPIOs, but for normal access, the Arduino wrapper seems to use the sysfs gpio interface.  We're simply exporting the GPIO pin in question - the arduino pin number to GPIO number was reverse engineered by looking at the g_APinDescription table in x86/variants/galileo_fab_g/variant.cpp file - and setting the value to be 0 or 1, making the LED blink.

We've peeled back one layer of abstraction, but I'm sure there are plenty more layers hiding underneath it all!

Sunday, August 3, 2014

Galileo and Arduino first impressions

The Galileo board is a lot smaller than I had imagined, and there's a lot more parts on it than I had expected.  Schematics helpfully provided shows them to be various I/O expanders, drivers, PWM generators, buffers, and muxes.  These are type of features that are integrated in a typical microcontroller, and it's interesting to see external parts being used to make up for the missing features in the Quark SoC.  On the flip side, the board has things like Mini PCI Express port so the PC heritage does help in other ways.

The other thing that I noticed - and it actually bothers me quite a bit for some reason - is that the Galileo PCB comes sans standoffs.  The board sits awkwardly on the Mini PCI Express connector and it teeters unnervingly whenever I need to hit the reset button.

You can't tell, but the LED's blinking, I swear!
Galileo runs full fledged Linux distribution under the hood, and the schematic makes it seem like the Linux boot console is likely to be UART1, which comes out of a header.  A bit of sleuthing shows that the original Galileo had the UART wired in to a microphone connector, so I'm glad that I can just get something like this instead of having to cut open a headset cable.  Mouser has the actual FTDI part with breakout board that I prefer, but having to track down some headers to solder on and additional female to female jumper cables makes it much more expensive to go that way.  Driver support for CP210x part seems decent from quick searching, but that's one of those things that you never know until you plug it in the USB port.

Galileo running with a full, modern operating system does present some interesting challenges on the Arduino side.  Looking at the Arduino tools and the example 'sketches', it really shows its microcontroller roots.  Arduino presents a very simple interface to the end users, providing them with a way to write C code divided into 'setup' and 'loop' routines.  Setup sets all the one-time options at the start of operation, and once the system is up and running, loop will run basically forever.  In the sample 'blink' sketch, functions like pinMode and digitalWrite in microcontroller world can probably map directly to a macro that does a simple memory write operation.

The blink magic happens here
Linux (or pretty much any other operating system) can't willy nilly give raw hardware access to any program that comes its way - it's a surefire way to have unstable unusable system once you have more than one program running at the same time.  Things like GPIOs are typically exposed as sysfs in embedded Linux, and sure enough, you dig down just a bit and you can see in places like hardware/arduino/x86/cores/arduino/wiring_digital.c how the magic happens.  Simple functions like pinMode and digitalWrite ends up poking into sysfs, writing strings like "high" and "low" into the sysfs tree to toggle the bits on and off.  This code then gets compiled into Linux ELF binary, and gets sent to the Galileo board using 80s technology over the serial link.

Intel provides Galileo Linux Examples that shows how to expose some of the power of Linux underneath the device, partly by running a series of system() calls.  One of the examples is how to redirect the console to the port used by Arduino tool, but it seems to get stuck for me on Galileo Gen2 - we'll see if I can get the issue figured out before I either dig out an old switch to connect to the ethernet port, or Amazon delivers the USB to UART converter.

What is clear is that Arduino platform is barely scratching the surface of what Galileo is capable of.  To really understand Galileo, I'm going to have to dive down much, much deeper.

Wednesday, July 30, 2014


When someone asks me what I do, I answer that I'm a software engineer.  When someone with computer or software background asks me what I do, though, my answer then becomes Embedded Systems.  It's not just that I've spent more of my professional career working in the field of embedded systems - I feel a certain level of affinity, comfort, and joy with embedded systems that I don't get with working on software at higher levels.  There is a certain creative freedom in working in highly constrained environments, not unlike solving a complicated puzzle of figuring out how to get the most out of limited hardware and tools on hand.

I've been pretty excited by the idea of connecting things to the Internet - I'm a big proponent of automating things, and automatic physical real-life things are especially exciting.  I've also been following with interest the efforts that Intel's been spending in the Maker movement, especially with the Galileo board.

I'm familiar with the genre of embedded systems, but not the specifics of Galileo - I have experience with Atmel devices, but I've never used an Arduino board before.  I'm also curious to see how working with x86 instruction set changes the experience of working with embedded systems.

Having spent last little while working on a smartphone and currently spending time thinking about over-the-top messaging platform, I'm excited to go back to basics and spend some personal time exploring the embedded space again.  I'm looking forward to understanding the Galileo board inside-out, and charting that journey and sharing what I learn.  Here we go!

Galileo waiting to be opened