Debugging (Hard To Replicate Issues) With Embedded Software Devices
I write software for various hardware devices. Finding bugs, and resolving them is usually relatively easy if the issue can be replicated in a test environment with development hardware and full logging equipment setup. The problem however is when:
- It is equipment using GSM, making use of various different phone networks, and cell towers that can not be easily replicated.
- The issue only occurs on location in the field (i.e. live customers using the device).
- It is an intermittent issue that can not be replicated on cue, ruling out simply going to the location for an hour setting up the relevant equipment and logging the issue.
- Devices are in locations where it not particularly convenient / safe to setup and leave expensive testing equipment / laptops etc.
- Testing equipment needs to run a Windows only application from a hardware manufacturer that needs “babysitting” (restarting if it stops logging).
I recently faced this issue. I tried a few options including blindly guessing at what the issue could be and implementing fixes for various possible issues, next tried working with the network operator (which got nowhere). Final solution I came up with was to build some custom / cheap (in comparison to the alternative) logging hardware to take the place of the hardware development board, Windows running laptop, hardware manufacturers software tool.
My first thoughts were that this could be done by using an Arduino with an SD Card data logging “shield” however after some further research I discovered the Spark Fun OpenLog. The OpenLog was a perfect fit since it:
- Can be powered from 3.3V – 12V (could just piggy back it on the existing devices power supply).
- Serial levels compatible with 2.8V CMOS levels of device.
- Already setup to automatically log serial data to files on an SD Card.
- Open source firmware that is freely available, meaning I could customise it to emulate the manufacturers software tool (eliminating the need for the extra hardware, and meaning I could deal with the reliability issues).
- Development environment (see section “Compile Version 2″) that is simple / relatively straight forward to obtain / setup / use (Ardino IDE – even though it is not “Arduino” branded hardware as such).
- Supports upto 16GB microSD cards (device outputs large logs).
There were a few steps involved in the process including, connecting the OpenLog to the device, building a programming cable for the OpenLog, and customising the OpenLog software to emulate the initialisation string of the Windows software tool.
I have never coded AVR C before however found it relatively straight forward to make the modifications I required to emulate the initialisation string of the Windows tool (primarily logging the string with Eltima Software’s Serial Port Monitor application, saving the data to a file on the SD Card and instructing OpenLog to read it back over the serial port, then drop back into logging mode.
A few issues I encountered along the way were:
- OpenLog stops reading files if it encounters a NULL byte, however this should be fixed in the latest release.
- Error trying to upload new software from the Arduino IDE to OpenLog due to the DTR pin needing to be connected to OpenLog to reset it just before programming. I substituted DTR for RTS (which my FTDI cable did have broken out).
- Device I was logging data to was outputting [CTRL]+[Z] 3 times – the OpenLog escape sequence – I therefore modified the escape sequence to be 9 times rather than just 3.
|TTL-232R-3V3 USB to serial FTDI cable||PPDEV-09717|
|Kingston USB Card Reader (19 in 1)||139230|
|1GB microSD Card (or alternate 8GB)||PPCOM-08163|
|Kingston 8gb Class 4 MicroSDHC Card||147731|
|4 pin header||PPPRT-08231|
|2 pin header||PPPRT-08233|
|4 pin housing||PPPRT-08097|
- Solder a x4 way header connector onto the logger GND, VCC, TXO, RXI (i.e. miss out the first pin labelled BLK, it is not required).
- Solder a x1 way header connector onto the logger for the GRN pin (I used a 2 way with the other pin overlapping off the end of the board as I could not easily obtain a 1 pin)
- Solder a x4 way housing connector onto the wires from the device (match up GND/VCC/TXO/RXI to appropriate pins on the device).
Note: A 5 or 6 way header soldered to all logger pins could just be used but I thought this approach best to avoid any mistakes (broken loggers) – that way there is a x4 way header to use when plugging into the printer and you are not left wondering which 4 pins should be connected.
FTDI Programming Cable
Modify the FTDI cable such that it has the following pinout (compatible with logger). Use a flat screw driver to release the pins from the existing socket.
|CTS||Brown – N/C|