Debugging Stack Traces
When Fruitymesh reboots, it tries to save the reason in the ramRetainStruct. The Stacked Program Counter (PC before e.g. the hardfaultHandler was called) will be saved in
Using this information it is possible to get the source code line where the error happened. Use the following:
<gccDir>\bin\arm-none-eabi-addr2line -e <fruitymeshDir>\_build\debug\NRF51_BOARD\FruityMesh.out 0x35BA5
The address must be given as a hex value, otherwise addr2line doesn’t print a valid result. FruityMesh must have been compiled with debug information using
If the line is not given and you just get sth. like ??:?, you can also use readelf or any online elf parser such as e.g. this one.
<gccDir>\bin\arm-none-eabi-readelf -a <fruitymeshDir>\_build\debug\NRF51_BOARD\FruityMesh.out
This will give you a list of addresses and sizes of where different methods are located in the binary. You can try to find an address that is a bit lower than the reported error address. Then check if your address is within the size of that method and you will know that the issue happened within this method.
Reading Flash Of A Broken FruityMesh Node
In order to analyze a broken node, the flash and UICR can be read back. A good idea is to first read back UICR and flash and then compare the .hex with either the last .hex flashed onto this beacon (from the fruitydeploy beacon directory) or flash the beacon again and read it back.
Read back code and UICR to a .hex file:
nrfjprog --readuicr --readcode E:\dump.hex
Convert both .hex files to a hex dump to view them with a standard text editor:
srec_cat C:\test\dump.hex -intel -o C:\test\dump.txt -hex_dump
Then, use The TortoiseDiff Tool to see the differences:
Page 0 is the master boot record and should not have been altered
The first ~120kb are the softdevice and should not have been altered
Afterwards comes the application that should not have any changes
The free space after the application can be any code or data
The settings are located 2 pages before the bootloader:
The bootloader is located at:
#define FRUITYLOAD_ADDRESS 0x3EC00(NRF51)
#define FRUITYLOAD_ADDRESS 0x78000(NRF52)
In order to check the correctness of the settings, convert the hex dump of the settings pages to a C array using Sublime and load it into the flash of a node in the simulator. This permits to debug the startup behaviour. If only the settings are broken, debugging a real node is also an option.
Debug Running Node
When the nodes are somewhere in the building and one of them is stuck in a hard fault or another endless loop, debugging is complicated. Follow these steps:
If the node is connected with its USB cable, put a battery in the node, then unplug the USB cable.
Take the node to the workstation while it is running on battery
Use two USB cables that are both plugged into the docking station (do not plug one into the laptop and the other in the docking station because they might have a different ground).
One of the cables is plugged into the DevKit/debugger.
The other cable is plugged into the node. Debugger and node are now powered from the same ground level. This is important.
Next, connect the debug cable to the node.
In Eclipse, select the debug configuration "Running Target". Connect to running target must be selected and no reset must be performed.
The node should not have restarted during this procedure.
Finding Problems In A Live System
Often, an error only occurs sporadically and in a system that is only accessible remotely. FruityMesh has some built-in functionality to find errors in these cases. First, live reports can be used (cf. StatusReporterModule). A number of live report statements may be added to the code and they will be reported through the mesh as soon as they get triggered. A drawback with live reports is that they may not be available if the trigger occurs while the mesh connection is lost. In this case, the error logging facility of the Logger class can be used. This way, the error is logged to RAM and can be requested once the node is available in the mesh again.
The nRF sniffer is used with Wireshark. A custom dissector for FruityMesh JOIN_ME packets has been written and is saved in the
util/wireshark folder. Integration into Wireshark described at the beginning of file
The nRF Sniffer doesn’t work with an updated Segger firmware. It is necessary to flash the Segger firmeware JLink_V498c onto the beacon with
JLinkConfig.exe, option: replace firmware. Afterwards, the sniffer should work together with the fixed sniffer application received by Nordic Semiconductor.
If the fixed sniffer application does not find the beacon, open the buggy official version until the beacon is found, then close it and reopen the fixed version.
Segger Virtual Com Port (VCOM)
A few words on the virtual COM port offered by some Segger debugger chips such as the on-board chips on the Nordic Development Kits: A few pins of the nRF controller are directly connected to some pins of the Segger debugger chip. The Segger chip will decode the UART signal and will then send it over USB to a host computer using a virtual com port. While this is a nice feature that means you do not have to use a dedicated UART-USB converter, such as an FTDI chip, it is very error prone because of the implementation in the Segger chip.
Problems that we have experienced so far:
If you send data without flow control and then switch flow control on, you must disconnect the board from power before you can send this data through. Apparently the Segger chip blocks the flow control in this case.
If you do not use flow control, many characters will get lost when sending a lot of data
If you use flow control e.g. a 1Mbit/s and send a lot of data, the Segger chip will raise the flow control lines half of the time and will severely limit the throughput
Even with flow control enabled, characters will be lost at high speeds. This was confirmed with a logic analyzer.