HackRF Shell Project

So, I got really far in the hackrf-stream project, with essentially everything working to control the HackRF and to manage the RX streams, and then ran into a fatal snag. Everything was based on the idea that I could use FIFO OS pipes to move the data from the C program to the processing program, but FIFO OS pipes turned out to be not fast enough for my needs. In the end, I was only able to move about 90% of the data live across the pipe, and profiling indicated the bottleneck was in system calls, rather than in number crunching. 90% is a lot of data, but I needed 100% going across.

So, I was faced with multiple options on how to proceed. On approach would be to switch instead to Shared Memory Objects for IPC. That I believe would have removed the i/o performance problem, but then all the control applications would need POSIX Shared Memory support, which rather went against the whole idea of having a simple, language agnostic interface.

Another idea was to abandon the idea of a simple, language agnostic interface, and just use POSIX Shared Memory Objects anyway. Unfortunately, Racket Scheme does not have a shared memory interface. Instead, it uses something called “places”, which is shared memory, but only allows sharing with other Racket programs. Chicken Scheme does have a POSIX shm interface, but I wasn’t sure if I really wanted to start using Chicken Scheme just for that reason.

Where I landed in the end was a different approach: embedding Guile Scheme inside my C program. This means when I start the C program, I’m dropped into a Guile Scheme shell, from which I can call whatever C functions I want (after writing the appropriate stubs). This takes me full circle back to what I originally had wanted to do with this project, but instead of using a Racket FFI, I am actually embedding Guile scheme inside the C application, with the C application providing all the core control and performance functionality. If it works, this should be the most fun approach.

I have renamed the project the “HackRF Shell” project.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

HackRF Stream: Update 6

I’m referring to the project as HackRF Stream rather than HackRF Racket Bindings because one could have any programming language tie into the stream interface.

It proved that having the data for all RX streams come through one pipe, and having the notifications go to the control stream, created too many complications. So I re-did the startrx and stoprx code to shunt each RX stream to a separate fifo pipe. Before each chunk of RX data sent to the pipe, I send a newline terminated string to the pipe, which indicates the amount of bytes following.

I needed a hashmap (a.k.a., associate array) to keep track of the streams internally. So, I added glib as a dependency, which has the well documented g_hash_table functions.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

HackRF Racket Bindings: Update 5

The current approach is working out well so far. There is a “hackrf-kernel” C program which receives commands over a control input stream (pipe file) and sends responses back to a control output stream. Data from RX is sent to the data output stream, after calling the startrx command. (The starttx command is not yet implemented.) Since hackrf supports multiple devices streaming simultaneously, there was a question of how to handle the multiple data streams. Rather than having separate pipes for each device, I opted instead to stick with one output pipe, and just have hackrf-kernel send a notification to control output, explaining how many bytes where coming down the pipe, and which device it is for. I used semaphores to ensure that the order of the incoming RX data matches the order of the control notifications. So the external program must simply pull the correct number of bytes from the stream.

A benefit of this approach is that the stream interface is language agnostic, so that I could use it with a Racket program, and somebody else could use it with a C program, or Python program, or anything else that supports reading and writing from file streams. Since I have not written any of the Racket code, for testing I have just been connecting to the pipes with the cat command and redirection arrows. I successfully downloaded about 500MB of RX data this way in a few seconds before sending the stoprx command.

So, I can do RX now, but the hackrf-kernel is not quite useful yet: I need to implement the commands for set the frequency, sample rate, etc.

HackRF Racket Bindings: Update 4

I had too much trouble, unfortunately, trying to bind the libhackrf functions directly to Racket functions with the FFI interface. The two hold ups were (1) I discovered I wasn’t actually supposed to be doing I/O inside a callout function using async-apply, and was having trouble trying to set up some kind of thunk queuing system instead; (2) for some reason the hackrf_stop_rx function would always hang when called from within Racket. On the second problem, I didn’t really have any way to debug the issue, but it looked liked pthread_join was not finishing from inside hackrf_stop_rx, perhaps something to do with the async-apply mechanics.

Feeling like I didn’t have way to move forward, because I don’t know how to debug C functions called from Rcket, I instead decided to back up and change my approach. Instead, I’m just going to use a separate small C program “kernel” to call the libhackrf functions, and communicate with that over named pipes. C programming is a lot of work, but I’ve gotten as far as writing the code in the kernel that creates the pipes and pulls a line from the input control pipe. There are four pipes: control input, control output, data input, and data output.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

Hackrf Racket Bindings: Update 3

tl;dr: I can set radio parameters and also transfer RX data to a file.

I created many of the bindings for setting radio parameters, such as setting the center frequency and the sample rate. Those bindings were easy.

Figuring out the binding for the hackrf_start_rx function, and figuring out how to make use of it, was difficult and took a lot of time. This is because internally hackrf_start_rx utilizes pthreads, which apparently does not play well with the Racket environment unless you pass in special parameters to _fun ctype. Once I figured that out, I had a lot of difficult figuring out how pull data from the hackrf_transfer buffer, and I kept trying to write the data to a file but couldn’t seem to pull it off.

Anyway, I figured that out for the most part, and was able to dump RX data to a file at a speed that seemed to match up with my set sample rate, which was 8 mhz. However, I’m not quite sure about the efficiency of that operation as I was calling write-byte for each byte in the buffer. I wanted to use write-bytes instead to transfer the whole buffer in one call, but I was having trouble getting that to work.

 racket@hackrf.rkt> (hackrf-init)
 racket@hackrf.rkt> (define h (hackrf-open))
 racket@hackrf.rkt> (sensible-defaults h)
 racket@hackrf.rkt> (define out (open-output-file "out.bin"))
 racket@hackrf.rkt> (define cb (cb-rx-to-port out))
 racket@hackrf.rkt> (hackrf-start-rx h cb)
christopher@nightshade:~/Repos/hackrf-rkt$ hexdump out.bin | head
 0000000 fa00 0201 f6ff 0303 f802 0100 fc00 ff04
 0000010 fff7 fa03 01f8 fdfe fefe fbfd 0000 f6fe
 0000020 0305 f9fe fb08 02fb f803 05fd fbff ffff
 0000030 fbf9 0100 faf9 ff03 fcfd f702 0200 f902
 0000040 0102 fc03 fefe fcfc 0203 fdf4 fe06 fff8
 0000050 f9fe 0202 f7fe 0104 f401 0000 feff fb04
 0000060 05fa f702 03fa fffc fe00 fdfc fe04 f6fb
 0000070 0105 fafd f906 02ff f500 0600 fdfc 0000
 0000080 fdfc fe01 fef6 fd04 fffc f500 0004 f7fb
 0000090 0103 fb01 fe01 fdfc 0003 02f5 fd05 01fb

HackRF Racket Bindings: Update 2

It took me a while to produce one more function, which is hackrf-open. This delay was mainly because I had a shaky understanding of some of the Racket FFI concepts and functions, so I had to go back and relearn several things, especially relating to handling C pointers in Racket. Another thing that is interesting: In C you are expected to manually allocate and deallocate memory, and signal errors via an integer return value, whereas in Scheme we are expecting the memory management to happen automatically, with errors being signaled by exceptions, so there is some work to be done bridging between to two ideas.

Anyway, hackrf-open was a good step as it gives you access to the hackrf_device pointer, which has to be passed to most of the other hackrf control functions.

racket@hackrf.rkt> (define dev (hackrf-open))
racket@hackrf.rkt> dev
#<cpointer:hackrf_device>

If you run (exit) or assign something else to dev, the program will run hackrf_close on #<cpointer:hackrf_device>.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

Hackrf Racket Bindings: Update

I code bindings for the info functions, the functions used to pull basic information about the HackRF devices that are plugged in to USB.

racket@hackrf.rkt> (hackrf-init)
0
racket@hackrf.rkt> (define dl (get-hackrf-device-list))
racket@hackrf.rkt> (hackrf-device-count dl)
1
racket@hackrf.rkt> (hackrf-serial-numbers dl)
'("000000000000000087c867dc29903e5f")
racket@hackrf.rkt> 
racket@hackrf.rkt> (hackrf-usb-board-ids dl)
'(USB_BOARD_ID_HACKRF_ONE)
racket@hackrf.rkt> (hackrf-usb-device-index dl)
'(0)
racket@hackrf.rkt> (hackrf-usb-devices dl)
'(#<cpointer>)

There are a few earlier boards that are also supported by this interface besides just the HackRF One.

I’m coding for the version of libhackrf in Debian Stretch, which I’ve discovered is a few years older than the master branch. Maybe after finishing these binds I’ll create a new Git branch for the newer library, and update the bindings for it.

Here is the repo:

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git