HackRF Shell: (Learning) Delays

I wanted to move on quickly from completing the FFT interface, on to FM demodulation. However, I realized that if the rest of the project was going to practical and modular, I would need some kind of block style I/O buffer management system, like GnuRadio has. And as I started to pursue that, I realized that I would need to properly modularize my code.

Up until now, all the functions were either being dumped into the default module by the C code, or loaded from one Scheme file in the source directory. I took all the external C code and organized it in sensible module divisions, and then took all the C-based Scheme primitives and put them in two modules ((sdr primitives) and (sdr hackrf primitives)). Those two modules are created and loaded when HackRF Shell is launched, while the other modules can be compiled independently.

That all required taking the time to actually learn how the module system works, as well as understanding how to create and fill-out modules through the C interfaces. So, not much amazing progress to the outside observer, but I learned a lot. The FFT demo function is still working:

One neat thing about programming in Guile is that you can jump into various modules at any time, to play around in them or see what variables are defined in them (private and public):

christopher@nightshade:~/Repos/hackrf-shell$ ./hackrf-shell 
HackRF Shell, copyright 2019 Christopher Howard

Initializing primitive functions.
Entering the Scheme shell.

GNU Guile 2.2.3
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (set! %load-path (cons "/home/christopher.qlfiles.net/Repos/hackrf-shell" %load-path))
scheme@(guile-user)> ,m (sdr hackrf util)
scheme@(sdr hackrf util)> ,b
hackrf-sensible-defaults #<variable 56314e6aaac0 value: #<procedure hackrf-sensible-defaults (d)>>
filename                #<variable 56314e6aa940 value: #<undefined>>
timed-read              #<variable 56314e6aa8a0 value: #<procedure timed-read (d file-path start-h start-m end-h end-m)>>
hackrf-cb-rx-to-file    #<variable 56314e6aa990 value: #<procedure hackrf-cb-rx-to-file (filename)>>
out                     #<variable 56314e6aa9e0 value: #<undefined>>
when-hour-min           #<variable 56314e6bd660 value: #<procedure when-hour-min (h m l)>>
hackrf-cb-rx-to-stream  #<variable 56314e6aaa30 value: #<procedure hackrf-cb-rx-to-stream (out)>>
cb-fft-to-gnuplot       #<variable 56314e6aa8f0 value: #<procedure cb-fft-to-gnuplot (center-freq samp-rate averaging-alpha)>>
scheme@(sdr hackrf util)> hackrf-
hackrf-cb-rx-to-file                  hackrf-open                           hackrf-set-sample-rate
hackrf-cb-rx-to-stream                hackrf-sensible-defaults              hackrf-set-vga-gain
hackrf-close                          hackrf-set-baseband-filter-bandwidth  hackrf-start-rx
hackrf-disable-amp                    hackrf-set-freq                       hackrf-stop-rx
hackrf-enable-amp                     hackrf-set-lna-gain                   
scheme@(sdr hackrf util)> ,m (sdr util)
scheme@(sdr util)> ,b
bv-complex-to-mag       #<variable 56314e6b4700 value: #<procedure bv-complex-to-mag (complex-float-bv n_complex_nums dest-bv)>>
freq-shifter            #<variable 56314e6b4670 value: #<procedure freq-shifter (freq samprate)>>

The next step in my project will be to work out my block-style I/O buffer management system, which will require me to re-learn GOOPS, Guile’s object oriented programming extension, so that will likely still take a while.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

HackRF Shell: FFT Averaging and Display Progress

I’m not quite sure I have all the bugs worked out yet, but I implemented simple averaging, as well and the frequency display. The latter required implementing fft-shift to move center frequency to the center. The above plot shows FM stations at 101.1, 102.5, 103.9, and 104.7 Mhz, matching local stations I know about.

These simple things took quite a while, mainly because of the busyness of other life responsibilities; but also because the process of implementation led to fixing some bugs, which led to cleaning up some code, which led to learning new things about Guile scheme and C.

In Guile scheme, I learned how to implement a procedure to auto-close unreachable ports:

(define ports-guardian #f)

(define (close-guarded-ports)
  (let f ()
    (let ((p (ports-guardian)))
      (when p (begin (close-port p) (f))))))

(define (initialize-ports-guardian)
  (when (not ports-guardian)
    (set! ports-guardian (make-guardian))
    (add-hook! after-gc-hook (lambda () (close-guarded-ports)))))

(initialize-ports-guardian)

(define (register-autoclose p) (ports-guardian p) p)

register-autoclose can simply be wrapped around the port, e.g.:

(define (hackrf-cb-rx-to-file filename)
  (hackrf-cb-rx-to-stream
   (register-autoclose (open-output-file filename))))

This uses Guile scheme “guardian” functionality, which is an object that can return registered objects if the have become unreachable everywhere else.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

HackRF Shell: FFT Plot Progress

I threw together a scheme procedure which dumps FFT data to Gnuplot.

You see, however, that I have more work to do as properly labeling the graph, as well as implementing averaging for a more practical spectrum display. The 512 points on the x-axis correspond to the 512 complex numbers in the FFT output buffer, but center frequency in this case is actually 103Mhz, at 8 msps.

scheme@(guile-user)> (define d (hackrf-open))
scheme@(guile-user)> (hackrf-sensible-defaults d)
scheme@(guile-user)> (hackrf-set-freq d 103000000)
scheme@(guile-user)> (hackrf-set-baseband-filter-bandwidth d 2000000)
scheme@(guile-user)> (hackrf-enable-amp d)
scheme@(guile-user)> (define c (cb-fft-to-gnuplot))
scheme@(guile-user)> (hackrf-start-rx d c)
scheme@(guile-user)> (hackrf-stop-rx d)

It was mostly easy to write the scheme -> Gnuplot interface as that is just sending text commands to Gnuplot standard input. However, it took me a while to fix an elusive bug in my C FFT function.

HackRF Shell: RX Restart Bug

I spent quite a while troubleshooting a bug in which RX would mysteriously not restart, if you did a hackrf-stop-rx followed by another hackrf-start-rx. The problem actually was not in my code, but due to some old libhackrf bugs that had not been patched in the old Debian 9 version of libhackrf which I am using.

This is a serious enough annoyance that you won’t want to be using HackRF Shell with the unpatched version. So, I added instructions to my git repo (debian9-libhackrf-patch directory, see commit 346c50e) on how to get a patched version of the Debian 9 packages. I think that the Debian 10 library version is actually not new enough to avoid all the bugs, either, so the info I have provided might be of value to Debian 10 users as well.

I would like to switch my home system from Debian to Gnu Guix, and use Guix package management for development, but I’m not sure how soon that will happen.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git

HackRF Shell: Conversion to Floating Point

As expected, the bottle neck has disappeared at the floating point conversion. At least, I can say that I didn’t have trouble pulling 8 million samples per second (msps), which with 32 bit floating point (and 2 floating point numbers per sample) is 64 MB/sec. (I am short on time this evening, so I haven’t had a chance yet to try 20 msps.) As before, I fed the data into a Gnu Radio FM demodulator and got clean FM radio station audio out of it.

GCC 6.3 with -O3 appears to do some SSE optimization on the byte buffer to float buffer conversion:

christopher@nightshade:~/Repos/hackrf-shell$ objdump hackrf-shell -x -D | less
<snip>
    172c:       66 0f 6f c1             movdqa %xmm1,%xmm0
    1730:       66 0f 68 cc             punpckhbw %xmm4,%xmm1
    1734:       66 0f 60 c4             punpcklbw %xmm4,%xmm0
    1738:       66 0f 6f f1             movdqa %xmm1,%xmm6
    173c:       66 0f 65 e8             pcmpgtw %xmm0,%xmm5
    1740:       66 0f 6f f8             movdqa %xmm0,%xmm7
    1744:       66 0f 61 fd             punpcklwd %xmm5,%xmm7
    1748:       66 0f 69 c5             punpckhwd %xmm5,%xmm0
    174c:       66 0f 6f eb             movdqa %xmm3,%xmm5
    1750:       0f 5b ff                cvtdq2ps %xmm7,%xmm7
    1753:       66 0f 65 e9             pcmpgtw %xmm1,%xmm5
    1757:       0f 17 3c 24             movhps %xmm7,(%rsp)
<snip>

The next thing, perhaps, should be to create a little demo program where it captures the data at certain times of day. Or I could work on the next stages of an FM receiver, i.e., frequency multiplication, a low pass filter, and the FM demodulator.

HackRF Shell: It’s alive! Wha ha HA HA HA!

The shell is functioning, including RX functionality (not TX). Currently the process looks like so:

christopher@nightshade:~/Repos/hackrf-shell$ ./hackrf-shell 
GNU Guile 2.2.3
Copyright (C) 1995-2017 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.

Enter `,help' for help.
scheme@(guile-user)> (load "hackrf-shell-lib.scm")
scheme@(guile-user)> (define d (hackrf-open))
scheme@(guile-user)> (hackrf-sensible-defaults d)
scheme@(guile-user)> (define c (hackrf-cb-rx-to-file "out.bin"))
scheme@(guile-user)> (hackrf-start-rx d c)
scheme@(guile-user)> (hackrf-stop d c)

hackrf-sensible-defaults is simply an alias for

(hackrf-set-freq d 99500000)
(hackrf-set-sample-rate d 8000000)
(hackrf-disable-amp d)
(hackrf-set-lna-gain d 16)
(hackrf-set-vga-gain d 16))

The hackrf-start-rx procedure must receive a callback function which will handle the data each time a block of data is received through libhackrf. The callback function receives a pointer object to the data buffer, which can be converted to a bytevector with pointer->bytevector, and also the byte length of the buffer (an int). User can put together their own callback functions, though I was planning to add more for common use cases. Here are the ones included now:

(define (hackrf-cb-rx-to-stream out)
  (lambda (b bl)
    (let ([bv (pointer->bytevector b bl)])
      (put-bytevector out bv))))

(define (hackrf-cb-rx-to-file filename)
  (hackrf-cb-rx-to-stream
   (open-output-file filename)))

I could expand this by adding more such scheme functions for common use cases (e.g., FM demodulation or frequency analysis) with helper functions written in C to handle intensive mathematical operations (e.g., FFT). You can see here the power of Guile scheme, allowing user to load in any Guile scheme code they want, while I can provide helper functions coded in C to allow for maximum efficiency in critical mathematical operations.

The current code simply provides the raw 8-bit IQ data, but the user may want to receive the IQ data as 32-bit floating point numbers. This should be an easy function to add, which I will code in C to allow for SSE/AVX style compiler optimizations. It still remains to be seen if I will run into a performance bottle neck with the byte to float conversions, but since I do not need to do any IPC with hackrf-shell, I do not expect this to be a problem.

To confirm received data is not junk, I saved about 1 GB of IQ data to “out.bin”, converted the data to floating point values using Gnu Radio, and then ran the data through an FM demodulator Gnu Radio program. I was able to tune into several different radio stations recorded in the save data, such as a local country music station, and I did not hear any distortion or skips.

git clone git://git.librehacker.com/pub/git/hackrf-rkt.git