The current approach is working out well so far. There is a “hackrf-kernel” C program which receives commands over a control input stream (pipe file) and sends responses back to a control output stream. Data from RX is sent to the data output stream, after calling the startrx command. (The starttx command is not yet implemented.) Since hackrf supports multiple devices streaming simultaneously, there was a question of how to handle the multiple data streams. Rather than having separate pipes for each device, I opted instead to stick with one output pipe, and just have hackrf-kernel send a notification to control output, explaining how many bytes where coming down the pipe, and which device it is for. I used semaphores to ensure that the order of the incoming RX data matches the order of the control notifications. So the external program must simply pull the correct number of bytes from the stream.
A benefit of this approach is that the stream interface is language agnostic, so that I could use it with a Racket program, and somebody else could use it with a C program, or Python program, or anything else that supports reading and writing from file streams. Since I have not written any of the Racket code, for testing I have just been connecting to the pipes with the cat command and redirection arrows. I successfully downloaded about 500MB of RX data this way in a few seconds before sending the stoprx command.
So, I can do RX now, but the hackrf-kernel is not quite useful yet: I need to implement the commands for set the frequency, sample rate, etc.