Jupyter client library

Posted by Simon Walker on Thu 06 December 2018

TL;DR I have been thinking about alternative frontends to the Jupyter notebook. The first step is to abstract communications with Jupyter kernels. I have started writing one of these.

I am a huge fan of the Jupyter notebook environment. It is a fantastic way to explore a new data analysis approach, and keep results together with the implementation and explanation.

I wanted to understand how the Jupyter notebook/lab frontend interacts with the Kernel backends, and as such have started writing a client library.

Architecture

The Jupyter suite consists of two parts:

  • Frontend
  • Kernel

Within the frontend is typically a kernel communication abstraction component, what I call the "client":

Jupyter schematic

The frontend handles nicely presenting the execution results and collecting input from the user (i.e. the notebook). The kernel is the execution engine that can be written in many languages. The client handles communicating with a kernel. It communicates over 41 ZeroMQ sockets:

  • shell: for client -> kernel requests (and responses)
  • iopub: for kernel -> client messages
  • heartbeat: for health checks
  • stdin: for kernel -> client requests (and responses from the frontend)

This abstraction is what allows the frontend to communicate with different kernels.

Writing a client library

The documentation on the Jupyter wire protocol is pretty good at describing how messages are sent to and from the kernel over the "wire" (the various ZeroMQ sockets). This documentation was used to build a WireMessage abstraction, which:

I hope that this library will form the beginning of a new Rust-based Jupyter frontend. I have some ideas but that's for another time.

Links


  1. Technically 5: a "control" socket is a duplicate of the "shell" socket but for high-priority messages. 

tags: rust