openssl/doc/designs/quic-design/quic-concurrency.md

QUIC Concurrency Architecture
=============================

Introduction
------------

Most QUIC implementations in C are offered as a simple state machine without any
included I/O solution. Applications must do significant integration work to
provide the necessary infrastructure for a QUIC implementation to integrate
with. Moreover, blocking I/O at an application level may not be supported.

OpenSSL QUIC seeks to offer a QUIC solution which can serve multiple use cases:

- Firstly, it seeks to offer the simple state machine model and a fully
  customisable network path (via a BIO) for those who want it;

- Secondly, it seeks to offer a turnkey solution with an in-the-box I/O
  and polling solution which can support blocking API calls in a Berkeley
  sockets-like way.

These usage modes are somewhat diametrically opposed. One involves libssl
consuming no resources but those it is given, with an application responsible
for synchronisation and a potentially custom network I/O path. This usage model
is not “smart”. Network traffic is connected to the state machine and state is
input and output from the state machine as needed by an application on a purely
non-blocking basis. Determining *when* to do anything is largely the
application's responsibility.

The other diametrically opposed usage mode involves libssl managing more things
internally to provide an easier to use solution. For example, it may involve
spinning up background threads to ensure connections are serviced regularly (as
in our existing client-side thread assisted mode).

In order to provide for these different use cases, the concept of concurrency
models is introduced. A concurrency model defines how “cleverly” the QUIC engine
will operate and how many background resources (e.g. threads, other OS
resources) will be established to support operation.

Concurrency Models
------------------

- **Unsynchronised Concurrency Model (UCM):** In the Unsynchronised Concurrency
  Model, calls to SSL objects are not synchronised. There is no locking on any
  APL call (the omission of which is purely an optimisation). The application is
  either single-threaded or is otherwise responsible for doing synchronisation
  itself.

  Blocking API calls are not supported under this model. This model is intended
  primarily for single-threaded use as a simple state machine by advanced
  applications, and many applications will be likely to disable autoticking.

- **Contentive Concurrency Model (CCM):** In the
  Contentive Concurrency Model, calls to SSL objects are wrapped in locks and
  multi-threaded usage of a QUIC connection (for example, parallel writes to
  different QUIC stream SSL objects belonging to the same QUIC connection) is
  synchronised by a mutex.

  This is contentive in the sense that if a large number of threads are trying
  to write to different streams on the same connection, a large amount of lock
  contention will occur. As such, this concurrency model will not scale and
  provide good performance, at least within the context of concurrent use
  of a single connection.

  Under this model, APL calls by the application result in lock-wrapped
  mutations of QUIC core objects (`QUIC_CHANNEL`, `QUIC_STREAM`, etc.) on the
  same thread.

  This model may be used either in a variant which does not support blocking
  (NB-CCM) or which does support blocking (B-CCM). The blocking variant must
  spin up additional OS resources to correctly support blocking semantics.

- **Thread Assisted Contentive Concurrency Model (TA-CCM):** This is currently
  implemented by our thread assisted mode for client-side QUIC usage. It does
  not realise the full state separation or performance of the Worker Concurrency
  Model (WCM) below. Instead, it simply spawns a background thread which ensures
  QUIC timer events are handled as needed. It makes use of the Contentive
  Concurrency Model for performing that handling, in that it obtains a lock when
  ticking a QUIC connection just as any call by an application would.

  This mode is likely to be deprecated in favour of the full Worker Concurrency
  Model (WCM), which it will naturally be subsumed by.

- **Worker Concurrency Model (WCM):** In the Worker Concurrency Model,
  a background worker thread is spawned to manage connection processing. All
  interaction with a SSL object goes through this thread in some way.
  Interactions with SSL objects are essentially translated into commands and
  handled by the worker thread. To optimise performance and minimise lock
  contention, there is an emphasis on message passing over locking.
  Internal dataflow for application data can be managed in a zero-copy way to
  minimise the costs of this message passing.

  Under this model, QUIC core objects (`QUIC_CHANNEL`, `QUIC_STREAM`, etc.) will
  live solely on the worker thread and access to these objects by an application
  thread will be entirely forbidden.

  Blocking API calls are supported under this model.

These concurrency models are summarised as follows:

| Model  | Sophistication | Concurrency           | Blocking Supported | OS Resources              | Timer Events    | RX Steering | Core State Affinity  |
|--------|----------------|-----------------------|--------------------|---------------------------|-----------------|-------------|----------------------|
| UCM    | Lowest         | ST only               | No                 | None                      | App Responsible | None        | App Thread           |
| CCM    |                | MT (Contentive)       | Optional           | Mutex, (Notifier)         | App Responsible | TBD         | App Threads          |
| TA-CCM† |                | MT (Contentive)       | Optional           | Mutex, Thread, (Notifier) | Managed         | TBD         | App & Assist Threads |
| WCM    | Highest        | MT (High Performance) | Yes                | Mutex, Thread, Notifier   | Managed         | Futureproof | Worker Thread        |

† To eventually be deprecated in favour of WCM.

Legend:

- **Blocking Supported:** Whether blocking calls to e.g. `SSL_read` can be
  supported. If this is listed as “optional”, extra resources are required to
  support this under the listed model and these resources could be omitted if an
  application indicates it does not need this functionality at initialisation
  time.

- **OS Resources:** “Mutex” refers to mutex and condition variable resources.
  “Notifier” refers to a kind of OS resource needed to allow one thread to wake
  another thread which is currently blocking in an OS socket polling call such
  as poll(2) (e.g. an eventfd or socketpair). Resources listed in parentheses in
  the table above are required only if blocking support is desired.

- **Timer Events:** Is an application responsible for ensuring QUIC timeout
  events are handled in a timely manner?

- **RX Steering:** The matter of RX steering will be discussed in detail in a
  future document. Broadly speaking, RX steering concerns whether incoming
  traffic for multiple different QUIC connections on the same local port (e.g.
  for a server) can be vectored *by the OS* to different threads or whether the
  demuxing of incoming traffic for different connections has to be done manually
  on an in-process basis.

  The WCM model most readily supports RX steering and is futureproof in this
  regard. The feasibility of having the UCM and CCM models support RX steering
  is left for future analysis.

- **Core State Affinity:** Which threads are allowed to touch the QUIC core
  objects (`QUIC_CHANNEL`, `QUIC_STREAM`, etc.)

Architecture
------------

To recap, the API Personality Layer (APL) refers to the code in `quic_impl.c`
which implements the libssl API personality (`SSL_write`, etc.). The APL is
cleanly separated from the QUIC core implementation (`QUIC_CHANNEL`, etc.).

Since UCM is basically a slight optimisation of CCM in which unnecessary locking
is elided, discussion from hereon in will focus on CCM and WCM except where
there are specific differences between CCM and UCM.

Supporting both CCM and WCM creates significant architectural challenges. Under
CCM, QUIC core objects have their state mutated under lock by arbitrary
application threads and these mutations happen during APL calls. By contrast, a
performant WCM architecture requires that APL calls be recorded and serviced in
an asynchronous fashion involving message passing to a worker thread. This
threatens to require highly divergent dispatch architectures for the two
concurrency models.

As such, the concept of a **Concurrency Management Layer (CML)** is introduced.
The CML lives between the APL and the QUIC core code. It is responsible for
dispatching in-thread mutations of QUIC core objects when operating under CCM,
and for dispatching messages to a worker thread under WCM.

![Concurrency Models Diagram](images/quic-concurrency-models.svg)

There are two different CMLs:

- **Direct CML (DCML)**, in which core objects are worked on in the same thread
  which made an APL call, under lock;

- **Worker CML (WCML)**, in which core objects are managed by a worker thread
  with communication via message passing. This CML is split into a front end
  (WCML-FE) and back end (WCML-BE).

The legacy thread assisted mode uses a bespoke method which is similar to the
approach used by the DCML.

CML Design
----------

The CML is designed to have as small an API surface area as possible to enable
unified handling of as many kinds of (APL) API operations as possible. The idea
is that complex APL calls are translated into simple operations on the CML.

At its core, the CML exposes some number of *pipes*. The number of pipes which
can be accessed via the CML varies as connections and streams are created and
destroyed. A pipe is a *unidirectional* transport for byte streams. Zero-copy
optimisations are expected to be implemented in future but are deferred.

The CML (`QUIC_CML`) allows the caller to refer to a pipe by providing an opaque
pipe handle (`QUIC_CML_PIPE`). If the pipe is a sending pipe, the caller can use
`ossl_cml_write` to try and add bytes to it. Conversely, if it is a receiving
pipe, the caller can use `ossl_cml_read` to try and read bytes from it.

The method `ossl_cml_block_until` allows the caller to block until at least one
of the provided pipe handles is ready. Ready means that at least one byte can be
written (for a sending pipe) or at least one byte can be read (for a receiving
pipe).

Note that there is only expected to be one `QUIC_CML` instance per QUIC event
processing domain (i.e., per `QUIC_DOMAIN` / `QUIC_ENGINE` instance). The CML
fully abstracts the QUIC core objects such as `QUIC_ENGINE` or `QUIC_CHANNEL` so
that the APL never sees them.

The caller retrieves a pipe handle using `ossl_cml_get_pipe`. This function
retrieves a pipe based on two values:

  - a CML pipe class;
  - a CML *selector*.

The CML selector is a tagged union structure which specifies what pipe is to be
retrieved. Abstractly, examples of selectors include:

```text
    Domain      ()
    Listener    (listener_id: uint)
    Conn        (conn_id:     uint)
    Stream      (conn_id:     uint, stream_id: u64)
```

In other words, the CML selector selects the “object” to retrieve a pipe from.

The CML pipe class is one of the following values:

- Request
- Notification
- App Send
- App Recv

The pipe classes available for a given selector vary. For example, the “App
Send” and “App Recv” pipes only exist on a stream, so it is invalid to request
such a pipe in conjunction with a different type of selector.

The “Request” and “App Send” classes expose send-only streams, and the
“Notification” and “App Recv” classes expose receive-only streams.

For any given CML selector, the Request pipe is used to send serialized commands
for asynchronous processing in relation to the entity selected by that selector.
Conversely, the Notification pipe returns asynchronous notifications. These
could be in relation to a previous Command (e.g. indicating whether a command
succeeded), or unprompted notifications about other events.

The underlying pattern here is that there is a bidirectional channel for control
messages, and a bidirectional channel for application data, both comprised of
two unidirectional pipes in turn.

Pipe handles are stable for as long as the pipe they reference exists, so an APL
object can cache a pipe handle if desired.

All CML methods are thread safe. The CML implementation handles any necessary
locking (if any) internally.

The `ossl_cml_write_available` and `ossl_cml_read_available` calls determine the
number of bytes which can currently be written to a send-only pipe, or read from
a receive-only pipe, respectively.

**Race conditions.** Because these are separate calls to `ossl_cml_write` and
`ossl_cml_read`, the values returned by these functions may become out of date
before the caller has a chance to read `ossl_cml_write` or `ossl_cml_read`.
However, such changes are guaranteed to be monotonically in favour of the
caller; for example, the value returned by `ossl_cml_write_available` will only
ever increase asynchronously (and only decrease as a result of an
`ossl_cml_write` call). Conversely, the value returned by
`ossl_cml_read_available` will only ever increase asynchronously (and only
decrease as a result of an `ossl_cml_read` call). Assuming that only one thread
makes calls to CML functions at a given time *for a given pipe*, this therefore
poses no issue for callers.

Concurrent use of `ossl_cml_write` or `ossl_cml_read` for a given pipe is not
intended (and would not make sense in any case). The caller is responsible for
synchronising such calls.

**Examples of pipe usage.** The application data pipes are used to serialize the
actual application data sent or received on a QUIC stream. The usage of the
request/notification pipes is more varied and used for control activity. There
is therefore a “control/data” separation here. The request and notification
pipes transport tagged unions. Abstractly, commands and notifications might
include:

- Request: Reset Stream (error code: u64)
- Notification: Connection Terminated by Peer

**Example implementation of `SSL_write`.** An `SSL_write`-like API might be
implemented in the APL like this:

```c
int do_write(QUIC_CML *cml,
             QUIC_CML_PIPE notification_pipe,
             QUIC_CML_PIPE app_send_pipe,
             const void *buf, size_t buf_len)
{
    size_t bytes_written = 0;

    for (;;) {
        /* e.g. connection termination */
        process_any_notifications(notification_pipe);

        /* state checks, etc. */
        if (...->conn_terminated)
            return 0;

        if (buf_len == 0)
            return 1;

        if (!ossl_cml_write(cml, app_send_pipe, buf, buf_len, &bytes_written))
            return 0;

        if (bytes_written == 0) {
            if (!should_block())
                break;

            ossl_cml_block_until(cml, {notification_pipe, app_send_pipe});
            continue; /* try again */
        }

        buf     += bytes_written;
        buf_len -= bytes_written;
    }

    return 1;
}
```

```c
/*
 * Creates a new CML using the Direct CML (DCML) implementation. need_locking
 * may be 0 to elide mutex usage if the application is guaranteed to synchronise
 * access or is purely single-threaded.
 */
QUIC_CML *ossl_cml_new_direct(int need_locking);

/* Creates a new CML using the Worker CML (WCML) implementation. */
QUIC_CML *ossl_cml_new_worker(size_t num_worker_threads);

/*
 * Starts the CML operating. Idempotent after it returns successfully. For the
 * WCML this might e.g. start background threads; for the DCML it is likely to
 * be a no-op (but must still be called).
 */
int ossl_cml_start(QUIC_CML *cml);

/*
 * Begins the CML shutdown process. Returns 1 once shutdown is complete; may
 * need to be called multiple times until shutdown is done.
 */
int ossl_cml_shutdown(QUIC_CML *cml);

/*
 * Immediate free of the CML. This is always safe but may cause handling
 * of a connection to be aborted abruptly as it is an immediate teardown
 * of all state.
 */
void ossl_cml_free(QUIC_CML *cml);

/*
 * Retrieves a pipe for a logical CML object described by selector. The pipe
 * handle, which is stable over the life of the logical CML object, is written
 * to *pipe_handle. class_ is a QUIC_CML_CLASS value.
 */
enum {
    QUIC_CML_CLASS_REQUEST,         /* control; send */
    QUIC_CML_CLASS_NOTIFICATION,    /* control; recv */
    QUIC_CML_CLASS_APP_SEND,        /* data; send */
    QUIC_CML_CLASS_APP_RECV         /* data; recv */
};

int ossl_cml_get_pipe(QUIC_CML                  *cml,
                      int                       class_,
                      const QUIC_CML_SELECTOR   *selector,
                      QUIC_CML_PIPE             *pipe_handle);

/*
 * Returns the number of bytes a sending pipe can currently accept. The returned
 * value may increase over time asynchronously but will only decrease in
 * response to an ossl_cml_write call.
 */
size_t ossl_cml_write_available(QUIC_CML *cml, QUIC_CML_PIPE pipe_handle);

/*
 * Appends bytes into a sending pipe by copying them. The buffer can be freed
 * as soon as this call returns.
 */
int ossl_cml_write(QUIC_CML *cml, QUIC_CML_PIPE pipe_handle,
                   const void *buf, size_t buf_len);

/*
 * Returns the number of bytes a receiving pipe currently has waiting to be
 * read. The returned value may increase over time asynchronously but will only
 * decrease in response to an ossl_cml_read call.
 */
size_t ossl_cml_read_available(QUIC_CML *cml, QUIC_CML_PIPE pipe_handle);

/*
 * Reads bytes from a receiving pipe by copying them.
 */
int ossl_cml_read(QUIC_CML *cml, QUIC_CML_PIPE pipe_handle,
                  void *buf, size_t buf_len);

/*
 * Blocks until at least one of the pipes in the array specified by
 * pipe_handles is ready, or until the deadline given is reached.
 *
 * A pipe is ready if:
 *
 *   - it is a sending pipe and one or more bytes can now be written;
 *   - it is a receiving pipe and one or more bytes can now be read.
 */
int ossl_cml_block_until(QUIC_CML *cml,
                         const QUIC_CML_PIPE *pipe_handles,
                         size_t num_pipe_handles,
                         OSSL_TIME deadline);
```