Skip to main content

EVENTful APIs

· 12 min read
Andrew Butson

Application Programming Interfaces (APIs) come in various forms, with synchronous and asynchronous being the primary types. Synchronous protocols like Hypertext Transfer Protocol (HTTP) represent RESTful implementations, while asynchronous protocols like Server Sent Events (SSE) and MQ Telemetry Transport (MQTT) represent EVENTful implementations.

EVENTful message design can be categorized into three main types: notifications, objects, and streams. These messages can represent past actions (events) or future actions (commands).

A Brief History of Asynchronous APIs​

Asynchronous APIs have been pivotal in the development of interactive and real-time web applications. The journey began with the establishment of standards like MQTT in 1999, which emerged as a lightweight messaging protocol ideal for low-bandwidth, high-latency environments. MQTT's publish-subscribe model was a departure from the synchronous HTTP protocol introduced in 1991, offering a more efficient way to handle real-time, bidirectional communication.

The term "EVENTful" APIs aptly encapsulates the nature of asynchronous communication, with EVENT standing for Efficient, Versatile, Nonblocking, and Timely. These characteristics are inherent to asynchronous APIs, which include message-based and event-driven architectures, providing a robust foundation for services that require real-time updates and interactions.

The concept of APIs has a storied history, dating back to the early days of computing as referenced in the 1951 book "The Preparation of Programs for an Electronic Digital Computer." Over the past 70 years, APIs have evolved dramatically, especially with the advent of web-based APIs. A significant milestone was the introduction of the Ajax pattern by Jesse James Garrett in 2005. Ajax leveraged asynchronous JavaScript and XML to enable web pages to dynamically fetch and display content without a full page reload, enhancing user experience and web application performance.

Ajax laid the groundwork for asynchronous web requests, primarily interacting with RESTful APIs. However, as the web evolved, so did the need for more efficient real-time communication methods. This led to the adoption of Server-Sent Events (SSE), a modern approach that allows servers to push updates to clients over a single, long-held HTTP connection. Unlike MQTT, which is a protocol designed for machine-to-machine communication, SSE is specifically tailored for web applications, providing a standardized way to stream updates from the server to the client.

RESTful Systems​

In RESTful systems, the consumer initiates communication, making a request to which the service responds with the appropriate data. For example, a RESTful service might maintain a dataset of all completed trades for an investment firm. As each trade is executed and finalized, this dataset is updated accordingly. When a consumer needs to retrieve information on completed trades, it can request the service, which then responds with the relevant data for all the trades completed that day.

To illustrate this with a practical example, imagine an investment firm that needs to keep its traders and consumers informed about the status of their stock market trades. The firm could utilize a RESTful API to manage this data flow. The service would have access to a completed-trades dataset, exposing an endpoint that consumers could request to obtain the list of trades completed on a given day. A consumer application might request a web address such as https://investment.arbs.io/completed-trades to retrieve this information.

The service's HTTP response to such a request could be structured as follows:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1234

{
  "trades": [
    {
      "id": "1",
      "stock": "AAPL",
      "volume": "100",
      "price": "150.00",
      "status": "completed"
    },
    {
      "id": "2",
      "stock": "MSFT",
      "volume": "50",
      "price": "250.00",
      "status": "completed"
    }
    // more trades here...
  ]
}

This JSON response provides a clear and structured representation of the completed trades, allowing consumers to easily parse and use the data as needed.

EVENTful Systems​

EVENTful systems stand in contrast to RESTful systems by adopting a model where services actively "push" messages to consumers. These messages are sent based on predefined filters or subscriptions made by the consumer applications. Unlike RESTful systems, which adhere to a strict request-response pattern, EVENTful systems blur the traditional roles of consumers and services, allowing consumers to act as both producers and consumers of messages. This model is beneficial in scenarios where real-time updates are crucial, and the constant polling of a service for updates would be inefficient.

For instance, an EVENTful system could notify consumers immediately when a trade is executed. Instead of consumers repeatedly querying the service for the latest trade statuses, the service could send a message to the relevant consumers as soon as a trade is completed. This ensures that consumers receive updates in real-time without the need to make redundant requests to the service.

Let's reimagine the previous example. We have a service that sends a message each time a stock trade is executed. The message could be sent to all interested parties, such as traders, investment managers, or consumers, as soon as the trade is confirmed.

The structure of the JSON payload for an trade_executed event might look similar to the following:

{
  "metadata": {
    "message_source": "trade-execution-system",
    "event-type": "trade-executed"
  },
  "message": {
    "trade_id": "3335dc20",
    "stock_symbol": "AAPL",
    "volume": "100",
    "price": "150.00",
    "trader_id": "butsona",
    "timestamp": "2023-03-13T12:00:00Z"
  }
}

In this payload, the metadata provides context about the source and type of the message, while the message body contains the details of the executed trade, such as the trade ID, stock symbol, volume, price, trader ID, and execution timestamp. Each message represents a single trade execution, and it's up to the receiving applications to maintain a record of all trades if they need a historical log.

By leveraging EVENTful systems, the investment firm can minimize latency and resource consumption, ensuring that all parties have the most current information as soon as it's available.

Types of EVENTful Messages​

EVENTful systems can utilize a variety of message types to communicate different kinds of information. While the terminology for these message types needs to be standardized, understanding their concepts is crucial for designing an effective event-driven architecture. The three primary types of messages are notifications, event-carried state, and event-source messages.

Message-Notifications​

Notifications are the simplest form of EVENTful messages, often containing minimal information about an event and a point where more detailed information can be found. In the context of an investment firm, a notification might be sent out to inform a trader or consumer about a significant event, such as a large trade execution or a market movement that triggers an alert. The notification could include basic details and a URL to access a more comprehensive report or to perform a follow-up action.

For example, a notification message in a trading environment might look like this:

{
  "metadata": {
    "event-type": "trade-alert",
    "message_source": "trading-platform"
  },
  "message": {
    "trade_id": "f18121e4",
    "timestamp": "2023-03-13T12:00:00Z",
    "trade_url": "https://investment.arbs.io/trade-notifications/f18121e4"
  }
}

A side-effect of using notifications is that it reduces the security requirements as the trade_url is protected, which means that only consumers with the correct permissions can request data, reducing the complexity of security.

Event-Carried State​

Event-Carried State Transfer (ECS) messages are used when a significant block of data needs to be transmitted, such as a summary of a trade or a batch of trades. These messages carry all the necessary information to update a data store or to provide a comprehensive view of a trade without requiring additional queries to the system. This approach is beneficial for consolidating data from multiple sources and ensuring that the recipient has immediate access to the full context of the event.

An ECS message in a trading system might include details about the trade, the trader, and the current status, as follows:

{
  "metadata": {
    "event-type": "trade-summary",
    "message_source": "trade-execution-system",
    "timestamp": "2023-03-13T14:13:12Z"
  },
  "message": {
    "trade": {
      "trade_id": "f18121e4",
      "stock_symbol": "AAPL",
      "volume": "100",
      "price": "150.00",
      "trader_id": "butsona"
    },
    "status": {
      "execution_time": "2023-03-13T13:13:13Z",
      "trade_status": "completed"
    }
  }
}

Event-Source​

Event-Source messages, or Delta messages, are designed to convey incremental changes or updates to data. They are useful for streaming real-time updates about the progress of a trade or a series of trades. These messages could inform consumers or internal systems about the progression of trade executions, price changes, or other relevant market events.

For example, a series of event-source messages might be sent to indicate the stages of trade execution:

[
  {
    "trade_id": "f18121e4",
    "stage": "initiated",
    "timestamp": "2023-03-13T10:11:12"
  },
  {
    "trade_id": "f18121e4",
    "stage": "executed",
    "timestamp": "2023-03-13T10:14:13Z"
  },
  {
    "trade_id": "f18121e4",
    "stage": "confirmed",
    "timestamp": "2023-03-13T10:14:15Z"
  },
  {
    "trade_id": "f18121e4",
    "stage": "settled",
    "timestamp": "2023-03-13T10:16:15Z"
  }
]

Each message in this array represents a discrete update to the trade's status, providing a granular view of the trade's lifecycle. By employing these different types of EVENTful messages, an investment firm can ensure efficient and timely communication within its trading ecosystem, facilitating quick decision-making and enhancing overall market responsiveness.

Events and Commands​

In an EVENTful system within an investment firm, messages are a crucial component of the communication process. They are categorized based on purpose and timing: command messages are sent to trigger an action, while event messages are sent to notify that an action has occurred. Understanding and distinguishing between these two types of messages is fundamental when designing an EVENTful API system.

Identifying Events and Defining Commands​

Defining Events​

Identifying events is a critical step in the architecture of asynchronous APIs. It involves pinpointing all significant activities within the domain of stock market trading that should be tracked. Once identified, these activities are documented as events and incorporated into the API's implementation. Events in a trading environment might include a trader placing an order, a consumer logging into their account, or executing a trade.

In addition to these user-centric events, process-centric events occur within the trading system itself. These could involve the internal progression of a trade order through various stages, such as order validation, execution, and settlement. Events may also signal issues or exceptions, such as a trade rejection due to insufficient funds or a discrepancy in a trade order.

A corresponding message is designed to convey the necessary information for each identified event. The nature of the event will determine the type of message required—whether it's a simple notification, a more detailed object-style message, or a continuous stream of updates. For instance, a notification might be sent when a consumer logs in, while an object-style message might be used to convey the details of a new trade order, and streaming messages could provide real-time updates as a trade progresses through various stages.

Discoverability is essential, and cataloguing these events is crucial, as well as listing critical information such as the event name, message type, triggering conditions, example messages, and any additional notes. This documentation becomes a valuable resource for designers, developers, and architects involved in the system's development.

Here's an example of how events might be documented for an investment firm:

  • Trade Initiated: A trader orders a stock trade through the investment firm's platform.
  • Trade Executed: A trade order has been successfully executed on the stock exchange.
  • Trade Settled: The executed trade has been settled, and the securities and funds have been exchanged.

After identifying the significant past events, the next step is defining the commands representing future actions. Commands in a trading system include placing a trade order, modifying an existing order, or transferring funds. These commands are integral to the system's interactivity, allowing users to initiate actions that the trading system will process.

By meticulously identifying events and defining commands, an investment firm can ensure that its EVENTful system is comprehensive, responsive, and capable of handling the complexities of stock market trading. This approach enables the firm to maintain high service and efficiency, providing consumers with timely information and the ability to act swiftly in a dynamic market environment.

Defining Commands​

In EVENTful systems, the distinction between events and commands is pivotal. Events indicate actions within the system, providing a historical record of transactions and activities. Conversely, commands instruct parts of the system to perform future actions, which may be executed immediately or after some delay, depending on the process involved.

For example, a command to execute a trade might be processed nearly instantaneously, reflecting a change in the status of a trade from pending to completed. However, specific commands, such as those involving complex payment processing or third-party bank verifications, may incur delays. In some cases, if there are issues with payment verification, the completion of a command might take significantly longer, ranging from minutes to hours.

The ability to define and initiate commands is essential for the functionality of an EVENTful system. Commands can encompass various actions, such as placing a new trade, modifying an existing trade, or cancelling a trade order. When designing such a system, it is crucial to meticulously describe all the commands necessary to support the desired activities within the trading environment.

The methodology for defining commands mirrors identifying events. A structured catalogue of each command detailing the command name, the type of message it will generate, an example of the message, and any supplementary notes. This documentation is a reference for the team members tasked with implementing the EVENTful system, ensuring clarity and consistency across the development process. This serves as a reference for consumers to understand how to interact with the solution. It is common to place commands behind RESTful APIs to improve security and improve discoverability (for example using the openapi-specifications).

For instance, the command list might include actions such as:

  • Place Trade: A command that initiates a new trade order within the trading platform.
  • Modify Trade: A command that alters an existing trade order's parameters.
  • Cancel Trade: A command that enables the cancellation of a previously placed trade order.

Upon finalizing the list of commands, the foundational elements of an EVENTful system are established, encompassing messages, events, and commands.

Summary​

We explore the fundamental aspects of an EVENTful API system, focusing on transmitting messages between machines based on past events and future commands. The three types of Message-Notifications, Event-Carried State, and Event-Source have been discussed, along with guidance on their appropriate use. Additionally, the processes for identifying events and defining commands have been outlined, with the recommendation to document these elements for ease of collaboration and implementation. This structured approach is instrumental in building an efficient and effective EVENTful system, facilitating real-time communication and action in an ever growing fast-paced world.