Event-based Communication between Microservices (integration events), Event Bus Implementation(s)

Event Bus Implementation(s)

Event-driven architecture pattern is a distributed asynchronous architecture pattern to create highly scalable reactive applications. The pattern suits for every level application stack from small to complex ones. The main idea is delivering and processing events asynchronously.

There are a couple of ways to implement event-driven pattern to your stack. One of them is using “Broker Topology”. In broker topology, you send events to a central broker and all subscribers of these broker receive and process events asynchronously.

img

img

Event Bus with multiple subscribers(green arrows) and notifiers(red arrows)

In this story, I shared four implementations of event bus and with their goods and drawbacks. In all implementations, all event generators send events to the event bus. Event processors(subscribers) subscribe to the event bus. And subscribers can also create new events and pass it to the event bus.

i1) Notify the Event to All Subscribers

img

img

Event Bus sends all incoming events(red arrows) to all subscribers(blue boxes) without any touch

Actors

Event Bus, Subscribers (Event Processor), Event Creators

Implementation

Whenever the event bus receives an event from event creators, it passes the event to all subscribers. Subscribers process the event data with their own logic. Subscribers can also create new events and pass it to the event bus too. Event bus does not care about if the receiver fails to receive the event message.

Functions Needed

Notify, subscribe, unsubscribe, subscribers are enough to implement such event bus.

Goods

It is very simple in terms of development, not too much responsibility.

Drawbacks

In this approach, you are replicating the event data to all subscribers, even if the subscribers do not have the power to consume this event data. This means that your memory demand may increase with the multiplication of the subscribers in time.

It also does not care about the assurance of the delivery. It tries to send the event to the subscriber only once but it does not care about the broken pipes, connection errors and other possible lacks.

This implementation notifies all subscriber without filtering. So, filtering should be done in subscribers.

On JS front-end world [reduxjs](http://redux.js.org/) library may be given as a sample of this implementation. When an event dispatched by actions, all reducers receive the same event type and event data and decides what to change.

Such implementation in Elixir lang: https://github.com/mustafaturan/event_bus/commit/08a955294c1d1c5bec2c7fef28ed255400e5f322

i2) Notify the Event Shadow to All Subscribers

img

img

Event Bus with Event Store and Event Watcher

Actors

Event Bus, Event Creators, Subscribers, Event Store, Event Watcher

What is Event Shadow?

Probably, you never heard this term before, do not worry I made it up. Event shadow is basically a reference data to your original event data.

Implementation

Whenever the event bus receives any event, it saves it to an Event Store. Then passes the event shadow to all subscribers. Subscribers process the event shadow and fetch the event data from Event Store when they start processing.

When an event comes to the event bus, it creates a watcher with the list of subscribers. Event Watcher is responsible for deletion of event data from the Event Store when all subscribers process the event.

Event bus does not care about if the receiver fails to receive the event message. But it automatically marks the Event Watcher as ‘skipped’ when any subscriber fails to receive event shadow.

Functions Needed

Notify, subscribe, unsubscribe, subscribers, save/delete/fetch_event_data, mark_as_completed/skipped are enough to implement such an event bus.

Goods

This approach comes with less memory consumption. It is possible to implement one more function to watch the current status of a specific event or all events.

Drawbacks

In this approach, each subscriber needs to call fetch_event_data, mark_as_completed, and mark_as_skipped functions. You need to implement a read-heavy event store. And you need to implement also an ‘event watcher’ to delete the event data when all subscribers processed the data. And sadly, event store write persistence should be blocking.

Like in the implementation i1, this implementation notify all subscriber without filtering. So, filtering should be done in subscribers. Which is not too bad, but if you are using the event bus over networks, this will cause unnecessary network traffic.

Such implementation in Elixir lang: https://github.com/mustafaturan/event_bus/releases/tag/v0.2.1

i3) Notify the Event Shadow to Filtered Subscribers

This implementation is same as the i2 but it is filtering the events before notifying the subscribers. In this approach, you need to register subscribers with their interests of topics. Note: It is good to have a regex filter on topics(event types) like in RabbitMQ.

It has the same goods and drawbacks as in the implementation i2, except i3 supports filtering on events.

Such implementation in Elixir lang: https://github.com/mustafaturan/event_bus

i4) Ordered Delivery to Subscribers

To guarantee the ordered event delivery, the idea is to partition your data before passing to interested in subscribers. The Kafka paper is a great way to start such an implementation. In basic, the producer decides for the event partition with a custom function like CRC32. And only one consumer is responsible only for that partition.

(Addition at 28 July 2017)In Kafka, consumers are subscribing to the given list of topics to get dynamically assigned partitions and then they poll the data from the topics or partitions specified using one of the subscribe/assign APIs.

Drawbacks

Increasing, decreasing the number of partition might be a pain. You need to implement your own partition function on the producer. The consumer implementation will be as hard as broker implementation.

Notes on Subscriber Implementations

In i1 and i2 implementations, all subscribers should match the event type(topic) first in a blocking manner to prevent unnecessary stacks in memory.

In i3 and i4 implementations, you should match event type(topic) in a non-blocking asynchronous way because only the subscribed topic data comes to the event processors.

In general, all subscribers should process the event asynchronously without any blocking manner. Also, subscribers can create new events and pass those events to the event bus.

In conclusion, event broker topology promises reactive, event driven, asynchronous systems. There are many implementation of event buses like AMQP (RabbitMQ is an implementation of AMQP), ZeroMQ and Kafka. There are all coming with different promises. For example, Kafka promises on the ‘guaranteed and ordered delivery’, AMQP promises on the ‘guaranteed delivery’ and ZeroMQ comes with patterns to develop your own custom broker.

If you are using Elixir or Erlang for the development, try the event_bus library which is an implementation of the i3.

Reference

https://medium.com/elixirlabs/event-bus-implementation-s-d2854a9fafd5

Implementing event-based communication between microservices (integration events)

Event Bus Implementation(s)

When you use event-based communication, a microservice publishes an event when something notable happens, such as when it updates a business entity. Other microservices subscribe to those events. When a microservice receives an event, it can update its own business entities, which might lead to more events being published. This is the essence of the eventual consistency concept. This publish/subscribe system is usually performed by using an implementation of an event bus. The event bus can be designed as an interface with the API needed to subscribe and unsubscribe to events and to publish events. It can also have one or more implementations based on any inter-process or messaging communication, such as a messaging queue or a service bus that supports asynchronous communication and a publish/subscribe model.

You can use events to implement business transactions that span multiple services, which gives you eventual consistency between those services. An eventually consistent transaction consists of a series of distributed actions. At each action, the microservice updates a business entity and publishes an event that triggers the next action. Figure 6-18 below, shows a PriceUpdated event published through and event bus, so the price update is propagated to the Basket and other microservices.

Diagram of asynchronous event-driven communication with an event bus.

Figure 6-18. Event-driven communication based on an event bus

This section describes how you can implement this type of communication with .NET by using a generic event bus interface, as shown in Figure 6-18. There are multiple potential implementations, each using a different technology or infrastructure such as RabbitMQ, Azure Service Bus, or any other third-party open-source or commercial service bus.

Using message brokers and services buses for production systems

As noted in the architecture section, you can choose from multiple messaging technologies for implementing your abstract event bus. But these technologies are at different levels. For instance, RabbitMQ, a messaging broker transport, is at a lower level than commercial products like Azure Service Bus, NServiceBus, MassTransit, or Brighter. Most of these products can work on top of either RabbitMQ or Azure Service Bus. Your choice of product depends on how many features and how much out-of-the-box scalability you need for your application.

For implementing just an event bus proof-of-concept for your development environment, as in the eShopOnContainers sample, a simple implementation on top of RabbitMQ running as a container might be enough. But for mission-critical and production systems that need high scalability, you might want to evaluate and use Azure Service Bus.

If you require high-level abstractions and richer features like Sagas for long-running processes that make distributed development easier, other commercial and open-source service buses like NServiceBus, MassTransit, and Brighter are worth evaluating. In this case, the abstractions and API to use would usually be directly the ones provided by those high-level service buses instead of your own abstractions (like the simple event bus abstractions provided at eShopOnContainers). For that matter, you can research the forked eShopOnContainers using NServiceBus (additional derived sample implemented by Particular Software).

Of course, you could always build your own service bus features on top of lower-level technologies like RabbitMQ and Docker, but the work needed to “reinvent the wheel” might be too costly for a custom enterprise application.

To reiterate: the sample event bus abstractions and implementation showcased in the eShopOnContainers sample are intended to be used only as a proof of concept. Once you have decided that you want to have asynchronous and event-driven communication, as explained in the current section, you should choose the service bus product that best fits your needs for production.

Integration events

Integration events are used for bringing domain state in sync across multiple microservices or external systems. This is done by publishing integration events outside the microservice. When an event is published to multiple receiver microservices (to as many microservices as are subscribed to the integration event), the appropriate event handler in each receiver microservice handles the event.

An integration event is basically a data-holding class, as in the following example:

C#Copy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class ProductPriceChangedIntegrationEvent : IntegrationEvent
{
    public int ProductId { get; private set; }
    public decimal NewPrice { get; private set; }
    public decimal OldPrice { get; private set; }

    public ProductPriceChangedIntegrationEvent(int productId, decimal newPrice,
        decimal oldPrice)
    {
        ProductId = productId;
        NewPrice = newPrice;
        OldPrice = oldPrice;
    }
}

The integration events can be defined at the application level of each microservice, so they are decoupled from other microservices, in a way comparable to how ViewModels are defined in the server and client. What is not recommended is sharing a common integration events library across multiple microservices; doing that would be coupling those microservices with a single event definition data library. You do not want to do that for the same reasons that you do not want to share a common domain model across multiple microservices: microservices must be completely autonomous.

There are only a few kinds of libraries you should share across microservices. One is libraries that are final application blocks, like the Event Bus client API, as in eShopOnContainers. Another is libraries that constitute tools that could also be shared as NuGet components, like JSON serializers.

The event bus

An event bus allows publish/subscribe-style communication between microservices without requiring the components to explicitly be aware of each other, as shown in Figure 6-19.

A diagram showing the basic publish/subscribe pattern.

Figure 6-19. Publish/subscribe basics with an event bus

The above diagram shows that microservice A publishes to Event Bus, which distributes to subscribing microservices B and C, without the publisher needing to know the subscribers. The event bus is related to the Observer pattern and the publish-subscribe pattern.

Observer pattern

In the Observer pattern, your primary object (known as the Observable) notifies other interested objects (known as Observers) with relevant information (events).

Publish/Subscribe (Pub/Sub) pattern

The purpose of the Publish/Subscribe pattern is the same as the Observer pattern: you want to notify other services when certain events take place. But there is an important difference between the Observer and Pub/Sub patterns. In the observer pattern, the broadcast is performed directly from the observable to the observers, so they “know” each other. But when using a Pub/Sub pattern, there is a third component, called broker or message broker or event bus, which is known by both the publisher and subscriber. Therefore, when using the Pub/Sub pattern the publisher and the subscribers are precisely decoupled thanks to the mentioned event bus or message broker.

The middleman or event bus

How do you achieve anonymity between publisher and subscriber? An easy way is let a middleman take care of all the communication. An event bus is one such middleman.

An event bus is typically composed of two parts:

  • The abstraction or interface.
  • One or more implementations.

In Figure 6-19 you can see how, from an application point of view, the event bus is nothing more than a Pub/Sub channel. The way you implement this asynchronous communication can vary. It can have multiple implementations so that you can swap between them, depending on the environment requirements (for example, production versus development environments).

In Figure 6-20 you can see an abstraction of an event bus with multiple implementations based on infrastructure messaging technologies like RabbitMQ, Azure Service Bus, or another event/message broker.

Diagram showing the addition of an event bus abstraction layer.

Figure 6- 20. Multiple implementations of an event bus

It’s good to have the event bus defined through an interface so it can be implemented with several technologies, like RabbitMQ Azure Service bus or others. However, and as mentioned previously, using your own abstractions (the event bus interface) is good only if you need basic event bus features supported by your abstractions. If you need richer service bus features, you should probably use the API and abstractions provided by your preferred commercial service bus instead of your own abstractions.

Defining an event bus interface

Let’s start with some implementation code for the event bus interface and possible implementations for exploration purposes. The interface should be generic and straightforward, as in the following interface.

C#Copy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public interface IEventBus
{
    void Publish(IntegrationEvent @event);

    void Subscribe<T, TH>()
        where T : IntegrationEvent
        where TH : IIntegrationEventHandler<T>;

    void SubscribeDynamic<TH>(string eventName)
        where TH : IDynamicIntegrationEventHandler;

    void UnsubscribeDynamic<TH>(string eventName)
        where TH : IDynamicIntegrationEventHandler;

    void Unsubscribe<T, TH>()
        where TH : IIntegrationEventHandler<T>
        where T : IntegrationEvent;
}

The Publish method is straightforward. The event bus will broadcast the integration event passed to it to any microservice, or even an external application, subscribed to that event. This method is used by the microservice that is publishing the event.

The Subscribe methods (you can have several implementations depending on the arguments) are used by the microservices that want to receive events. This method has two arguments. The first is the integration event to subscribe to (IntegrationEvent). The second argument is the integration event handler (or callback method), named IIntegrationEventHandler<T>, to be executed when the receiver microservice gets that integration event message.

Additional resources

Some production-ready messaging solutions:

Reference

https://docs.microsoft.com/en-us/dotnet/architecture/microservices/multi-container-microservice-net-applications/integration-event-based-microservice-communications

赏

Thank you for your candy!😃

扫一扫，分享到微信

微信分享二维码