For the past couple of years I have been playing around with the idea of a hard divide between outgoing communication (transactional emails, push notifications, newsletter contact list synchronization, marketing automation, slack notifications, etc) and the business logic of my applications.
It never made sense to me that a backend system that manages ticket sales for an should would also implement a mailer queue; but also it doesn’t make sense that the copywriter who wants to change the content of an email would have to talk to a developer to do so.
In 2017 I started working on removing all logic for these kind of actions from my websites and replace them with a single event driven microservice that would handle anything related to outgoing communication for all projects I build.
My goals where to:
- Be vendor independent: Services like Mailchimp, Moosend, Sendpulse etc have basically already built what I wanted to have, but given my DIY nature I wanted to stay independent from them.
- Base all actions from a stream of events sent by the business logic.
- Manage all GDPR related permissions and preferences from a central location.
- Cut all links between the app domain and this microservice: no accessing data that has not been provided. Push only.
- Be able to synchronize lists of users (= segments) according to relationships and synchronize these to newsletter services. (ie: automatically make an email list of all registered users who are attending a certain quiz).
- Provide a management panel where non coders could change the content and the logic of the outgoing notifications.
That is what I started building. I even setup a website to ask feedback from smarter people to make sure I wasn’t reinventing a square wheel, but in the end I kept building nevertheless: Eukles was born.
The Slack notification
I started with our ticking website CatLab Events, and my first goal was pretty modest: send a slack notification to our channel every time someone bought a ticket for one of our quizzes. I setup a MySQL database (probably not the best fit, but as I wanted quick results I stayed with the tools I knew best) and a Laravel/Charon REST API and also created a separate API client package for PHP to easily implement in existing projects. A few days later I registered my first event:
/**
* @param OrderConfirmed $e
* @throws \GuzzleHttp\Exception\GuzzleException
*/
public function onOrderConfirmed(OrderConfirmed $e)
{
$order = $e->order;
$attributes = [
'event' => $order->event,
'order' => $order,
'ticketCategory' => $order->ticketCategory
];
if ($order->group) {
$attributes['group'] = $order->group;
$attributes['subject'] = $order->group;
} else {
$attributes['subject'] = $order->user;
}
// Track on ze eukles.
$euklesEvent = \Eukles::createEvent(
'event.order.confirmed',
$attributes
);
$euklesEvent->unlink($order->user, 'registering', $order->event);
if ($order->group) {
$euklesEvent->link($order->group, 'attends', $order->event);
foreach ($order->group->members as $member) {
if ($member->user) {
$euklesEvent->setObject('member', $member->user);
}
}
}
\Eukles::trackEvent($euklesEvent);
}To break it down: what I wanted to communicate to Eukles was:
- An event.order.confirmed event has happened
- This event has some related properties (event, order, ticketCategory, group and subject (which can be user or group, depending on the event type).
- Unlink the ‘registering’ relationship between the active user and the event (= the user is not registering anymore, as the registration is already done). The idea is that these relationship actions could be defined either in the business logic or triggers on Eukles itself.
- Link the ‘attends’ relationships between the group and the event. This would allow us to create a segments of all groups that attend a specific event and synchronize that to Mailchimp later.
- Add all members of the group as properties of the event (= when a group has 4 members, send an email to every single one of them)
(Note that all models used as properties would have to implement an ‘EuklesModel’ interface that included methods to serialize the content that is send to Eukles.)
The content of the event properties is stored in two places:
- As part of the event, so that we always know the state of a model at the time of the event creation.
- (If a unique identifier is provided) the model it is also synchronized with a centralized project-model database that contains the relationships between project-models and always contains the last known properties. Every time an event is sent, the content of these models get updated.
Now I just had to add a SlackService, store authentication details for these services in the database and setup a few Action and Trigger models that would listen for the event and cary out actions. And using the Jobs queue in Laravel everything was handled without impacting the user flow.
To keep things simple I used Laravels Mustache syntax to parse the arguments that were used by the service clients.
GDPR consents
As Eukles would become responsible for all outgoing communication, I found it fitting to store communication preferences and consents on Eukles instead of in the business logic. Eukles doesn’t really have the concept of a ‘user’, as user project models are sent in events and follow the same logic as any other event property. So I created an API endpoint that would show me all consents in a given project by a given project model. By adding this call to my registration process I was able to ask all users that logged in (existing or new) to give the required and optional consents.
By adding these as filters to segments and actions, I was able to block communications to users who opted out of these messages.
Segments
The goal of Segments was to make dynamic or static lists of project models and synchronize those lists with newsletter services like Mailchimp. For example: by defining segment rules and setting a parent model in the segment schema, Eukles is able to create a segment for each ‘event’ projectmodel, containing all ‘user’ projectmodels that have the ‘attending’ relationship set to it AND have given consent to receive newsletters.
Once every hour these segments are then synchronized with Moosend or Mailchimp, and whoever is in charge of sending out the newsletters can rely on those lists being up to date at all times.
This synchronization also checks for users who are marked as ‘unsubscribed’ in Mailchimp, and the applicable consent is removed from those specific users, resulting in remove them from that segment.
Templates
As all emails in QuizWitz use the same basic template, I have added the ability to define mustache-like ‘templates’ for each project. Anywhere parameters are parsed, these templates can be used (but they only really make sense in html-like contexts)
Wait 1 hour. if !subject.hasLicense(event.license): then: send email.
And that is pretty much where the project stayed for a few years. My simple event -> trigger -> action model worked fine for sending transactions emails, notify us about certain events and even sending tweets with giphies based on event attributes (implementing services is trivial since the base architecture for defining actions and storing credentials is already provided).
All was well in Eukles world… until I found myself with a bit of time on my hands and I decided to implement some dark design patterns to try to raise the conversion rates in QuizWitz: when someone initalizes a license purchase but in the end decides not to complete the purchase, send them an email asking for feedback.
A perfect use-case for Eukles, where it not that I had gone for the quick event -> trigger -> actions design instead of a fully fledged ‘workflow’ state machine. So that is what I started building.
Instead of triggering an action, all events now check for ‘workflow triggers’ that initializes states in the workflow machine. Depending on the trigger configuration, Eukles also checks for duplicates to make sure that recurring events don’t cause too much triggers. A state then follows the configurable steps in the workflow and triggers actions that are defined in steps. Each trigger also has the ability to set one of the event properties as ‘subject’ that can be used anywhere in the workflow, to reduce the dependency on the initial event.
A simple shunting yard algorithm offers me the ability to define conditions in string format and makes it possible to setup decision trees in the workflow, while a special ‘wait’ step adds a delay to the state machine processing. The whole thing is using Laravel Job queues, so it was all fairly trivial to setup.
The table based interface is obviously a horrible choice for setting up workflows like this, but as for the moment I am the only one using this, it will do. No need to pour more hours in things that only marginally improve my life 😉
And that basically is the current state of the Eukles project. My goal is to clean it up and release it under a GPL license in the future, but in order to do that I first want to improve the admin panel interface and write down some documentation. If that ever happens, I’ll definitely post it here.
