Receiving data from Dotdigital Omnichannel using Python

This tutorial will show you how to receive data using Dotdigital Omnichannel's webhooks and Python. It covers off the basics for receiving and validating that the data has come from Dotdigital Omnichannel and has not been tampered with.

The tutorial's navigation controls are located above this text. Use to move to the next step of the tutorial. Use to move to the previous step of the tutorial (if there is one). Use to see an overview of all the steps in the tutorial.

To begin with you will need the following prerequisites installed:

• Python 3.x

Setup and configure the example web site

This tutorial solution is a simple web site created using Python's Flask framework that provides a POST URL for Dotdigital Omnichannel to forward events to via it's webhooks. The website also supports a GET request to allow for easy testing to ensure the web site is available.

To setup this project do the following:

• Clone the Github repo for the tutorial
• Change directory in the repo to the folder: OneAPI\Python\Webhook
• Run the command pip install Flask-API to install Flask

Next we will configure the web site for your webhook registration in your Dotdigital Omnichannel API Space

Setup your shared secret

When you setup a webhook registration with Dotdigital Omnichannel you will need to specify a secret which is a phrase or password used as the key when creating a HMAC SHA-1 hash for your forwarded notifications.

You need to update the code where it has >>>YOUR SECRET<<< to match the secret you want to use in your webhook registration, so that the receiving page can validate any data received by comaring the hash it generates with that Dotdigital Omnichannel set and stored in the X-Comapi-Signature HTTP header.

If the HMAC hash does not match then the data should not be trusted and rejected with a HTTP 401 - Unauthorized response.

Hosting your webhook page

You must now deploy the webhook website to a server and ensure it is publicly accessible.

Good options for this are:

• ngrok, allows you to run locally for development and open a publicly accessible tunnel with SSL support to that local site
• Azure, as trial accounts are free
• Heroku, allows easy creation of web containers with hosting and SSL support

Test by browsing to the webhook page which should return a basic web page, and make a note of your URL so it can be configured as a webhook URL.

Locally the webhook page will be: http://localhost:port/webhook , note change the port in the URL to your local hosts port number.

We will use Heroku for the tutorial, as accounts are free and you can spin up a Java Spring framework project in minutes, see below for instructions.

Hosting in Heroku

To setup the webhook quick start in Heroku for free follow these instructions:

• Sign-up for a free account with Heroku

• Download and install Heroku
• Open a CLI console
• Ensure Heroku has your credentials by running the following command and entering them:

heroku login

• Clone the following Github repo:

git clone https://github.com/dotmailer/ec-cpaas-quickstarts

• Change directory to your cloned repo folder
• Change directory to the sub folder OneAPI/Python/Webhook
• Create a Heroku app by running:

git init
heroku create

• Update the file webhook.py where it has >>>YOUR SECRET<<< to match the secret you want to use in your webhook registration
• Publish the code to Heroku with these commands:

git add .
git commit -m "Updated secret"
git push heroku master

• Now test the deployment worked by running:

heroku open

• If you have issues you can see the logs by running:

heroku logs

Running locally

You can run the project locally by following these steps:

• Open the project in your IDE
• Build to install dependencies
• Click the run button in the IDE

If all has worked you can browse to http://127.0.0.1:3000/ and the test page for the webhook will be served.

Alternatively you can run in Heroku by running:

heroku local

Again if all has worked you can browse to http://127.0.0.1:3000/ and the test page for the webhook will be served.

All done

Configuring your SSL certificates (Optional)

Dotdigital Omnichannel will forward data to both HTTP and HTTPS URLs. The tutorial is configured to use HTTP but for production HTTPS is recommended. You can use a 3rd party host such as Heroku who will do SSL offloading for you.

Any SSL certificates used need to be valid for the URL your webhook site is hosted at, and issued by a public certificate authority such as Verisign.

Registering your webhook

To register or modify a webhook registration you will need to call our Webhook service. We detail how to do this here and to make this easier we recommend using Postman which is a great tool for exploring web APIs.

When registering your webhook for this tutorial we will be subscribing to the message events so you can receive receipts and inbound messages via the webhooks. An example request to register a webhook with the message events is as follows:

 {
   "name": "test-webhook",
   "url": "https://webhooks.acme.com",
   "secret": "a strong secret",
   "subscriptions": [
     {
       "type": "message.sent",
       "filters": []
     },
     {
       "type": "message.delivered",
       "filters": []
     },
     {
       "type": "message.read",
       "filters": []
     },
     {
       "type": "message.expired",
       "filters": []
     },
     {
       "type": "message.failed",
       "filters": []
     },
     {
       "type": "message.inbound",
       "filters": []
     }
   ]
 }

Testing the webhook

For this test please ensure you have ticked the message.sent event on your webhook subscription.

We need to send a message via the Omnichannel API to trigger a message.sent event to be sent to our webhook. To do this you can either use a tool such as Postman and our API docs or follow our simple SMS sending tutorial.

Once you have sent your message check the console output on your server hosting your webhook page; you should see the event data similar to the image below. If you used Heroku then you can do this by running:

heroku logs --tail

Understanding the code

Exposing the raw body data

In order to calculate the HMAC SHA-1 hash to compare with one passed in the x-comapi-signature header we need to have access to the raw HTTP body data. To do this we read the body stream to a string, whilst UTF-8 decoding it.

The highlighted code perfroms this task.

Adding a GET handler

Officially Dotdigital Omnichannel only requires your web page to handle HTTP POSTs, but to make it easier to test whether your webhook page is available we have added a basic HTTP GET handler that just returns a basic static web page.

Adding a POST handler

This is the web method that is used to receive the forwarded events from Dotdigital Omnichannel in JSON format. It job is:

• Validate that the data is from Dotdigital Omnichannel and hasn't been tampered with using the HMAC validation
• Store the received event for processing ayschronously to ensure the fastest response possible
• Acknowledge the data receipt within 10 seconds, otherwise Dotdigital Omnichannel will assume a timeout

Note: We strongly advise doing any data processing of events asynchronously to ensure that the data can be passed to your systems as fast a possible. Suitable technologies to use for passing the received data into system are queues and distributed caches such as RabbitMQ, MSMQ and Redis.

HMAC Validation

Dotdigital Omnichannel uses HMAC SHA-1 validation to ensure you can tell that data has come from Dotdigital Omnichannel and hasn't been tampered with. This is achieved by Dotdigital Omnichannel creating a hash value using the HMAC SHA-1 algorithm using a secret password or phrase as the encryption key. This secret key can be anything you like and must be configured against your webhook registration.

The highlighted code takes the hash values from the HTTP header x-comapi-signature and then calculates an equivalent hash using the HMAC SHA-1 algorithm with the raw body data and the same secret key you entered in the webhook registration. The two hash values will be identical if the data came from Dotdigital Omnichannel and hasn't been tampered with.

If the hash values do not match, do not accept the data and return a HTTP 401 Unauthorized return code. If the values match you can trust the data is from Dotdigital Omnichannel and hasn't been tampered with, and continue to store it for processing.

Event storage

In this basic example code we are simply dumping the event data to log file in the systems temp folder, but for real implementations you would store the data for processing in a queue or distributed cache typically like RabbitMQ, MSMQ or Redis.

The queue processors will be responsible for checking the revision property of the events to see if the event should be discarded or not as it is no longer valid. A good example of this would to track the messageId and revision when receiving receipts for message sends, as messages go through multiple statuses such as: sent to delivered to read, and as the ordering of these events cannot be guarantied the revision property can be used to ensure obsolete events can be recognized. Find out more about revision processing and deduping in our docs here.

In the message receipt use case the revision property is guarantied to be higher in later events, so the revision property value for the read event would be higher than the revision property values on the sent and delivered events.

All done

Thanks for taking the time to look through the tutorial, to find out more visit our full documentation.