Building the AppNexus Alert Hub

Or, finally Retiring Smokey the Bear...

The AppNexus Alert Hub was born from an idea which a colleague of mine had over 2 years ago: What if we could utilise our Public Service Announcements (PSA) system for more than just serving “Smokey the Bear”-esque ads? Instead, could we use it to distribute critical public safety alerts to the right people in the right place at the right time?

In simpler terms, could we go from something like this:

Smokey the Bear.

To something like this:

The current dynamic public safety alert.

The answer, of course, is yes!

In order to realise this vision, we built a direct integration with the Federation for Internet Alerts (FIA). The Federation for Internet Alerts is a non-profit organization that collects public safety alerts from governments and other organisations around the globe, using the internet to aggregate and disseminate these alerts in real time. The FIA would be our source of alerts, and the integration would provide a direct link between these alerts and AppNexus’ global advertising platform.

There were three challenges that the new application had to solve:

  1. The application had to easily allow lots of different alert types to be processed for any given alert provider. For example, from the National Weather Service we can receive anything from a variety of Weather Alerts, to Air Quality Alerts, to Local Civil Emergencies that we need to broadcast correctly.

  2. The application had to easily allow lots of different alert providers to integrate. This is made somewhat easier by the FIA’s usage of the CAP format (the Common Alerting Protocol). However, organizations sometimes interpret the format differently resulting in very different message formats!

  3. The application had to easily scale to handle the volume of incoming alerts. We had to reconcile the time to process a new alert on our platform (typically 5-10 seconds) with the variable nature of the incoming alert feeds. Usually the feed was a slow drip, but it could at peak spike to 10-15 alerts per second or more.

To tackle these issues, we architected the system to have an asynchronous Receiver and Processor system separated by a task queue. This design allowed us to have:

  1. Modularity (to deal with different alert provider CAP formats).
  2. Configurability (to allow the system to easily handle new alert types).
  3. Scaleability (to tackle the very "spikey" alert volume).

The AppNexus Alert Hub architecture

For the receiver, we used Flask with Apache to provide a lightweight endpoint for the FIA to push to. The receiver is also responsible for translating the incoming format into our version of the CAP data structure which all downstream processes then rely on. For each new alert provider, we connect a custom ingestion module which provides the translation of the format.

After ingestion, the alert is placed onto the message queue (RabbitMQ), allowing the receiver to return a success acknowledgement to the FIA (a success acknowledgement is important, as the FIA will attempt to re-deliver failed alerts).

The worker takes tasks off the task queue to process. Its job is to create the creatives and campaigns within our platform which will then be responsible for serving alerts. It looks up the alert type configuration, which describes the template to use, text placement and text style. It then generates the correct HTML5 and Javascript snippets to serve as a creative. These elements are configurable so that it is simple to add new alert types or creatives. Therefore, activating a new alerting service becomes as easy as plugging in a new ingestion module and adding new configuration rows!

The processor generates campaigns using the AppNexus Console APIs (read our API documentation for more information about that process).

After the campaign is created, our ad serving platform takes over. Alerts are delivered like a standard advertising campaign targeted to the people impacted. Instead of seeing Smokey’s stern warning about preventing forest fires, a woman in Omaha could be warned of a tornado threat or a family in Boston could be alerted to impending blizzard conditions.

Since beginning the beta test in December, the system has received approximately 100,000 alerts, and actioned just over 20,000 of them. This has resulted in us serving over 200 million alert impressions and garnering over 60,000 clicks thus far! You can find the latest activity stats on FIA’s website.

Alerting map of the US

We also have exciting plans to expand functionality: we will soon be releasing alerting capabilities with Environment Canada, and will upgrade our geo-targeting to reach mobile devices in arbitrary areas defined by latitude / longitude polygons. We are also looking at ways to open source as much of the application as possible – because we believe in an open and free internet, but also because we hope it will lower the barrier to entry for other entities to get involved. For instance, we have already released a US ZIP code to US FIPS6 mapping script that anyone can use.

Speaking from a personal perspective, using technology in this way to make an impact has been and continues to be one of the most rewarding career experiences I have had to date, and I am grateful that we have the kind of open and encouraging culture at AppNexus that gives “spare-time” projects like these room to grow and thrive and become something. As someone who isn’t a software engineer by training or trade, working on this project has also once again highlighted the generosity of my fellow AppNexians who were happy to give time, knowledge and guidance to help make it a success.

Of course, no project of this scale gets to where it is based on a one-man effort, and so there are a number of people I would like to give thanks to. Firstly, the colleague who pioneered and prototyped the idea of dynamic PSAs at AppNexus, Simon Kempner. The folks at the FIA – especially Jason Bier and Mickey Schwab, for their enthusiasm and partnership. Fellow developer Greg Field for his vision of the current integration, invaluable code contributions and effort without which the project would very much be dead, and last but not least, David Greenhouse for continuing to run the ‘Technology for Good’ group within AppNexus, of which this project is but a part.

George Zhao’s day job is as a Client Insights Analyst, responsible for performing big data analytics and using the resulting data driven insights to guide clients and internal stakeholders. By night, he is the Lead Developer on the AppNexus Alert Hub project.

George Zhao

Read more posts by this author.

Subscribe to AppNexus Developer Blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!