Event data pipeline

The UDP's event data pipeline accepts and transforms event data in the IMS Global Caliper format. The event data pipeline aggregates, computes, and stores event-data and metrics in the UDP's Event store and data marts through a set of loosely-coupled application and infrastructure components.

The ingress point for behavioral data is the UDP Caliper endpoint, a web service that accepts and authenticates Caliper events posted over the web.

Once authenticated and accepted by the UDP Caliper endpoint, events are enriched via the UDP Enricher. The enrichment process incorporates data sourced from the Context store into the event payload. This enables forms of downstream event-data processing that are not otherwise possible or efficient with the data native to an event.

Once enriched, events are routed to various messaging queues (Google PubSub topics) where event-processing services consume events to create and maintain the UDP's Event store and event-based data marts.

The UDP Event store is a date-partitioned Google BigQuery table that serves as the archive for all-time behavioral data. Alongside the UDP Event store are behavioral- and context-derived data marts.

The key components of the Event data pipeline are:

1. UDP Caliper endpoint, which accepts, authenticates, and routes Caliper events from learning tools

2. UDP Event enricher, which adds cached context data to streaming behavioral data

3. UDP Event store, the data stores that undergird the UDP's data services