Unizin Product Documentation
ProductsSupport and TrainingPolicies
  • Unizin Product Documentation
  • Products
    • Content
      • Unizin Engage
        • eReader User Guide
          • Notes, Highlights, and Citations
          • Appearance Settings
          • Download for Offline
          • eReader Layout
          • Keyboard Shortcuts
          • Navigating Your eBook
          • Print
          • Text to Speech
          • Copy and Paste
          • Creating Flashcards
          • Collaboration and Note Sharing
          • Pearson Titles
        • Institution Support
          • Disabled Student Services / Alt-Format
            • Best Practices for Republishing Course Content
            • Disabled Student Services
            • Requesting eTextbook Files for Accessibility Purposes
            • WCAG 2.0 AA evaluation for Engage
            • WCAG 2.0 AA evaluation for EPUB for Engage
          • Institution's Support Responsibilities
        • Caliper 1.1 sensor
        • Release Notes
          • 2.28.22
          • 2019-09-17
          • 2019-05-29
          • 2.26.8
          • 2.26.0
          • 2.25.0
          • 2.22.0
          • 2.21.6
          • 2.21.5
          • 2.20.8
          • 2.20.5
          • 2.20.3
          • 2.19.1
          • 2.18.0
          • 2.17.0
          • 2.14.0
          • 2.12.0
          • 2.11.0
          • 2.9.0
          • 2.8.3
          • 2016-03-17
          • 2016-02-11
          • 2016-01-28
        • Using Analytics (New)
      • Unizin Order Tool
        • Overview of the User Interface
        • Key Concepts
          • Profiles
          • Ordering periods
          • Coordinator permissions
          • Program administrator permissions
        • Courses & Ordering
          • Course filtering
          • Place an order
          • Add sections to a placed order
          • Edit a placed order
          • Cancel an order
          • Reordering
        • Order History
          • Instructor Order History
          • Coordinator and Program Administrator Order History
          • Order Activity
        • Student Choice
          • Student Choice (Program Administrators)
          • Student Choice (Students)
        • Entitlements
          • Entitlements (Program Administrators)
          • Entitlements (Students)
        • Catalog Tool
        • Schedule of Classes
        • Content Request Tool
        • Order Tool Dashboard
        • Vendor Sandbox Tenant
        • Institution Support
        • Implementation
          • SIS Data Integration
            • 1.0 - SIS Integration
            • 2.0 - SIS Integration
          • SSO integration
          • UI customizations
          • Order Feed
            • 1.0 - Order Feed
            • 2.0 - Order Feed
            • 3.0 - Order Feed
          • Publisher report
          • Final declined offers feed
          • Institutional (SIS) Catalog Import
          • Student Price
          • Historical Entitlements Import
        • Release Notes
          • Order Tool Bug Fixes and Enhancements
          • Order Tool Bug Fixes
          • Order Tool Accessibility Improvements
          • Order Feed Improvements
          • Content Request Form Update and Minor Bug Fix
          • Flat Markup Fee Update
          • Ordering Email Receipt Update & Minor Bug Fix
          • Bug Fix for Public Catalog Feature
          • Catalog Search Enhancements
          • Reordering Reminder Email Notifications
          • UX Improvements & Minor Bug Fixes
          • Historical Entitlements Import
          • Student Prices
          • Reordering Feature
          • Email Enhancements
          • Ordering Enhancements
          • Bug Fix for the Institutional Catalog Import
          • Bug Fix for the Final Declined Offers Feed (FDOF)
          • Order Activity Feature and Other Enhancements
          • Bug Fixes for Order History and Report an Issue Features
          • Public catalog feature
          • Minor Bug Fixes for Ordering and Student Choice
          • Entitlements Production Release, Bug fixes, and Minor updates
          • Minor Updates and Bug Fixes for Ordering Workflows
          • Catalog Search Optimization
          • Student Choice
          • Archive Terms Feature and Integration Improvements
          • Introduces the Program Administrator role, Catalog Tool, and Schedule
          • User interface updates and improvements
          • Order feed improvements
          • Order history, UI enhancements
          • Email notification upgrades, UI improvements
          • Order feed changes
          • New features for Course coordinators and upgrades to the UI
          • Changes to the Term, Course, and Section models; introduces a Session
          • Bug fixes, import improvements, and validation improvements
          • Tracking Order History
          • Publisher Reporting
          • Fixes the order feed, automates SIS data importing, and automates the generation of order feed repor
    • Data & Analytics
      • Unizin Data Platform
        • Key concepts
          • Platform overview
          • Data categories
          • Data models
          • Loading schemas
          • Keymap
        • Unizin Common Data Model
          • Academic structures (ERD)
          • Learners (ERD)
          • Course structures (ERD)
          • Course resources (ERD)
          • Learner activities (ERD)
          • Quizzes (ERD)
          • Social (ERD)
          • Course outcomes (ERD)
        • System overview
          • Context data pipeline
            • Context data ingress
            • Batch-ingest application
            • Batch-ingest db server
            • Context store
          • Event data pipeline
            • UDP Caliper endpoint
            • Approval process for implementing Caliper compliant tools
            • UDP Event enricher
            • Event store
        • Data stores
          • Data lake
            • UDP Context store
            • UDP Event store
              • Accessing the Event store
              • Expanded table
                • Expanded table: Canvas edApp mapping
            • Synthetic Data [beta]
              • Viewing Synthetic Data datasets within the BigQuery UI
              • Query Synthetic Data via client libraries
          • Data marts
            • UDP Distributions
            • Interaction sessions
            • Learning Environment Organization
            • File Interaction
            • Last Activity
            • Long Inactivity
            • Course Status
            • Daily Course Grade Record
            • LTI Tool Use
            • LMS Tool Use
            • Tool Usage Metrics
            • Links
            • Taskforce
              • Level 1 Aggregated
              • Level 2 Aggregated
              • Level 2 Course Weekly Distribution Summary
              • Student Term Profile
              • Course Profile
            • Student Activity Score
              • Student Course Metrics
              • Student Course Section Metrics
              • Final
              • Course Final
              • Course Section Final
        • Data integrations
          • Context data integration
            • Loading schema
            • Keymap support
            • Manifest file
            • File requirements
            • Integration mechanics
          • Event data integration
          • SIS data integration
          • LMS data integration
            • Instructure Canvas
        • Release Notes
          • UDP Marts Release Notes
            • 1.0.83
            • 1.0.80
            • 1.0.79
            • 1.0.78
            • 1.0.77
            • 1.0.72
            • 1.0.67
            • 1.0.58
            • 1.0.51
            • 1.0.44
            • 1.0.42
            • 1.0.32
            • 1.0.31
            • 1.0.0
            • Level 2 Taskforce data marts now available
          • 2.0.167
          • 2.0.152
          • 2.0.138
          • 2.0.137
          • 2.0.113
          • 2.0.112
          • 2.0.111
          • 2.0.110
          • 2.0.99
          • 2.0.98
          • 2.0.83
          • 2.0.80
          • 2.0.71
          • 2.0.66
          • 2.0.59
          • 2.0.58
          • 2.0.53
          • 2.0.47
          • 2.0.25
        • Miscellaneous
          • Canvas Data additions, ~Fall 2021
          • Canvas Live Events: from SQS to HTTPS
          • Canvas New Analytics vs. UDP
          • Course Section Enrollment Role Status Mappings
          • Migrating from UDW to UDP
      • Unizin Data Warehouse
        • Implementation Guide
        • Scope of Services
        • Access Provisioning
        • Access Revocation
        • Connecting to the UDW
      • Raw Canvas Data 2
        • Flat Files
        • BigQuery Datasets
    • Hosted Services
      • My Learning Analytics
        • Install MyLA via LTI 1.3
        • Custom configure MyLA
  • Support and Training
    • Professional Development
      • Stepping Stones: A Faculty Development Curriculum for Learning Analytics Use
      • Structured Conversations initiative
    • UDP Self-paced Training
    • Resources Site Broken Links
    • Status Pages
  • Policies
    • General policies
      • Sponsor Teams
      • Browser Support Policy
      • Opt-Out & Invoicing Policy (Order Tool)
    • Support Policy
      • Unizin Engage - SP
      • Unizin Order Tool - SP
      • Unizin Data Platform - SP
      • Unizin Data Warehouse - SP
      • Unizin Data Analysis - SP
      • Pressbooks Hosting - SP
    • Privacy Policy
      • Unizin Engage - PP
      • Unizin Order Tool - PP
      • Unizin Data Platform - PP
      • RStudio service - PP
    • End User License Agreements
      • Unizin Engage - EULA
      • Unizin Order Tool - EULA
    • Terms of Use
      • Unizin Data Platform - ToU
    • Incident Reports
Powered by GitBook
LogoLogo

Unizin Homepage

  • unizin.org

Data & Analytics

  • Unizin Data Platform
  • Unizin Data Warehouse

Content

  • Unizin Engage
  • Unizin Order Tool

Hosted Services

  • My Learning Analytics

Copyright © 2023, Unizin, Ltd.

On this page
  • Schemas
  • Entity schema
  • Keymap schema
  • Zero downtime updates (via "publisher" schemas)
  1. Products
  2. Data & Analytics
  3. Unizin Data Platform
  4. System overview
  5. Context data pipeline

Context store

PreviousBatch-ingest db serverNextEvent data pipeline

Last updated 1 year ago

In the final phase of the UDP's Context data pipeline, the UDP Context store is updated to present newly imported context data. The UDP Context store serves as an aggregate, consolidated data store for all context data in a UDP instance.

The context store database is the database resource to which institutions will connect to use context data, including the keymap.

The purpose of the context_store database is to present the keymap and entity data that is the result of the context data ETL process. The context_store database must also be updated at regular, periodic intervals (usually once daily) without interrupting current connections to the database. Both of these tasks are accomplished through a small set of schemas.

The context store database has a relatively simple design to reflect its single function: to present the latest version of ingested context data to institutions.

However, there is some nuance in the design to enable the zero-downtime replication process that is part of the “publish” phase of the ETL.

Schemas

The two primary database schemas (namespaces) of the context store are entity and keymap. Each schema is used to present the two essential parts of the UDP Context store:

  1. The consolidated, normalized, entity-by-entity context data in a relational schema, which is housed in the entity schema.

  2. The entity-by-entity keymaps that unify source data identifiers with UDP identifiers, which are housed in the keymap schema.

In the sections below, we briefly describe the organization of views and data in these schemas. For a fuller description of how to get started using the UDP Context data, see our .

Entity schema

The entity schema of the "context_store" database contains all context data from all data sources.

The schema is composed of views (not tables), where each view corresponds to a distinct entity of the Unizin Common Data Model. For example, the "person" view corresponds to the Person entity; the "academic_organization" view corresponds to the Academic organization entity; and, the "person__academic_degree" view corresponds to the Person-Academic degree entity.

Singular UCDM entities

For views in the entity schema that corresponds to a singular entity, a primary key (which corresponds to a UDP ID key in the keymap) is provided whose name corresponds to the entity. For example, the primary key for a Person record in the "person" view is "person_id;" the primary key for an Academic organization record in the "academic_organization" view is "academic_organization_id."

Composite UCDM entities

Views in the entity schema that correspond to a composite entity do not have primary keys. For example, Person-Academic degree records in the "person__academic_degree" table do not have a primary key. Instead, each record has both a "person_id" and an "academic_degree_id," which jointly define a unique record.

Keymap schema

The keymap schema of the "context_store" database contains all of the entity keymaps in a UDP instance.

The schema is composed of a set of views (not tables), where each view corresponds to a distinct, singular UCDM entity. For example, the "person" view corresponds to the Person entity. In contrast to the entity schema, where every UCDM Entity is represented, the keymap schema only contains views for singular UCDM entities. Composite UCDM entities do not have keymap tables, since no UDP IDs are ever required to define them. Instead, composite UCDM entities are composed of the UDP IDs of their composite entities.

All keymap views have the same definition. The first column in the view corresponds to primary key for a singular UCDM entity record. Every subsequent column corresponds to the native identifies associated with the UDP from a native data system. In the diagram below, for example, we see a keymap view where native identifies from the SIS, LMS, and a learning tool are associated with a particular primary key. As context data from different systems is added to a UDP instance, the keymap tables widen to accommodate more native keys from each source data system.

Zero downtime updates (via "publisher" schemas)

One of the features of the UDP batch-ingest process is the zero-downtime update of the UDP Context store.

Zero downtime is achieved by maintaining two sets of context stores behind the scenes, one of which serves the views of the entity and keymap schemas, the other that is updated during the next run of the batch-ingest DAG. When an import process is complete, the views in the entity and keymap schemas are pointed to the newly-updated tables.

The swap between two sets of the context store is achieved through what's called the "publisher" schemas of the UDP Context store. These schemas also contain tables that are definitionally equivalent to the entity and keymap schemas of the "context_store" database. However, for each entity, two tables are defined in the publisher_keymap and publisher_entity schemas. They are the A and B tables. For example, the Person entity tables in the “publisher_entity” schema are called “person__A” and “person__B.” Note that in both of these cases, the “person” and the identifier are separated by double-underscores. This is part of the naming convention.

The UDP keeps track of which publisher tables –A or B– are associated with a corresponding entity or keymap view. In the publish step of the context data ETL, the UDP will replicate the newly ingested data into the publisher table that is not currently used by the entity and keymap schemas. Once the replication of an entity’s data in the publisher tables is complete, the UDP will redefine the corresponding entity view in either the keymap or entity schema to point to the table – A or B – with the freshest data.

For a comprehensive overview of the UDP Keymap concept, .

primer on using the UDP Context store
please review our keymap documentation
A brief description of the Person table definition, highlighting that singular Entities are defined by a primary key.
A brief description of the Person-Academic degree table definition, highlighting that composite Entities are defined by two or more UDP IDs of distinct entities.
An example keymap table definition, highlighting that a surrogate identifier (i.e., a UDP ID) maintains relationships with native identifiers from source data systems such as the SIS and LMS