This topic applies to ANGEL administrators who are migrating to Learn, have implemented an XEI integration, and are interested in utilizing the SIS Framework for their Learn integration. This topic provides an overview of the similarities and differences between XEI and the SIS Framework with a specific focus on the Snapshot Flat File integration type.
There are many similarities between ANGEL and Learn in how SIS integration is executed. There are however some key differentiators that must be considered when making the move. The SIS Framework and Snapshot Flat File are designed to facilitate reuse of existing SIS data extraction processes as much as possible to reduce the effort required to migrate your integration from ANGEL to Learn.
ANGEL integration summary
ANGEL's Extended Enterprise Integration (XEI) agent framework automates the interchange of data from your Student Information System (SIS) database to ANGEL. Using one of three available XEI Agents, XEI reduces integration costs and improves reliability and accuracy by eliminating manual data entry. It reduces operating costs by enabling integration to ANGEL from more than one data source, automatically keeping your databases in synch, and tracking error messages for easier debugging. Furthermore, XEI reduces maintenance costs and disruption as ANGEL and your other databases change.
With XEI, course and user data used in ANGEL, including account data and course enrollment drops and adds, is automatically exchanged with the SIS database (i.e., student information system, Datatel, SCT Banner, Peoplesoft, or other ERP system).
Figure 1. ANGEL XEI Diagram
The XEI process can also be fully automated to execute on specified intervals. Each unique XEI mapping created in ANGEL is defined as an XEI Agent. When executed, the XEI Agent performs the specified tasks as outlined in the agent. XEI is a one way data feed from an SIS.
Review SIS Framework Overview before continuing.
Blackboard Learn SIS Framework summary
Blackboard modeled the SIS Framework after XEI, thus it serves the same purpose for enabling the integration of your SIS to Learn with the same benefits offered by XEI. The SIS Framework supports event driven real-time data integration and file-based models (see below section: ODBC vs. Flat File) allowing multiple simultaneous active integrations. The SIS Framework also supports in UI creation, configuration, and monitoring of integrations.
Figure 2. Learn SIS Framework Diagram.
Finally, the SIS Framework supports multiple integration types:
- LIS 2.0: Learner Information Services, the IMS standard for event driven SIS integration
- Snapshot Flat File: Learn defined file-based data format
- Enterprise 1.1: Legacy IMS integration type
- Snapshot XML: Legacy Learn defined data format modeled after Enterprise 1.1
More on the SIS Framework and the supported integration
As you develop your integration strategy it is in your interest to follow the LIS or Snapshot Flat File integration models as these are the forward looking integration types, support current data objects as added to Learn, and will receive object support updates as required. Enterprise 1.1 and Snapshot XML are intended to support in-field need and will not receive object support updates in the future.
System requirements
The Learn SIS Framework is delivered as a core component of Learn starting with 9.1 SP6-SP8 added Snapshot Flat File capabilities.
Blackboard recommends that 9.1 SP 8 be considered the minimal target Learn version for implementing your SIS integration as it offers the greatest level of choice in integration types. It is further recommended that 9.1 SP12 be considered the production target as SP12 greatly improves on the ability of multiple integrations to manage shared data in Learn.
Review SIS Framework Overview for the latest vendor versions required to support your data integration choice.
ODBC/Messages vs. Flat File
The Learn SIS Framework Snapshot Flat File integration type is most closely aligned with the XEI File System Agent. The Snapshot Flat File topics will provide you with comprehensive coverage of the capabilities, data formats, file headers, and examples for implementing the Snapshot Flat File integration type.
If you currently implement your integration via an ODBC or Messaging Agent for your ANGEL SIS integration please consider the SIS Framework LIS integration type for event driven integration or the Learn Web Services for developing a message agent replacement.
Object types and files
Learn support for object type may be viewed in the SIS Framework Overview document. The Snapshot Flat File integration type supports the following data objects for the Course Delivery License:
- Users (aka Person)
- Courses
- Enrollments (aka Course Membership or just Membership)
- Course Categories
- Course Category Membership
- Observer Association
- Goals Associations (aka Course Standard Association)
- Terms
With the Community Engagement license the following additional object types are supported:
- Secondary Institution Role Associations (aka User Secondary Institution Role)
- Organizations
- Organization Enrollments
- Organization Categories
- Organization Category Membership
- Hierarchy Nodes
- User Associations (links Users to Hierarchy Nodes)
- Course Associations (links Courses to Nodes)
- Organization Associations (links Orgs to Nodes)
For differences between ANGEL and Learn data specifications, see the following Data Format Differences section. For more detailed descriptions of Learn data elements, see Snapshot Flat File Data Format. For a description of Snapshot Flat File Headers, see Snapshot Flat File Header Description.
File formats and headers
Learn header elements are in some cases different between Learn and ANGEL however the files can remain entirely unchanged and function as expected. However, there are some additional header attributes and flexibility of which you should be aware. As you review the SIS Framework interface, you should explore the features under Advanced Configuration to understand what the Header values should be and which attributes are required.
By selecting the Field Mapping interface you can see the fields that are required by looking at the "Required for Insert" column. By selecting Custom Headers you can see the full list of default header values.
The SIS Framework allows you do define the data header names to suit your needs. Not only can you rename an existing header attribute, but you can add additional headers. You will be able to select these headers from the Source Field drop downs on the Field Mapping page and in custom scripting.
A quick look at header differences
Users
With some configuration changes you may use your current user data feed for creating/updating/deleting user accounts in Learn. Note that you will minimally need to create the following custom header mappings:
Learn Header | Angel Header |
---|---|
user_id | LoginName |
lastname | LastName |
firstname | FirstName |
passwd | Password |
external_person_key | SourceId |
system_role | See below note. |
Also required is the addition of the SYSTEM_ROLE Header which is typically set to NONE. This may be added to your feed or set as a default on the integration using Advanced Settings Field Mapping.
Courses
With some configuration changes you may use your current course data feed for creating/updating/deleting course accounts in Learn. Note that you will minimally need to create the following custom header mappings:
Learn Header | Angel Header |
---|---|
external_course_key | SourceId |
course_id | CourseId |
course_name | Title |
Memberships
With some configuration changes you may use your current course membership data feed for creating/updating/deleting courses in Learn. Note that you will minimally need to create the following header mappings:
Learn Header | Angel Header |
---|---|
external_course_key | CourseSourceId |
external_person_key | UserSourceId |
Role: Student, Instructor, Teaching_Assistant, Grader, Course_Builder | UserRole or UserRights
Note: if using User Rights you must use a Custom Script to properly map the numerical role value to the matching Learn role. |
row_status | N/A |
available_ind | Disabled |
A full ANGEL to LEARN data map is shown in the Data Format Differences between Learn and ANGEL section. To learn more about using custom headers and field mappings, see Snapshot Flat File Configuration.
Setup of Snapshot Flat File
Detailed instructions and options for setting up the Snapshot Flat File integration type may be found in Snapshot Flat File Configuration.
Figure 4: Portion of the Snapshot Flat File configuration page accessible when creating a new integration or by selecting 'edit' from the integration's menu.
As you can see in Figure 4, this properties page captures configuration information.
You should also notice this is the interface that maintains the user/password for the integration; as well as controls the integration status. The status allows you to run your integration in testing mode, where you can see the results in the log without actually committing the data changes to Learn.
Snapshot Flat File operation modes
In this section, we are going to expand on how the Framework relies less on the use of Data Source Keys for managing groups of data.
Operation modes
The Framework allows you to execute the processing of data files in three modes. They are:
- "Store": performs explicit add and update operations on the data records that appear in the file.
- "Full Replace": performs add and update operations on the data records in the file AND it implicitly will execute a remove operation on records that are not in the file but are members of the integration.
- "Delete": executes explicit remove operations on the records included in the file.
Snapshot Flat File data ownership and data source keys
Data Source Keys (DSKs) serve as a means of identifying groupings of like data. The data source key does not represents the 'ownership' of that data, meaning a single SIS Framework may operate on multiple sets of data simultaneously.
Specifically within the SIS Framework Complete Refresh operation, the Data Source Key does NOT control the boundaries of what's safe to change, these boundaries are controlled by whether the record was created by a specific integration. We may state that a record is "owned" by the integration that created it.
To provide DSK level data management you must provide a unique Framework integration per data source key.
Note, it remains possible to run disables and purges in Learn by Data Source Key, rather than integration data feeds, by using the Data Sources Admin Panel tool.
How to use the data source keys
The implication for your integration will depend on your use of multiple Data Sources. You will have to figure out the best configuration for your institution, but we will use a common deployment configuration as an example to help your planning.
It is common for customers to use the name of the data file to define the Data Source Key of the records. A typical set of integration files and corresponding DSKs might look like this:
Data Type | Contents | Filename by Data Source Key |
---|---|---|
Users | Active user accounts | Users.txt |
Secondary Institution Roles | 2ndary role Assignments | 2ndary_roles.txt |
Courses | Fall 2012 Courses | FA2012_CRS.txt |
Student Enrollments | Fall 2012 Enrollments | FA2012_ENR.txt |
Instructor Assignments | Fall 2012 Instructor Assignments | FA2012_STAFF.txt |
Courses | Winter 2013 Courses | WI2013_CRS.txt |
Student Enrollments | Winter 2013 Enrollments | WI2013_ENR.txt |
Instructor Assignments | Winter 2013 Instructor Assignments | WI2013_STAFF.txt |
Table 1: Example DSKs and Files
If you want to use the "Full Replace" method in the SIS Framework, you will need to define separate integration deployments that segregate groups of data of the same type. With the example above, we would recommend that you deploy three integrations - one for the data types that are not term based and one for all the files that belong to a particular term. The result might look like this:
Figure 5: Example Integration Deployments
This would allow you execute a "Full Replace" on the Fall 2012 enrollment records without inadvertently disabling any other enrollments on the system. And note, do to this, you don't NEED to have separate integrations for every file.
Data/field ownership control
In the Framework, identification of the attributes that should be controlled or "owned" by Learn vs the integration is accomplished by a checkbox in the Field Mapping interface. In the capture below, the green arrow indicates the column "Change on Update?" When the box is checked, the values in the file will overwrite whatever is in Learn. With the box unchecked, the attribute will be populated during an INSERT, but ignored for all subsequent UPDATEs.
Figure 6: Ownership Checkbox
For example: Let's assume that you pre-populate Learn with users' passwords based on some known convention. Your data file includes this initial value for each user. However, you encourage users to change their password when they first log-in and give them the option to change their password whenever they want. If you un-check the box, subsequent runs of the integration will NOT reset the password.
Course copy/template changes
The SIS framework supports the use of Course Copy and Course Templates. This enables the reuse or templating of courses at the time of course creation.
Course copy
Course Copy configuration is done using the SIS Framework admin user interface. Course Copies are completed as part of the processing of STORE or FULL REPLACE requests. The feed files must contain the TEMPLATE_COURSE_KEY (Source Course Copy Key in the UI).
In all cases using the COPYINTO is only meant for populating new Courses. Using this method with existing Courses will not copy the data. Settings for Course Copy are found in the Advanced Configuration Settings for the integration under Courses:Course Copy Settings.
To learn more about Course COPYINTO and using Templates, see Snapshot Flat File Examples.
Automation and scheduling of Snapshot Flat File processes
As discussed, the SIS Framework includes HTTP "end points" which are the programmatic interfaces for delivering the data files to the Learn system. To automate batch integration, you will need to establish your own method for securely posting your files to these endpoints that does not require human intervention. Customers are encouraged to work with technology that is familiar and easily supported at your institution.
Also, you will have to decide the exact location from which the files are going to be posted. Consider both the security and stability of the system(s) to ensure that your integration runs and that no unauthorized people have access to the data.
Figure 7: Shows the endpoints for a fictitious integration. Notice how each entity/mode has a unique URL. Also, it is important to recognize that it is the integration Username that uniquely identifies this integration from others. All integrations of the type Snapshot Flat File use the same URLs - the username determines which integration is executed.
Figure 7: HTTP End Points
Automation scripting: A cURL example
The Snapshot Flat Files topics contain information about automation. The following provides an overview.
Processing for the Snapshot Flat File integration type is based on the concept of HTTPS POSTing of SIS data to Learn provided integration specific endpoints. This allows you to re-use existing data generation processes to generate your data files and then utilize a server-side process to POST your data to Learn. Learn supports different endpoints for different data objects (For example: Courses, Users, Memberships) and actions (For example: Store, Complete Refresh, Delete).
With the SIS Framework, the data is introduced directly to Learn by POSTing files to defined, secure HTTPs "end points." Files POSTed to Learn are immediately processed by the integration, so all automation of the integration process is external to the Learn server.
To give you an example of how this can be done, we are providing some instructions for how to use cURL, a command line tool for transferring data with URL syntax. You can download the version of cURL you want here:http://curl.haxx.se/download.html. Make sure to use a version that supports SSL.
The following example is for a Windows system. cURL must be downloaded and installed. The appropriate binary may be acquired at the curl project website: http://curl.haxx.se/latest.cgi?curl=win64-ssl-sspi
Linux systems may or may not require installation of cURL. Downloads are available from the curl project website:http://curl.haxx.se/download.html
Select the latest stable version (highlighted) for your version of Linux.
To learn more about automating a Snapshot integration type, see Snapshot Flat File Automation.
With the curl.exe file on the PC, we execute a post using the following command syntax:
curl -k -H "Content-Type:text/plain" -u 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/course/store --data-bin @courses_fall2012.txt
The syntax is explained below:
- -k = allow connection without verifying the SSL certificate.
- -H = determines this a required HTTP header line for Learn
- "Content-Type:text/plain" tells Learn that we are sending a plain text file.
- If you were using XML, it would be "Content-Type:text/xml"
- -u = username and password for authentication.
- 2d2f8833-90b7-444e-8bea-514111d5be70:password is from our setup example (see Figure 1)
- --url = the page to which you will post the file.
- End-point URLs are listed in Learn on the "HTTP Information" page for all integrations. For flat files this is specific to the Data Type (users, course, enrollments, etc.) and mode (store, complete refresh or delete) we want the data
- --data-binary = HTTP post of binary data
- @courses_fall2012.ext = the path to the file that is being sent to Learn. In this example the command is being run while in the directory where the file is located.
At its most simple, you can build a batch or shell script file that contains the commands you want to execute. Make sure you follow the logical order of creating data objects before you try to assign them to something. (For example: run your Users and Courses files before you run Enrollments).
This is the most basic example of a batch file that would execute the upload of three files. The assumption is that you would name this something like integration_post.bat and then create a scheduled job to run automatically at times when you know the files will be there. Clearly, you can be more sophisticated in your scripting and scheduling based on your skills and requirements. For shell script (Linux) and other automation strategies, see Snapshot Flat File Examples.
@ECHO OFF
REM.-- Simple script to run three integration files to Learn REM. --USERS posted in STORE mode curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/person/store --data-bin @users.txt REM. --COURSES posted in STORE mode curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/course/store --data-bin @courses_fall2012.txt REM. --ENROLLMENTS posted in STORE mode curl -k -H "Content-Type:text/plain" -U 2d2f8833-90b7-444e-8bea-514111d5be70:password --url https://learn.mycollege.edu/ webapps/bb-data-integration-flatfile-BBLEARN/endpoint/membership/store --data-bin @enrollments_fall2012.txt |
Responses
Note that the cURL commands will only return standard HTTP status codes from Learn as an indication of the success or failure of the file post. Code 200 is the only one that indicates a fully successful file post.
The successful post of the file does not indicate the successful processing of the data. For that you will have to examine the log entries on the server - in the GUI.
Strategies for monitoring integration processing progress are outlined in Snapshot Flat File Examples.
Data format differences between Blackboard Learn and ANGEL
The Snapshot Flat File Data Format and Header information is contained Snapshot Flat File Data Format and Snapshot Flat File Header Description topics, respectively. The following is provided to identify and map ANGEL data elements to Learn. For a complete description of the header element and data usage, see the other topics.
User
Note in cases where the ANGEL field is one of multiples you may choose to use the one which suits your data model in use. For example, you may use Tel3 instead of Tel4 for mobile if you are not using it for a business number.
LEARN Field | ANGEL Field | Notes |
---|---|---|
external_person_key | SourceId | Learn char limit of 50. |
new_data_source_key | N/A | Learn char limit of 256. |
firstname | FirstName | Learn char limit of 100. |
lastname | LastName | Learn char limit of 100. |
passwd | Password | Learn char limit of 32. |
user_id | LoginName | Learn char limit of 50. |
external_node_key | N/A | |
available_ind | AccountStatus | Learn char limit of 1.
Accepted values: 'Y', 'N' For a conversion example, see Snapshot Flat File Custom Field Mapping. |
birthdate | N/A | |
city | Adr1City | Learn char limit of 50. |
company | N/A | Learn char limit of 100. |
country | Adr1Country | Learn char limit of 50. |
department | Organization Hierarchy Node | Learn char limit of 100. |
educ_level | N/A | This is a user's education level. 0 = None 8 = K-8 12 = High school 13 = Freshman 14 = Sophomore 15 = Junior 16 = Senior 18 = Graduate School 20 = Post Graduate School |
Learn char limit of 100. | ||
gender | N/A | Learn char limit of 1.
Accepted values of 'M' or 'F' |
h_fax | N/A | Learn char limit of 50. |
h_phone_1 | Tel1 | Learn char limit of 50. |
h_phone_2 | Tel2 | Learn char limit of 50. |
job_title | Title | Learn char limit of 100. |
middlename | MiddleName | Learn char limit of 100. |
m_phone | Tel4 | Learn char limit of 50. |
othername | Nickname | Learn char limit of 100. |
pwencryptiontype | N/A | |
institution_role | Role | %nbsp; |
new_external_person_key | N/A | Learn char limit of 50. |
row_status | N/A | This is the row_status of the user. 0 = Enabled, 2 = Disabled. |
role_id | N/A | |
state | Adr1State | Learn char limit of 50. |
street_1 | Adr1Line1 | Learn char limit of 100. |
street_2 | Adr1Line2 | Learn char limit of 100. |
student_id | SSN | Learn char limit of 100. |
suffix | Suffix | Learn char limit of 100. |
system_role | AccountRights | The user's administrative role, describing the user's level of system administration privilege. The role of "none" has no system administration or Course creation privileges associated with it, and is the most commonly assigned role.
The following strings are acceptable: User Administrator: "account_admin", "accountadmin" or "user_admin" System Support: "system_support" or "syssupport" Course Creator: "course_creator" or "creator" Support: "course_support" or "support" Guest: "guest" None: "none" Observer: "observer" Portal Administrator: "portal_admin" or "portal" System Administrator: "sys_admin", "sysadmin" or "system_admin" eCommerce Administrator: "ecommerce_admin" Card Office Administrator: "card_office_admin" Store Administrator: "store_admin" User defined system roles can be used. Just match the "Role ID" found in Administrator Panel > System Role If one is not given the system defaults to none. |
title | Prefix | Learn char limit of 100. |
webpage | Homepage | Learn char limit of 100. |
b_fax | N/A | Learn char limit of 50. |
b_Phone _1 | Tel3 | Learn char limit of 50. |
b_phone_2 | N/A | Learn char limit of 50. |
zip_code | Adr1PostalCode | Learn char limit of 50. |
ANGEL User Fields with no corresponding LEARN Field |
---|
AccountExpires |
AccountSourceId |
AccountType |
Adr1Security |
Adr2City |
Adr2Country |
Adr2Line1 |
Adr2Line2 |
Adr2Line3 |
Adr2Pobox |
Adr2PostalCode |
Adr2Security |
Adr2State |
Attributes |
Division |
EmailSecurity |
GeoSecurity |
HintAnswer |
HintPrompt |
Latitude |
Logo |
Longitude |
Organization |
OrgSecurity |
OrgUnitPath -OR- OrgUnitSourceId |
Photo |
PhotoSecurity |
PwdChanged |
PwdForceChange |
PwdNeverExpires |
Security |
SkipIntro |
Sortstring |
Soundex |
Tel1Security |
Tel1Type |
Tel2Security |
Tel2Type |
Tel3Security |
Tel3Type |
Tel4Security |
Tel4Type |
UtcOffset |
WwwSecurity |
Courses
Learn Header | Angel Header | Notes |
---|---|---|
external_course_key | SourceId | Learn char limit of 64 |
course_id | CourseId | Learn char limit of 50 |
course_name | Title | Learn char limit of 255 |
new_data_source_key | N/A | |
classificationId | N/A | |
desc_page_ind | N/A | |
days_of_use | N/A | |
service_level | CourseType | Learn char limit of 1.
Accepted values: 'F', 'C', 'R', 'T', 'S' ANGEL: Course, Group, or Library. For a conversion example, see Snapshot Flat File Custom Field Mapping. |
allow_guest_ind | N/A | |
allow_observer_ind | N/A | |
available_ind | Disabled | Learn char limit of 1.
Accepted values: 'Y', 'N' For a conversion example, see Snapshot Flat File Custom Field Mapping. |
master_course_key | N/A | |
description | Description | Learn char limit of 4000. |
duration | Duration | Learn limit. String. Accepted Values: "Continuous", "Range", "Fixed", "Term"
For a conversion example, see Snapshot Flat File Custom Field Mapping. |
pace | N/A | |
end_date | ClassEnds | String. yyyymmdd |
locale_enforced | N/A | |
enroll_access_code | EnrollmentPin | |
enroll_end | EnrollmentEnds | String. yyyymmdd |
enroll_start | EnrollmentBegins | String. yyyymmdd |
enroll_option | N/A | |
fee | Cost | Numeric string.
11 characters, 2 decimal places. |
institution_name | N/A | |
locale | N/A | |
lockout_ind | N/A | |
abs_limit | N/A | |
soft_limit | N/A | |
upload_limit | N/A | |
external_association_key | N/A | These are each related to hierarchy - not sure how that would map to ANGEL data |
primary_external_node_key | Org_Unit | |
new_external_course_key | N/A | |
row_status | N/A | |
catalog_ind | N/A | |
template_course_key | N/A | |
start_date | ClassBegins | String. yyyymmdd |
term_key | Semester | Learn char limit 256. |
use_term_availability_ind | N/A |
ANGEL Course Fields with no corresponding LEARN Field |
---|
BillingId |
Category |
Contact1 |
Contact2 |
CourseField |
CourseTime |
CourseUploadUrl |
Credits |
Hidden |
Instructor |
InstructorId |
Keywords |
Locked |
OrgUnitPath -OR- OrgUnitSourceId |
RedirectTarget |
RedirectUrl |
ScheduleDays |
ScheduleInterval |
ScheduleOccurrence |
ScheduleType |
Section |
Semester |
Stylesheet |
ViewableBy |
Memberships
Learn Header | Angel Header | Notes |
---|---|---|
external_course_key | CourseSourceId | Learn char limit of 64. |
new_data_source_key | N/A | |
external_person_key | UserSourceId | Learn char limit of 64. |
available_ind | Disabled | Learn char limit of 1.
Accepted values: 'Y', 'N' For a conversion example, see Snapshot Flat File Custom Field Mapping. |
Image URL | N/A | |
Include_In_Roster | Hidden | Learn char limit of 1.
Accepted values: 'Y', 'N' For a conversion example, see Snapshot Flat File Custom Field Mapping. |
intro | N/A | |
notes | N/A | |
Personal Information | N/A | |
receive_email_ind | N/A | |
role | UserRole or UserRights | String. The valid values for the Course Role are: Instructor, teaching_assistant, course_builder, Grader, Student, guest, none.
For a conversion example, see Snapshot Flat File Custom Field Mapping. |
row_status | N/A | |
link_description_1 | N/A | |
link_name_1 | N/A | |
link_url_1 | N/A | |
link_description_2 | N/A | |
link_name_2 | N/A | |
link_url_2 | N/A | |
link_description_3 | N/A | |
link_name_3 | N/A | |
link_url_3 | N/A |
ANGEL Membership Fields with no corresponding LEARN Field |
---|
BillingId |
EndDate |
Locked |
StartDate |
Statu> |
UserGroup |
UserRights |