Snapshot Flat File

It is important to note that for the purposes of this document the use of the term 'snapshot' pertains to the general process of taking point-in-time data from the SIS that is stored in a flat file that is then consumed by Learn - not the snapshot controller.

Snapshot provides the ability to integrate using point-in-time data as provisioned by the SIS with two supported integration types (read: data formats) within the SIS Framework for Snapshot - Snapshot XML and Snapshot Flat File (or delimited data). The latter, Snapshot Flat File, is the 'go-forward' supported format and will receive updating to support new data objects. Snapshot XML will only receive maintenance updates.

Processing for both snapshot integration types is via an HTTP post of the data file to the appropriate endpoint.

Data supported

Requirements

Ability of the SIS to generate data formatted for the chosen integration type - Snapshot XML or Snapshot Flat File with appropriate delimiter.

A point worth reiterating - the Snapshot Controller and the Snapshot XML data format should not be the integration option of first choice as they are only receiving maintenance updates not new object support or feature development. It should be expected that at some time in the future an announcement for support removal will be made with the provision of ample time for client adjustment.

See the Enterprise 1.1 examples section for data samples, noting that the data format is slightly different between the two supported flavors of Enterprise 1.1 (standard and vista).

Setup: Snapshot integration types

In addition to an understanding of the data involved in any SIS integration, setup of the Snapshot Integration integration types requires knowledge of the SIS capabilities and endpoints with an eye on whether the data objects supported in the selected Snapshot type meet the client's integration goals. Additionally an understanding of the specific file format is necessary. You may create a Snapshot integration via the following steps in the Learn System Administrator SIS Integration UI: Data Integration > Student Information System Integrations

The following steps are the same for both Snapshot Integration types - the key difference in selection of the integration type is whether the data is XML (choose Snapshot XML) or delimited (Snapshot Flat File).

1. Select Create Integration.

2. Select Snapshot XML or Snapshot Flat File. On the next screen, select the configuration options. 

   To perform advanced configuration (Data mapping for example), an integration must be set to Testing or Active status

3. Integration Name: enter Snapshot XML or Snapshot Flat File.
4. Description (optional): enter whatever you want to describe the integration; for example: Test Snapshot XML or Test Snapshot Flat File.
5. Shared Username and Shared Password are used for data post operations.
6. Set the Integration Status:
   1. Inactive — Integration performs no processing.
   2. Active — Integration performs processing and updates Learn data.
   3. Testing — Integration performs processing and does not update Learn.
7. Set the log verbosity levels — for Testing, we recommend you select All Diagnostic and Debug Messages. For Active, we recommend a lower setting be used, such as Errors Only or Errors and Warnings.
8. Select the Data Source Key (DSK) to be used for this integration. You may select to use a specific existing DSK, create a new key, or to use the data source provided in the feed. NOTE:  In all cases a DSK must be created before the data feeds may be processed.
9. Only Snapshot Flat File supports the management of parent nodes. Select appropriately to meet your goals.
10. Select the objects handled via this integration and how to handle parity. Generally, the defaults are fine. NOTE:  You may have multiple integration types each handling a unique subset of available integration objects.
11. Select Submit and your new Snapshot integration appears in the list of integrations.

One more step must be completed to support Snapshot integration and that is either configuring the SIS system to post data to Learn or to configure an app server side cron or batch job to post the data to Learn. Note that Snapshot endpoints are specific to the required task to be performed.

Application server side posting to learn is accomplished by using CURL or similar application triggered by a cron or batch job:

curl -k -w %{http_code} -H "Content-Type:text/plain" -u SHARED_USERNAME:SHARED_PASSWORD --data-binary @PATH_TO_TEST_FILE ENDPOINT_URL

Replacing:

SHARED_USERNAME with the shared username

SHARED_PASSWORD with the shared password

PATH_TO_TEXT_FILE with the path the the xml data file

Replace ENDPOINT_URL with the endpoint url.

You can see a list of the endpoints on the server by going to the HTTP Information page off of the context menu of your flat file integration:

System Admin (Building Blocks) > Data Integration > Student Information System Integrations > (Your integration's context menu) > Http Information)

If it is supported by a middle tier application or by the SIS, data may be posted directly to the endpoint following the above protocol.

Once the above has been completed and the SIS has been enabled to generate data and post or save to file for posting to Learn you should begin to see logs on the integration.

Assess the data and determine if you need to make adjustments using the Advanced Configuration/Field Mapping on the available data objects. See section on Field Mapping at end of examples section for description of this functionality.

To learn more about processing data files, see Snapshot Flat File Automation.

Snapshot integrations

Snapshot integrations are based on sets of data that meet specific purposes. For example, to manage Users or Courses in the system. Of the two supported types - snapshot XML and Snapshot Flat File - only Snapshot Flat File should be considered most current with supported Learn data objects.

As with all the "file based" integration types (Snapshot and Enterprise 1.1 types) there are two processes for consuming data in those files - the use of the GUI Upload Feed File or the use of an http data transfer (For example: cURL). In the case of http data transfer a cron or batch job may be used to trigger the operation. Additionally provided it is supported by the SIS or middle-tier a data stream may be posted directly.

Each operation is achieved via a specific end point. These end points for the application server are available via the integrations HTTP Information link from the menu.

Create a data file

1. In Administrator Tools, under Integrations, select Data Integration.
2. Select Student Information System Integrations.
3. From the dropdown menu Create Integration, select Snapshot Flat File if you are creating a file manually.
4. Enter an Integration Name (required). You have the option of entering a description.
5. You can add security by entering a Shared Username and Shared Password. This is more often used for publishing to the endpoint.
6. In Feed File Delimiter, select the character that separates fields in your data file.
7. In Integration Status, select inactive, active, or testing.
8. In Log Verbosity, select how detailed you want your logs to be.  

The Data Support area allows you to do logical data segmentation.  For example, if you have two different student information systems for two different campuses, you can segment for those two separate campuses. Or if you want to segment data for different academic terms.  

1. In Learn Data Source, you can select whether the data source is to be defined in the inbound data or use the same Learn Data Source for all new inbound data.  
2. You have the option to enter a Batch Uid Prefix. This is used to prefix identifiers for objects created by this integration to avoid id conflicts. It is not recommended that this be modified after data has been created for this integration.
3. Select node placement in Parent Hierarchy Node.

In Advanced Configuration, you have an array of options to, for example, handle situations when specific records are not present in a batch. For this use case, we’ll leave the default selections in place.

When you’ve entered your data, select Submit. This will create your file for integration.

Upload a data file for integration:  

1. In the Administrator Panel, under Integrations, select Data Integration.
2. Select Student Information System Integration.
3. Select the menu for the file you want to upload. Select Upload Feed File.
4. In the Data Type menu, select your data type. The most common for integrations are Hierarchy Node (if your file represents your hierarchy), Course Association, Organization Association, and User Association.  
5. For Operation Type, the most common option is Store. 
   • Use Complete Refresh when doing full data comparisons, and you want to delete something that’s not in the file. Use caution with this option. 
   • Use Delete to delete the records you’re uploading.  
6. Select Submit.

You can check the SIS Log to confirm it is added. For larger files, it may take a couple minutes to upload. Select Logs under Tools and Utilities, then select SIS Logs. 

Snapshot endpoints

Snapshot operation endpoints for Flat File

Use this information to set up your SIS to POST Snapshot Flat File feed files to your integration. Stored objects will continue to exist, and retain their data. Deleted objects are removed. Complete Refresh will remove any existing objects that are not in the feed file. Posting a file of a specific object type to a non-matching endpoint will fail. For example: posting a Course data file to a Person endpoint will fail.

Disabling a course will disable memberships in that course. This is new behavior.

You can find an operation's endpoint URL in the Blackboard Learn user interface as well. Navigate to Administrator Panel > Data Integration > Student Information Systems Integrations. Locate the integration, open its menu, and select HTTP Information. The integration's endpoint URLs are listed.

If you are familiar with legacy Snapshot the SIS Framework Flat File type brings no change to the format other than the addition of headers for the new data objects supported. To learn more, see Snapshot Flat File Data Format and Snapshot Flat File Examples.

Snapshot operation endpoints for XML

Use this information to set up your SIS to POST XML feed files to this integration. Stored objects will continue to exist, and retain their data. Deleted objects are removed. Complete Refresh will remove any existing objects that are not in the feed file. Posting an object containing an incorrect object type will fail.

Learn more

SIS Framework Overview

Data Source Key Overview