Field mapping
The SIS Framework supports customizing incoming data before it is added to Blackboard Learn on a record-by-record basis. This is useful when the data from your SIS system is not aligned with your Learn data requirements and cannot be altered in the SIS because it is prohibitive or the information the data represents is used by other systems. This document covers the use of the use of the 'Field Mapping' option, specifically how to use the Custom Script field mapping option.
This ability to customize the incoming data is called Field Mapping and is accessed in the UI under the integration's Advanced Configuration menu.
The Advanced Configuration screen provides a listing of the objects supported for that object in Learn. These align with the integration type will vary from integration type to integration type.
When you select Field Mapping on a Learn Object associated with the integration type, in this case Courses, you will be directed to the Field Mapping page for the integration.
In addition to useful information regarding the field, such as whether it is required for insert or required to be unique, you may also select several settings which control how incoming data is managed by the integration and whether you choose to map between incoming objects.
Custom scripts and field mapping
On the field you wish to provide a field mapping you may select to use a different incoming field. For example: say the incoming Course Id was not suitable for your use in Learn, you could map it to the Course External Key instead. Because this tutorial is about using a custom script to perform the mapping you would select 'Use a custom script' from the 'Source Field' pull down.
Custom scripts are based on JavaScript. All incoming data is mapped to an internal object model which provides the ability to access each data element for its data and subsequent use in or manipulation using a custom JavaScript.
Use integer comparators in Javascript switch statements. If you must use strings or chars as comparators, then use if/else statements rather than switch statements.
Two excellent JavaScript references are provided by the Mozilla Developer Network https://developer.mozilla.org/en-US/...ript/Reference and the World Wide Web Consortium (W3C) http://www.w3schools.com/jsref/.
Selecting 'Use a custom script' provides you with a text area to enter your javascript code.
It is in this text area that you will enter your custom script to edit the incoming data.
In-Learn documentation
Documentation regarding Custom Field Mapping scripts is found on the Sample Documents Page linked from the top-level Student Information System Integrations page in the Learn Sys Admin Panel.
The Sample Documents link opens a page that contains several links to SIS Integration Framework documents covering all integration types. The two documents of interest in the context of the Snapshot Flat File integration type and Custom Field Mapping are linked by Data Dictionary: Snapshot Flat File Integration and Custom Field Mapping Script Examples respectively.
Custom scripts
Access to each Integration type's object model is different and based on the underlying data structure of the integration type's data specification. From an understanding of the integration data specification, you can derive a pattern for accessing all the integration's objects.
In the case of Snapshot Flat File this pattern is based on a root object of 'data' so we can derive the following pattern:
data.getValue(<FlatFileDataObject>);
where <FlatFileDataObject> is replaced by the header element name from the data feed.
For example:
data.getValue("firstname");
data.getValue("lastname");
data.getValue("course_name");
Additionally, where the data feed takes a 'Y' or 'N' you must conditionally replace as the script result with the boolean values true or false respectively. For example:
var avail_ind = false;
if (data.getValue("available_ind").toUpperCase() == 'Y') {
avail_ind=true;
avail_ind;
or alternatively written a using a statement using a ternary operator:
condition ? execute if condition true: execute if condition false :
For example:
data.getValue("available_ind").toUpperCase( )=='Y' ? avail_ind = true : avail_ind = false;
avail_ind;
the above may be further reduced to just the ternary statement as it is the final value of the expression on the last line of the script that is returned as the data point saved in Learn.
So we can use:
data.getValue("available_ind").toUpperCase() == 'Y' ? avail_ind = true : avail_ind = false;
and get the same result.
Given custom scripts are processed on a per record basis integration performance may be impacted if computationally intensive scripts (such as encryption) are performed. Also note that accessing external data is not supported - you may only perform operations on existing feed data or computationally derived data.
Debugging scripts
Debugging your Scripts is relatively easy as the errors or data issues you may encounter are logged.
For example:
Given the development of a custom script on the 'Available' Field:
data.getValue("available_ind");
may successfully run, but if the value is not passed in the data feed the script will produce the following entry in the Log:
Null value returned for non-nullable attribute: isAvailable.
Invalid data for attribute: isAvailable. Value: null. Using default Learn value.
Where as (assuming available_ind is provided in the feed):
data.getValue("available_ind");
may successfully run, but produce the following entry in the Log:
Invalid value type for attribute: isAvailable. Value: y
Invalid data for attribute: isAvailable. Value: Y. Using default Learn value.
that is because we need to conditionally alter the incoming data to the expected boolean value - in this case "TRUE", so we could change the script to:
data.getValue("available_ind") == 'y'?true:false;
But to be on safe side we want to capture Upper and Lower case comparisons so we tweak the script:
data.getValue("available_ind").toUpper() == 'Y' ? true : false;
This will result in an actual script error and print to the log:
Error in script execution for attribute: isAvailable.
blackboard.platform.script.ScriptingException: javax.script.ScriptException: sun.org.mozilla.javascript.internal.EcmaError: ReferenceError: "toUpper" is not defined. (<Unknown Source>#2) in <Unknown Source> at line number 2
This is because JavaScript has no toUpper(), but it does have toUpperCase() so editing the script...
data.getValue("available_ind").toUpperCase() =='Y' ? true : false;
now the script will successfully run.
Helper scripts: Logging in custom scripts
Learn provides helper scripts for providing common capabilities such as logging for your scripts. Adding the following to your scripts will include the output in the integration logs:
helper.logError(msg)
helper.logWarning(msg)
helper.logInfo(msg)
helper.logDebug(msg)
The output of the helper.log* methods will be visible in logs based on the integration configuration for logging levels.
Additional helper scripts available (as described on the Custom Field Mapping Script Examples page linked from the Sample Documents Page) are:
- helper.getBatchUid( String id ) - This constructs an identifier that is prefixed with the Batch Uid Prefix specific to the SIS Integration that the mapping is running on. This should be used whenever generating "unique" ids as the prefix can assist in avoiding id conflicts.
- helper.getHelper( String helperName ) - This returns a SIS Integration Type-specific helper object that contains helper methods applicable to that integration type. Documentation for these helpers will be provided in the JavaDocs for each SIS Integration Type.
- helper.skipAttribute() - This returns a value that when returned from a mapping script causes the field being mapped to be skipped (not changed.)
- helper.skipAttributeIfNull( Object value ) - If the value passed in is null, this returns the same thing as skipAttribute. If it is not null, the value is returned.
- helper.skipRecord() - This returns a value that when returned from a mapping script causes the entire record currently being processed to be skipped.
- helper.skipRecordIfNull( Object value ) - If the value passed in is null, this returns the same thing as skipRecord. If it is not null, the value is returned.
Continuing with our Sample script we will add an info log message to indicate that the value used in Learn is derived from a custom script, the base data object referenced for the inserted data, and the calculated values that were used. We will also add a log message indicating an error condition based on the lack of a suitable
Revisiting the above isAvailable script let's say that the AVAILABLE_IND is set inconsistently in course data feeds - meaning it is not always present in the data feed - and that you want to set it to true when a value for AVAILABLE_IND is not present or otherwise evaluate appropriately? The script now becomes:
var outInd = false;
var inInd = data.getValue("available_ind")
if (inInd == "") {
helper.logInfo("INCOMING AVAILABLE_IND UNDEFINED for "+data.getValue("course_id")+": setting isAvailable to true");
outInd=true;
} else {
data.getValue("available_ind").toUpperCase()=='Y'?outInd=true:outInd=false;
}
helper.logInfo("OUTGOING AVAILABLE_IND: ["+outInd+"]");
outInd;
The log now displays:
May 31, 2013 4:03:56 PM - Course [create/update, testCourse1]
INCOMING AVAILABLE_IND UNDEFINED for TEST_COURSE_1: setting isAvailable to true
OUTGOING AVAILABLE_IND: [true]
Course 'testCourse1' processed successfully.
testCourse1|TEST_COURSE_1|Test Course 1|
...
remaining log entries
...
Field mapping successful.
Examples
- Course Names
- Passwords
- Emails
Course name: Add term and year
Use case
It is desired that the display of courses contain the term and year the course is offered. Currently the data provided by the SIS does not append this information to the course_name. Courses are displayed using the course name. For example:
Precondition
The SIS provides the term and year as part of the course_id string and provides the required course_name field.
For example:
external_course_key|course_id|course_name|available_ind
ARTHIST.202.01|36202010114|Art History 202: Renaissance Architecture|Y
In the above example the course_id is comprised of:
the department: 36
the course: 202
the section: 01
the term month: 01
the year: 14
Requirements
1. Terms are determined by the month designated. For example:
01 = Winter
04 = Spring
06 = Summer
09 = Fall
2. Append the the programmatically derived term and year to the course name separated by spaces and enclosed in parenthesis "()". For example:
Art History 202: Renaissance Architecture (Winter 2014)
3. Ignore appending information if suitable term is not provided.
Postcondition
1. Courses where the course_id provides suitable term/year identifiers will have their course_name data appended with (TERM YEAR) prior to creating or updating the course record in Learn
2. Courses without suitable term/year identifiers in the course_id will not have their course_name changed.
Script
The course name is provided in the SIS data feed course_name field and the required term information is contained in the SIS data feed elements course_id field in numerical format. To meet the requirements we need to
a) determine the year, and
b) determine the term by mapping to a range and then properly format the resulting data.
This we will do by providing javascript functions to return the desired format for the course name display and place the output of those functions in the proper format.
Tutorials for writing javascript functions are provided on the HTMLGoodies (http://www.htmlgoodies.com/beyond/ja...d-classes.html) and the World Wide Web Consortium (W3C) (http://www.w3schools.com/js/js_functions.asp) sites.
a) Determine the year.
Given the course_id contains a two number year at the end of the character sequence we can write a function to pull and return the last two characters:
function courseYear(crn) {
return crn.substring(9);
}
b) Determine the term.
The course_id also provides us with the data to determine the term and we can use the javascript substring function to pull the MM data and map that to a range of months representing a term label - For example: 09=Fall, 01=Winter, 04=Spring, 06=Summer
This is a simplistic case - the data could also provide start dates which could be used in conjunction with the course_id to conditionally determine the term.
function getTerm(crn) {
var termCode=crn.substring(7, 9);
var term="";
if (termCode == "01") {
term="Winter";
} else if (termCode == "04") {
term="Spring";
} else if (termCode == "06") {
term="Summer";
} else if (termCode == "09") {
term="Fall";
} else {
term="";
}
return term;
}
function getYear(crn) {
return crn.substring(9);
}
Because we are working with integers here we could write the getTerm function using a switch statement instead of the above one which uses if/else...
function getTerm(crn) {
var termCode = parseInt(crn.substring(7,9));
var term = "";
switch (termCode)
{
case 09: term ="Fall"; break;
case 01: term ="Winter"; break;
case 04: term ="Spring"; break;
case 06: term ="Summer"; break;
default: term ="";
}
return termString ;
}
The switch statement works only because we are able to generate an integer for the comparator.
Now that we have functions to determine the term and year let's write the script which appends the course_name:
function getTerm(crn) {
var termCode=crn.substring(7, 9);
var term="";
if (termCode == "01") {
term="Winter";
} else if (termCode == "04") {
term="Spring";
} else if (termCode == "06") {
term="Summer";
} else if (termCode == "09") {
term="Fall";
} else {
term="";
}
return term;
}
function getYear(crn) {
return crn.substring(9);
}
var crn = data.getValue("course_id");
var year = getYear(crn);
var term = getTerm(crn);
var courseName = data.getValue("course_name");
var newCourseName = "";
if (term!="") {
newCourseName = courseName + " (" + term + " 20" + year + ")";
} else {
newCourseName = courseName;
}
helper.logInfo("INCOMING COURSE_NAME " + data.getValue("course_name"));
helper.logInfo("INCOMING COURSE CRN " + crn);
helper.logInfo("INCOMING COURSE YEAR " + year);
helper.logInfo("INCOMING COURSE TERM " + term);
helper.logInfo("INCOMING COURSE_NAME " + data.getValue("course_name"));
helper.logInfo("OUT NEW NAME " + newCourseName);
newCourseName;
Entering the above into the Course field mapping for Course Name and, using the integration Upload file option, manually uploading (storing) the course feed example from the above precondition, you will see the below data posted to the logs for our sample course:
And the course name will now display with the term and year as derived from the course_id:
Passwords
Use case
You use LDAP to authenticate Learn Users, but Learn requires passwords for creating User accounts and your SIS does not provide passwords. You need to create a random password per user.
Precondition
The data feed provides the basic user information which may be used to build the password. For example: firstname and lastname
Requirements
Passwords should be the combination of firstname+lastname+random number
Must be able to specify a range for the randomization
Postcondition
Password is created for user For example:
For user Barney Rubble the password could be barneyrubble102464
Sample data
external_person_key|user_id|passwd|firstname|lastname|email|system_role
testPerson1|aanderson_test|changeme|Alpha|Anderson||none
testPerson2|bvonbrown_test|changeme|Beta|Von Brown||none
testPerson3|ddavis_test|changeme|Delta|Davis!||none
testPerson4|ggardner_test|changeme|Gamma|G'Ardner||none
Script
function rand (min, max) {
var argc = arguments.length;
if (argc === 0) {
min = 0;
max = 2147483647;
}
return Math.floor(Math.random() * (max - min + 1)) + min;
}
var password = "";
var regex = new RegExp(" ", 'g');
helper.logInfo("INCOMING PASSWORD: " + data.getValue("passwd"));
helper.logInfo("INCOMING FIRSTNAME: " + data.getValue("firstname"));
helper.logInfo("INCOMING LASTNAME: " + data.getValue("lastname"));
password = (data.getValue("firstname") + data.getValue("lastname") + rand() + rand()).toUpperCase();
password = password.replace(regex, '');
helper.logInfo("GENERATED PASSWORD: " + password);
password;
Entering the above into the User field mapping for Password and manually uploading (storing) the User (Person) feed example from the above sample data, you will see the below data posted to the logs:
Emails
Emails can be complicated to properly validate and some would argue that one should never do this with a simple regular expression or in a javascript. The below is a high level example that while probably correctly matching 95% or more of passed email addresses may miss some edge case examples and thus is intended more as an example of how one would approach construction of email addresses and the use of error capturing in a custom field mapping script than an exercise in email validation.
Use case
Your SIS does not store student email accounts or it stores personal email accounts, but not those issued by your institution - you want emails in Learn to be the institution issued email accounts.
Precondition
Data required to generate the local-part of the desired email address exists in the SIS data feed. For example: firstname and lastname
Requirements
Institution issued email accounts follow the pattern of [email protected] where '-yy' represents the anticipated year of graduation for the student. These elements are passed in the SIS feed in the following data:
FIRSTNAME - the student's first name
MIDDLENAME - the student's middle name
LASTNAME - the student's last name
OTHERNAME - the year of graduation
The local-part of acceptable email Addresses should contain only alphanumerics, '-','_','.' and apostrophes. For example: Gamma.G'[email protected]
Postcondition
A properly formatted email address is generated and passed to Learn.
Sample data
external_person_key|user_id|passwd|firstname|lastname|email|system_role
testPerson1|aanderson_test|changeme|Alpha|Anderson||none
testPerson2|bvonbrown_test|changeme|Beta|Von Brown||none
testPerson3|ddavis_test|changeme|Delta|Davis!||none
testPerson4|eedwards_test|changeme|E...nstitution.edu|none
testPerson5|ggardner_test|changeme|Gamma|G'Ardner||none
Script
var emailAddress = "";
var instDomain = "institution.edu";
function buildIt() {
var fname = data.getValue("firstname");
var lname = data.getValue("lastname");
var regex = new RegExp(" ", 'g');
emailAddress = fname +"."+lname+"@"+instDomain;
emailAddress=emailAddress.toLowerCase();
emailAddress = emailAddress.replace(regex, '');
}
function validateIt(eAddress) {
var emailRegEx = /^([a-zA-Z0-9_\.\-\'])+\@(([a-zA-Z0-9\-])+\.)+([a-zA-Z0-9]{2,4})+$/;
if (!emailRegEx.test(eAddress))
throw new Error("Email Validator Error: Cannot Validate Email Address");
}
emailAddress = data.getValue("email");
if (emailAddress == "" || emailAddress == null) {
buildIt();
} else if ( emailAddress.indexOf(instDomain) === -1) {
buildIt()
}
try {
validateIt(emailAddress);
} catch (err) {
helper.logError("INCOMING EMAIL ADDRESS: " + emailAddress);
helper.logError("INCOMING FIRSTNAME: " + data.getValue("firstname"));
helper.logError("INCOMING LASTNAME: " + data.getValue("lastname"));
helper.logError(err +" for User (" + data.getValue("user_id") +", Email (" + emailAddress + "). No Email Address Saved for this user. )");
emailAddress="";
}
helper.logInfo("emailAddress: " + emailAddress);
emailAddress;
Entering the above into the User field mapping for Password and manually uploading (storing) the User (Person) feed example from the above sample data, you will see an error posted to the logs.
This is because email addresses cannot contain exclamation points per our regular expression.
Note that the user is still created as a valid email address is not required for creating or updating a user record.
Changing the entry for Delta Davis from:
testPerson4|ddavis_test|changeme|Delta|Davis!||none
to
testPerson4|ddavis_test|changeme|Delta|Davis||none
removing the '!' in Delta's lastname field and re-uploading the file removes the error and updates Delta's record with the new email address.
Skipping a record or attribute on error
Helper provides two additional methods which allow even greater control over the processing in your scripts:
helper.skipAttribute() - skips a non-critical attribute that appears to be incorrectly set
helper.skipRecord() - to skip the entire record and move on to the next one
The above email script logs an error when the email address cannot be validated, but also continues to create the record anyway. If we did not want to create users with empty emails, but instead log the validation error and move to the next record we would add helper.skipRecord() in a conditional based on the emailAddress at the end of the script. For example:
instead of closing the script with emailAddress=""; close it with:
(emailAddress=="") ? helper.skipRecord(): emailAddress;
This conditionally skips the record based on the contents of the emailAddress which we set to be an empty string when the validation failed.
Add the helper.skipRecord() method to the script and change the entry for Delta Davis from:
testPerson4|ddavis_test|changeme|Delta|Davis||none
to
testPerson4|ddavis_test|changeme|Delta|Davis!||none
Manually uploading (storing) the revised - with exclamation mark - User (Person) feed example, you will see data posted to the logs.
Note that the error is logged and that at the end of the log it notes that the record was skipped as a result from the mapping script and the attribute which returned the result.
Conditionally applying helper.skipAttribute() will have the similar impact - the script will process the feed data and based on the presence of a conditional as the final line calling helper.skipAttribute() the integration will either throw an error and skip the record or if the attribute is not required the mapping configuration takes control and handles the attribute as configured.
