Load Balancing

To improve performance, Blackboard Learn supports using a hardware or software Load Balancer and multiple Application Servers.

Understanding load balancing

Load-balanced configurations include multiple Application Servers, a Database Server (or failover cluster), a File System server, and a Collaboration Server. A typical configuration looks like this:

A network switch is used to handle communications from the client machines to the Application Servers and the Collaboration Server. A separate switch is used to handle a secure, private connection between the Application Servers and the File System and Database Servers. The database and file system must be on a secure, private network.

The diagram also shows integration with an LDAP server or servers to handle authentication and an SIS system to share data with Blackboard Learn. Integrating these components with a load-balanced configuration is not difficult, but does require that each Web/Application Server shares the same settings.

Benefits and drawbacks

Adding additional Application Servers distributes the computing workload among multiple servers, which improves both performance and the availability of the overall service. Down time is reduced because most single points of failure are removed. If a server goes offline, other servers pick up the load dynamically, allowing the application to continue servicing clients.

However, having too many Application Servers can degrade performance. Blackboard Learn uses application caching mechanisms which work better when there are multiple users working on an Application Server. Avoid having more Application Servers than are required to support peak service load.

The load balancer and Blackboard Support

Blackboard does not recommend a specific Load Balancer vendor, make, or model. As long as the vendor supports the requirements laid out in this document, Blackboard Support will do its best to assist in these configurations.

In the past Blackboard has recommended hardware Load Balancers over software based Load Balancers but software and Virtual Machine (VM) based Load Balancers have improved significantly and Blackboard Support sees no reason to exclude them.

Port requirements

Depending on the configuration chosen Blackboard Learn will use the following ports:

HTTP: 80
HTTPS: 443
Collaboration Server: 8010, 8011
Collaboration Server TLS: 8012

Intra-server communication between Learn servers behind the load balancer is essential for proper operation of the learn environment. The port range 1025 - 65536 is required for communication between app servers.

For service such as Cloud services, SafeAssign, RSS feeds, and some of the 'mashups' like YouTube embedding to work successfully, all of the Application Servers need access to the Internet.

Load balancing/distribution methods

Different algorithms can be used to determine distribution methods. "Round robin" and "least connections" are the two most common load balancing methods used. To avoid overloading any single server in the cluster, select the method that will ensure that each Application Server receives approximately the same amount of traffic.

TLS traffic encryption

Learn more about TLS, which is the successor to SSL.

To meet industry best practices for protecting Internet communications, Blackboard recommends enabling TLS system-wide. To that end, TLS Choice was removed and is no longer available.

The following are the three most common high-level TLS configurations:

TLS Offloading is where the Load Balancer communicates with the client using TLS but decrypts the sessions and communicates with the Blackboard Application Servers using HTTP.
TLS Re-encryption is where the Load Balancer communicates with the client using TLS (HTTPS), decrypts the sessions so that it can read the payload (cookies etc.), and then re-encrypts the session and communicates with the Blackboard Application Servers using SSL (HTTPS).
TLS Pass-through is where the Load Balancer communicates with the client using TLS (HTTPS) but does not decrypt the TLS session at all and just passes the session onto the Application Servers. As the Load Balancer cannot read the payload, it has no access to cookies; it can only persist sessions to the Application Servers using IP based persistence.

Blackboard Learn recommends using either TLS Offloading or TLS Re-encryption. This is because Blackboard applications require session persistence which is achieved by the use of cookie insertion. (See Persistence Requirements.) Without TLS Re-encryption or TLS Offloading, the Load Balancer can't read the encrypted payload of the packets and so cannot read the persistence cookie which tells the Load Balancer which Application Server it should be forwarding the request to.

TLS Pass-through and/or IP persistence can be used but if any clients access the service using 'Mega Proxies' or NAT (Network Address Translation) they will see failures or errors. For some areas of Blackboard to work successfully, all client connections from a user must 'persist' to the same Application Server. All connections that originate from large enterprises or megaservices such as AOL or MSN pass through client-side proxies so the same session can originate from different IP Addresses. This issue can be mitigated by sending all external (off campus) sessions to the same Application Server but only if the amount of expected external traffic does not exceed the capacity of a single Application Server.

Configure TLS re-encryption

To configure TLS Re-encryption:

Configure the Load Balancer with an TLS certificate signed by a certificate authority.
Do not configure the Load Balancer to inject the X-Forwarded-Proto header.
The Application Servers can be configured with self-signed certificates and even use a lower security encryption key size or key length to reduce processing overhead, especially if the Application Servers are on a secure network.
If you use self signed certificates on Application Servers, the Load Balancer needs to be set to trust the self-signed certificates. Otherwise, the File Upload applet will not work correctly. Please refer to the Load Balancer's vendor's documentation related to importing and trusting TLS certificates.

Configuring TLS offloading

To configure TLS Offloading:

Ensure that all the Blackboard Learn Application Servers' HTTP ports cannot be accessed directly from outside the firewall and implement a rule on the Load Balancer to redirect any http requests to https for the site.
In Learn 9.1 April 2014 and later:
- For Learn to be aware that TLS is offloaded to the Load Balancer the following headers need to be added: X-Forwarded-Proto: httpsX-Forwarded-For
  Refer to the load balancer vendor's documentation for details on adding X-Forwarded headers.
For prior Learn versions:
- In the bb-config.properties file on each Application Server set:
  bbconfig.appserver.ssl.offloaded=true
Some features within Blackboard Learn (such as hostname based branding and authentication provider hostname rules) require that the hostname be specified by the user. The parameter bbconfig.appserver.ssl.offloaded.hostname can be used to hard code the name when the Load Balancer does not pass through the host header from the users' requests. For example:
bbconfig.appserver.ssl.offloaded.hostname= loadbalancer.blackboard.com

(Replace loadbalancer.blackboard.com with the name of your Load Balancer)

However some features will be unavailable. Normally this parameter should be empty.
It is best practice to install TLS certificates on the application nodes even though they will not be accessed by end users. This is so administrators can access the application severs directly, bypassing the Load Balancer for testing and troubleshooting.

Configure TLS choice in Blackboard Learn

In Learn 9.1 April 2014 and later, TLS is not optional. Full TLS is enabled by default.

Learn more about TLS, which is the successor to SSL.

For prior Learn versions, if SSL is used (whether via SSL Re-encryption or SSL Offloading), Blackboard Learn must be configured to use system-wide SSL from the Admin Panel. See SSL Choice.

Health checking

Blackboard recommends monitoring the system for liveliness to ensure that the Load Balancer is sending traffic to Application Servers that are able to handle the traffic.

Blackboard includes a Health Check page designed for Load Balancers and IT infrastructure monitoring systems such as NAGIOS and NetIQ AppManager. This page can be requested by the health check service every 15-20 seconds. To avoid skewing the statistics or adding an additional load, this page does not create a Blackboard session. The Health Check page is:

http://appservername/webapps/portal/healthCheck

(Note the capital "C" in "healthCheck".)

If the server is up and running a response will be displayed similar to:

Hostname: appserver01.sup.local
Status: Active - Database connectivity established
Running since: Tue, Jul 26, 2011 - 06:25:45 AM EDT
Time of request: Tue, Jul 26, 2011 - 09:18:25 AM EDT
Clustering: disabled

A successful call to this page with return a HTTP status code 200.

An unsuccessful call will return a HTTP status code 5xx with 503 being the most common.

If the Application Server is active but a database issue has occurred, the HTTP status code 503 will be returned and the response will be similar to:

Hostname: appserver01.sup.local
Status: Service unavailable. An exception occurred either while trying to connect to or querying the database
Running since: Tue, Jul 26, 2011 - 10:25:45 AM EDT
Time of request: Tue, Jul 26, 2011 - 10:27:25 AM EDT
Clustering: disabled
Exception: [java stack trace...]

Health checking interval

When performing a health check, it is important to set a health check interval that is frequent enough to maintain the health of the application, but not so frequent that it affects performance. When a health check fails for a specific Application Server, the Load Balancer redistributes the traffic that was bound for that server to the remaining available servers. This interval should be optimized to ensure that healthy nodes do not inadvertently fail the health check during normal processing. Blackboard considers intervals such as one minute or 0.1 seconds as extreme values and recommends that they be avoided. Blackboard recommends starting with a health check interval of 20 seconds and blocking traffic to a node after three failures (60 seconds).

Persistence requirements

Blackboard applications require session persistence to ensure all requests from the same session are forwarded to the same Application Server. This is achieved by the use of cookie insertion.

Having a persistence cookie on HTTP is not required if you use system-wide TLS and redirect all HTTP request to HTTPS. Otherwise, the same cookie (name) should be used for both HTTP and HTTPS protocols to ensure correct cross protocol persistence.

Blackboard recommends configuring the Load Balancer for cookie-insert based session affinity. The Blackboard session cookie should not be used as there are separate cookies session_id for HTTP and s_session_id for HTTPS.

Idle timeout

To avoid active sessions being sent to the wrong Application Server, Blackboard recommends that the idle timeout of the persistence cookie (cookie-insert) inserted at the Load Balancer should be set to greater than the Blackboard session timeout by at least 1 minute.

The default session timeout for Blackboard is 3 hours. This is defined in the invalid property in the bb-tasks.xml file. (10800000 milliseconds = 3 hours.)

<task-entry key="bb.session.invalidation" version="60">
    <task classname="blackboard.platform.session.impl.SessionInvalidationTask">
        <property name="delay" value="60000" />
        <property name="period" value="3600000" />
        <property name="invalid" value="10800000" />
    </task>
</task-entry>

X-Forwarded-For header

Blackboard has some features that use the client's IP address as part of their function. For example, IP restriction in Tests and Assessments, and Session Fingerprinting.

In a load balanced environment the IP address passed is usually the IP address of the Load Balancer. To preserve the original client IP address most Load Balancers support the insertion of an X-Forwarded-For header. This should be added when configuring the Load Balancer.

If you are using another proxy in the chain such as ARR confirm it is not setting a X-Forwarded-For header as this would overwrite the X-Forwarded-For set by the load balancer.
e.g. on ARR to avoid overwriting the header change the name of the header set by ARR. Application Request Routing Cache->Server Proxy Settings>Preserve client IP in the following header-> X-Forwarded-For-xx

Blackboard Learn will automatically (9.1 SP 10+) use the forwarded client IP.

Configuring Tomcat Valve

Some clients have found the IP address not persisting correctly due to an idiosyncrasy of the Tomcat Valve framework. If, after setting the X-Forwarded-For header in the Load balancer, the blackboard/logs/bb-access-log.nnnnnn.txt is still missing the client's IP address, then Tomcat may need the IP of the Load Balancer added to the configuration as an allowed proxy. In this situation open the blackboard/config/tomcat/conf/server.xml.bb file and locate the line:

Add in the IP address of the Load Balancer (the one listed in bb-access-log.nnnnnn.txt) using a backslash (\) to escape the periods (.). For example:

If the Load Balancer is in an active/active or active/passive pair configuration, add both separated by a comma. For example:

Setting the bbconfig.frontend.fullhostname

On all Application Servers in a load balanced environment, the bb-config.properties file needs to be edited and the bbconfig.frontend.fullhostname setting updated to reflect the hostname of the Load Balancer. This is the hostname that end users use to access the Blackboard Service. (If the bbconfig.frontend.fullhostname is not set to the Load Balancer service address, URLs may point directly to the Application Server and, if the Application Servers are secured with a firewall or are on a private network, users will experience http 404 responses.)

The bbconfig.appserver.fullhostname and bbconfig.appserver.machinename must be unique for each Application Server.

Set the local hosts file

To avoid installer errors when running a load balanced environment, Blackboard recommends that you add an entry to the hosts file on each Application Server to redirect the Load Balancer hostname to localhost. This is done so that the Application Server makes any DNS resolution internally to itself rather than trying to communicate through the Load Balancer. For example:

On Unix, add a line like this to /etc/hosts:

127.0.0.1 localhost loadbalancer.blackboard.com

On Windows, add a line like this to C:\WINDOWS\SYSTEM32\DRIVERS\etc\hosts:

127.0.0.1 localhost loadbalancer.blackboard.com

(Replace loadbalancer.blackboard.com with the name of your Load Balancer)

Setting a local host file may prevent some features, such as the Math Editor, from functioning as expected. In this case, you may want to comment out this line in the file.

The primary Application Server

Certain Blackboard system tasks need to be executed on only one server, while other tasks need to be executed on all servers. Since 9.1 Sp10 Blackboard introduced setting to easily control which Application Server is the 'primary' backend processor and avoid the need for manual configuration of the tasks.xml files.

The setting is controlled by this line in the bb-config.properties file:

bbconfig.server.backend.processor=true

This setting should be set to true on one Application Server only and false on all the others. The one where it is set to true is referred to as the Primary Application Server. Ideally this server should be started first and stopped last when administering the cluster. Any Application Server can be the primary, so if the primary fails another server should be reconfigured as the primary as soon as possible. (Ideally, within 3 hours.)

If you change the setting in bb-config.properties, run PushConfigUpdates(.bat|sh) afterwards to activate or deactivate processing of the system tasks.

System tasks and command line scripts take additional resources such as processing and memory. So it is best practice to have the Primary Application Server outside the load balanced pool, i.e. not serving pages to end users. On larger services, you may use the Primary Application Server purely for administration and management. However, the Collaboration Server can also double up as the Primary Application Server.

Configure the Collaboration Server

The Collaboration Server is a standalone service that provides Chat and Virtual Classroom services.

The Collaboration Server is not related to Blackboard Collaborate applications such as Blackboard Collaborate web conferencing.

Only one Collaboration Server can be active in a load balanced environment. Attempting to load balance multiple Collaboration Servers will result in users being isolated from each other on different servers.

Turn an Application Server into a Collaboration Server

To make any Application Server into the Collaboration Server:

In the bb-config.properties file on the Application Server, set bbconfig.collabserver.run.on.localhost to true.
Execute the command ServiceController.(bat|sh) services.collabserver.start.

Specifying the Collaboration Server

There are two ways to specify the Collaboration Server:

You can choose one Blackboard Application Server to run the Collaboration Services. This can be a dedicated Collaboration Server or one of the Application Servers.
Or, you can make the Collaboration Server available through the Load Balancer This can mitigate the Collaboration Server as a single point of failure. This can be useful because the standard configuration might make it necessary to change IP configurations or wait for DNS updates in the event of a host failure.

Standard Collaboration Server configuration

In the standard configuration, the Collaboration Server can be connected to directly through an external hostname. In this configuration end users must have access directly to the Collaboration Server hostname on ports 8010, 8011 and 8012.

Use the standard configuration

On each Blackboard Learn Application Server, set the following parameters in the bb-config.properties file:
1. Set bbconfig.config.collabserver.fullhostname.default to the fully qualified hostname of the Collaboration Server.
2. Set bbconfig.collabserver.run.on.localhost to false.
In the bb-config.properties file on the Collaboration Server, set bbconfig.collabserver.run.on.localhost to true.
Run PushConfigUpdates.(bat|sh).

Alternate Collaboration Server configuration

The Collaboration Server can be configured to be available through the Load Balancer allowing for simple manual failover.

How to use the alternate configuration

On the Load Balancer, configure the Collaboration Server to be available on ports 8010, 8011 and 8012 as IP layer 4 services.
On each Blackboard Learn Application Server, set the following parameters in the bb-config.properties file:
1. Set bbconfig.config.collabserver.fullhostname.default to the fully qualified hostname of the Blackboard Load Balancer
2. Set bbconfig.collabserver.run.on.localhost to false.
In the bb-config.properties file on the Collaboration Server, set bbconfig.collabserver.run.on.localhost to true.
Run PushConfigUpdates.(bat|sh).

Handle the failure of the Collaboration Server

If the Collaboration Server or its host fails, set bbconfig.collabserver.run.on.localhost to true in the bb-config.properties file of one of the other Application Servers and run PushConfigUpdates.(bat|sh). This will restore the Collaboration service.

If the Collaboration Service fails over to another host, all existing sessions will be lost.

Shut down an Application Server

Before shutting down an Application Server, any queued tasks should be allowed to complete. If this is not possible, then the tasks should be re-assigned manually to another Application Server so that they may complete.

How to tell if there are queued tasks

In Blackboard Learn, select System Admin to open the Administrator Panel.
In the Tools and Utilities section, select Logs.
Select System Tasks Status.
In the filter, make sure that both Type and Status are set to Show All.
Any tasks with "Running" or "Assigned State" in the Status column have not yet been completed.

Manually assign queued tasks to another Application Server

To reassign the tasks to another node execute the following SQL Query:

update queued_tasks set process_node = 'name_of_server_to_assume_queued_tasks' where process_node = 'name_of_server_being_shut_down' and status in ( 'A', 'R' )

("A" is the assigned state, and "R" is running state.)

Язык