If you are a Managed Hosting customer, this topic doesn't apply to you.

Looking for the latest release notes? View release notes for Redis Cache.

About application caching

Application caching is critical to running any large-scale web application. Understanding how application caching works and what some of the options you have to help your Blackboard Learn instance run efficiently, helps you make decisions about investing in hardware and software.

More on caches in computing

Ehcache

Ehcache is an open source cache library that Blackboard Learn uses by default. Its configurations for the core caches are defined in your Blackboard Learn installation inside the config/cache-settings.properties file.

More on cache-settings.properties Parameters

Redis Cache

The Redis Cache Building Block enables Blackboard Learn's cache framework to use Redis as an additional type of memory store outside the JVM heap. Redis is an open source, high-performance, networked advanced key-value store. Other companies that use Redis include Twitter, Craigslist, Github, Stackoverflow, and Instagram.

To get started with the Redis Cache Building Block, you must have Blackboard Learn 9.1 SP 11 or later and a Redis server.

Performance, scalability, and availability highlights

  • Increases request throughput and response times
  • No risk of performance loss
  • Cache is shared across all application servers and the hottest subset of data is kept local
  • Cache is available under JVM heap pressure
  • Enables the system to scale with a manageable JVM heap size
  • Bigger cache and faster overflow storage
  • Fast and efficient object graph serialization
  • Asynchronous puts and updates via write-behind strategy
  • Increases off-heap caching opportunity
  • Reduces on-heap cluster invalidation overhead
  • Caches that survive application restarts and crashes
  • Rich monitoring features for preventive maintenance and troubleshooting

Redis Cache in action

Discover how to use Redis Cache to boost your efficiency when diagnosing and troubleshooting your Blackboard Learn servers.

Video: Introduction to Redis Cache

When do you need it?

The Redis Cache building block is recommended if one or more of these scenarios are observed:

  • High cache eviction rate
  • System resource pressure on the database server
  • Pressure on JVM heap utilization
  • High usage of Ehcache disk overflow
  • Horizontally large scale Blackboard Learn deployments with high cluster cache invalidation traffic (needs additional step to reduce the Ehcache size to reduce the traffic volume)

Install and activate Redis Cache

You can use Software Updates to install the Redis Cache building block directly from the Administrator Panel. Alternatively, you can visit Behind the Blackboard Downloads and find the Redis Cache building block to download and install manually.

Software Updates

Software Updates are located on the System Admin panel. When a new building block is available or has been updated, you are notified by a change in the icons. Select the Feature Building Blocks Available icon and locate the Redis Cache building block. Select Install. You will need to make the building block available after it has been installed.

Behind the Blackboard

Download the Redis Cache building block from Behind the Blackboard Downloads home, located under Featured Building Blocks for Learn. Save the file on a local drive. Unzip the package to access the .war file.

  1. On the Administrator Panel, go to Building Blocks > Installed Tools.
  2. Select Upload Building Blocks.
  3. Browse for the Redis Cache building block .war file.
  4. Select Submit.
  5. You will get a success message when the building block is installed. Set the building block to Active and select Approve.

Enable Redis Cache

After you install a Redis server, go to the Redis Cache building block settings page on the Administrator Panel.

  • Select the list of caches to enable Redis.
  • Provide Redis connection settings in the Host, Port, and Password fields. In newer versions of Blackboard Learn, these settings are in the bb-config.properties file.
  • Select Submit.
  • Make sure that the Redis server port and the RMI ports are open among Redis server and the Blackboard Learn application servers.
  • Restart all Blackboard Learn application servers. A rolling restart is recommended for production environments.
  • Navigate to the Redis connection status page on the Administrator Panel.
  • Verify that all application servers are connected to Redis.

Install Redis server

These steps assume that you are installing Redis on Red Hat Enterprise Linux 6.

  1. Prepare a Linux server. A dedicated machine or VM instance is highly recommended. Don't run Redis on Windows or Solaris. Blackboard Learn on Windows and Solaris can use Redis on Linux.
    • Set the Linux kernel overcommit memory setting to 1 by adding vm.overcommit_memory=1 to /etc/sysctl.conf. Reboot the server or run the command sysctl vm.overcommit_memory=1 for the change to take effect. More information is available in the Redis FAQ.

      Redis is a single-threaded server. It favors fast CPUs with large caches instead of many cores. Intel CPUs are currently known to be the best fit. Redis documentation states that it is not uncommon to get only half the performance on an AMD Opteron CPU compared to similar Nehalem EP, Westmere EP, or Sandy bridge Intel CPUs.

    • Redis throughput is limited by the network well before the CPU. Use fast and/or multiple NICs.
    • (Optional) Redis runs slower on a VM, so they recommend running Redis on a physical box.
  2. Download and build the latest stable version of Redis.

    wget http://download.redis.io/redis-stable.tar.gz
    tar xvzf redis-stable.tar.gz
    mv redis-x.x.xx /usr/local/redis
    cd redis-stable
    make

  3. Prepare the Redis server configuration file by copying $REDIS_HOME/redis.conf. Set the following properties:

    daemonize yes # Use "no" when run under daemontools.
    pidfile /var/run/redis/redis_6379.pid
    port 6379
    timeout 300 # Reclaim inactive connections (seconds).
    tcp-keepalive 0 # Keep connections live from network tools likefirewall.
    loglevel notice # Keep it moderately verbose.
    logfile /var/log/redis/redis_6379.log
    databases 1
    #save 900 1 # Disable snapshotting.
    #save 300 10 # Disable snapshotting.
    #save 60 10000 # Disable snapshotting.
    requirepass xxxx # Use at least 16 characters with mix lowercase and uppercase
    # characters, numbers, and symbols. Try not to use words or
    # phrases in your password.
    maxclients 100000
    maxmemory xxxx # Make sure Redis doesn't use swap (bytes).
    maxmemory-policy volatile-lru # We use expire.
    maxmemory-samples 3
    appendonly no
    slowlog-log-slower-than 50000 # Execution time threshold (microseconds).
    slowlog-max-len 128 # Length of the slow log.

  4. Add a Redis home directory to root.

    # Redis
    export REDIS_HOME=/usr/local/redis

  5. Create a Redis user.

    useradd -d /home/redis -s /bin/sh redis
    chown -R redis:redis $REDIS_HOME
    chmod 700 $REDIS_HOME

  6. Prepare the Redis init script.

    cp utils/redis_init_script /etc/init.d/redis_6379

  7. Configure the Redis init script.

    # chkconfig: - 85 15
    # description: Redis is a persistent key-value database
    # processname: redis
    REDISUSER="redis"
    REDISPORT=6379
    EXEC=/usr/local/redis/src/redis-server
    CLIEXEC=/usr/local/redis/src/redis-cli
    PIDFILE=/var/run/redis/redis_6379.pid
    CONF="/usr/local/redis/6379.conf"
    $EXEC $CONF ==change to==> /bin/su - $REDISUSER -c "$EXEC $CONF"

  8. Activate the Redis service.

    mkdir /var/run/redis
    mkdir /var/log/redis
    chown redis:adm /var/run/redis /var/log/redis
    sudo chmod 750 /var/log/redis
    cd /etc/init.d
    chkconfig --add redis_6379

  9. Start the Redis server.

    service redis start

Redis Cache Server Monitoring

Version 1.2 and above: Redis Cache Server Monitoring is accessible through the Redis Cache link in Tools and Utilities on the Administrator Panel.

Version 1.1 and below: Redis Cache Server Monitoring is accessible through the Admin Console. Look for Redis Cache navigation link in the main menu. If you do not have Admin Console installed, you can access Redis Cache Server Monitoring by manually pointing your browser to the URL path, /webapps/bb-redis-cache-BBLEARN/execute/redisStats.

Security

Harden the Redis server

Two components secure the Redis server:

  • Adding proper security configurations to the Redis application
  • Adding secure configurations to the server running Redis

Secure Redis settings make sure that the Redis application is as secure internally as it can be. Server configurations secure the server and the Redis application without involving the Redis application itself. The following sections describe two types of configurations.

Secure Redis settings

The two important settings for the Redis server itself are:

  • Using a secure password for Redis
  • Blocking access to some Redis commands

More on Redis' recommended security configuration

Password

The Redis server is very efficient at serial string lookups, which is exactly what the Redis password lookup request is. Additionally, the Redis password is required only by the Blackboard application. Users do not need to remember it, so the password used should be much stronger than a traditional password. Normally you should follow the owasp password length and complexity guidelines for setting password length, but because end users do not need to remember this password, the convention of choosing passwords based on users' ability to remember the password can be ignored.

Guideline Reason
Use at least 16 characters, instead of 8 A brute force attack would take n to the 8th power times as long as an 8 character password, which is the normal minimum.
Mix lowercase and uppercase characters, numbers, and symbols As with normal passwords, mixing case, numbers, and symbols reduces the likelihood of a successful dictionary attack and adds complexity for a brute force attack because all types of characters need to be checked.
Try not to use words or phrases in your password Because this password does not have to be remembered, you can remove dictionary attacks entirely by not using any words or phrases.

The password setting itself must be added to the Redis configuration file, using the requirepass directive, such as:

requirepass ^a@$4J0|-|k!4P+f%

Block Redis commands

Because Blackboard is using Redis just as a cache, a number of commands are not being used. Some of these commands could be used by a malicious user to cause Denial of Service via cache misses, denial of service by running the server out of space, as well as information exposure by allowing malicious users to obtain the entire set of cache data. Redis provides an interface for altering or dropping these commands entirely, which Blackboard recommends for a few specific commands:

Key Removal recommended because
APPEND Allows modification of keys to force cache misses
BGSAVE Command that saves the dataset to disk, which could allow information exposure if a user obtains the file
RENAME Renames a key, which would force cache misses
SAVE Same as BGSAVE
SPOP Removes a random member from the set
SREM Removes one or more members from the set

To block these commands, edit the Redis configuration file to add the following:

rename-command APPEND ""
rename-command BGSAVE ""
rename-command RENAME ""
rename-command SAVE ""
rename-command SPOP ""
rename-command SREM ""

By renaming the command to the empty string, you block anything from accessing it. Additionally, any new Redis commands that are created should also be blocked in this way because the building block will not be using them.

Secure server configuration

Securing the server running Redis means taking steps to ensure that only users who need access to the Redis server can access it, as well as preventing unauthorized access to the Redis application through file and network permissions.

Run Redis without root context

The user root, on Linux machines, has full access to any file on the system, regardless of permissions or access control. This is why it is dangerous to use the root user to run applications because well-crafted attacks could allow a malicious user to control the account running that application. Because Redis does not need that level of access to the system, a user account with decreased permissions, just enough to run Redis, should be used to install and run the Redis application.

Blackboard recommends not using this account for anything except running the Redis server. It should not have any permissions to the operating system or other applications or daemons running on this server.

Secure the Redis configuration file

The password used by the Redis server, which was described in an earlier section, is stored in cleartext within the configuration file. This is outside of Blackboard's control because this is how Redis functions at this time. So, you need to limit the users who have access to the file. It should be limited only to the user that runs the Redis application.

This can be accomplished using the following command, assuming that the redis configuration file is redis.conf:

chown 600 redis.conf

Block unnecessary traffic to Redis server

The Redis server, assuming the only application running on it is Redis, should not be accessible to the open internet. It should be able to be reached only in the following circumstances:

  • An administrator needs to do work on the Redis server
  • Blackboard communications over the Redis port

This will prevent unauthorized users from attempting to bring down the Redis server, run unauthorized commands in Redis, and so on. Assuming that the default Redis port, 6379 is in use, add the following types of rules to the firewall for this server:

Rule Explanation
Block all traffic to and from the server This should be the default behavior, except in the case of the exceptions below.
Create an exception for loopback requests The server should be able to access itself, if for no other reason than monitoring incoming requests to Redis using the redis-cli commandmonitor.
Create an exception for traffic from and to the Blackboard servers only over Redis port The Blackboard server should be able to communicate with Redis over port 6379, and Blackboard should be able to get responses from the Redis server.
Create an exception allowing for remote administration from a specific machine Administrators may need access to the server for the purposes of upgrading Redis, checking logs, monitoring performance, and so on. Unless administrators have direct access to the machine, they will need to be able to access the server remotely. SSH traffic should be limited to a single machine, or a small set of machines, so normal users can't communicate to the server over port 22.

Administrators will also need to create exceptions for environment specific items, such as required shared resources. These will be specific to each institution.

Implement traffic monitoring

In case of attack against the Redis server from an outside source, some form of intrusion detection should be in place. A few items that can be monitored to flag traffic as possibly malicious, or drop the traffic entirely:

  1. If anything other than a Blackboard application server or system administration server tries to access the Redis server, it should be flagged.
  2. If a bad password is entered, it should be flagged.
  3. If too many requests are made to Redis server from a single location, flag that traffic. Monitoring typical usage patterns, as well as speculating on edge cases (say, during finals week), can help determine how many requests per minute are expected.

Implementing these intrusion detection rules will help determine whether anyone is trying to penetrate the Redis server or application.

Building Block permissions

Java Type Name Actions Why is this permission necessary?
persist * *
  • The Settings page needs to persist and load configuration data into SYSTEM_REGISTRY table
  • Persists and loads Redis server and cache statistics for the specified duration to the Stats database for trend analysis
java.io.FilePermission 1. BB_HOME/-

2. BB_CONTENT/-

3. "${java.home}/-

 1. read, write, delete

2. read, write, delete

3. read
  • Reads properties files under Blackboard Learn home directory
  • Saves a hash value unique to a Blackboard Learn instance, shared by all Blackboard Learn application servers within a cluster. It is used in Redis cache key to avoid collision from other Redis cache usage by another Learn instance or other applications.
  • Ehcache diskoverflow feature
java.lang.reflect.
ReflectPermission		 suppressAccessChecks * Used by Spring framework and an object graph serialization framework, Kryo
java.lang.
RuntimePermission		 * * Used by various Blackboard Learn components including log services and service managers
java.net.
SocketPermission		 * connect, accept, resolve, listen Connection to Redis server and peer Blackboard Learn application nodes
javax.management.
MBeanServerPermission		 * * To expose cache statistics to MBean
javax.management.
MBeanPermission		 * * To expose cache statistics to MBean
attribute user.authinfo get Gets user roles for operations