If you are a Managed Hosting customer, this topic doesn't apply to you.

Oracle Real Application Clusters (RAC) provides advantages such as high availability and failover by clustering multiple database instances. The Oracle RAC architecture helps clients support high-volume and critical applications.

It is strongly recommended by Blackboard that for purposes of system management Blackboard Learn be installed on a single instance of Oracle. If for reasons of Institutional Policy you must run Blackboard Learn in a RAC environment you may do so, but Blackboard will only support the installation at the Learn-Oracle JDBC layer - not at the Oracle RAC layer.

The Blackboard Learn 9.1 installer uses a thin Java Database Connectivity (JDBC) driver and performs software upgrades through a single Oracle instance. The configuration of the Oracle Call Interface (OCI) driver is easier to use in Blackboard Learn 9.1 than in previous releases. Clients running Blackboard Learn against Oracle RAC can run Blackboard Learn in a single database instance mode, against one of the Oracle RAC instances immediately after installation, by switching Blackboard Learn to Oracle RAC mode.

To upgrade, you must first switch Blackboard Learn to single instance mode, perform the upgrade, and then switch back to Oracle RAC mode.

Because the Oracle RAC architecture has been set up to support Blackboard Learn, this section explains how to switch Blackboard Learn from single database instance mode to Oracle RAC mode and vice versa. This section also explains how to install and upgrade Blackboard Learn against Oracle RAC.

Blackboard Learn 9.1 installs with a full set of libraries for 64-bit systems, which allows for a seamless configuration to connect to a thick driver.

The procedures in this section have been successfully tested against Oracle 10g, 11gR1, and 11gR2.

Prerequisites

  • Blackboard Learn 9.1 has been set up, configured, and tested against a single database instance.
  • Blackboard Learn 9.1 has been installed, upgraded, and configured. It has also been tested and updated with any customizations or advanced configurations.
  • Oracle RAC has been properly installed and set up according to Oracle's documentation. Blackboard recommends that you use Oracle RAC 11.1.0.7+ for better performance and availability.
  • You are using a 64-bit OS, Oracle client, Oracle RAC, and JVM.
  • Failover has been tested at the database level with the Oracle RAC environment according to Oracle's documentation.

Change from a single instance mode to RAC mode

The alias for the bbconfig.database.type.oracle.tns parameter must match the one defined in the tnsnames.ora.bb file.

  1. Stop all application server services using the following command:

    #/usr/local/blackboard/tools/admin/ServiceController.sh services.stop

  2. Make a backup copy of your Blackboard Home directory using the tar command as follows:

    #tar -cvf blackboardnorac.tar /usr/local/Blackboard Home

    exclude content directory (bbcontent)

  3. Check the Oracle environment variables for both root and bbuser. These properties are needed for testing the connection to SQL Plus and allowing Java to find the OCI library path.

    export ORACLE_BASE=/usr/local/Blackboard Home/apps
    export ORACLE_HOME=$ORACLE_BASE/oracle-client
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib64:$LD_LIBRARY_PATH
    export PATH=$ORACLE_HOME/bin64:/usr/local/Blackboard Home/tools/admin:$PATH

  4. Perform a backup of your database in case a recovery is required according to Oracle's documentation.

  5. Make sure Blackboard Learn 9.1 is running in single instance mode against one of the Oracle RAC instances.

  6. Make a backup of the tnsnames.ora file located in the Oracle Home/network/admin directory. (If your installation includes a tnsnames.ora.bb file in the same directory, also back this up.)

    #cp tnsnames.ora tnsnames.ora.single_$YYMMDD

    where $YYMMDD represents the backup date (for example, 120104 for January 4, 2012).

    Blackboard Learn 9.1 has embedded the Oracle client. The Oracle Home directory is defined in the bbconfig.database.local.oracle.home property in the bb-config.properties file. The default Oracle Home directory is located in the /usr/local/Blackboard Home/apps/oracle-client directory.

  7. Configure the Oracle RAC entry in the tnsnames.ora file by navigating to Oracle Home/network/ adminand adding the tns entry to the tnsnames.ora.bb file according to your RAC environment configuration. For example, if your RAC environment has two instances on nodes rac01.xyz.com and rac02.xyz.com with a service name of bb-rac-db, the tns entry would be as follows:

    bbrac =
    (DESCRIPTION=
    (LOAD_BALANCE=on)
    (FAILOVER=on )
    (ADDRESS=
    (PROTOCOL=tcp)(HOST = rac01.xyz.com)(PORT=1521)
    )
    (ADDRESS=
    (PROTOCOL=tcp)(HOST = rac02.xyz.com)(PORT=1521)
    )
    (CONNECT_DATA=
    (SERVICE_NAME= bb-rac-db)
    (FAILOVER_MODE=(TYPE=select)(METHOD=basic))
    )
    )

    The alias in the code example above should be the alias for the Virtual IP (VIP) rather than the public or private address.

  8. Test Oracle RAC with Oracle Plus by connecting to the Oracle RAC tns datasource. This connection should be successful. If not, do not proceed. Continue troubleshooting this issue until a connection to the database is successful.

    #sqlplus bblearn_admin@bbrac

  9. Update the bb-config.properties file to specify the JDBC driver for OCI and the TNS for Oracle RAC as follows:

    bbconfig.oracle.client.drivertype=oci
    bbconfig.database.type.oracle.tns=bbrac

  10. Because the content system is sharing the same database as Blackboard Learn, its connections can be changed to Oracle RAC as follows:

    sqlplus bblearn_admin@bbrac
    SQL> select DB_USERNAME,DB_PASSWORD,JDBC_CONNECTION_URL from $cms_schema.XY_FILE_SYSTEMS;
    SQL> update $cms_schema.xy_file_systems set jdbc_connection_url='jdbc:oracle:oci:@bbrac '
    2 /
    2 rows updated.
    SQL> commit;
    Commit complete.

    In the example above, the services name bbrac is equal to the Oracle RAC TNS entry you made earlier in this procedure. Substitute the $cms_schema variable with the actual CMS schema name (for example, bbleam_cms).

  11. Run PushConfigUpdates to activate all the changes. To learn more, see PushConfigUpdates.

  12. Confirm that Blackboard Learn and the Oracle RAC are working as expected by testing load balancing and failover.

Change Blackboard Learn from Oracle RAC mode to single instance mode

  1. Ensure that Blackboard Learn is running well in Oracle RAC mode.
  2. Stop all services on the application server using the following command:

    #/usr/local/Blackboard Home/tools/admin/ServiceController.sh services.stop

  3. Update the bb-config.properties file to set the JDBC driver to thin, and then unset this as follows:

    bbconfig.oracle.client.drivertype=thin
    bbconfig.database.type.oracle.tns=

  4. Also in the bb-config.properties file make sure the bbconfig.database.server.* properties point to one of the Oracle RAC instances. For example:

    bbconfig.database.server.name= rac01
    bbconfig.database.server.fullhostname= rac01.xyz.com
    bbconfig.database.server.instancename= bb-rac-db01
    bbconfig.database.server.portnumber=1521
    bbconfig.database.server.systemuserpassword=oracle

  5. Change the content system from RAC mode to single instance mode by changing the jdbc_connection_url field setting in the XY_FILE_SYSTEMS table under the CMS schema, shown below:

    jdbc:oracle:thin:@$bbconfig.database.server.fullhostname:$bbconfig.database.
    server.portnumber:$bbconfig.database.server.instancename

    Using the sample configuration above, change your mode as follows:

    SQL> select DB_USERNAME,DB_PASSWORD,JDBC_CONNECTION_URL from $cms_schema.XY_FILE_SYSTEMS;
    SQL> update $cms_schema.xy_file_systems set jdbc_connection_url='jdbc:oracle:thin:@rac01:1521: bb-rac-db01'
    2 /
    2 rows updated.
    SQL> commit;

  6. Run PushConfigUpdates to activate all the changes. To learn more, see PushConfigUpdates.
  7. Confirm that Blackboard Learn is working properly.

Install Blackboard Learn on Oracle RACs

  1. Be sure you have completely installed and set up your Oracle RAC environment according to Oracle's documentation before installing Blackboard Learn.
  2. Install Blackboard Learn, and connect to one of the Oracle RAC nodes.
  3. Follow the steps provided in Changing From a Single Instance Mode to RAC Mode to switch Blackboard Learn to Oracle RAC mode.
  4. Use redundant interconnect to optimize stability, reliability, and scalability of Oracle RAC. To learn more, see http://docs.oracle.com/cd/B19306_01/rac.102/b14197/monitor.htm.

Upgrade Blackboard Learn on Oracle RACs

  1. Be sure the database server properties are set properly to point to one of the Oracle RAC nodes. To learn more, see Changing Blackboard Learn From Oracle RAC Mode to Single Instance Mode.
  2. Stop all application server services using the following command:

    #/usr/local/blackboard/tools/admin/ServiceController.sh services.stop

  3. Upgrade Blackboard Learn, and connect to one of the Oracle RAC nodes.
  4. Follow the steps provided in Changing From a Single Instance Mode to RAC Mode to switch Blackboard Learn to Oracle RAC mode.
  5. Confirm that Blackboard Learn and the Oracle RAC are working as expected by testing load balancing and failover.
  6. Use redundant interconnect to optimize stability, reliability, and scalability of Oracle RAC. To learn more, see http://docs.oracle.com/cd/B19306_01/rac.102/b14197/monitor.htm.

Operating system, JVM, Oracle client library, and bb-config.properties file configurations for Oracle RAC

The Blackboard Learn 9.1 installer, by default, ensures the correct properties are set for the OS bit, JVM bit, Oracle client library, and the bbconfig file. The installer dynamically checks the OS and JVM for the Oracle RAC environment and selects the correct file properties for bbconfig and the Oracle client library files as follows:

bbconfig.platform.64bit=true
## must mirror the above property and be either lib32 or lib64 ##
bbconfig.platform.libdir=lib64

Otherwise, you will encounter the following error when running PushConfigUpdates.sh:

"Error message:
[java] Error initializing Blackboard services: /usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so: /usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so: wrong ELF class: ELFCLASS64 (Possible cause: architecture word width mismatch)
[java] blackboard.base.InitializationException: Error initializing Blackboard services: /usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so: /usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so: wrong ELF class: ELFCLASS64 (Possible cause: architecture word width mismatch)
checking lib:
file /usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so
/usr/local/blackboard/apps/oracle-client/lib64/libocijdbc11.so: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped"

Troubleshooting

Oracle RAC 10.2.0.4 or 11.1.0.7

6.1 ORA-00600: internal error code

Problem:

ORA-00600: internal error code, arguments: [kkocxj : pjpCtx] while complex sqls.

Solution:

This is an Oracle RAC bug. To resolve it, log on to Oracle RAC as sysdba and alter the following system setting:

"_optimizer_push_pred_cost_based" = false scope =both;

Reference

https://community.oracle.com/thread/838121