Several NFS appliances and big-iron cluster filesystems are very common in large cluster database deployments. We will use OCFS2 here to practice 11gR2 RAC with a filesystem.

1. As the oracle user, launch the grid installer. At the first screen, choose Install and Configure Grid Infrastructure for a Cluster and click NEXT. [oracle@collabn1 ~]$ /mnt/cdrom*/grid/runInstaller
   

   
   

   
2. Choose Advanced Installation and click NEXT.
   
3. Accept the default language (English) and choose NEXT.
   
4. Name the cluster collab and make sure that the SCAN name is collab-scan with port 1521, then click NEXT.
   
5. Add node collabn2 with VIP collabn2-vip and choose NEXT to validate the cluster configuration.
   
6. Verify that eth0 on subnet 192.168.78.0 is PUBLIC and that eth1 on subnet 172.16.100.0 is PRIVATE, then click NEXT.
   
7. Choose to store the Clusterware Files in Shared File System and choose NEXT.
   
8. For the OCR, choose External Redundancy and type the path /u61/cluster/ocr. (This is the NFS location from the BUG WORKAROUND.) Click NEXT to continue.
   
9. For the Voting Disk, do the same – choose External Redundancy and type the path /u61/cluster/vdsk. (Again, this is the NFS location from the BUG WORKAROUND.) Click NEXT to continue.
   
10. Do not use IPMI. Click NEXT.
    
11. Set the OSDBA group to asmdba, the OSOPER group to asmoper and the OSASM group to asmadmin. Then click NEXT.
    
12. Accept the ORACLE_BASE location of /u01/app/oracle and use the ORACLE_HOME location of /u01/grid/oracle/product/11.2.0/grid_1. Then click NEXT.
    
13. Accept the default inventory location and choose NEXT
    
14. The prerequisite checks will execute. A warning will be issued saying that three checks failed: physical memory, swap size and network time protocol. Click the CHECK BOX to Ignore All, then click NEXT.
    
15. SAVE a response file called grid.rsp in the oracle user home directory. Then click FINISH to install grid infrastructure.
    

    
    

    
16. When prompted, open a terminal as the root user and run the two root.sh scripts. Make sure to run BOTH SCRIPTS on BOTH NODES!
    
    [oracle@collabn1 ~]$ su - Password: racattack [root@collabn1 ~]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@collabn1 ~]# /u01/grid/oracle/product/11.2.0/grid_1/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/grid/oracle/product/11.2.0/grid_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/bin Copying dbhome to /usr/bin ... Copying oraenv to /usr/bin ... Copying coraenv to /usr/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2011-03-30 16:49:13: Parsing the host name 2011-03-30 16:49:13: Checking for super user privileges 2011-03-30 16:49:13: User has super user privileges Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting CRS-2672: Attempting to start 'ora.gipcd' on 'collabn1' CRS-2672: Attempting to start 'ora.mdnsd' on 'collabn1' CRS-2676: Start of 'ora.mdnsd' on 'collabn1' succeeded CRS-2676: Start of 'ora.gipcd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'collabn1' CRS-2676: Start of 'ora.gpnpd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'collabn1' CRS-2676: Start of 'ora.cssdmonitor' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'collabn1' CRS-2672: Attempting to start 'ora.diskmon' on 'collabn1' CRS-2676: Start of 'ora.diskmon' on 'collabn1' succeeded CRS-2676: Start of 'ora.cssd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'collabn1' CRS-2676: Start of 'ora.ctssd' on 'collabn1' succeeded clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-2672: Attempting to start 'ora.crsd' on 'collabn1' CRS-2676: Start of 'ora.crsd' on 'collabn1' succeeded Now formatting voting disk: /u61/cluster/vdsk. CRS-4603: Successful addition of voting disk /u61/cluster/vdsk. ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 59f9f62cc0f44f35bff552ced06e35ee (/u61/cluster/vdsk) [] Located 1 voting disk(s). CRS-2673: Attempting to stop 'ora.crsd' on 'collabn1' CRS-2677: Stop of 'ora.crsd' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'collabn1' CRS-2677: Stop of 'ora.asm' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'collabn1' CRS-2677: Stop of 'ora.ctssd' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'collabn1' CRS-2677: Stop of 'ora.cssdmonitor' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'collabn1' CRS-2677: Stop of 'ora.cssd' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'collabn1' CRS-2677: Stop of 'ora.gpnpd' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'collabn1' CRS-2677: Stop of 'ora.gipcd' on 'collabn1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'collabn1' CRS-2677: Stop of 'ora.mdnsd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.mdnsd' on 'collabn1' CRS-2676: Start of 'ora.mdnsd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'collabn1' CRS-2676: Start of 'ora.gipcd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'collabn1' CRS-2676: Start of 'ora.gpnpd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'collabn1' CRS-2676: Start of 'ora.cssdmonitor' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'collabn1' CRS-2672: Attempting to start 'ora.diskmon' on 'collabn1' CRS-2676: Start of 'ora.diskmon' on 'collabn1' succeeded CRS-2676: Start of 'ora.cssd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'collabn1' CRS-2676: Start of 'ora.ctssd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'collabn1' CRS-2676: Start of 'ora.asm' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'collabn1' CRS-2676: Start of 'ora.crsd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'collabn1' CRS-2676: Start of 'ora.evmd' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'collabn1' CRS-2676: Start of 'ora.asm' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.DATA.dg' on 'collabn1' CRS-2676: Start of 'ora.DATA.dg' on 'collabn1' succeeded CRS-2672: Attempting to start 'ora.registry.acfs' on 'collabn1' CRS-2676: Start of 'ora.registry.acfs' on 'collabn1' succeeded collabn1 2011/03/30 16:59:35 /u01/grid/oracle/product/11.2.0/grid_1/cdata/collabn1/backup_20110330_165935.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 968 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful.

    
    

    [root@collabn1 ~]# ssh collabn2 root@collabn2's password: racattack -bash: oraenv: No such file or directory [root@collabn2 ~]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. [root@collabn2 ~]# /u01/grid/oracle/product/11.2.0/grid_1/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/grid/oracle/product/11.2.0/grid_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/bin Copying dbhome to /usr/bin ... Copying oraenv to /usr/bin ... Copying coraenv to /usr/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2011-03-30 17:04:26: Parsing the host name 2011-03-30 17:04:26: Checking for super user privileges 2011-03-30 17:04:26: User has super user privileges Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node collabn1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster CRS-2672: Attempting to start 'ora.mdnsd' on 'collabn2' CRS-2676: Start of 'ora.mdnsd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'collabn2' CRS-2676: Start of 'ora.gipcd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'collabn2' CRS-2676: Start of 'ora.gpnpd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'collabn2' CRS-2676: Start of 'ora.cssdmonitor' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'collabn2' CRS-2672: Attempting to start 'ora.diskmon' on 'collabn2' CRS-2676: Start of 'ora.diskmon' on 'collabn2' succeeded CRS-2676: Start of 'ora.cssd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'collabn2' CRS-2676: Start of 'ora.ctssd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.drivers.acfs' on 'collabn2' CRS-2676: Start of 'ora.drivers.acfs' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.asm' on 'collabn2' CRS-2676: Start of 'ora.asm' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'collabn2' CRS-2676: Start of 'ora.crsd' on 'collabn2' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'collabn2' CRS-2676: Start of 'ora.evmd' on 'collabn2' succeeded collabn2 2011/03/30 17:12:32 /u01/grid/oracle/product/11.2.0/grid_1/cdata/collabn2/backup_20110330_171232.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 1205 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /u01/app/oraInventory 'UpdateNodeList' was successful.

    
    

17. After running both scripts, return to the installer window and click OK to continue running configuration assistants.
    
18. The Cluster Verification Utility will fail because NTP is not running. If you want to, check the error message at the very end of the logfile. Then click OK to close the messagebox and click NEXT to continue.
    
19. You should now see the final screen! Click CLOSE to exit the installer.