Gunther Oracle APEX und Datenbank Wiki

Diese Seite ist nicht editierbar. Sie können den Quelltext sehen, jedoch nicht verändern. Kontaktieren Sie den Administrator, wenn Sie glauben, dass hier ein Fehler vorliegt.
===== Real Application Cluster - Infiniband  für den Interconnect ===== 


Konfiguration:
  * Infiniband Treiber einrichten
  * Oracle Rac mit UDP testen
  * Oracle Kernel auf RDS Umstellen


==== Kernel parameter überprüfen ====
^Parameter^Value^
|net.ipv4.ip_local_port_ range| 1024 65000|
|Net.core. rmem_default|262144|
|Net.core. rmem_max|262144|
|Net.core. wmem_default|262144|
|Net.core. wmem_max|262144|

===== Was für eine Karte ist installiert

<code>
$ lspci | grep Infini

...
 47:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0GT/s] (rev a0)
...
</code>

Treiber: \\
  * http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=26&menu_section=34#tab-three


==== Infiniband Treiber einrichten ==== 

Download des Treiber unter der SilverStorm website.\\


=== Installation ===
Oracle verwendet RDS als natives InfiniBand Protokol. \\
RDS benötigt IP over IB beide Protokolle müssen ausgewählt werden.


=== Konfiguration ===
Datei ipoib.cfg (vom Installer angelegt) prüfen.


=== Start / Stop ===
Prüfen ob RDS nach dem Boot läuft \\
Starten der RDS driver über "/etc/init.d/rds start"\\
\\
oder über:\\
<code>
# iba_start {rds | ics_init | ipoib…}
# iba_stop {rds | ics_init | ipoib…}
</code>

mit lsmod prüfen ob der Treiber im Kernel gebunden ist:
<code>
lsmod | grep rds
</code>

Infiniband Connect zwischen den Knoten überprüfen
<code>
#1. Check the list of connect hosts

[root@GPIrac1 ~]# ibnetdiscover -l
Ca       : 0x0002c99999999996 ports 2 devid 0x634a vendid 0x2c9 "GPIrac3 HCA-1"
Ca       : 0x0023788888888880 ports 2 devid 0x634a vendid 0x2c9 "GPIrac2 HCA-1"
Ca       : 0x002377777777778 ports 2 devid 0x634a vendid 0x2c9 "GPIrac1 HCA-1"
Switch   : 0x00227777777777a8 ports 32 devid 0xbd36 vendid 0x2c9 "Infiniscale-IV Mellanox Technologies"

#2. Get GuID of  connected Ports to ping

[root@GPIrac1 ~]# ibnetdiscover -p | grep GPI | awk '{print  $11 " ->  "  $17 }' | grep GPI
0x0002c9999999999 ->  'GPIrac3
0x0002c8888888888 ->  'GPIrac2
0x0002c7777777777 ->  'GPIrac1

#3. Start ping Server on the target hosts in a new ssh session

[root@GPIrac3 ~]# ibping -S
and
[root@GPIrac2 ~]# ibping -S
and
[root@GPIrac1 ~]# ibping -S

#4. Ping the target from all Hosts 

example for host 1
[root@GPIrac1 ~]# ibping -G 0x0002c9999999999
[root@GPIrac1 ~]# ibping -G 0x0002c8888888888
[root@GPIrac1 ~]# ibping -G 0x0002c7777777777

or
ibping -G 0x0002c9999999999 -f -c 10
-f flood, -c 10 roundtrips
</code>
==== Oracle Rac mit UDP testen ==== 

Verwendung des Infiniband Devies als Interconnect sicherstellen:

<code>
oifcfg getif –global
oifcfg delif –global <ifname, ex. bond0>
oifcfg setif –global ib1/192.168.1.0:cluster_interconnect
</code>

==== Oracle Kernel auf RDS Umstellen ==== 

Was wird akuelle für ein Protokoll für den Interconnect verwendet?\\
Siehe alert.log der aktuellen DB:\\
<code>
...
Cluster communication is configured to use the following interface(s) for this instance
  10.10.10.118
cluster interconnect IPC version:Oracle UDP/IP (generic)
...
</code>

Datenbank stoppen.\\
!! Im Cluster auch prüfen ob wirklich alle Instancen gestoppt sind!!! \\
sonst :  ***ORA-27550: Target ID protocol check failed. tid vers=1, type=1, remote instance number=3, local instance number=1***\\
\\
Durch linken des Oracle Kernels mit der rds library RDS aktivieren.
<code>
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ipc_rds

[oracle@c7000rac1 lib]$ make -f ins_rdbms.mk ipc_rds
# ... rm -f $ORACLE_HOME/lib/libskgxp11.so
# ...  cp $ORACLE_HOME/lib//libskgxpr.so /u01/app/oracle/product/11.2.0/dbhome_1/lib/libskgxp11.so

</code>

==== Oracle Linux 6.1 bzw. Scientific Linux (SL) Problematik  ====


Installation der Mellanox Treiber schlägt fehl:
\\
\\
Problem:
<code>
 ./mlnxofedinstall
The 2.6.32-100.34.1.el6uek.x86_64 kernel is installed, but do not have drivers available.
Cannot continue.
</code>

**Lösung**:\\
Mellanox Treiber manuel installieren\\
(Sorry, in Englisch da für englischsprachigen Kunden .-) )\\

Ablauf:

  * Mount ISO Image of Mellanox Driver to /mnt/mellanox
  * Create directory /tmp/build_driver_mellanox
  * Copy from src directory from iso image the source code to /tmp/build_driver_ mellanox
  * Copy Conf File to /tmp/build_driver_ mellanox (from failed installation!)
  * Start Kernel driver installation ( ./install.pl –c ofed.conf)
  * Install Tools from rpm directory of iso cd
 
Mount the Infiniband Setup ISO Image via the command
<code bash>
# mkdir /mnt/mellanox

# mount -o loop -t iso9660 /installfiles/06_mellanox/MLNX_OFED_LINUX-1.5.3-3.0.0-rhel5.7-x86_64.iso /mnt/mellanox
</code>

Building RPMs for un-supported kernels.
<code bash>
mkdir /tmp/build_driver_mellanox

cd /mnt/mellanox/src

cp MLNX_OFED_SRC-1.5.3-3.0.0.tgz /tmp/build_driver_mellanox 

cd /tmp/build_driver_mellanox

tar zxvf MLNX_OFED_SRC-1.5.3-3.0.0.tgz

cd MLNX_OFED_SRC-1.5.3-3.0.0
</code>

Copy ofed.conf to the /tmp/build_driver_mellanox/MLNX_OFED_SRC-1.5.3-3.0.0 directory\\
Deinstall package  scsi-target-utils\\

<code bash>
yum remove  scsi-target-utils
</code>

Start the build and installation of the kernel module.
<code bash>
./install.pl -c ofed.conf 
Below is the list of OFED packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
kernel-ib
kernel-ib-devel
kernel-mft

Uninstalling the previous version of OFED
Build ofed-scripts RPM

… …… ………… …… …..


Device (15b3:634a):
        47:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)
        Link Width: 8x
        PCI Link Speed: 2.5Gb/s


Installation finished successfully.

</code>

If you get the output "Installation finished successfully" the driver installation is finished. \\
Go back to iso image and install the Mellanox Support  tools from RPM directory\\

<code bash>
cd /mnt/mellanox/RPMS/


yum --nogpgcheck install opensm-3.3.9.MLNX_20111006_e52d5fc-0.1.x86_64.rpm rds-tools-2.0.4-1.x86_64.rpm infiniband-diags-1.5.8.MLNX_20110906-0.1.x86_64.rpm ibutils2-2.0-0.34.g9d3133a.x86_64.rpm ibutils-1.5.7-0.1.g05a9d1a.x86_64.rpm opensm-libs-3.3.9.MLNX_20111006_e52d5fc-0.1.x86_64.rpm libibmad-1.3.7.MLNX_20110814-0.1.x86_64.rpm libibumad-1.3.7.MLNX_20110814-0.1.x86_64.rpm

</code>

Please check in the configuration files for Infiniband, that the RDS driver is loaded at startup.
<code bash>
$ cat /etc/infiniband/openib.conf | grep RDS
# Load RDS module
RDS_LOAD=yes
If not, please change Value for parameter RDS_LOAD to yes on each node. If whole parameter is missing, add the parameter.
</code>

See also this : http://www.hpcadvisorycouncil.com/events/2011/switzerland_workshop/pdf/Presentations/Day%201/2_InfiniBand_Training.pdf

==== Quellen ===

  * http://www.qlogic.com/SiteCollectionDocuments/Solutions/HP/hp_silverstorm_10g_rac_rds_v8_public.pdf 
  * http://www.servercare.com/assets/files/downloads/2008_121_RAC_11g_BEST_PRACTICES_AND_TUNING.doc 
  * http://www.oracle.com/technetwork/database/features/availability/s281216-tsien-131087.pdf 
  * http://www.openfabrics.org/archives/spring2010sonoma/Monday/8.30%20Tim%20Shetler%20Oracle/Sonoma_Workshop_2010_Oracle-final.pdf
  * http://www.hpcuserforum.com/presentations/April2009Roanoke/OFAOPENFABRICSSpring2009OFA.ppt 
  * http://www.voltaire.com/Solutions/Database_Applications/oracle_10g_and_11g_real_application_clusters_rac 
  * http://www.voltaire.com/download/VOLT-OracleSolutionGuide-092108.pdf 
  * http://www.dell.com/downloads/global/power/ps2q07-20070279-Mahmood.pdf
  * http://www.unyoug.com/uploads/files/20090313/UNYOUG_20090313_Leveraging_Infiniband_v2.ppt 
  * http://www.texmemsys.com/files/oracle_performance_tuning_with_ssd.pdf 
  * http://www.oracle.com/global/de/events/2007/locals/germany/odd_hochverfuegbarkeit/ORACLE_ODD_IT_Betrieb_RAC_ASM_Solbach.pdf