IB Switch Configuration¶
Setup the xCAT Database¶
The Mellanox Switch is only supported in xCAT Release 2.7 or later.
Add the switch ip address in the
Define IB switch as a node:
chdef -t node -o mswitch groups=all nodetype=switch mgt=switch
Add the login user name and password to the switches table:
tabch switch=mswitch switches.sshusername=admin switches.sshpassword=admin switches.switchtype=MellanoxIB
The switches table will look like this:
If there is only one admin and one password for all the switches then put the entry in the xCAT passwd table for the admin id and password to use to login.
tabch key=switch passwd.username=admin passwd.password=admin
The passwd table will look like this:
Setup ssh connection to the Mellanox Switch¶
To run commands like xdsh and script to the Mellanox Switch, we need to setup ssh to run without prompting for a password to the Mellanox Switch. To do this, first you must add a configuration file. This configuration file is NOT needed for xCAT 2.8 and later.
mkdir -p /var/opt/xcat/IBSwitch/Mellanox cd /var/opt/xcat/IBSwitch/Mellanox cp /opt/xcat/share/xcat/devicetype/IBSwitch/Mellanox/config .
The file contains the following:
[main] [xdsh] pre-command=cli post-command=NULL
Then run the following:
rspconfig mswitch sshcfg=enable
For Mellanox switch in manufacturing defaults, the user may need to log in once and answer no to the configuration wizard as
rspconfig will fail when prompted against the wizard.
Setup syslog on the Switch¶
Use the following command to consolidate the syslog to the Management Node or Service Nodes, where ip is the address of the MN or SN as known by the switch.
rspconfig mswitch logdest=<ip>
Configure xdsh for Mellanox Switch¶
To run xdsh commands to the Mellanox Switch, you must use the –devicetype input flag to xdsh. In addition, for xCAT versions less than 2.8, you must add a configuration file, see Setup ssh connection to the Mellanox Switch section.
For the Mellanox Switch the
IBSwitch::Mellanox. See xdsh man page for details.
Now you can run the switch commands from the mn using xdsh. For example:
xdsh mswitch -l admin --devicetype IBSwitch::Mellanox \ 'enable;configure terminal;show ssh server host-keys'
Commands Supported for the Mellanox Switch¶
Setup the snmp alert destination:
rspconfig <switch> snmpdest=<ip> [remove]
where “remove” means to remove this ip from the snmp destination list.
Enable/disable setting the snmp traps.
rspconfig <switch> alert=enable/disable
Define the read only community for snmp version 1 and 2.
rspconfig <switch> community=<string>
Enable/disable snmp function on the switch.
rspconfig <switch> snmpcfg=enable/disable
Enable/disable ssh-ing to the switch without password.
rspconfig <switch> sshcfg=enable/disable
Setup the syslog remove receiver for this switch, and also define the minimum level of severity of the logs that are sent. The valid levels are: emerg, alert, crit, err, warning, notice, info, debug, none, remove. “remove” means to remove the given ip from the receiver list.
rspconfig <switch> logdest=<ip> [<level>]
For doing other tasks on the switch, use xdsh. For example:
xdsh mswitch -l admin --devicetype IBSwitch::Mellanox 'show logging'
Interactive commands are not supported by xdsh. For interactive commands, use ssh.
Send SNMP traps to xCAT Management Node¶
First, get http://www.mellanox.com/related-docs/prod_ib_switch_systems/MELLANOX-MIB.zip , unzip it. Copy the mib file MELLANOX-MIB.txt to
/usr/share/snmp/mibs directory on the mn and sn (if the sn is the snmp trap destination.)
To configure, run:
monadd snmpmon moncfg snmpmon <mswitch>
To start monitoring, run:
monstart snmpmon <mswitch>
To stop monitoring, run:
monstop snmpmon <mswitch>
To deconfigure, run:
mondecfg snmpmon <mswitch>
For more details on monitoring the cluster: TODO Monitoring_an_xCAT_Cluster/#snmp-monitoring