Switch-based Discovery

For switch based hardware discovery, the servers are identified through the switches and switchposts they are directly connected to.

In this document, the following configuration is used in the example

Management Node info:

MN Hostname: xcat1
MN NIC info for Management Network(Host network): eth1, 10.0.1.1/16
MN NIC info for Service Network(FSP/BMC network): eth2, 50.0.1.1/16
Dynamic IP range for Hosts: 10.0.100.1-10.0.100.100
Dynamic IP range for FSP/BMC: 50.0.100.1-50.0.100.100

Compute Node info:

CN Hostname: cn1
Machine type/model: 8247-22L
Serial: 10112CA
IP Address: 10.0.101.1
Root Password: cluster
Desired FSP/BMC IP Address: 50.0.101.1
DHCP assigned FSP/BMC IP Address: 50.0.100.1
FSP/BMC username: ADMIN
FSP/BMC Password: admin

Switch info:

Switch name: switch1
Switch username: xcat
Switch password: passw0rd
Switch IP Address: 10.0.201.1
Switch port for Compute Node: port0

Configure xCAT

Configure network table

Normally, there will be at least two entries for the two subnet on MN in networks table after xCAT is installed:

#tabdump networks
#netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,staticrange,staticrangeincrement,nodehostname,ddnsdomain,vlanid,domain,mtu,comments,disable
"10_0_0_0-255_255_0_0","10.0.0.0","255.255.0.0","eth1","<xcatmaster>",,"10.0.1.1",,,,,,,,,,,,,
"50_0_0_0-255_255_0_0","50.0.0.0","255.255.0.0","eth2","<xcatmaster>",,"50.0.1.1",,,,,,,,,,,,,

Run the following command to add networks in networks table if there are no entries in it:

makenetworks

Setup DHCP

Set the correct NIC from which DHCP server provide service:

chdef -t site dhcpinterfaces=eth1,eth2

Add dynamic range in purpose of assigning temporary IP address for FSP/BMCs and hosts:

chdef -t network 10_0_0_0-255_255_0_0 dynamicrange="10.0.100.1-10.0.100.100"
chdef -t network 50_0_0_0-255_255_0_0 dynamicrange="50.0.100.1-50.0.100.100"

Update DHCP configuration file:

makedhcp -n
makedhcp -a

Config passwd table

Set required passwords for xCAT to do hardware management and/or OS provisioning by adding entries to the xCAT passwd table:

# tabedit passwd
# key,username,password,cryptmethod,authdomain,comments,disable

For hardware management with ipmi, add the following line:

"ipmi","ADMIN","admin",,,,

Verify the genesis packages

The xcat-genesis packages should have been installed when xCAT was installed, but would cause problems if missing. xcat-genesis packages are required to create the genesis root image to do hardware discovery and the genesis kernel sits in /tftpboot/xcat/. Verify that the genesis-scripts and genesis-base packages are installed:

  • [RHEL/SLES]: rpm -qa | grep -i genesis

  • [Ubuntu]: dpkg -l | grep -i genesis

If missing, install them from the xcat-deps package and run mknb ppc64 to create the genesis network boot root image.

Predefined Nodes

In order to differentiate one node from another, the admin needs to predefine node in xCAT database based on the switches information. This consists of two parts:

  1. Predefine Switches

  2. Predefine Server Node

Predefine Switches

The predefined switches will represent devices that the physical servers are connected to. xCAT need to access those switches to get server related information through SNMP v3.

So the admin need to make sure those switches are configured correctly with SNMP v3 enabled. <TODO: The document that Configure Ethernet Switches>

Then, define switch info into xCAT:

nodeadd switch1 groups=switch,all
chdef switch1 ip=10.0.201.1
tabch switch=switch1 switches.snmpversion=3 switches.username=xcat switches.password=passw0rd switches.auth=sha

Add switch into DNS using the following commands:

makehosts switch1
makedns -n

Predefine Server node

After switches are defined, the server node can be predefined with the following commands:

nodeadd cn1 groups=powerLE,all
chdef cn1 mgt=ipmi cons=ipmi ip=10.0.101.1 bmc=50.0.101.1 netboot=petitboot installnic=mac primarynic=mac
chdef cn1 switch=switch1 switchport=0

[Optional] If more configuration planed to be done on BMC, the following command is also needed.

chdef cn1 bmcvlantag=<vlanid>                 # tag VLAN ID for BMC
chdef cn1 bmcusername=<desired_username>
chdef cn1 bmcpassword=<desired_password>

In order to do BMC configuration during the discovery process, set runcmd=bmcsetup.

chdef cn1 chain="runcmd=bmcsetup"

[Optional] More operation plan to do after hardware discovery is done, ondiscover option can be used.

For example, configure console, copy SSH key for OpenBMC, then disable powersupplyredundancy

chdef cn01 -p chain="ondiscover=makegocons|rspconfig:sshcfg|rspconfig:powersupplyredundancy=disabled"

Note: | is used to split commands, and : is used to split command with its option.

Set the target osimage into the chain table to automatically provision the operating system after the node discovery is complete.

chdef cn1 -p chain="osimage=<osimage_name>"

For more information about chain, refer to Chain

Add cn1 into DNS:

makehosts cn1
maekdns -n

Discover server and define

After environment is ready, and the server is powered, we can start server discovery process. The first thing to do is discovering the FSP/BMC of the server. It is automatically powered on when the physical server is powered.

Use the bmcdiscover command to discover the BMCs responding over an IP range and write the output into the xCAT database. This discovered BMC node is used to control the physical server during hardware discovery and will be deleted after the correct server node object is matched to a pre-defined node. You must use the -w option to write the output into the xCAT database.

To discover the BMC with an IP address range of 50.0.100.1-100:

bmcdiscover --range 50.0.100.1-100 -z -w

The discovered nodes will be written to xCAT database. The discovered BMC nodes are in the form node-model_type-serial. To view the discovered nodes:

lsdef /node-.*

Note: The bmcdiscover command will use the username/password from the passwd table corresponding to key=ipmi. To overwrite with a different username/password use the -u and -p option to bmcdiscover.

Start discovery process

To start discovery process, just need to power on the PBMC node remotely with the following command, and the discovery process will start automatically after the host is powered on:

rpower node-8247-42l-10112ca on

[Optional] If you’d like to monitor the discovery process, you can use:

makegocons node-8247-42l-10112ca
rcons node-8247-42l-10112ca

Verify node definition

The following is an example of the server node definition after hardware discovery:

#lsdef cn1
Object name: cn1
    arch=ppc64
    bmc=50.0.101.1
    cons=ipmi
    cpucount=192
    cputype=POWER8E (raw), altivec supported
    groups=powerLE,all
    installnic=mac
    ip=10.0.101.1
    mac=6c:ae:8b:02:12:50
    memory=65118MB
    mgt=ipmi
    mtm=8247-22L
    netboot=petitboot
    postbootscripts=otherpkgs
    postscripts=syslog,remoteshell,syncfiles
    primarynic=mac
    serial=10112CA
    supportedarchs=ppc64
    switch=switch1
    switchport=0