Ubuntu Infiniband

From Teknologisk videncenter
Revision as of 05:11, 27 April 2012 by Heth (talk | contribs) (Checking hardware)
Jump to: navigation, search

Ubuntu 10.04 LTE do not support Infiniband. NO opensm package.

Upgrading from 10.04 LTE to 10.10

install the update-manager-core package if it is not already installed:

sudo apt-get install update-manager-core

Edit /etc/update-manager/release-upgrades and set Prompt=normal;

Launch the upgrade tool with the command

sudo do-release-upgrade -d

and follow the on-screen instructions.

Checking hardware

root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>
root@DKAM1:~# <input>apt-get -y install opensm</input>
root@DKAM1:~# <input>modprobe ib_sa</input>
root@DKAM1:~# <input>modprobe ib_cm</input>
root@DKAM1:~# <input>modprobe ib_umad</input>
root@DKAM1:~# <input>modprobe ib_addr</input>
root@DKAM1:~# <input>modprobe ib_uverbs</input>
root@DKAM1:~# <input>modprobe ib_ipoib</input>
root@DKAM1:~# <input>modprobe ib_ipath</input>
<notice>May report device busy. Then it's already loaded</notice>
root@DKAM1:~# <input>modprobe ib_qib</input>
root@DKAM1:~# <input>ifconfig ib0</input>
ib0       Link encap:UNSPEC  HWaddr 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00
          BROADCAST MULTICAST  MTU:4092  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
root@DKAM1:~# <input> ibstat</input>
CA 'qib0'
        CA type: InfiniPath_QLE7340
        Number of ports: 1
        Firmware version:
        Hardware version: 2
        Node GUID: 0x0011750000788b2c
        System image GUID: 0x0011750000788b2c
        Port 1:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 65535
                LMC: 0
                SM lid: 65535
                Capability mask: 0x07610868
                Port GUID: 0x0011750000788b2c
root@DKAM1:~# <input>ibstatus</input>
Infiniband device 'qib0' port 1 status:
        default gid:     fe80:0000:0000:0000:0011:7500:0078:8b2c
        base lid:        0xffff
        sm lid:          0xffff
        state:           1: <notice>DOWN</notice>
        phys state:      2: <notice>Polling</notice>
        rate:            <notice>10 Gb/sec (4X)</notice>

Getting the Infinitiband hardware up

To initialize the infinitiband hardware you need a Infiniband Subnet Manager and Administration entity, then the adapters on that subnet should initialize.

root@node2:~# <input>ibstatus</input>
Infiniband device 'qib0' port 1 status:
        default gid:     fe80:0000:0000:0000:0011:7500:0078:8c6c
        base lid:        0x1
        sm lid:          0x5
        state:           4: <notice>ACTIVE</notice>
        phys state:      5: <notice>LinkUp</notice>
        rate:            <notice>40 Gb/sec (4X QDR)</notice>

Configuration

add the following modules to /etc/modules

ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Install opensm

root@DKAM1:~# <input>apt-get -y install opensm</input>
root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>

Add ib0 interface to /etc/network/interfaces

auto ib0
iface ib0 inet static
  address 10.0.1.12
  netmask 255.255.255.0
  gateway 10.0.1.1

Links