Difference between revisions of "Ubuntu Infiniband"

From Teknologisk videncenter
Jump to: navigation, search
m (→‎Links)
m (→‎Links)
 
(14 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
sudo apt-get install update-manager-core
 
sudo apt-get install update-manager-core
 
</source>
 
</source>
−
Edit /etc/update-manager/release-upgrades and set '''Prompt=normal;'''
+
Edit '''/etc/update-manager/release-upgrades''' and set '''Prompt=normal;'''
  
 
Launch the upgrade tool with the command
 
Launch the upgrade tool with the command
Line 12: Line 12:
 
</source>
 
</source>
 
and follow the on-screen instructions.
 
and follow the on-screen instructions.
 +
 
=Checking hardware=
 
=Checking hardware=
 
<source lang=cli>
 
<source lang=cli>
 +
root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>
 +
root@DKAM1:~# <input>apt-get -y install opensm</input>
 
root@DKAM1:~# <input>modprobe ib_sa</input>
 
root@DKAM1:~# <input>modprobe ib_sa</input>
 
root@DKAM1:~# <input>modprobe ib_cm</input>
 
root@DKAM1:~# <input>modprobe ib_cm</input>
Line 21: Line 24:
 
root@DKAM1:~# <input>modprobe ib_ipoib</input>
 
root@DKAM1:~# <input>modprobe ib_ipoib</input>
 
root@DKAM1:~# <input>modprobe ib_ipath</input>
 
root@DKAM1:~# <input>modprobe ib_ipath</input>
−
FATAL: Error inserting ib_ipath (/lib/modules/2.6.35-32-generic/kernel/drivers/infiniband/hw/ipath/ib_ipath.ko): Device or resource busy
+
<notice>May report device busy. Then it's already loaded</notice>
 
root@DKAM1:~# <input>modprobe ib_qib</input>
 
root@DKAM1:~# <input>modprobe ib_qib</input>
 
root@DKAM1:~# <input>ifconfig ib0</input>
 
root@DKAM1:~# <input>ifconfig ib0</input>
Line 52: Line 55:
 
         base lid:        0xffff
 
         base lid:        0xffff
 
         sm lid:          0xffff
 
         sm lid:          0xffff
−
         state:          1: DOWN
+
         state:          1: <notice>DOWN</notice>
−
         phys state:      2: Polling
+
         phys state:      2: <notice>Polling</notice>
 
         rate:            <notice>10 Gb/sec (4X)</notice>
 
         rate:            <notice>10 Gb/sec (4X)</notice>
  
 +
</source>
 +
==Getting the Infinitiband hardware up==
 +
To initialize the infinitiband hardware you need a [[InfiniBand|Infiniband Subnet Manager and Administration]] entity, then the adapters on that subnet should initialize.
 +
<source lang=cli>
 +
 +
root@node2:~# <input>ibstatus</input>
 +
Infiniband device 'qib0' port 1 status:
 +
        default gid:    fe80:0000:0000:0000:0011:7500:0078:8c6c
 +
        base lid:        0x1
 +
        sm lid:          0x5
 +
        state:          4: <notice>ACTIVE</notice>
 +
        phys state:      5: <notice>LinkUp</notice>
 +
        rate:            <notice>40 Gb/sec (4X QDR)</notice>
 +
root@node2:~# <input>ifconfig ib0</input>
 +
ib0      Link encap:UNSPEC  HWaddr 80-00-00-03-FE-80-00-00-00-00-00-00-00-00-00-00
 +
          inet addr:10.0.1.11  Bcast:10.0.1.255  Mask:255.255.255.0
 +
          inet6 addr: fe80::211:7500:78:8c6c/64 Scope:Link
 +
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
 +
          RX packets:3734630 errors:0 dropped:0 overruns:0 frame:0
 +
          TX packets:3725918 errors:0 dropped:7 overruns:0 carrier:0
 +
          collisions:0 txqueuelen:256
 +
          RX bytes:4634441688 (4.6 GB)  TX bytes:4649148404 (4.6 GB)
 +
 +
</source>
 +
 +
=Configuration=
 +
add the following modules to '''/etc/modules'''
 +
<source lang=cli>
 +
ib_sa
 +
ib_cm
 +
ib_umad
 +
ib_addr
 +
ib_uverbs
 +
ib_ipoib
 +
ib_ipath
 +
ib_qib
 +
</source>
 +
Install '''opensm'''
 +
<source lang=cli>
 +
root@DKAM1:~# <input>apt-get -y install opensm</input>
 +
root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>
 +
</source>
 +
Add '''ib0''' interface to '''/etc/network/interfaces'''
 +
<source lang=cli>
 +
auto ib0
 +
iface ib0 inet static
 +
  address 10.0.1.12
 +
  netmask 255.255.255.0
 +
  gateway 10.0.1.1
 
</source>
 
</source>
  
Line 61: Line 113:
 
*[http://www.unixmen.com/how-to-upgrade-from-ubuntu-1010-to-ubuntu-1104-natty-desktop-a-server/ Upgrading Ubuntu]
 
*[http://www.unixmen.com/how-to-upgrade-from-ubuntu-1010-to-ubuntu-1104-natty-desktop-a-server/ Upgrading Ubuntu]
 
*[http://davidhunt.ie/wp/?p=2291 Infiniband on Ubuntu 11.10]
 
*[http://davidhunt.ie/wp/?p=2291 Infiniband on Ubuntu 11.10]
 +
*[http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html Infiniband howto 4]
 +
*[http://en.wikipedia.org/wiki/InfiniBand Information about Infiniband]
 
{{Source cli}}
 
{{Source cli}}
−
[[Category:Ubuntu]]
+
[[Category:Ubuntu]][[Category:Infiniband]]

Latest revision as of 05:08, 28 April 2012

Ubuntu 10.04 LTE do not support Infiniband. NO opensm package.

Upgrading from 10.04 LTE to 10.10

install the update-manager-core package if it is not already installed: 

sudo apt-get install update-manager-core

Edit /etc/update-manager/release-upgrades and set Prompt=normal;

Launch the upgrade tool with the command 

sudo do-release-upgrade -d

and follow the on-screen instructions.

Checking hardware

root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>
root@DKAM1:~# <input>apt-get -y install opensm</input>
root@DKAM1:~# <input>modprobe ib_sa</input>
root@DKAM1:~# <input>modprobe ib_cm</input>
root@DKAM1:~# <input>modprobe ib_umad</input>
root@DKAM1:~# <input>modprobe ib_addr</input>
root@DKAM1:~# <input>modprobe ib_uverbs</input>
root@DKAM1:~# <input>modprobe ib_ipoib</input>
root@DKAM1:~# <input>modprobe ib_ipath</input>
<notice>May report device busy. Then it's already loaded</notice>
root@DKAM1:~# <input>modprobe ib_qib</input>
root@DKAM1:~# <input>ifconfig ib0</input>
ib0       Link encap:UNSPEC  HWaddr 80-00-00-02-FE-80-00-00-00-00-00-00-00-00-00-00
          BROADCAST MULTICAST  MTU:4092  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
root@DKAM1:~# <input> ibstat</input>
CA 'qib0'
        CA type: InfiniPath_QLE7340
        Number of ports: 1
        Firmware version:
        Hardware version: 2
        Node GUID: 0x0011750000788b2c
        System image GUID: 0x0011750000788b2c
        Port 1:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 65535
                LMC: 0
                SM lid: 65535
                Capability mask: 0x07610868
                Port GUID: 0x0011750000788b2c
root@DKAM1:~# <input>ibstatus</input>
Infiniband device 'qib0' port 1 status:
        default gid:     fe80:0000:0000:0000:0011:7500:0078:8b2c
        base lid:        0xffff
        sm lid:          0xffff
        state:           1: <notice>DOWN</notice>
        phys state:      2: <notice>Polling</notice>
        rate:            <notice>10 Gb/sec (4X)</notice>

Getting the Infinitiband hardware up

To initialize the infinitiband hardware you need a Infiniband Subnet Manager and Administration entity, then the adapters on that subnet should initialize. 

root@node2:~# <input>ibstatus</input>
Infiniband device 'qib0' port 1 status:
        default gid:     fe80:0000:0000:0000:0011:7500:0078:8c6c
        base lid:        0x1
        sm lid:          0x5
        state:           4: <notice>ACTIVE</notice>
        phys state:      5: <notice>LinkUp</notice>
        rate:            <notice>40 Gb/sec (4X QDR)</notice>
root@node2:~# <input>ifconfig ib0</input>
ib0       Link encap:UNSPEC  HWaddr 80-00-00-03-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:10.0.1.11  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::211:7500:78:8c6c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:3734630 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3725918 errors:0 dropped:7 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:4634441688 (4.6 GB)  TX bytes:4649148404 (4.6 GB)

Configuration

add the following modules to /etc/modules 

ib_sa
ib_cm
ib_umad
ib_addr
ib_uverbs
ib_ipoib
ib_ipath
ib_qib

Install opensm 

root@DKAM1:~# <input>apt-get -y install opensm</input>
root@DKAM1:~# <input>apt-get -y install infiniband-diags</input>

Add ib0 interface to /etc/network/interfaces 

auto ib0
iface ib0 inet static
  address 10.0.1.12
  netmask 255.255.255.0
  gateway 10.0.1.1

Links

Retrieved from "http://mars.merhot.dk/w/index.php?title=Ubuntu_Infiniband&oldid=22158"