Revision as of 13:52, 9 November 2011

Hardware

Each group installs Ubuntu 10.10 32 bit on four Dell PowerEdge 1750 bladeserver with 36,5GB harddisk.

Rack

Position in Rack

1. Common Switch for all groups
2. Head node Group 1
3. Node 1 Group 1
4. Node 2 Group 1
5. Node 3 Group 1
6. Head node Group 2
7. Node 1 Group 2
8. Node 2 Group 2
9. Node 3 Group 2
10. Head node Group 3
11. Node 1 Group 3
12. Node 2 Group 3
13. Node 3 Group 3

Switch configuration

1. Group 1 VLAN 10 port 1 to 5
2. Group 2 VLAN 20 port 6 to 10
3. Group 3 VLAN 30 port 11 to 15
4. Common VLAN 40 port 16 to 24 - Connected to 192.168.139.0/24 net

Filesystem

• / filesystem 2 GB primary
• /tmp filesystem 5 GB logical
• /var filesystem 5 GB primary
• /usr filesystem 6 GB logical
• swap filesystem 2 GB logical
• /home filesystem remaining space

Updating Ubuntu

Update

sudo bash
apt-get update
apt-get upgrade

List installed packages

Number of installed packages

root@newclusterh:~# <input>dpkg --get-selections | wc -l</input>
962

Searching installed packages

root@newclusterh:~#  <input>dpkg --get-selections | grep nfs</input>
libnfsidmap2                                    install
nfs-common                                      install
nfs-kernel-server                               install
root@newclusterh:~# <input>dpkg -L nfs-common</input>
/.
/etc
/etc/init.d
/etc/init
/etc/init/statd.conf
/etc/init/statd-mounting.conf
/etc/init/rpc_pipefs.conf
/etc/init/gssd.conf
<notice>...OUTPUT OMITTED...</notice>

Routing IPv4 and NAT-ing

Configuring additional NIC's

• Cluster networks between Headnode and Nodes.
  • Group 1: 10.100.1.0/24
  • Group 2: 10.100.2.0/24
  • Group 3: 10.100.3.0/24
• Headnode public IP address
  • Group 1: 192.168.139.21
  • Group 2: 192.168.139.22
  • Group 3: 192.168.139.23
  • Switch: 192.168.139.24

edit the file /etc/network/interfaces. Example below

auto eth0
iface eth0 inet static
 address 192.168.139.50
 netmask 255.255.255.0
 gateway 192.168.139.1
 network 192.168.139.0
 broadcast 192.168.139.255

auto eth1
iface eth1 inet static
        address 10.0.0.1
        netmask 255.255.255.0

Head node

Routing IPv4

• Add or uncomment the line net.ipv4.ip_forward=1 in /etc/sysctl.conf to allow Routing after boot
• Issue the command sysctl -w net.ipv4.ip_forward=1 to allow routing in flight.

NAT with iptables

#!/bin/bash
#
# Start firewall
#
# Tillader ping indefra og ud.
# Lukker for al anden trafik.
# Diverse erklæringer
FW=iptables
INT_NET="10.0.0.0/24"
EXT_NET="172.16.4.0/24"
EXT_IP="172.16.4.99"
EXT_IF="eth0"

# Fjern alle tidligere regler
$FW -F INPUT
$FW -F OUTPUT
$FW -F FORWARD
$FW -F -t nat


# Sæt default politik til afvisning
$FW -P FORWARD ACCEPT

# Tillad ping indefra
$FW -A FORWARD -s $INT_NET -p icmp --icmp-type echo-request -j ACCEPT

# Tillad pong udefra
$FW -A FORWARD -d $INT_NET -p icmp --icmp-type echo-reply -j ACCEPT

# Source NAT på udgående pakker
$FW -t nat -A POSTROUTING -o $EXT_IF -s $INT_NET -j SNAT --to-source $EXT_IP

NFS server

install NFS package

 root@headnode:~# <input>apt-get install nfs-kernel-server</input>

Add export to /etc/export (Group 1 net shown)

/home   10.100.1.0/24(rw,sync,no_subtree_check,no_root_squash)

Restart the NFS-Server

root@headnode:~# <input>/etc/init.d/nfs-kernel-server restart</input>

Node clients

NFS client

Install NFS client package

root@node1:~# <input>apt-get install nfs-common</input>

Mounting /home from headnode

First change the nodes own /home partition to /home2

mkdir /home2

Change /etc/fstab to mount the local /home partition on the new /home2 directory. (See below)

UUID=d3bdcead-df52-d5ba7df3a1e4 /home2           ext4    defaults        0       2

Add a line to /etc/fstab to mount the headnodes /home on the local nodes directory /home

10.100.3.1:/home  /home    nfs          rw<notice>,vers=3</notice>            0    0

/etc/hosts

Add machines to /etc/hosts for example

10.100.3.1      head3
10.100.3.31     node31
10.100.3.32     node32
10.100.3.33     node33

Auto login

To login from ssh without password between the machines in the cluster. Issue the following commands from the headnode.

su -c "ssh-keygen" <notice>USERNAME</notice>
cat /home/<notice>USERNAME</notice>/.ssh/id_rsa.pub > /home/<notice>USERNAME</notice>/.ssh/authorized_keys
chown <notice>USERNAME</notice>:<notice>USERNAME</notice> /home/<notice>USERNAME</notice>/.ssh/authorized_keys

Running processes on CPU's

Use mpstat -P ALL' to see current utilization of CPU's. See the CPU mask of any given process with taskset -p <PID>. With taskset it's possible to change the CPU mask of running processes and start commands on any given CPU or allow a range of CPU's the process can run on.

The Linux Scheduler decides which processes runs on which CPU's. See man sched_setscheduler.

SAN på Compute Nodes

Install a SAN in the backend of the compute nodes. For example FreeNAS

TO-DO liste

• Installere SSH på alle heads og noder
• Automatisk login vha. distribution af certifikater
• Installere scripts til oprettelse af brugere
• Installere og konfigurere NFS-server på heads
• installere og konfigurere NFS-Clients på noder
• installer dfs (Distributed Shell) på heads