Matraex, Inc Blog

Over the last couple of weeks I have been working on doing some in depth “System Discovery” work for a client.

The client came to us after a major employee restructuring,  during which they lost ALL of the technical knowledge of their network.
The potentially devestating business move on their part turned into a very intriguing challenge for me.

They asked me to come in and document what service each of their 3 Linux servers. 
As I dug in I found that their network had some very unique, intelligent solutions:

• A reliable production network
• Thin Client Linux printing stations,  remotely connected via VPN
• Several Object Oriented PHP based web applications

Several open source products had been combined to create robust solutions

• LAMP (Ubuntu and CentOS)
• CUPS
• Subversion
• PHP, PEAR
• MySQL
• openvpn
• hylafax

It has been a very rewarding experience to document the systems and give ownership of the systems, network and processes back to the owner.

The  documentation I have provided included

• A high level network diagram as a quick reference overview for new administrators and developers
• An overall application and major network, server and node object description
• Detailed per server/node description with connection documentation,  critical processes , important paths and files and dependencies
• Contact Information for the people and companies that the systems rely on.

As a business owner myself,  I have tried to help the client recognize that even when they use an outside consultant,  it is VERY important that they maintain details of their critical business processes INSIDE of their company.  Their might not be anything in business that is as rewarding as giving ownership of a “lost” system back to a client.

Debian Lenny (and etch) both donot include the drivers for the broadcom network cards that come with many dell servers,  I use the Dell 2650 alot but I have also had issues with the 1750 and 1950,   I am sure their are other
I hear it is a licensing problem with debian notbeing able to distribute the drivers or something. 
Here is the solution I have come up with from my end.

On my pxeboot server (I refer to pxe booting but I dont describe how to set one up,  check this howto out)

I cd into the directory that my lenny installation is to be setup in (based on the pxelinux.cfg/default file)

  #cd /tftpboot/debian/lenny/i386/

I am going to download all of the network installation files for debian lenny on i386, this should apply 64 bit too though.

  # wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz
  # wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/debian-installer/i386/initrd.gz
  # wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/debian-installer/i386/linux

download the broadcom drivers package and extract it to a folder called bnx2

  # wget http://ftp.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14+lenny1_all.deb
  # dpkg-deb  -x firmware-bnx2_0.14+lenny1_all.deb bnx2

Create a temp working directory 

  # mkdir temp
  # cd temp

extract all of the installation files from the initrd.gz file so you can manipulate them (include the driver) 

  # zcat ../initrd.gz |cpio -iv

Copy all of the firmware drivers from the extracted bnx2 directory into the root of the extracted initrd.gz kernel directory

  # cp ../bnx2/lib/firmware/* ../bnx2/usr/share/initramfs-tools/hooks/firmware_bnx2 .

Since the kernel initrd.gz is only used during installation of the OS,  this fix so far hasn’t addressed installing the broadcom driver package for the OS after installation.
To do this you will need to customize the kernel to select and install this package during installation using “preseeding” 
Create and edit a file called preseed.cfg in the root of the extracted kernel directory ( 

  # vi preseed.cfg

Place the following contents in that file (I have aso included the ssh server since I typcially do a minimum install without ANY packages but I need ssh)

#automatically select these packages when installing the server
#d-i pkgsel/include string openssh-server firmware-bnx2
base-config     apt-setup/non-free      boolean true
d-i     preseed/late_command            string apt-install firmware-bnx2; apt-install openssh-server;

As another shortcut that can shave a tiny bit of time of of your installation,  if you do not use USB storage during your installation,  there is no need to wait for the delay and errors that occur during the system’s search for those USB storage devices. Deleting the USB drivers from the installation kernel will prevent these errors

  # rm -rf lib/modules/2.6.26-2-486/kernel/drivers/usb/storage

Now it is time to put the extracted kernel directory back together in the location that the pxe boot is looking for it.

  # find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz

that is  it!  you have customized and rebuilt your installation kernel for network boot. 
Simply pxeboot to this installation with your Dell or broadcom server and the drivers will be included.
The concepts used above can also help you to setup and customize a net boot which has packages already selected or otherwise speed your install along with drivers or other customizations already selected,  look into preseeding for more options here

This blog describes the commands neccessary to add a disk to existing RAID V array in the case where you have an empty available slot where you can add a disk.
The actual manual for afacli can be found here:
http://docs.us.dell.com/support/edocs/storage/57kgr/cli/en/index.htm

Accessing the CLI from the UNIX Prompt

To access the CLI from the UNIX prompt, display a window and type afacli in any directory. The system displays the FASTCMD> prompt, which indicates you can now use CLI commands. The path in the startup file (.login or .cshrc) must include the directory where the software is installed for the command to work in any directory. See your UNIX documentation for information on setting up directory paths in the .login and .cshrc files.

To view all controllers use ‘controller list’
To connect to the controller with the command line utility, execute:

FASTCMD> open afa0

AFA0>

To show the status of all disks in all arrays and get an overview of the disks in the RAID, execute ‘enclosure show status’.

AFA0> enclosure show status

Executing: enclosure show status

Enclosure
ID (B:ID:L) UpTime D:H:M   PowerCycle Interval Door     Alarm
———– ————– ———- ——– ——– —–
 0  0:06:0         0:00:00        0       10   UNLOCKED OFF   

Enclosure
ID (B:ID:L) Fan Status
———– — ————-

Enclosure
ID (B:ID:L) Power State        Status
———– —– ———— ——-

Enclosure
ID (B:ID:L) Slot scsiId Insert  Status
———– —- —— ——- ——————————————
 0  0:06:0   0   0:00:0     1   OK ACTIVATE
 0  0:06:0   1   0:01:0     1   OK ACTIVATE
 0  0:06:0   2   0:02:0     1   OK ACTIVATE
 0  0:06:0   3   0:03:0     1   OK ACTIVATE
 0  0:06:0   4   0:255:0     0   OK UNCONFIG EMPTY I/R READY NOTACTIVATE

Enclosure
ID (B:ID:L) Sensor Temperature Threshold Status
———– —— ———– ——— ——–
 0  0:06:0   0       73 F         120    NORMAL 
 0  0:06:0   1       69 F         120    NORMAL 

Above, there is no disk in slot 4, insert the disk again and execute the command again to see it again.

AFA0> enclosure show status
Executing: enclosure show status

Enclosure
ID (B:ID:L) UpTime D:H:M   PowerCycle Interval Door     Alarm
———– ————– ———- ——– ——– —–
 0  0:06:0         0:00:00        0       10   UNLOCKED OFF   

Enclosure
ID (B:ID:L) Fan Status
———– — ————-

Enclosure
ID (B:ID:L) Power State        Status
———– —– ———— ——-

Enclosure
ID (B:ID:L) Slot scsiId Insert  Status
———– —- —— ——- ——————————————
 0  0:06:0   0   0:00:0     1   OK ACTIVATE
 0  0:06:0   1   0:01:0     1   OK ACTIVATE
 0  0:06:0   2   0:02:0     1   OK ACTIVATE
 0  0:06:0   3   0:03:0     1   OK ACTIVATE
 0  0:06:0   4   0:04:0     1   OK UNCONFIG ACTIVATE

Enclosure
ID (B:ID:L) Sensor Temperature Threshold Status
———– —— ———– ——— ——–
 0  0:06:0   0       73 F         120    NORMAL 
 0  0:06:0   1       73 F         120    NORMAL

 You can see that the disk in slot 4 is waiting to be configured.  lets take a look at the RAID V container that we are going to add the new disk to,  execute “container list”

AFA0> container list
Executing: container list
Num          Total  Oth Chunk          Scsi   Partition
Label Type   Size   Ctr Size   Usage   B:ID:L Offset:Size
—– —— —— — —— ——- —— ————-
 0    RAID-5  101GB       64KB Valid   0:00:0 64.0KB:33.8GB
 /dev/sda                                         0:01:0 64.0KB:33.8GB 
                                                      0:02:0 64.0KB:33.8GB
                                                      0:03:0 64.0KB:33.8GB  

Even though it is visible in the enclosure list above,  you will need to execute a “controller rescan” to find the new disk. 

AFA0> controller rescan
Executing: controller rescan

Now initialize the disk so it can be used.

AFA0> disk initialize 4 
Executing: disk initialize (ID=4)

Finally you can add the disk to the container,  you will simply run the container reconfigure command with the container number (in our case 0) and the device number (in our case 4)

AFA0> container reconfigure 0 4
Executing: container reconfigure 0 (ID=4)
 

Now, wait for the disk to rebuild. You can view the rebuild process with ‘task list’.

AFA0> task list
Executing: task list

Controller Tasks

TaskId Function  Done%  Container State Specific1 Specific2
—— ——– ——- ——— —– ——— ———
  101   Reconfg   8.4%      0      RUN   00000000  00000000

 Also while adding a new disk to this array I found that the existing array was only  using 25.6 GB of ech disk and not the full 36 GB 

I issued a “container reconfigure” command to utilize the more space on each disk

AFA0>  container reconfigure /partition_size=36388763000 0
Executing: container reconfigure /partition_size=36,388,763,000  0