Category: Server Administration
Commanddump – remove all kernel header packages
Servers fill up with kernels that are not in use.
Use this single command to remove them on ubuntu / debian.
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get purge -y
AIDE huge daily reports
AIDE huge daily reports
AIDE is giving me HUGE daily reports on moderately used sites.
It was originally installed simply using ‘apt-get isntall aide’, and then I ran /usr/sbin/aideinit.
This generated 144 files in the /etc/aide/aide.conf.d folder and has been sending me an email every morning since with such a huge volume of changes, that it is useless.
The first item to cover is determining exactly what is in the reports. the way that AIDE works, it is not incremental by day, so if any file has changed since the day that you originally ran the aideinit command, it will be reported daily. So if you have not run the aideinit report since you made configuration changes or installed something, you need to do that, otherwise those configuration changes or installs will show up.
Use the -y switch to avoid having to wait and type ‘y’ in order to overwrite the old database
Then run the daily cron which should show you that no changes have occured.
/usr/sbin/aideinit -y /etc/cron.daily/aide
If your report is huge with log files or other standard, acceptable and planned for changes, you may want to exclude those from being monitored.
One school of thought might have been to remove EVERYTHING and then only put in place the items you want specifically to track, however I thought it would be better to explicitly remove the items which do not need to be monitored as they are sent to me. It seems more prudent, as we could potentially miss items we are not thinking about.
(Be sure to update your policy documents when making this change)
To address this on the command line, I went into the /etc/aide/aide.conf.d folder and started to remove the items I do not need.
Changing IP Addresses on a XenServer 6.5 Pool
Changing IP Addresses on a XenServer 6.5 Pool
To change the ip addresses on a XenServer 6.5 pool, start with the slaves, and use the following xe commands:
Remember: Slaves first, then the Master
NOTE: There is no need to change the IP from the Management Console.
Find the UUID of the Host Management PIF:
xe pif-list params=uuid,host-name-label,device,management
You will see a big list. Find the UUID for the slave that you’re working on. Use the “more” pipe if the UUID for your particular slave scrolls off the screen:
xe pif-list params=uuid,host-name-label,device,management | more
Change the IP Address on the first slave:
xe pif-reconfigure-ip uuid=<UUID of host management PIF> IP=<New IP> gateway=<GatewayIP> netmask=<Subnet Mask> DNS=<DNS Lookup IPs> mode=<dhcp,none,static>
Then:
xe-toolstack-restart
Verify the new address with ifconfig, and/or ping it from a workstation.
Point the slave to the new Master IP Address:
xe pool-emergency-reset-master master-address=NEW_IP_OF_THE_MASTER
Repeat the commands above on all slaves.
On the Master:
xe pif-list params=uuid,host-name-label,device,management
xe pif-reconfigure-ip uuid=<UUID of host management PIF> IP=<New IP> gateway=<GatewayIP> netmask=<Subnet Mask> DNS=<DNS Lookup IPs> mode=<dhcp,none,static>
xe-toolstack-restart
DO NOT run the emergency-reset-master command on the Master.
Reboot the Master, then reboot the Slaves and verify that they can find the Master.
Matt Long
04/06/2015
Bulk Domain NS, MX and A record lookup tool
Summary: We have two tools to help you lookup information on domains quickly
- quick-domain-research.php – See the NS, MX, A records and IPs for multiple domains in one table
- nameserver-compare.php – Compare NS, MX, A records for multiple domains, against multiple Name Servers
Bulk Domain NS, MX and A record lookup tool
Occassionally, we come across some sort of project in which we have to work through a list of multiple domain names and make some sort of changes.
In some cases we simply have to update contact records, in other cases we have to determine ownership, hosting and mail setups so we can assist with an ownership transfer.
There are a plethora of domain tools out there which help one at a time, But we were hard pressed to find a tool that could do a bulk lookup of multiple domains with table based out put.
So, we built the tool
https://www.matraex.com/quick-domain-research.php
This tool has the
- A records for the root domain (@) and the (www) domain.
- MX records for the root domain
- NS records for the root domain
This tool was thrown together quickly to help us identify whether an OLD but active nameserver, which had dozens of domain names on it, was actually being used for the domains.
We were able to delete more than 20 domains cluttering up the DNS entries.
Additionally we were able to clean up associated webservers that had not been cleaned of hosting accounts after a client left the account.
Some future ideas which will make their way in next time:
- Display whois information for the domain
- Optionally group the domains based on which name servers, whois records or www C class they are hosted at
Update 11/28/2015 by Michael Blood
Since this original post, we have added several new features including the ability to upload a file with a large batch upload, and download a CSV file with the results. You can see all of the details in this Enhanced Bulk Domain NS, MX and A record lookup tool post.
Linux System Discovery
Linux System Discovery
Over the last couple of weeks I have been working on doing some in depth “System Discovery” work for a client.
The client came to us after a major employee restructuring, during which they lost ALL of the technical knowledge of their network.
The potentially devastating business move on their part turned into a very intriguing challenge for me.
They asked me to come in and document what service each of their 3 Linux servers.
As I dug in I found that their network had some very unique, intelligent solutions:
- A reliable production network
- Thin Client Linux printing stations, remotely connected via VPN
- Several Object Oriented PHP based web applications
Several open source products had been combined to create robust solutions
It has been a very rewarding experience to document the systems and give ownership of the systems, network and processes back to the owner.
The documentation I have provided included
- A high level network diagram as a quick reference overview for new administrators and developers
- An overall application and major network, server and node object description
- Detailed per server/node description with connection documentation, critical processes , important paths and files and dependencies
- Contact Information for the people and companies that the systems rely on.
As a business owner myself, I have tried to help the client recognize that even when they use an outside consultant, it is VERY important that they maintain details of their critical business processes INSIDE of their company. Their might not be anything in business that is as rewarding as giving ownership of a “lost” system back to a client.
Matraex Upgraded Mail Client From Squirrelmail to Roundcube
Matraex Upgraded Mail Client From Squirrelmail to Roundcube
Matraex has officially upgraded our web based mail client from Squirrelmail to Roundcube.
Roundcube is a modern mail client utilizing newer technologies for faster and more feature rich mail interaction. Roundcube runs on our Linux webservers, utilizing Apache, PHP and MySQL. The software connects to the mail server using the IMAP protocol.
All address book contacts and preferences were imported to Roundcube from Squirellmail at the time of the transition.
As well as updating and implementing their own technologies, Matraex provides server administration, open source production implementation and software customizations to business as a service.
Users with questions about the new mail service or Matraex Consulting Services should contact:
Michael Blood
Matraex, Inc
208.344.1115
www.matraex.com
Network Boot Server with Linux Install, Debian Etch and Lenny, CentOS and KNOPPIX
Network Boot Server with Linux Install, Debian Etch and Lenny, CentOS and KNOPPIX
I just LOVE my dedicated PXE boot server at the office with several flavors of linux install on it.
I can bring a new server online with a base install in as few as five minutes with Debian or CentOS
I can debug workstations and servers with a quickbooting KNOPPIX install.
I even have some kernel installations customized to install network drivers for the Dell 2650 so that the installs I do for those are quick and simple. (basically the broadcom network drivers and the openssh-server packages are preseeded to be installed with the default package)
Here are the contents my pxelinux.cfg/default file:
DISPLAY boot.txt
#DEFAULT etch_i386_install
LABEL etch_i386_install
kernel debian/etch/i386/linux
append vga=normal initrd=debian/etch/i386/initrd.gz —
LABEL etch_i386_expert
kernel debian/etch/i386/linux
append priority=low vga=normal initrd=debian/etch/i386/initrd.gz —
LABEL etch_i386_rescue
kernel debian/etch/i386/linux
append vga=normal initrd=debian/etch/i386/initrd.gz rescue/enable=true —
LABEL knoppix
kernel knoppix/vmlinuz
append secure myconfig=scan nfsdir=192.168.0.1:/srv/diskless/knoppix nodhcp lang=us ramdisk_size=100000 init=/etc/init apm=p
ower-off nomce vga=791 initrd=knoppix/miniroot.gz quiet BOOT_IMAGE=knoppix
LABEL centos5_install
kernel centos/5/vmlinuz
append ks=nfs:192.168.0.1:/srv/diskless/centos/5/ks_prompt.cfg initrd=centos/5/initrd.img ramdisk_size=100000 ksdevice=eth0
ip=dhcp url –url http://mirror.centos.org/centos/5/os/i386/CentOS/
LABEL centos5_raid_install_noprompt
kernel centos/5/vmlinuz
append ks=nfs:192.168.0.1:/srv/diskless/centos/5/ks_raid.cfg initrd=centos/5/initrd.img ramdisk_size=100000 ksdevice=eth0 ip
=dhcp url –url http://mirror.centos.org/centos/5/os/i386/CentOS/
LABEL centos5_hda_install_noprompt
kernel centos/5/vmlinuz
append ks=nfs:192.168.0.1:/srv/diskless/centos/5/ks_hda.cfg initrd=centos/5/initrd.img ramdisk_size=100000 ksdevice=eth0 ip=
dhcp url –url http://mirror.centos.org/centos/5/os/i386/CentOS/
LABEL centos5_install_noprompt
kernel centos/5/vmlinuz
append ks=nfs:192.168.0.1:/srv/diskless/centos/5/ks.cfg initrd=centos/5/initrd.img ramdisk_size=100000 ksdevice=eth0 ip=dhcp
url –url http://mirror.centos.org/centos/5/os/i386/CentOS/[dfads params=’groups=221&limit=1′]
LABEL lenny_i386_install
kernel debian/lenny/i386/linux
append vga=normal initrd=debian/lenny/i386/initrd.gz —LABEL lenny_amd64_install
kernel debian/lenny/amd64/linux
append vga=normal initrd=debian/lenny/amd64/initrd.gz —LABEL etch_amd64_install
kernel debian/etch/amd64/linux
append vga=normal initrd=debian/etch/amd64/initrd.gz —LABEL etch_amd64_linux
kernel debian/etch/amd64/linux
append vga=normal initrd=debian/etch/amd64/initrd.gz —LABEL etch_amd64_expert
kernel debian/etch/amd64/linux
append priority=low vga=normal initrd=debian/etch/amd64/initrd.gz —LABEL etch_amd64_rescue
kernel debian/etch/amd64/linux
append vga=normal initrd=debian/etch/amd64/initrd.gz rescue/enable=true —LABEL etch_amd64_auto
kernel debian/etch/amd64/linux
append auto=true priority=critical vga=normal initrd=debian/etch/amd64/initrd.gz —PROMPT 1
Here are the contents of my boot.txt file (so that I know what to type at the command line when booting)
– Boot Menu –
=============etch_i386_install  –  Debian Stable
etch_i386_expert   –  Debian Stable (Shows install menu every step)
etch_i386_rescue   –  Debian Stable Rescue
lenny_i386_install — has Broadcom net card customization
lenny_amd64_install — has Broadcom net card customization
etch_amd64_install
etch_amd64_linux
etch_amd64_expert
etch_amd64_rescue
etch_amd64_auto
centos5_install –Â CentOS 5 (Will prompt for disks)
centos5_install_noprompt –Â CentOS 5 (Will auto install without prompts)
centos5_hda_install_noprompt –Â CentOS 5 (Will auto install without prompts)
centos5_raid_install_noprompt –Â CentOS 5 (Will auto install on raid 1 without prompts)
knoppix
Hope someone out there can find some use from this.
We of course can help people having trouble with their own TFTP and PXE Boot Server .
Installed PERC management software afaapps and created simple mirror
Installed PERC management software afaapps and created simple mirror
I just installed Debian Lenny on a Dell 2650 with an OLD PERC 3 RAID controller.
I then installed the afaapps package from Dell’s website (http://support.us.dell.com/support/downloads/download.aspx?c=us&l=en&s=gen&releaseid=R85529&formatcnt=1&libid=0&fileid=112003)
Use this link or just search for ‘afaapps’ under the Drivers and Downloads section of the Dell support site.
After extracting the rpm from the downloaded file I ran alien against the file to turn it into a debian file
#apt-get install alien
#alien -d –scripts afaapps-2.8-0.i386.rpm
Now just install the created debian package
#dpkg -i afaapps_2.8-1_i386.deb
Now that you have installed the afacli you can run it at the command line prompt which will open the PERC command line “FASTCMD>”
Then you’ll open / connect to the RAID controller using “open afa0”
#afacli
FASTCMD> open afa0
Executing: open “afa0”
A simple ‘disk list’ command to find out what your disk situation looks like
AFA0> disk list
Executing: disk list
B:ID:L Device Type Blocks Bytes/Block Usage Shared
—— ————– ——— ———– —————- ——
0:00:0 Disk 35566478Â 512 Initialized NO
0:01:0 Disk 287132440 512 Initialized NO
0:02:0 Disk 287132440 512 Initialized NO
you may have to initialize your disks by typeing ‘disk initialize 1’ and ‘disk initialize 2’ to make sure that the container can access them, you can see in my example above that my two disks are already initialized.
Now I will create a volume on disk 1 and mirror that disk to disk 2
AFA0> container create volume 1
AFA0> container create mirror 1 2
At the bottom of your screen you should see the status of the mirroring Job, something like.
Stat:OK!, Task:100, Func:MSC Ctr:1, State:RUN 16.2%
Once the job completes you can partition and format the disk. Check the label on the disk by running:
AFA0> container list
Executing: container list
Num Total Oth Chunk Scsi Partition
Label Type Size Ctr Size Usage B:ID:L Offset:Size
—– —— —— — —— ——- —— ————-
0 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1 Mirror 136GB Valid 0:01:0 64.0KB: 136GB
/dev/sdb 0:04:0 64.0KB: 136GB
From this I can see that I will need to partition and format disk “/dev/sdb”
Have fun! And if I can help you on it let me know.
Print jobs on Remote Desktop wont delete from print queue
Print jobs on Remote Desktop wont delete from print queue
A remote user was logging into their server to do some work from afar.
When they went to print their work some sort of connection problem occurred and when they reconnected they had two local printers in their “Printers and Faxes” window with different sessions, one with a print job (from the disconnected session) and one with no print jobs (from the current session).
When attempting to print again the job would not print and they would disconnect and try again only to find that they now have 3 printers (2 from disconnected session). This went on for a while they tried to delete the print jobs from the queue but the job never deleted, it just stayed in the status of “Deleting” or “Deleting – Paused”. Eventually they ended up with 6 of these printers each with one print job (I am sure they could have put more jobs in a print queue if they really wanted to).
This is the point at which I came in to clear up the issue.
I knew we had to clear out the print queue or that printer would never work.
Here are the steps we had to go through in order to fix it,
- Stop the “Print Spooler” Service. You can do this two ways
- From the Command Prompt (Start -> Run -> Type ‘cmd’ -> Click ‘OK’)
This opens a Command Line Interface, type “net stop spooler” - Open the Services Console (Start -> Control Panel -> Administrative Tools -> Services)
Select “Print Spooler” from the list of Services by double clicking and then click “Stop”
- From the Command Prompt (Start -> Run -> Type ‘cmd’ -> Click ‘OK’)
- Delete all print jobs. You can also do this two ways
- In the open Command Prompt window type del c:windowssystem32spoolprinters*
Type ‘y’ if it asks you if you are sure - In your Windows Explorer window goto C:windowssystem32spoolprinters
Delete all files
- In the open Command Prompt window type del c:windowssystem32spoolprinters*
- Restart the print spooler. Either:
- In the open Command Prompt window type “net start spooler”
- Click the “Start” button on the open “Print Spooler” Service Properties window
There you have it. For those of you that know how you can create a batch file which will quickly do all of this
net stop spooler
sleep for 5
del /Q /F c:windowssystem32spoolprinters*
net start spooler
The “/Q” switch tells the “del” command to delete the files without prompting if it is “ok to delete on global wildcard”
The “/F” switch tells the “del” command to delete the files even if they are marked read only
To get this batch file to work you will need to get the “sleep” program to pause for a few seconds after the service stops so that the system can release the lock it has on the files. If you do not have the sleep program you may find that running the batch file the first time errors out saying it can not access the file. But run it a second time and it works.
Debian Lenny Network Boot on Dell 2650 (Broadcom Network Card)
Debian Lenny Network Boot on Dell 2650 (Broadcom Network Card)
Debian Lenny (and etch) both do not include the drivers for the broadcom network cards that come with many dell servers, I use the Dell 2650 a lot but I have also had issues with the 1750 and 1950, I am sure their are other
I hear it is a licensing problem with debian not being able to distribute the drivers or something.
Here is the solution I have come up with from my end.
On my pxeboot server (I refer to pxe booting but I dont describe how to set one up, check this howto out)
I cd into the directory that my lenny installation is to be setup in (based on the pxelinux.cfg/default file)
#cd /tftpboot/debian/lenny/i386/
I am going to download all of the network installation files for debian lenny on i386, this should apply 64 bit too though.
# wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/netboot.tar.gz
# wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/debian-installer/i386/initrd.gz
# wget http://ftp.nl.debian.org/debian/dists/lenny/main/installer-i386/current/images/netboot/debian-installer/i386/linux
download the broadcom drivers package and extract it to a folder called bnx2
# wget http://ftp.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14+lenny1_all.deb
# dpkg-deb -x firmware-bnx2_0.14+lenny1_all.deb bnx2
Create a temp working directory
# mkdir temp
# cd temp
extract all of the installation files from the initrd.gz file so you can manipulate them (include the driver)
# zcat ../initrd.gz |cpio -iv
Copy all of the firmware drivers from the extracted bnx2 directory into the root of the extracted initrd.gz kernel directory
# cp ../bnx2/lib/firmware/* ../bnx2/usr/share/initramfs-tools/hooks/firmware_bnx2 .
Since the kernel initrd.gz is only used during installation of the OS, this fix so far hasn’t addressed installing the broadcom driver package for the OS after installation.
To do this you will need to customize the kernel to select and install this package during installation using “preseeding”
Create and edit a file called preseed.cfg in the root of the extracted kernel directory (
# vi preseed.cfg
Place the following contents in that file (I have aso included the ssh server since I typcially do a minimum install without ANY packages but I need ssh)
#automatically select these packages when installing the server
#d-i pkgsel/include string openssh-server firmware-bnx2
base-config apt-setup/non-free boolean true
d-i preseed/late_command string apt-install firmware-bnx2; apt-install openssh-server;
As another shortcut that can shave a tiny bit of time of of your installation, if you do not use USB storage during your installation, there is no need to wait for the delay and errors that occur during the system’s search for those USB storage devices. Deleting the USB drivers from the installation kernel will prevent these errors
# rm -rf lib/modules/2.6.26-2-486/kernel/drivers/usb/storage
Now it is time to put the extracted kernel directory back together in the location that the pxe boot is looking for it.
# find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz
that is it! you have customized and rebuilt your installation kernel for network boot.
Simply pxeboot to this installation with your Dell or broadcom server and the drivers will be included.
The concepts used above can also help you to setup and customize a net boot which has packages already selected or otherwise speed your install along with drivers or other customizations already selected, look into preseeding for more options here.