HowTo Install Dell OpenManage 5.3 on Citrix XenServer 4.1 and have it listed in a central OpenManage Management System

Update 10/3/2008: I have successfully followed these directions to install OpenManage 5.4 under XenServer 5.0.

A few months ago I posted an article on how to install Dell OpenManage on a Citrix XenServer. Since then I've been wanting to have the Xen servers appear in our central OpenManage Management System so we can get central alerting of hardware issues. I've discovered a few gotcha's along the way, and figured it'd be a good idea to post one consolidated how-to for anyone interested.

This article pulls heavily on vendor documentation, but let me stress that as of the posting of this how-to, neither Citrix nor Dell officially support running OpenManage on a XenServer.

For reference, here are the documents I've used:

• Citrix's KnowledgeBase article on how to get IT Assistant, the client-side component also called "OpenManage Managed Node", also called "OpenManage Server Administrator", also known as "Dell OpenManage Server Administrator Managed Node". Seriously, different names names all depending on the readme, package file, or Dell's support site.
• Dell's page from the 5.3 manual on configuring SNMP under Linux

And the software you'll need is:

• Citrix XenServer 4.1
• Dell OpenManage Management Station installed on a Windows server somewhere and properly configured/working, with your SNMP traps/destinations/communities configured for the IP range the XEN server is on.
  
• Dell OpenManage Managed Node 5.3 for RedHat Enterprise
  

Okay, now lets begin. I suggest doing all of these changes on a test system first. At the end, I'd suggest consolidating all of the changed files and make a new .tar.gz file to deploy to your other servers.

Step 1: Modify the OpenManage installation package
Remember, this isn't an officially supported solution, so if you try to run the Server Administrator Managed Node installation right away, you'll get an error that it doesn't recognize Xen's linux distribution (CentOS, fyi).

1. Copy the tar.gz file to your Xen host. I use WinSCP for this.
   
2. Extract the tar.gz you downloaded to a temporary folder:
   tar -xzf OM_5.3.0_ManNode_A00.tar.gz
   
3. Edit the file "setup.sh":
   vi setup.sh
   
4. Change the lines 2972 and 2973 as follows:
   2972: GBL_OS_TYPE=${GBL_OS_TYPE_UKNOWN}
   2973: GBL_OS_TYPE_STRING="UKNOWN"
   Become:
   2972: GBL_OS_TYPE=${GBL_OS_TYPE_RHEL5}
   2973: GBL_OS_TYPE_STRING="RHEL5"
5. Save the file

Step 2: Aquire "lockfile"
I noticed originally when running the "setup.sh" script, that as it tried to start the services the inventory service would fail with an error

Warning: The lockfile utility is not found in PATH or /usr/bin.
This utility prevents concurrent executions of setup.sh
which can lead to unexpected or invalid installation results.

Or with:

no lockfile in (/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin)
invcol Error: Cannot find utilities on the system to execute Inventory
Collector.
Make sure the following utilities are in the path: tar gzip tail rm mkdir
chmod ls basename wc lockfile stat

I figured the easiest way to get this file would be to run YUM, the CentOS repository system which Xen helpfully left in. This, of course, had its own set of issues. From my previous article on YUM under Xen, you may need to do the following. I'd suggest first testing if "lockfile" exists by typing "which lockfile". If you don't have "lockfile" installed, then proceed:

Xen is using CentOS for their host OS, but they disable the CentOS repository and enable what appears to be a non-existent/functional Xen repository. Whenever you try to use Yum, it throws an error that the repository checksum is invalid:

http://updates.xensource.com/XenServer/4.0.96/domain0/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from xensource: [Errno 256] No more mirrors to try.

To resolve this temporarily, and only on my test system, I edited /etc/yum.repos.d/XenSource.repo and changed the following line:

enabled=1

to:

enabled=0

I then edited /etc/yum/repos.d/CentOS-Base.repo and changed the following line under the top "[base]" section

enabled=0

to:

enabled=1

"yum whatprovides lockfile" now returned a bunch of results. It seems "lockfile" is part of the postfix package. To get this file I:

1. ran "yum install postfix"
2. copied "/usr/bin/lockfile" to a temporary location
3. ran "yum remove postfix"
4. copied "lockfile" back to "/usr/bin/lockfile"

Now starting the services ("/opt/dell/srvadmin/omil/supportscripts/srvadmin-services.sh restart") no longer shows an error and I can see the inventory in the GUI

Step 3: Configure SNMP
This can actually be fairly complex depending on your SNMP setup in your environment. I'm going to assume you're doing a simple configuration here, where you have two communities set up in your OpenManage Management System for this IP range: "MY_READ" and "MY_WRITE" configured for read and write access, respectively. In this case, just copy in what I have below, after making a backup of the original.

If you want an explanation of the options and conventions used below, try "man snmpd.conf", Google "snmpd.conf", or read Dell's page from the 5.3 manual on configuring SNMP under Linux. If you get stuck, feel free to leave a comment here and I'll help you as best I can.

1. Rename the original snmpd.conf file:
   mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.orig
2. Copy the following into a new /etc/snmp/snmpd.conf file, changing the "MY" strings to your own:

# Map users to community strings
# sec.name source community
com2sec U_ReadOnly default MY_READ
com2sec U_ReadWrite default MY_WRITE

# Map users to groups
# groupName securityModel securityName
group G_ReadOnly any U_ReadOnly
group G_ReadWrite any U_ReadWrite

# create view
# name incl/excl subtree mask(optional)
view all included .1

# grant rights to the above views
# group context sec.model sec.level prefix read write notif
access G_ReadOnly "" any noauth exact all none none
access G_ReadWrite "" any noauth exact all all none

# Set trap destination
trapsink my.openmanage.server MY_MANAGEMENT

# Allow localhost access
rocommunity MY_READ 127.0.0.1
rwcommunity MY_WRITE 127.0.0.1

# Allow remote hosts access
rocommunity MY_READ my.openmanage.server
rwcommunity MY_WRITE my.openmanage.server


# Below has been left in from the default snmpd.conf file:
syslocation Unknown (edit /etc/snmp/snmpd.conf)
syscontact Root (configure /etc/snmp/snmp.local.conf)

# Added for support of bcm5820 cards.
pass .1.3.6.1.4.1.4413.4.1 /usr/bin/ucd5820stat

# Allow Systems Management Data Engine SNMP to connect to snmpd using SMUX
smuxpeer .1.3.6.1.4.1.674.10892.1

Now, restart SNMP by typing:
service snmpd restart

Step 4: Modify IPTABLES
XenServer contains a firewall that needs to be modified before OpenManage can see this node.

Below are my modifications to the iptables file. You only really need the lines for UDP 161 and UDP 162. The TCP 1311 is if you choose to install the "Web Administration Console", which I don't because it takes up 100M of RAM.

1. Make a backup copy of the current iptables file, just in case:
   cp /etc/sysconfig/iptables /etc/sysconfig/iptables.orig
2. Edit the file as below, adding in the BOLD lines in the same place. This is important as the order of the file determines what is allowed. If you simply append these lines to the end, traffic will be blocked:

# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 161 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 162 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1311 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 694 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Now, restart iptables by typing:
service iptables restart

Step 5: Install OpenManage
Finally :)

1. Change to the directory where you extracted "setup.sh"
2. Execute the setup script:
   ./setup.sh
3. You'll be prompted with a screen asking you which options to select. I generally only install 1,3,4,5. Option 2, "Server Administrator Web Server" will consume at least 100M of the XenServer RAM so I usually leave it off. After you selected your options, press "i" to install
4. Follow the prompts checking for errors. I usually take all the defaults from here
5. When prompted, go ahead and start the services. Watch for errors

Thats it!

Now go to your OpenManage Management System console and initiate a "Discovery and Inventory" of the IP range for this XenServer. It should show up.

For reference, you can stop/start the OpenManage services by running:
/opt/dell/srvadmin/omil/supportscripts/srvadmin-services.sh stop
/opt/dell/srvadmin/omil/supportscripts/srvadmin-services.sh start

And you can uninstall it by running:
/opt/dell/srvadmin/omil/supportscripts/srvadmin-uninstall.sh

Step 6 (optional): Repackage all of this as a new tar.gz
I did the following to ease deployment, as this is a lot of steps.

1. I copied the snmpd.conf, iptables, and lockfile to my extracted "openmanage" folder:
   cp /etc/snmp/snmpd.conf /root/openmanage/
   cp /etc/sysconfig/iptables /root/openmanage/
   cp /usr/bin/lockfile /root/openmanage/
2. I wrote a wrapper for setup.sh to make the above changes for me prior to running setup.sh. The contents of the wrapper script are below.
3. After creating the wrapper script, I made sure it was executable:
   chmod +x /root/openmanage/setup_wrapper.sh
4. I made a new tarball of all of this:
   cd /root
   tar -czf modified_om_5.3_mannode.tar.gz openmanage/

Pay attention, there's a space between "gz" and "openmanage/"

My wrapper script:

#!/bin/sh
# version 8.5.20 Aaron Dodd
# back up existing config files
cp /etc/sysconfig/iptables /etc/sysconfig/iptables.orig
cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.orig

#copy in new config files
cp iptables /etc/sysconfig/iptables
cp snmpd.conf /etc/snmp/snmpd.conf

cp lockfile /usr/bin/lockfile

# restart affected daemons
service snmpd restart
service iptables restart

./setup.sh

Importing / Exporting filters in Thunderbird 2.0, copying filters between folders

Thunderbird 2.0 has an annoying limitation of only being able to specify rules for folder roots and cannot apply these rules to other sets of folders. That would be less annoying if Thunderbird offered a way to move or copy rules between folder roots.

For example, I use IMAP to speak to our corporate Exchange server. I have a set of rules configured there. I also have "Local Folders" which contains my "Archive" folder, where I put all the IMAP emails I want to keep.

Sometimes I add a filter to the IMAP folders that I want to then process retroactively against my archive (i.e. to add a certain tag to be picked up in the Save Searches folders I have). Using just the Thunderbird interface I'd have to manually recreate this rule.

There are a few extensions on addons.mozilla.org for exporting and importing rules. I haven't had any success with these. I get errors that the folder doesn't exist or such.

You can fairly easily copy rules between folders by editing the underlying rules file. To do this:

1. Find out your root folder locations. For each of the sets of folders you want to copy rules between, right-click the folder in Thunderbird and choose "Properties".
2. Remember the location listed under "Local Directory".
   
3. Close Thunderbird. If Thunderbird is open, any changes you make to the rules file appear to get overwritten as soon as you open the Filters window again
4. Navigate to the locations from step 2.
5. In each is a file called "msgFilterRules.dat". This is a plain text file. Make a backup copy, then open the original in your favorite text editor
6. There is no explicit delimeters between rules, but for me a rule seems to always start with the line "name=" and end with the line "condition=". Just copy that block from one msgFilterRules.dat" file to the other.
7. Launch Thunderbird and you should now see that rule in the Filters window.

Issue with Dell OpenManage 5.3 inventory under Xen Server 4.1

I wrote in this article how to install Dell OpenManage 5.3 on a Xen 4.1 Host. However, I noticed that when logging into the test systems' GUI that I wasn't seeing an inventory of hardware. Performing the installation again, I noticed a few errors I didn't see originally.

When first running setup.sh, there is a brief display of the following error:

Warning: The lockfile utility is not found in PATH or /usr/bin.
This utility prevents concurrent executions of setup.sh
which can lead to unexpected or invalid installation results.

This appeared again when starting the services

To resolve this, I decided to install lockfile via YUM. The problem is, YUM seems broken by default in 4.1.

Xen is using CentOS for their host OS, but they disable the CentOS repository and enable what appears to be a non-existent/functional Xen repository. Whenever you try to use Yum, it throws an error that the repository checksum is invalid:

http://updates.xensource.com/XenServer/4.0.96/domain0/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from xensource: [Errno 256] No more mirrors to try.

To resolve this temporarily, and only on my test system, I edited /etc/yum.repos.d/XenSource.repo and changed the following line:

enabled=1

to:

enabled=0

I then edited /etc/yum/repos.d/CentOS-Base.repo and changed the following line under the top "[base]" section

enabled=0

to:

enabled=1

"yum whatprovides lockfile" now returned a bunch of results. It seems "lockfile" is part of the postfix package. To get this file I:

1. ran "yum install postfix"
2. copied "/usr/bin/lockfile" to a temporary location
3. ran "yum remove postfix"
4. copied "lockfile" back to "/usr/bin/lockfile"

Now starting the services ("/opt/dell/srvadmin/omil/supportscripts/srvadmin-services.sh restart") no longer shows an error and I can see the inventory in the GUI

To avoid this cumbersome process on our other Xen hosts I make a new tar.gz of the extracted and modified OpenManage installation which included this file, which I make sure to copy to /usr/bin prior to running "setup.sh" above.

Using LDAP in Thunderbird 2.0 to query Microsoft Exchange for addresses

I've recently switched from Outlook to Thunderbird/Lightening for my email and calendaring needs. I use IMAP to talk to our Exchange server, but was missing being able to querying the global address book.

Thunderbird supports LDAP, but it took some digging around to get this to work right. Here's what I did.

1. In Thunderbird, I clicked "Go" and choose "Address Book"
2. I clicked "New" and "LDAP Directory"
3. In the "Directory Server Properties" window that appears, on the "General" tab I gave the "Name" a meaningful value (in this case, our Exchange server).
4. In the "Hostname" field I entered the FQDN of our Exchange server (i.e. mail.mycompany.com)
5. For Base DN, since I know we're not doing anything fancy I entered in "DC=mycompany,DC=com" since I know that's our AD root DN.
6. For "Port Number" enter 3268. I was a little confused at first as I could telnet to our Exchange server over both LDAP and LDAP/SSL ports. But, Thunderbird couldn't seem to query them. But after googling Exchange and LDAP, apparently Exchange doesn't use the standard LDAP ports, it uses 3268.
7. For "Bind DN" simply enter your username in domain\username format.

Seems obvious in retrospect but I figured I'd post it in case anyone else needed help.

Manual "P2V" migration of a Windows server into a XEN VM instance

Xen doesn't officially offer a P2V solution for migrating existing physical Windows instances into XEN VMs. Their answer is a link to two vendors that offer XEN support for their P2V products. Both companies, however, charge a few hundred bucks per-migration, which makes it a bit expensive for the one-off migrations here and there (in both vendors' defense, the P2V functions aren't the main purpose of their products). I've used the following steps to perform migrations of Dell-based Windows 2003 Server instances into XEN virtual machines simply using NTBackup, without having to purchase any "P2V" tools/licenses.

These steps basically mimic how one can recover a Windows server onto new hardware after a failure, with some extra precautions taken to avoid interrupting the existing physical instance until it's ready to be replaced.

The steps:

1. From the original server, I perform an NTBACKUP-based backup of the system state and the harddrives to a file share on the network.
2. On the XEN host I provision a new machine of the same OS type and SP level as the physical machine to be moved. For example: if I'm moving a multi-CPU Windows 2003 Standard 32-bit server with Service Pack2, I provision a VM with Windows 2003 Standard 32-bit with Service Pack2 and multiple VCPUs. (Note: the HAL of the VM must match the original server's, so if you're migrating a single, non-hyperthreaded CPU machine, provision the VM with only one VCPU).
   
3. This new VM must have the same drive layouts as the original server. I.e. if I have the OS on "C:" and the applications on "D:" and the CDROM mapped to "Z:" on the original server, I do the same on this VM. The only exception can be to create additional VM disk images that don't exist on the original server as long as they're mounted on a drive letters not used on the original server. I create "E:" for temporary space for below as I want the .BKF file local for the restore.
   
4. On the new VM I copy the following files off the C drive to the "E:" local disk:
1. c:\boot.ini
2. c:\windows\repair
5. I copy in the .BKF file created in step 1 to the new VM's "E:" disk as I want to disconnect the NIC before restoring.
   
6. In the XEN console, I switch the VM's NIC to a non-used network so that it won't be able to talk to our network or domain controllers. This is to avoid interrupting the physical machine's functionality until I'm done, as the VM will take its SID, domain credentials, and possibly IP address.
7. On the new VM I run NTBACKUP and choose "Restore".
   
8. Under "Tools / Options" I choose the "Restore" tab and click the radio button "Always replace the file on my computer", as otherwise NTBACKUP will skip any existing newer files.
   
9. I then restore the local harddrives and system state from the backup
10. BEFORE REBOOTING I copy the boot.ini and windows\repair folders back in from the copies made in step 4 above
11. I run the Xen Tools installer to install XEN's PV, NIC, and SCSI drivers
12. I reboot. Sometimes at this point I get a blue-screen saying helpfully "An error has occurred." If this happens I reboot into safe mode and:
1. Go into the device manager
2. Delete the existing SCSI card (do this even if it shows up as a Citrix XEN SCSI card)
   
3. Install Xen Tools again
   
4. Reboot
13. When the machine comes up, I verify all the settings match the live server's. If they do, I then:
1. Turn off the physical instance
2. In the Xen console, switch the NIC to be on the proper network, allowing it to talk to our domain controller
3. I assign it the same IP address as the physical server, if appropriate
4. I reboot and allow it to log into the domain

If there are any failures logging into the domain, its most likely because the physical computer reset its security channel between the time the backup was made and the time the VM replaced the phyiscal server. This can usually be fixed by typing the following commands, or demoting the machine to a workgroup server and then re-joining it to the domain:

1. NLTEST /SC_RESET:

2. netdom reset Destination computer /domain:domain_name usero:admin_user /passwordo:admin_user_password

Note: these commands require the Windows Support Tools to have been installed. You can find them on the Windows install CD.

There have been a few instances where I've wanted to clone a physical machine but not actually replace it with the VM. This has been mostly in cases where I wasn't100% sure how the physical machine is set up or functions (inheriting old applications) but need to start the migration process. The modifications to this procedure below allow me to have the new VM running in tandem with the old physical so that testing can be done to verify functionality.

To keep the physical running in tandem with the VM, before starting the process above, place a copy of NewSid from Microsoft on the VM. Then, instead of step 13 above, do the following:

1. While the VM is still off the network, demote it from a domain member to a workgroup machine
2. Reboot
3. Run NewSid and as part of running it, change the name to something unique
4. Reboot
5. Give the VM a new IP address, different than the physical
   
6. Now you can bring the server onto the network and join it to the domain under the new name, IP, and SID