November 26, 2009

How to Customize the vSphere ESX 4.0 Installation ISO

Filed under: vmware — Tags: , , , , , , , , , — Tim Lefler @ 4:41 pm

This is my own approach on customizing the VMWare vSphere ESX 4.0 installation DVD.

The vSphere ESX 4 install process uses an updated linux boot release commonly referred to as Syslinux. From the Syslinux boot process vSphere launches an initrd.img kernel instance which is a custom VMware/Linux kernel containing a multitude of VMware ESX 4 drivers and components. The custom VMware kernel incorporates Linux kickstart scripting functionality to invoke automated installations. The script location is defined as part of the Syslinux functionality and is available as a menu at boot time. A control file located on the boot media provides these variable control elements.

So this is my attempt to documenting my approach on modifying the VMWare syslinux boot menu and putting my kickstart config file “ks.cfg” on the boot media. This makes an install clean and only requires the installation DVD instead of a USB thumb drive to hold the ks.cfg.

For more information my particular ks.cfg file and how I prompt for dynamic ESX host properties like network information and FQDN see my previous post here.

To put a custom “ks.cfg” file on the CDROM’s filesystem so you need a couple of prerequisites….I use the mkisofs tool to modify the default ESX ISO because it is a Unix command line tool and I’m familiar with using it.  You can use whatever you are comfortable with but the document process below will need to be adjusted.

Do all of this signed in as root, of course. Install the mkisofs tool on an ESX Host with a lot of space, you can download the version I used here

# rpm -i ./mkisofs-2.0-6.i386.rpm

Download the DVD iso from VMware and upload to your ESX Service console on one your ESX hosts.

Mount the iso

# mount -o loop ./esx-DVD-4.0.0-164009.iso /mnt/cdrom

Create a Directory to hold the contents of this iso in a filesystem that has at least 3 GB of free space.

# mkdir /home/root/newiso

Copy the contents of the mounted iso to this new newiso directory:

# cp -rp /mnt/cdrom/* /home/root/newiso

Unmount the ESX DVD iso

# cd /root/newiso
# umount /mnt/cdrom

To save space you can now delete the original iso file. We don’t need it anymore.
In the new location change into the isolinux sub-directory.

# cd /root/newiso/isolinux

First we need to modify the isolinux.cfg file. This controls the menu that is displayed when the isolinux image is booted.

The default menu looks something like this:

And is defined in the isolinux.cfg like this:
(more…)

November 25, 2009

Notes on How to perform a scripted install of VMware ESX 4.0

Filed under: vmware — Tags: , , , , , , , — Tim Lefler @ 5:32 pm

First a little backgroud and a few links.  You will find all kinds of information on automated installation on the VMWare User Group community.  VMWare also has some paltry level of documentation on the subject.  This best beginning source of information for me was this blog….. and it is from here that the following excerpt was taken to help remind me of the basiss behinfd the ESX scripted installation:

The vSphere ESX 4 install process uses an updated linux boot release commonly referred to as Syslinux. This Syslinux release version 3.63 supports a variety of popular protocols to facilitate a remote central install repository. FTP, HTTP, NFS and gPXE are all available options for provisioning network attachment to a remote install repository.

From the Syslinux boot process vSphere launches it’s initrd.img kernel instance which is a custom VMware/Linux kernel containing a multitude of VMware ESX 4 drivers and components. The custom drivers allow for a more closely integrated VMware ESX 4 install process that targets an improved ESX 4 server configuration result.

The custom VMware kernel incorporates Linux kickstart scripting functionality to invoke automated installations. The script location is defined as part of the Syslinux functionality and is available as a menu at boot time. A control file located on the boot media provides these variable control elements. Depending on the media type Syslinux uses a respective cfg file to implement this function. The various available Syslinux boot methods that I am aware of are USB, CD, DVD, PXE and gPXE. In this blog I will demonstrate an ISO CDROM method to perform the automated boot cycle. Any of the boot methods mentioned will all work and have varying levels of complexity to achieve.

The ISO CD and DVD based Syslinux configuration uses a config file named isolinux.cfg, USB boot images would use isolinux.cfg as well as gPXE based boot services can use either depending on the final gPXE target image.

This is a great source of reference, although it did not meet all my needs because it is a network based installation.

So what exactly am I trying to do?  I want to use the built in kickstart scripting mechanism to consistently install or reinstall ESX Hosts in our VMWare vSphere 4.0 environment.  I want the installation script to prompt for things that change from ESX host to ESX host like:  Service console IP  information, iSCSI IP information, and the FQDN.  I also want the installation script to download an additional configuration script from a centralized repository after the installation is complete and execute it.

This is probably complicated a little in our environment because all of the network connections to our ESX hosts are 802.1Q VLAN trunks to give us the greatest flexibility in production.  Unfortunately, the ESX kickstart/syslinux kernel used for the ESX 4.0 installation does not support VLAN trunking, so no networking is available unless we switch the port before and after the installation.  I don’t want to do that……

So what are the advantages to how I approached this automated installation problem:

  1. Installation is done totally from media so network not required during the scripted installation
  2. One configuration script for any ESX host being installed that prompts for dynamic elements.
  3. A post installation download of an additional script that can be easily updated without changing the installation media.
  4. The downloaded script can configure anything that can be configured on a running ESX host.
  5. Complete install/reinstall of ESX environment in 25 minutes.

We used to do all this with ESX 3.5.  By omitting the “network” section in the kickstart “ks.cfg” the installation would prompt us, allowing us to setup the dynamic elements.  Unfortunately, I had to build my own prompting mechanism into my ESX4 kickstart script and it was a little tricky.

I’ve found the most straight forward approach is to simply use the same old boring default DVD iso downloaded from VMWare.  Download it, burn it to a DVD and we are 25% of the way there already.  The default installtion DVD, contains a non default option to do a  “ESX Scripted Install using USB ks.cfg”.

Select

ESX Scripted Install using USB ks.cfg

So now, all I need to do is place my modified kickstart “ks.cfg” file on a USB memory stick and it will install from the CD but because I’ve modified the %pre and %post sections of the “ks.cfg” in the manner I’m about to describe, the ks.cfg will prompt me for the dynamic networking components.  If you don’t want to use a USB drive then you can see my other post that talks about a complicated method of burning your custom ks.cfg on to the DVD media.

So now lets look at my customized kickstart “ks.cfg”

We start with some of the basics……

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
accepteula
 
keyboard us
 
# Read network information from a file we are going to populate in the %pre section
# of this kixstart script.
%include /tmp/networkconfig
 
auth  --enablemd5 --enableshadow
 
install cdrom 
 
rootpw --iscrypted $1$M7fvhjkgghkyguysdfjhgjoqg.
 
timezone --utc 'US/Eastern'
 
# Read partition information from a file we are going to populate in the %pre section
# This allows me to create local vmfs with "hostname:storage1" and I've adjusted the sizes
# on my service console partitions.
%include /tmp/partconfig
 
%pre --interpreter=bash

Now the “%pre” section is where it begins to get interesting. We need to collect information for three things:

  • Service Console IP
  • iSCSI Service Console IP
  • iSCSI Kernel IP

I have my script fill in all the other stuff, like subnet, gateway, VLAN excetera because we set up all of our hosts consistently and that stuff doesn’t change. You could easily expand on this part to prompt for other items. I’ve XX’d out my specific settings you will need to plug in your own if you model after this script.

So here is my “%pre” section….. (more…)

November 20, 2009

vSphere 4.0 Service Console in a local VMFS volume

Filed under: vmware — Tags: , , , , — Tim Lefler @ 9:54 am

WTF? Why would you stick the service console or Console Operating System (COS) on the VMFS volume?

Previous versions of ESX held the /Boot within the Red Hat formatted disk partition’s. Vsphere differs, it now has the Service Console stored within a VMDK which can be either held on a local VMFS partition or from a Shared Storage VMFS LUN. I guess some of the advantages to this might be:

  • Maybe to remove the way the COS in previous releases is tied to CPU0, with it being a VM it can be scheduled.
  • Being able to perform a snapshot the service console and have the opportunity to rollback if ever needed after upgrading/updating due to issues on the changes
  • Probably means you can now provide dedicated resource throttling for the VM running the service console so it doesn’t have a chance to run away with the whole CPU

The Service console VM is stored on the VMFS volume as host-XX-XXXXXXXXXX and contains a vmdk labeled cos-vmdk.  This vmdk includes the /boot, /,/var and swap partitions for the service console.

Service Console (COS) on local VMFS volume

Service Console (COS) on local VMFS volume

But the downside is the VMFS volume can never be deleted and recreated therefore they cannot be upgraded. After upgrading to vSphere 4 I now have VMFS volumes at 3.21, 3.31 and 3.33 and I’m stuck with them unless I reinstall.

So why do I care about upgrading my VMFS partitions?  For locally attached storage, I guess it isn’t a big deal, everything still works, but for shared SAN vmfs volumes or NFS volumes the newer 3.33/3.31 vmfs version introduces distributed locking optimization and can significantly reduce metadata IOs on the shared storage.  Read here for a post that touches upon the advantages -  VMFS 3.31 distributed locking optimization

This is also a problem if you wanted to ensure that the service console is on a separate VMFS partition away from other VMs.  So what if we do a manual install instead of an upgrade?  When performing a manual install of Vsphere from DVD/ISO it formats under an extended partition all of the local spare disk space with VMFS.  If you want to change this behavior you need to use the scripted install.

So for me I don’t really care if it is isolated but I would like to try to keep all of my VMFS volumes at the most current level to take advantage of performance improvements and bug fixes.  To do this I have to reinstall.

Other vSphere 4 Storage Improvements

November 18, 2009

How to Determine the FSMO Roles in use for a Windows 2003 Active Directory Domain

Filed under: Information Technology — Tags: , , , , , , , , — Tim Lefler @ 5:08 pm

Windows 2003 Active Directory domains utilize a Single Operation Master called FSMO (Flexible Single Master of Operations)

The five FSMO roles are:

  • Schema master – Forest-wide and one per forest.
  • Domain naming master – Forest-wide and one per forest.
  • RID master – Domain-specific and one for each domain.
  • PDC Emulator- domain-specific and one for each domain.
  • Infrastructure master – Domain-specific and one for each domain.

In most cases, an administrator can keep the FSMO role holders (all 5 of them) on the same DC configured by the Active Directory installation process. However, there are scenarios where an administrator would want to move one or more of the FSMO roles from the default holder DC to a different DC.  For example, if you are decommissioning old equipment you would want to move the FSMO role to one of the other DC to prevent disruption of services.

I’m going to document two separate methods for determining each of the FSMO roles:

Using the GUI each of the FSMO Roles

(more…)

November 9, 2009

Notes on How to Setup VMWare Consolidated Backup with BackupExec

Filed under: vmware — Tags: , , , — Tim Lefler @ 12:34 pm

How to Setup VCB in SAN Mode with BackupExec

For a brief overview on VMWare’s Consolidated Backup and it’s modes of operation check out this earlier post.

This time I’m going to get into the nitty gritty of the installation/implementation. First thing is to downloaded the latest version from VMWare. I’m using VCB version 1.5 U1 and the “Integration Module for Symantec Backup Exec for Windows” (BEIM) both can be downloaded here.

I’m using BackupExec Version 12.5 with all of the service packs.

On your VCB proxy, install a supported version of Microsoft Windows. All versions of Windows, except Windows 2003 Enterprise Edition and Windows 2003 Datacenter Edition, assign drive letters to each visible NTFS and FAT volumes. To be able to use Consolidated Backup, you need to clear any unused drive letter assignments.

To disable assignment of drive letters

  1. Shut down the VCB proxy.
  2. Disconnect the VCB proxy from the SAN or mask all the LUNs containing VMFS volumes or RDMs.
  3. Boot the proxy and log in as administrator.
  4. Open a command‐line interface.
  5. Run the Diskpart utility by typing:
    diskpart

    The Diskpart utility starts up and prints its own command prompt.

  6. Disable automatic drive‐letter assignment to newly seen volumes by typing at the Diskpart command prompt:
    automount disable
  7. Clean out entries of previously mounted volumes in the registry by typing at the Diskpart command prompt:
    automount scrub
  8. Reattach the VCB Proxy to the SAN LUNs and reboot just for good measure.

To function,VCB needs to be able to communicate with the VMWare vCenter Server using TCP port 443 and needs to be able to communicate with the ESX Hosts on TCP port 902. Make sure you make any firewall adjustments required to allow this communication.
(more…)

VMWare Consolidated Backup Overview

Filed under: vmware — Tags: , , — Tim Lefler @ 11:05 am

VMWare Consolidated Backup Overview

VMWare Consolidated Backup (VCB) along with a third‐party backup software like Symantec’s BackupExec, allow you to perform backups of virtual machine disks. Consolidated Backup centralizes backup on the VCB proxy, which can be used to protect many virtual machines.

The following are the key features of Consolidated Backup:

  • Most major backup applications integrate with Consolidated Backup providing you with a fast and efficient way of backing up data in virtual machines.
  • Consolidated Backup eliminates the need for having a backup agent installed in each virtual machine you want to protect.
  • Consolidated Backup can read virtual disk data directly from your SAN storage device using Fibre Channel or iSCSI, or through the ESX Server I/O stack or use a network connection to an ESX Server host to access virtual disk data.
  • Consolidated Backup can run in a virtual machine.
  • Consolidated Backup supports file‐level full and incremental backups for virtual machines running Microsoft Windows operating system and image‐level backups for virtual machines running any operating system.

VCB can operate in 3 different modes: SAN Mode, SCSI Hot-Add Mode, and LAN or NBD Mode. I’ll primarily focus on SAN Mode but want to include information about the differences here for clarity. The VCB Host can not mix and match modes.

SAN Mode

You select this mode when your ESX Server stores its virtual machine disks on Fibre Channel SAN or iSCSI SAN. When using this mode, you can completely off‐load backups to a physical VCB proxy.
(more…)

November 2, 2009

Using Enhanced VMotion Compatibility (EVC) to Overcome VMotion compatibility problems

Filed under: vmware — Tags: , , — Tim Lefler @ 6:19 pm

When migrating machines using VMotion across multiple you may get the dreaded error:

Host CPU is incompatible with the virtual machine’s requirements at CPUID level 0×1 register ‘ecx’

CPUError

For VMotion purposes, compatible CPUs means that source and target CPUs must have the same manufacturer (AMD or Intel) and be members of the same basic processor family (for example, Pentium 4 or Core), and have a common set of features implemented.

I’m going to try to get around this problem by enabling EVT on my cluster.  Enhanced VMotion Compatibility (EVC) simplifies VMotion compatibility issues across CPU generations. (more…)

How to enable Virtualization Technology(VT) on Dell 1950

Filed under: Information Technology — Tags: , , , — Tim Lefler @ 3:48 pm

A quick post to docment how to enable the Virtualization Technology (VT) BIOS setting from the command line.  We had a situation where we could not add a 64bit VM on our ESX Host without having the BIOS setting set.  Unfortunately, this ESX host was in production and downtime could not be scheduled.

Install OpenManage and then run the following:

omconfig chassis biossetup attribute=cpuvt setting=enabled
(more…)

Powered by WordPress