According to VMware support, ESX will not  go any further in trying to shut down your VM – it will not just kill the process of that VM running in memory. If ESX can’t perform a graceful shutdown of a VM it will stop trying to get that VM process shut down.

That’s an important thing to understand and remember and directly speaks to what we saw: when some of our ESX hosts regained network connectivity, we had the classic split-brain scenario. We would open the console to a VM, see a BSOD, and power off the VM – and then almost immediately see the running operating system in that same console window. What we were watching was the BSOD process on the formerly-isolated ESX host being stopped, replaced by the healthy process of the VM when it was brought up on one of the surviving hosts.

So some take-away from this event?

• Once we realized that we had lost both the Service Console connection and the NFS datastore connection, we simply should have taken note of what VMs were still running in memory by using esxtop and then powered off the ESX host until the network outage was resolved. The other surviving hosts were already booting up the VMs. The key here, obviously, is that this is a manual process – if this outage occurred and resolved itself before we could get to the isolated hosts we’d still end up with a split-brain problem.
• We might want to reconsider our use of NFS and it’s dependence on network connectivity to the datastores. This major outage interrupted TCP/IP traffic at many different layers of the network, to the point where NIC redundancy and failover did not occur. It’s been my experience that FC SAN, while more expensive, also has been more robust and less prone to outages – the vast majority of our outages involving ESX have been Ethernet-related problems.

• Support for deleting snapshots while the VM is running
• Support for multi-monitor guest setups in the GUI for Windows guests
• RDP video acceleration
• Memory ballooning to dynamically in- or decrease the amount of RAM used by a VM (64-bit hosts only)

As you might already know, VirtualBox is my favorite virtualization product for the desktop because of it’s rich feature set and support for both Windows and Mac. I recently scored over a 90% on my VCP exam in no small part to the ability to run a Windows VM on my Mac, allowing me to mobilize my study environment with ease. If you’re looking to simply get into virtualization with ease, and for free, VirtualBox is definitely worth the download and minimal install footprint.

http://www.virtualbox.org/

/usr/bin/vmware-cmd /vmfs/volumes/datastore/vmname/vmname.vmx getid

This should return something like “getid() = 192″

Using that ID you can then cancel that tools install:

/usr/bin/vmware-vim-cmd vmsvc/tools.cancelinstall

Voila! No more tools install, and migration should work normally. Huge thanks, again, to Bob over at The Lone Sysadmin!

So, found my snapshot and all the files of the VM (one line):

/vmfs/volumes/NFSSTORE/.snapshot/nightly.2010-01-31_0000/SERVERNAME

I then found my destination, or where the VM currently lives:

/vmfs/volumes/NFSSTORE/SERVERNAME

I’m particularly interested in SERVERNAME.vmdk, which is the C: in this case. In order to not run into myself, I renamed the current, not-booting SERVERNAME.vmdk using vmfsktools while in the working directory of the current VM:

vmkfstools -E SERVERNAME.vmdk SERVERNAME_old.vmdk

Then, this is as simple as importing the .vmdk from the .snapshot directory to the current working directory of the VM. Yes, you can use cp to do this, but vmfsktools is hella faster (one line):

vmkfstools -i /vmfs/volumes/NFSSTORE/.snapshot/nightly.2010-01-31_0000/SERVERNAME/SERVERNAME.vmdk /vmfs/volumes/NFSSTORE/SERVERNAME/SERVERNAME.vmdk

Since the .vmx file is still configured with the SERVERNAME.vmdk and SERVERNAME_1.vmdk disk files, there shouldn’t be any need to reconfigure the VM before powering it on. Finally, I verified that the machine was booting normally and deleted the _old file to save disk space.

…when I flipped the page over on my desk calendar this morning and noticed that the January 31st Virtualization Tip of the Day from the fine folks at PrintedOwl.com has this to say:

It’s a best practice to keep the number of COS software installations to a minimum. Here’s a case that’s justified for some environments to ensure graceful host shutdowns. Mike at motogobi has done some leg work & put up a procedure: http://www.motogobi.com/2008/11/13/setting-up-powerchute-network-shutdown-in-vmware-esx-35/

I’m flattered

It’s good to see Lifehacker focusing on backup tools. After literally years of struggling with backing up various computers and laptops I’ve owned over the years, and having learned my disaster-recovery lesson the hard way when I lost the hard drive on my desktop in 1996 or something, I’ve recently settled into a very cozy relationship with Mozy. I signed up for the yearly Home Unlimited plan ($50/yr per machine) and have been nothing short of pleased with it’s performance.

Check out other options over at Lifehacker, but whatever you do? Make sure you’re backing up your data…

Click for larger, more confusing view

Virtual Geek has an amazingly insightful post combining multiple presentations and points from representatives from EMC, MVWare, NetApp, Dell/Equalogic, HP/Lefthand, and some other folks in the business. As you can see by the names involved, we’re talking about most every major virtualization storage player in the industry today – and they’re here to help us make heads or tails of iSCSI storage. You’ll even be able to make sense of that diagram above after reading:

As discussed earlier, the ESX 3.x software initiator really only works on a single TCP connection for each target – so all traffic to a single iSCSI Target will use a single logical interface. Without extra design measures, it does limit the amount of IO available to each iSCSI target to roughly 120 – 160 MBs of read and write access.

This design does not limit the total amount of I/O bandwidth available to an ESX host configured with multiple GbE links for iSCSI traffic (or more generally VMKernel traffic) connecting to multiple datastores across multiple iSCSI targets, but does for a single iSCSI target without taking extra steps.

Here are the questions that customers usually ask themselves:

Question 1: How do I configure MPIO (in this case, VMware NMP) and my iSCSI targets and LUNs to get the most optimal use of my network infrastructure? How do I scale that up?

Question 2: If I have a single LUN that needs really high bandwidth – more than 160MBps and I can’t wait for the next major ESX version, how do I do that?

Question 3: Do I use the Software Initiator or the Hardware Initiator?

Question 4: Do I use Link Aggregation and if so, how?

Here are the answers you seek…

One word: NAS. (okay, so it’s an acronym for Network Attached Storage)

In our case, we need some large, fast, short-term backup space to handle convenience-backups, or backups that I think are not there for disaster-recovery, but are there to avoid having to re-download gigabytes of data from repositories like MSDN or Linux distros. Should irreplaceable data go to tape and get rotated offsite? Sure. Can I still offer several terabytes off disk-to-disk backups just in case? You bet. It was surprisingly easy, actually.

Since the vast majority of the administration on this network is at the Windows-comfort level, the three things I used were: old hardware, Windows 2003 Server, and the most recent distro of Openfiler.

Step one: wrangle your hardware

Using an Optiplex GX270, I was able to mount three IDE drives in it – two 500gig and a 750gig. I booted the machine to make sure that the BIOS and pins were set correctly. On another Optiplex, I installed Server 2003, brought it up-to-date, and added it to our domain.

Step two: install Openfiler

Download the latest release of Openfiler from their website and burn to CD. Boot the machine with all the disks to that CD and install Openfiler. Openfiler is an open-source product built on rPath linux. It’s it most complex setup, it can be deployed as a full-featured filer/file server, with useful things like NFS, CIFS, FTP, and rsync, as well as snapshots and software RAID. In this case, I enabled the iSCSI service in Openfiler, added all three disks in that old Optiplex as a single physical group, then created a single iSCSI volume on that group. That’s a 1.67TB iSCSI target using disks that would have otherwise ended up spread around in desktops.

Step three: mount the Openfiler iSCSI target in Windows, format, fill

On the second Optiplex, I installed the free Microsoft iSCSI Software Initiator, which comes installed by default in Server 2008. Without going into all the technical aspects of iSCSI (since I’m not fully aware of them, really), the iSCSI initiator is a simple Windows service that allows a Server 2003 machine to connect to an iSCSI target – in this case, the Openfiler machine. In it’s most basic setup, you simply add the IP address of the Openfiler in the iSCSI initiator setup to point to the Openfiler. It’ll find the target that’s presenting those physical disks, and Server 2003 will now believe it’s got one big additional hard drive to play with when you open up Disk Management. It’s cool.

Some things I’ve learned about using this setup (and iSCSI in general):

• DO NOT use an iSCSI target as a dynamic disk – only use it as a basic disk. If you create it as a dynamic disk, and then do something like, oh, set the flag that doesn’t allow Windows to fully start up until it’s reconnected all of it’s iSCSI targets… well, you’ll be waiting a long, long time. Ask me how I know this.
• On your file server, or whatever is presenting the iSCSI target, it’s highly useful to have two gigabit NICs in use. In a basic setup like this with low traffic usage it’s not as noticeable. But as traffic increases, you want your file traffic to users going out through one interface, then your iSCSI traffic to your NAS going out another interface (and across a different switch or network segment).

Finally, the flexibility of using something like Openfiler really creates a ton of pretty cool opportunities. Say you have the desktop hardware to present the disks but want to use a VMWare guest as the file server? Easy: since Openfiler can act as an iSCSI target, and since VMWare ESX can initiate sessions to iSCSI, you could add an Openfiler NAS as an additional datastore for your ESX server – and utilize all that disk space using a regular old VM setup.

Some additional reading on iSCSI:

Wikipedia entry on iSCSI
Microsoft Storage Technologies – iSCSI
Openfiler.com
Connect VMware ESX Server to a free iSCSI SAN using Openfiler

Here’s how I did it – mostly for my own future reference:

Step 1 – get the software

Obtain the PCNS 2.2.1 rpm (not 2.2.3, which won’t install) from APC:

ftp://ftp.apcc.com/apc/public/software/unix/linux/pcns/221vmware

Upload this to your ESX server using some utility like FileZilla or WinSCP over port 22. I just placed it in /tmp

Step 2 – open ESX firewall for PCNS

Now, open up some ports in the ESX firewall after opening up an SSH session to ESX (please note that the quotes in these statements will not copy over, it seems, in PuTTY. Make sure the config changes you submit have quotes in them):

esxcfg-firewall -o 80,tcp,out,”APC PowerChute Port 80″
esxcfg-firewall -o 3052,tcp,out,”APC PowerChute Port 3052″
esxcfg-firewall -o 3052,tcp,in,”APC PowerChute Port 3052″
esxcfg-firewall -o 3052,udp,out,”APC PowerChute Port 3052″
esxcfg-firewall -o 3052,udp,in,”APC PowerChute Port 3052″

Step 3 – Install PNCS and configure

Switch the to directory you placed the rpm in and install using the following command:

rpm -ihv pcns-2.2.1-100.i386.rpm

Start the PCNS configuration script using the following command:

/usr/local/bin/PowerChute/PCNSConfig.sh

Some things to look out for when setting up PCNS using that command-line wizard:

• I picked option 3 when running the script since I had 2 APCs to configure. Presumably you could pick option one for a single APC setup.
• The script asks for the IP of the management card on the APC, the user name and password (I used the admin username/password) and then the Authentication Phrase. This phrase needs to be set up in the APC’s management interface prior to trying to connect this PCNS client – setup uses this phrase to register the client with the APC.
• Once you finish the install script, it’ll ask you if you’d like to start the PCNS service – do so, then go to the URL of the PCNS web console for your ESX server: http://yourvmhostname:3052 and log in using the admin credentials you specified prior.

Step 4 – disable UPS shutdown

The final step that I chose based on this excellent writeup of installing PCNS was to disable the default behavior of shutting down the UPS once the client has shut down during a power event. Since I have several physical servers connected to these UPSes, I really didn’t want to mess with actually shutting down the APC itself automatically. To disable this, simply connect to the client web management console (the port 3052 link in step 3), navigate to “Configure Shutdown,” uncheck the box next to “Turn off the UPS after the shutdown finishes” and apply.

Further thanks for Stefan Schuller’s post on setting up PCNS and the commented link to the 2.2.1 rpm…