Esxcfg command help

Thank you, Eric Siebert!

During VMworld 2010 in San Francisco one of the big messages being pushed out to the attendees is that VMware ESX will no longer have a future. The 4.1 release will be the last release of VMware ESX, future releases will only come with ESXi. Dauglas Phillips interviewed Sean and Charu from the ESXi team about what this will mean for customers.

Our 4.1 upgrade just changed a wee bit, I think.

The vCenter Server’s vpxd logs contain entries similar to:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) “ODBC error: (23000) – [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint ‘PK_VPX_GUEST_DISK’. Cannot insert duplicate key in object ‘dbo.VPX_GUEST_DISK’.” is returned when executing SQL statement “INSERT INTO VPX_GUEST_DISK (VM_ID, PATH, CAPACITY, FREE_SPACE) VALUES (?, ?, ?, ?)”

The link to the VMware knowledgebase article is here: vCenter Server service fails with the error: Violation of PRIMARY KEY constraint ‘PK_VPX_GUEST_DISK’

The curious part? That issue is addressed specifically by vCenter Update 1, Build 208111 and we’re running 208111. In any case, shutting down the vCenter service (ok, it wouldn’t run anyway) and running the SQL script supplied in the KB article did the trick and fixed that which should probably already have been fixed. Still super happy we’re not running on Oracle anymore, though…

If you haven’t heard about disk alignment and you’re using virtual machines, you owe it to yourself and your most-likely-growing infrastructure to understand what alignment is all about. On a small scale it’s almost unnoticeable. But I can tell you that on a large scale it becomes a major pain for you or your storage infrastructure team.

One quick tip: to check your disk offset on Windows systems, simply launch msinfo32.exe from the Run menu. See the thumbnail of this post for a screenshot.

From VirtualGeek’s excellent post:

The purpose of alignment is to minimize extraneous internal array operations. All arrays have internal constructs that are generally a function of the RAID model (and also the filesystem alignment, and in some cases logical page table constructs in virtually provisioned models).

<snip>

All the funky goodness is done via either filesystem or another (pages commonly) abstraction on TOP of the RAID abstraction. Think of a 4K NTFS IO operation in a Guest making it’s way down to the array. Once it gets there, let’s say the array has a 64K stripe, but a 1MB “page” used for these fancy features. Falling into two 1MB logical memory pages as an example – where statistically it’s much more likely to land on a boundary if the volume is aligned on a 4K boundary.

It’s very worth your time to delve into this article and find out how your environment is set up, like right now. You might find your templates are mis-aligned, or in our case, that VMware Converter does not properly align disks on conversion (wonder-app Platespin Migrate does, in fact, give you properly-aligned disks).

According to VMware support, ESX will not  go any further in trying to shut down your VM – it will not just kill the process of that VM running in memory. If ESX can’t perform a graceful shutdown of a VM it will stop trying to get that VM process shut down.

That’s an important thing to understand and remember and directly speaks to what we saw: when some of our ESX hosts regained network connectivity, we had the classic split-brain scenario. We would open the console to a VM, see a BSOD, and power off the VM – and then almost immediately see the running operating system in that same console window. What we were watching was the BSOD process on the formerly-isolated ESX host being stopped, replaced by the healthy process of the VM when it was brought up on one of the surviving hosts.

So some take-away from this event?

• Once we realized that we had lost both the Service Console connection and the NFS datastore connection, we simply should have taken note of what VMs were still running in memory by using esxtop and then powered off the ESX host until the network outage was resolved. The other surviving hosts were already booting up the VMs. The key here, obviously, is that this is a manual process – if this outage occurred and resolved itself before we could get to the isolated hosts we’d still end up with a split-brain problem.
• We might want to reconsider our use of NFS and it’s dependence on network connectivity to the datastores. This major outage interrupted TCP/IP traffic at many different layers of the network, to the point where NIC redundancy and failover did not occur. It’s been my experience that FC SAN, while more expensive, also has been more robust and less prone to outages – the vast majority of our outages involving ESX have been Ethernet-related problems.

You have two hosts with VMs running in an HA cluster. Host 1 gets completely isolated from the network – no service console, no guest net connections, nothing – but the VMs are still running on it. If you have your HA settings set up to not power down those virtual machines on isolation, HA has already started up those VMs on Host 2 – and when Host 1 reconnects to the network (with it’s never-shut-down-VMs) you’ve now got two VMX processes running in memory on two hosts. This is not good.

The current solution is to make sure that your HA settings, as shown in the attached picture, are such that a host will power off it’s virtual machines when it finds itself isolated,  thereby making sure that only one VMX process is running as. But there is still a window of time – 12 seconds by default – where a reconnection of that isolated hosts can mean that you’ll end up with those two VMS processes running on separate hosts. In my experience so far, if you’re running HA you’re going to run into this situation sooner rather than later. Some notes on what we saw last night:

• HA was set up correctly to shut down VMs on isolation. Yet no VMs were shut down on isolated hosts, even after several minutes of isolation.
• We started to manually take note of running VMs on isolated hosts using esxtop, shut down the hosts by the command line through iLO, and re-registered them manually on good hosts.
• Before we could finish this, network connectivity was restored and we found ourselves in a split-brain situation. It was fairly easy to tell what was going on by looking at the listing for VMs in the cluster. In the column for which host a VM was registered on, you’d see it change back and forth every few seconds.
• The best course of action, unfortunately, was to right-click the VM and power it off. In some cases we’d check the VM’s console and see a BSOD, power it off, and immediately the console would switch to booted Windows login screen – that meant that one of the VMX processes was the BSOD (the “old” process) and that the new process was the one where HA had taken over and properly booted up the VM.

So the largest concern on our part right now is that isolation response – why didn’t these hosts know to start shutting down their VMs? We’re running 4.0U1, possibly 3-4 patches back, though. We’re working with VMware’s support personnel to determine the root cause on that. The good news, though, is that vSphere 4.0 Update 2 should include code to directly address this long-standing problem with High Availability. Currently, if you know enough to connect directly to each ESX host during this split-brain outbreak, you’ll be greeted with a dialog box to release the lock on a VM and resolve the split-brain situation for that VM. In Update 4, apparently, this will be auto-answered for you. After last night, I can’t wait…

zombies tag applies ’cause that’s what it feels like is going on when you’re in the thick of it

– mike

/usr/bin/vmware-cmd /vmfs/volumes/datastore/vmname/vmname.vmx getid

This should return something like “getid() = 192″

Using that ID you can then cancel that tools install:

/usr/bin/vmware-vim-cmd vmsvc/tools.cancelinstall

Voila! No more tools install, and migration should work normally. Huge thanks, again, to Bob over at The Lone Sysadmin!

If you want to clear that saved connections dialog, it’s a simple registry edit in XP. Navigate to:

HKEY_CURRENT_USER\Software\VMware\VMWare Infrastructure Client\Preferences\RecentConnections

and simply delete the connections you don’t want (DO NOT delete the entire key, just the connections in that reg key). Enjoy your clutter-free VI Client…

So, found my snapshot and all the files of the VM (one line):

/vmfs/volumes/NFSSTORE/.snapshot/nightly.2010-01-31_0000/SERVERNAME

I then found my destination, or where the VM currently lives:

/vmfs/volumes/NFSSTORE/SERVERNAME

I’m particularly interested in SERVERNAME.vmdk, which is the C: in this case. In order to not run into myself, I renamed the current, not-booting SERVERNAME.vmdk using vmfsktools while in the working directory of the current VM:

vmkfstools -E SERVERNAME.vmdk SERVERNAME_old.vmdk

Then, this is as simple as importing the .vmdk from the .snapshot directory to the current working directory of the VM. Yes, you can use cp to do this, but vmfsktools is hella faster (one line):

vmkfstools -i /vmfs/volumes/NFSSTORE/.snapshot/nightly.2010-01-31_0000/SERVERNAME/SERVERNAME.vmdk /vmfs/volumes/NFSSTORE/SERVERNAME/SERVERNAME.vmdk

Since the .vmx file is still configured with the SERVERNAME.vmdk and SERVERNAME_1.vmdk disk files, there shouldn’t be any need to reconfigure the VM before powering it on. Finally, I verified that the machine was booting normally and deleted the _old file to save disk space.