The vCenter Server’s vpxd logs contain entries similar to:

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) “ODBC error: (23000) – [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint ‘PK_VPX_GUEST_DISK’. Cannot insert duplicate key in object ‘dbo.VPX_GUEST_DISK’.” is returned when executing SQL statement “INSERT INTO VPX_GUEST_DISK (VM_ID, PATH, CAPACITY, FREE_SPACE) VALUES (?, ?, ?, ?)”

The link to the VMware knowledgebase article is here: vCenter Server service fails with the error: Violation of PRIMARY KEY constraint ‘PK_VPX_GUEST_DISK’

The curious part? That issue is addressed specifically by vCenter Update 1, Build 208111 and we’re running 208111. In any case, shutting down the vCenter service (ok, it wouldn’t run anyway) and running the SQL script supplied in the KB article did the trick and fixed that which should probably already have been fixed. Still super happy we’re not running on Oracle anymore, though…

You have two hosts with VMs running in an HA cluster. Host 1 gets completely isolated from the network – no service console, no guest net connections, nothing – but the VMs are still running on it. If you have your HA settings set up to not power down those virtual machines on isolation, HA has already started up those VMs on Host 2 – and when Host 1 reconnects to the network (with it’s never-shut-down-VMs) you’ve now got two VMX processes running in memory on two hosts. This is not good.

The current solution is to make sure that your HA settings, as shown in the attached picture, are such that a host will power off it’s virtual machines when it finds itself isolated,  thereby making sure that only one VMX process is running as. But there is still a window of time – 12 seconds by default – where a reconnection of that isolated hosts can mean that you’ll end up with those two VMS processes running on separate hosts. In my experience so far, if you’re running HA you’re going to run into this situation sooner rather than later. Some notes on what we saw last night:

• HA was set up correctly to shut down VMs on isolation. Yet no VMs were shut down on isolated hosts, even after several minutes of isolation.
• We started to manually take note of running VMs on isolated hosts using esxtop, shut down the hosts by the command line through iLO, and re-registered them manually on good hosts.
• Before we could finish this, network connectivity was restored and we found ourselves in a split-brain situation. It was fairly easy to tell what was going on by looking at the listing for VMs in the cluster. In the column for which host a VM was registered on, you’d see it change back and forth every few seconds.
• The best course of action, unfortunately, was to right-click the VM and power it off. In some cases we’d check the VM’s console and see a BSOD, power it off, and immediately the console would switch to booted Windows login screen – that meant that one of the VMX processes was the BSOD (the “old” process) and that the new process was the one where HA had taken over and properly booted up the VM.

So the largest concern on our part right now is that isolation response – why didn’t these hosts know to start shutting down their VMs? We’re running 4.0U1, possibly 3-4 patches back, though. We’re working with VMware’s support personnel to determine the root cause on that. The good news, though, is that vSphere 4.0 Update 2 should include code to directly address this long-standing problem with High Availability. Currently, if you know enough to connect directly to each ESX host during this split-brain outbreak, you’ll be greeted with a dialog box to release the lock on a VM and resolve the split-brain situation for that VM. In Update 4, apparently, this will be auto-answered for you. After last night, I can’t wait…

zombies tag applies ’cause that’s what it feels like is going on when you’re in the thick of it

My lovely wife might disagree, but I’ve been on Facebook for a while – since you had to petition to have your non-.edu account opened up to it (she’s been on since, like, day 2). At one point I completely destroyed my FB account over drama and “friends” and whatever. Now I’m just getting more and more concerned with privacy and how much Facebook seems to be willing to erode what little of it we have left. From hard-to-understand privacy settings to the latest:

Today, Facebook removed its users’ ability to control who can see their own interests and personal information. Certain parts of users’ profiles, “including your current city, hometown, education and work, and likes and interests” will now be transformed into “connections,” meaning that they will be shared publicly. If you don’t want these parts of your profile to be made public, your only option is to delete them.

So I’ve now deleted all that info from my profile. But I’m getting that uneasy feeling that Facebook is becoming more of a worry as far as my privacy is concerned vs. how much I get out of it. The consideration of deleting my account continues, I guess.

You’ve got to wonder how badly you foobered up your product release when the Electronic Frontier Foundation has this to say about Buzz:

If you’re going to use Google Buzz, we recommend that you opt-out during profile creation. If you have already created a profile,¬†change it to private immediately. Then go through the suggested list, and edit it as appropriate before making it public again. PC World has a helpful¬†privacy checklist to help users understand the privacy implications of Google Buzz options.

How many people are going to want to read through such checklists and really tweak their profile/settings in order to actually take advantage of Buzz while maintaining some semblance of privacy? I just turned it off and removed anything I could find related to Buzz in my profile. If anything, it made me take a closer look at how open my profile was on Google.

Fail, Google. Really really really huge fail.

Didn’t realize that Google Buzz gets turned on with some really, really huge privacy-nuking options. WTF, Google?

From the Silicon Valley Insider:

A Google spokesperson asked us to phrase this claim differently. Like this:¬†“In other words, after you create your profile in Buzz, if you don’t edit any of the default settings, someone could visit your profile and see the people you email and chat with most (provided you didn’t edit this list during profile creation).”

I guess I could take some time to figure out how to protect myself, but instead I’ll be sitting it out for now with a tech tip from Consumerist:

You can go into Buzz and selectively follow/unfollow certain people to avoid this kind of incident, but the best evasive maneuver is to scroll down to the bottom of the screen and click “Turn off Buzz.”

Wow… just wow, Google.

via Gizmodo: AT&T Comes in Last in Consumer Reports Study That Surprises No One

Here’s some news anyone with an iPhone could have told you: AT&T delivers crappy service that its customers hate. But this news comes from a reputable source, Consumer Reports, instead of the usual whiny friends.

Carrier exclusivity and all that.

So I’m interested in this ’cause I run a Mac at home for sanity’s sake. Gizmodo’s reporting about a particularly nasty (possible) result of hoovering that new copy of iWork ’09 that was announced a few weeks ago: OSX.Trojan.iServices.A.

This may be a first for the Mac software world, and it’s not cool at all: ill-gotten copies of iWork ’09 circulating on Torrent sites contain OSX.Trojan.iServices.A, which is something you don’t want.

The Trojan parks itself in your /System/Library/StartupItems folder with read-write-execute root privileges—from there it can phone home to a remote server and install additional nasties throughout your system. Right now, the only true fix is a full format and re-install, since its residual pieces can be spread far and wide. You can spot if your particular warez iWork is infected by searching for the iWorkServices.pkg inside the installer.

You know, Macs being invulnerable and all…