Record Replay, the technology that allows you to reproduce what's going on in a virtual machine with machine-level instructions, has been shown off at VMworlds past, but is just now coming into its own. You could experiment with it a bit in Workstation 6.0, but it is now available in a useful way in VMware Workstation 6.5, (in beta but has a new Release Candidate). Let's let E Lewis introduce it in his new blog. Link: Better Software Development with Replay Debugging: VMware Workstation 6.5: Reverse and Replay Debugging is Here!.
I'm proud to announce that VMware Workstation 6.5 includes new experimental features that provide replay debugging for C/C++ developers using Microsoft Visual Studio. Replay debugging allows developers to debug recordings of programs running in virtual machines, and it is valuable for finding, diagnosing, and fixing bugs that are not easily reproduced, a particularly challenging class of bugs. Once the manifestation of a bug has been recorded, it can be replayed (and debugged) over and over again, and it is guaranteed to have instruction-by-instruction identical behavior each time. In addition, Workstation includes a feature that simulates reverse execution of the program, making it easier to pin point the origin of a bug.
Aside from being insanely cool and perhaps the end of the heisenbug, I think this shows how VMware's 10 years of experience manifests itself in innovation. Virtualization is about more than server consolidation, and once you are virtualized, the really interesting things can start to happen.
Here's E demonstrating how this works. I think the UI has changed a bit since we filmed this. We're running Visual Studio on the host, outside the VM, and attaching to a process inside the VM and putting in triggers and whatnot in the debugger as it replays until we track down the bug we're looking for. If we go too far, we can always hit rewind.
Oh, and there's a Lenovo laptop to be won: VMware Record and Replay Challenge
I got asked this today, because this post is now out of date and the location of Andrew Kutz's VC plug-ins have moved from his original site lostcreations. I'm throwing this up here so Google can lead others to the right place. Specfically, he's got the Console, Invoke, and SVMotion plug-ins up there. His guide to his back-door plug-in methodology (which remember is unsupported and is being superceded by official mechanisms) is on his publications page.
Link: Can’t find my VMware utilities? « akutz’s blog.
You can access all of my VMware utilities at http://code.google.com/p/akutz/. There have been several requests for these utilities, and I simply do not have the time right now to create binaries for all of them. I am also waiting to see what will happen at VMworld and how it affects the plug-in architecture. Once that is more definitively known then I will resume my work on plug-ins.
VMware Fusion 2 Release Candidate 1 is now freely available for download.
Just like VMware Fusion 2 Beta 1 and VMware Fusion 2 Beta 2, the RC is free to download. VMware Fusion 2 will be a free, downloadable upgrade for all VMware Fusion 1.x customers, when released.
Building on great features like Multiple Snapshots and AutoProtect automatic, timed snapshots that keep your Windows-on-Mac experience as safe as possible, VMware Fusion 2 RC also includes an embedded complimentary 12-month subscription to McAfee VirusScan Plus to make sure your Windows XP and Vista stay squeaky clean.
You can install VirusScan Plus right from the VMware Fusion menu bar, from the moment you create a new virtual machine, meaning you’re always protected.
VMware Fusion 1.x ships in English, French, German, Japanese, and Simplified Chinese. VMware Fusion 2 RC adds Italian and Spanish to that list, making sure the VMware Fusion team will be welcomed with open arms on our next team offsite to Cancun, Barcelona, and Rome.
3D performance keeps getting stronger and more polished, as seen recently in the launch of Microsoft’s new Photosynth 3d photo visualization product.
Lots of Mac users were disappointed to see that Photosynth shipped without native Mac support. The good news is, it runs just fine in VMware Fusion 2 RC. You can read our blog post here about Photosynth for Mac.
Nuts and Bolts for Web Developers
And for you Mac web developers out there who enjoy using Mac-based web development editors like Panic Software’s Coda or Bare Bones’ BBEdit, but need to verify your creations on Windows-based browsers too, VMware Fusion 2 RC modifies the way it treats Windows-based web browsers shared using VMware Fusion 2’s application sharing.
Now you can you can launch your web page projects right into any Windows browser you like, not just the Windows-default browser, directly from the development environment. Windows-based Internet Explorer, Firefox, Opera, and even Safari for Windows, all on the same virtual machine, all accessible from your development editor!
As always, you can read about all the new additions, bug fixes, and more in the Release Notes.
Have I mentioned how VMware tends to slip more features into its products during the beta cycle? It's like Christmas every day around here. You think you've poked and shaken all the boxes in your parents' closet and know what's coming, and then the shiny new bike shows up in front of the tree. Today's shiny bike is a free year of virus protection for your XP or Vista VMs that you can invoke from the Fusion toolbar. There are other new features in the RC, which you can read about here: VMware: Team Fusion: VMware Fusion 2 Goes Release Candidate: Embedded Antivirus Software, Enhanced Localization, and More.
VMware Fusion 2 Release Candidate 1 is now freely available for download.
Just like VMware Fusion 2 Beta 1 and VMware Fusion 2 Beta 2, the RC is free to download. VMware Fusion 2 will be a free, downloadable upgrade for all VMware Fusion 1.x customers, when released.
Building on great features like Multiple Snapshots and AutoProtect automatic, timed snapshots that keep your Windows-on-Mac experience as safe as possible, VMware Fusion 2 RC also includes an embedded complimentary 12-month subscription to McAfee VirusScan Plus to make sure your Windows XP and Vista stay squeaky clean.
You can install VirusScan Plus right from the VMware Fusion menu bar, from the moment you create a new virtual machine, meaning you’re always protected.
EMC's Chad Sakac goes into SRM failback (essentially running SRM back from the Recovery Site to your original site), but in the midst of this talks about the value proposition of SRM when, after all, many of the required steps can just be scripted, right?
Link: Virtual Geek: A Few Technical Threads - Part 3: SRM Failback.
Click through for the details, but here are the highlights. Link: VMware: VI Team Blog: Top Tips for Deploying VI, part 2.
1. If you have an active/passive FC storage array (most mid-range arrays fall into this bucket), be careful about setup. Firstly, be sure to have redundant paths from FC switches to your arrays’ storage processors. Secondly, be sure to use “MRU” (the default) for the path-selection policy and not “fixed”.
2. When configuring your VI environment for VMotion, make sure that your physical network switches are configured properly; in particular, make sure that each port has the right network (e.g. VLAN) visibility.
3. When using VMware HA, take note of how memory reservations are specified and used to reserve cluster failover capacity. Using more consistent reservations or disabling admission control are both appropriate workarounds if the calculations are overly conservative in your environment.
4. When sizing your LUNs, a medium-sized LUN (~500GB) seems best for most situations.
Also don't miss Top Tips for Deploying VI, Part 1
There are two new pdf’s online…
Timekeeping in VMware Virtual Machines:
VMware Management and Automation products provide datacenter automation solutions in IT service delivery and business continuity. These products can be deployed across a shared VMware Infrastructure platform. This paper presents technical guidance and considerations when deploying these products together on a shared VI platform. [Link]
Management and Automation Products: Deployment Approaches and Considerations:
This paper describes how timekeeping hardware works in physical machines, how typical guest operating systems use this hardware to keep time, and how VMware products virtualize the hardware. [Link]
There’s a new book launched at VMworld Vegas 2008! This book is written by John Arrasjid(VMware), Daniel Conde(VMware), Karthik Balachandran(VMware), Gary Lamb(INX) and Steve Kaplan(INX).
You can pre-order the book via Digital-Guru, and here’s there description just to give you an idea: Deploying the VMware Infrastructure is an introduction to the world of VMware virtualization technology written by hands-on experts in the field. This booklet explains virtualization, enumerates its many benefits, and describes a set of real-world use cases. Overviews are provided for each of the VMware server virtualization, desktop virtualization, and automation and infrastructure management products; and detailed information is given about virtualizing your infrastructure-from building a VMware Infrastructure and selecting candidates for migration, to managing and optimizing the virtual environment. Other topics include security and disaster recovery, virtual desktop infrastructure, and advanced capabilities such as high availability, virtual machine snapshots and consolidated backups. This booklet is highly recommended for anyone interested in virtualizing their IT infrastructure or expanding their knowledge of VMware Infrastructure, including managers, technical staff, and those new to virtualization or VMware technology.

If you're interested in the newly-free ESXi, check out the new VMware ESXi Community Portal, which pulls together some of the available resources as well as being the best place to get up and running with ESXi.
Here is VMware CTO Steve Herrod on ESXi as a free download and some of the ecosystem and infrastructure surrounding it. Link: VMware: VMware: Virtually There: VMware Everywhere, part II.
The Scalable Platform for ApplicationsBut for the new virtualization user, I want to emphasize the core
technology that has long made ESX, and now ESXi, the most trusted
hypervisor in the industry today. Both hypervisors offer features,
such as 4 virtual CPUs, 256GB hosts, NIC teaming, and memory
de-duplication, for running the most resource intensive applications
simultaneously. We have invested significantly in engineering features
and optimizing the core kernel to provide the highest consolidation
ratios while still offering near-native performance for each
application.
Importantly, these features work equally well
regardless of your guest operating system, be it Windows, Linux,
Netware, etc. VMware has the broadest support for guest operating
systems and we do not play favorites. VMware does not have an
operating system agenda. VMware ESXi is offered no strings attached –
with no expensive operating system upgrade required. The same can’t be
said for our competition…
Aside from availability as a single server hypervisor, ESXi is also a component of the VMware Infrastructure suite. Customers deploy VMware Infrastructure to transform their computing environments from individual servers into a shared pool of CPU, memory, network, and storage resources that can be dynamically allocated to applications as demand changes.
Link: VMware Communities: ESX 3.5U2 Patch experiences? ....
Dear VMware Customers,
In addition to the express patch and the re-issued ESX/ESXi 3.5 Update 2 release, we now have an alternative installation process for customers who haven't applied either to hosts that were affected by the product expiration issue.
Below are the details we list at http://www.vmware.com/landing_pages/esxexpresspatches.html:
*Known VMware ESX 3.5 Update 2 Express Patch Installation Challenges
The following message is applicable ONLY for customers who had
installed the impacted release of ESX 3.5 Update 2 (build number
103908), but not yet applied the express patch.
We are aware that you may encounter the following challenges installing the express patches needed to correct the problem:
If you experience one of the challenges listed above, please contact
your support provider and indicate you need assistance with the U2
Alternative Install Process (U2 AIP). The support team can assist
customers with this alternative installation procedure.
Thank you,
The VMware ESX Product Team
I’ve never seen this before. I wrote an article about root SSH access to a ESXi system. Today I noticed a blog entry that describes how you can disable root access for SSH and create users which can use “su” to become root! Cool stuff.
Check the article here! Here’s the procedure:
Log in to the console,
edit the inetd.conf:
vi /etc/inetd.conf
search for the following line (type: “/ssh”) (This is the line you uncommented to enable SSH in the first place.)
ssh stream tcp nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -i
add -w to the end of this line: (type: “i” for insert mode):
ssh stream tcp nowait root /sbin/dropbearmulti dropbear ++min=0,swap,group=shell -i -w
Exit and save the file (press escape, type “: x”)
Create a /home directory
mkdir /home
Create a new unprivileged user:
useradd your_name
Change the password for this user:
passwd your_name
Reboot the server
reboot
Once rebooted,
Log in with SSH using your new unprivileged user
Use
su -
to change to the root user.
Tested on:
VMware ESXi 3.5.0_Update_2-103909
As everyone probably already knows by now there’s a problem with 3.5 U2. VMware is working on a patch as we speak. There has been a KB article released, but it seems like everyone is clicking on the same link at the same moment cause it’s hard to get a decent respond.
The error message that appears:
This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
————–
Module License Power on failed
In short, the workaround is simple just set the date back and you will be able to start VM’s again. As soon as I know more about the new 3.5 U2 update I’ll let you guys know!
Last week I gave a webinar on managing VMware with PowerShell. If you missed it or didn't know about it, you can view it now. The webinar is mostly geared at people who haven't used the toolkit at all, but has a few real-time demos of some of the really powerful stuff you can do with it. Even if a basic overview doesn't sound all that interesting, the demos had a few examples you might not have known about, such as the example that shows how to set up VMotion. For the examples, please refer to the corrected slides I posted to slideshare.
There was also a Q and A session, which I've attached here with answers to the questions.
Enjoy!
Last night, we became aware of a code issue with the recently released update to ESX 3.5 and ESXi 3.5 (Update 2).
When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire. The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2. When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:
The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.
In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.
I am sure you’re wondering how this could happen. We failed in two areas:
We are doing everything in our power to make sure this doesn’t happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.
I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.
Thank You,
Paul Maritz
President and CEO
VMware
I’ve always been interested in ubuntu for the fact that in my opinion they have the capabilities to become a real Windows Desktop replacement. But on the other hand I’ve always been strugling to get everything working as it should. Video drivers and Wireless drivers always were a problem. This “new” alpha release is a huge step forward. Without any problems my wireless card got recognized and the correct video drivers were loaded. I connected to the internet within a few seconds, which took me half a day with 8.0.4. I know it’s far from being a release candidate but so far 8.10 is worth looking into.
1. If you have an active/passive FC storage array (most mid-range arrays fall into this bucket), be careful about setup. Firstly, be sure to have redundant paths from FC switches to your arrays’ storage processors. Secondly, be sure to use “MRU” (the default) for the path-selection policy and not “fixed”.
The best way to explain the first issue is with a picture. What’s wrong with the following configuration?
Although you might believe that you have full redundancy between the hosts and the switches, and specifically that you can survive one HBA failure on each host, the reality is that you don’t have enough redundancy. Here’s one failure scenario that won’t be handled properly:
The reason is that, with active/passive storage arrays, a given LUN can only be presented on one storage processor at a given time. The LUN can shift from one storage processor to another, but such a shift takes many seconds (potentially up to 30 seconds). If both HBA’s have failed (as in the above diagram), then the ESX hosts won’t be able to access to the same LUN at the same time. Host 1 attempts to access the LUN on storage processor 1; host 2 attempts to access the same LUN on storage processor 2; and you end up with a ping-pong effect, or a “path thrashing” effect due to the active/passive array shifting the LUN back and forth between the two storage processors. Performance of VM’s on both hosts will be erratic and penalized.
The solution is simple: create redundant connections from the FC switches to the array storage processors, as shown below.
There is a second noteworthy issue with active/passive arrays related to this same path thrashing effect: make sure that you use the “MRU” path selection policy (the default) rather than the “fixed” path selection policy. If you use “fixed”, you may make the mistake of forcing the use of a particular storage processor for one host… but a different storage processor for another host… and thus end-up in a similar LUN ping-pong or path thrashing situation.
For more details about path thrashing see, the SAN Configuration Guide.
2. When configuring your VI environment for VMotion, make sure that your physical network switches are configured properly; in particular, make sure that each port has the right network (e.g. VLAN) visibility.
VMotion requires that the destination ESX host have similar network connectivity to the source ESX host (so that, for example, the VM can continue access to its assigned VLAN after the VMotion). VirtualCenter checks for correct virtual switch configuration on the source and destination ESX; however, VirtualCenter does not for correct configuration of the physical network switches. In a larger VI deployment where many network switch ports are involved, a single misconfiguration of a single physical switch port can be hard to detect. The symptom will be as follows: when the particular VM relying on a particular VLAN id VMotion migrates to the particular ESX host with the misconfigured switch port, the VM loses all network connectivity. Solution: when adding new ESX hosts to a network, take the time to double-check your network switch port configurations to make absolutely sure that all the VLANs are correctly configured.
3. When using VMware HA, take note of how memory reservations are specified and used to reserve cluster failover capacity. Using more consistent reservations or disabling admission control are both appropriate workarounds if the calculations are overly conservative in your environment.
How VMware HA works: If a VMware ESX host fails, VMware HA will restart the VMs affected by that failure on alternate hosts in the cluster. In order to do so, HA must reserve failover capacity within the cluster. HA currently achieves this by implementing an “admission control” policy that prevents (or warns against) the powering on of VMs that would encroach upon the failover capacity being reserved. In some cases, however, the admission control calculations may be too conservative.
Example scenario: Suppose you have 19 VMs, each with a 300 MB memory reservation. To power-on all of these VM's, you need 5.7GB of RAM (=19*0.3) (total within the cluster, after allocating space for potential host failures, and not accounting for memory sharing in ESX). Since all reservations are equivalent, HA defines an average VM to require 300 MB of memory.
Now, let's suppose you power-on a 20th VM with a 2 GB memory reservation. Instead of calculating memory requirements as 7.7 GB (=19 x 0.3 + 1 x 2), HA takes a more conservative approach and redefines the average VM to be the biggest reservation observed. With the higher reservation specified, HA will cautiously assume that every VM need 2 GB of memory, and will ask for 40GB (=20*2) of RAM to be set aside for total runtime and failover capacity within the cluster. These calculations are intended to be conservative to ensure that sufficient spare capacity is available, without fragmentation across hosts within a cluster.
In many cases (such as clusters with widely varying sizes of hosts and VMs), however, these calculations can be more conservative than desirable, and can lead to “insufficient failover capacity” warnings when powering on more VMs.
Two potential approaches are recommended if you are observing these warnings, or want to avoid them within a heterogeneous cluster configuration:
Approach 1: Either lower the reservations on your most demanding VM’s, or remove the reservations skewing the calculations and rely upon “shares” instead. See the resource management guide for differences between reservations and shares.
Approach 2: Alternatively, configure HA to disable strict admission control. Host failures will still be detected and acted upon, but VMware HA will not prevent the starting of new VMs due to insufficient failover capacity.
4. When sizing your LUNs, a medium-sized LUN (~500GB) seems best for most situations.
Small LUN’s (and VMFS volumes) can result in SAN management complexity (too many LUNs to manage). Very large LUN’s can result in performance issues (due to VMFS lock-contention for certain operations), too coarse a granularity for troubleshooting and performance tuning, and failure/error isolation. The below chart summarizes some of the considerations. Details are provided on page 72 of the VI 3 SAN Design Guide.
| Smaller LUN's | ||||
| VMFS: Metadata overhead | Some overhead (0.5%) | Negligible overhead (<0.1%) | Negligible overhead (<0.1%) | |
| VMFS: Lock-contention during VM creation operations, or during VCB-based backup operations (*) | Near zero contention | Some contention | Much contention | |
| Impact of a failure or error, difficulty of troubleshooting | Affects a few VM's | Affects 20-30 VM's | Affects many VM's | |
| Ease of SAN mgmt | Hard (many LUN's to manage) | Medium | Easy (just 1 LUN to manage) | |
| Ease of tuning performance (**) | High (tunable per the few VM's on a LUN) | Medium (tunable for 20-30 VM's at a time) | Low (one setting for many, many VM's) | |
| Flexibility in specifying value-added services (***) | High (different LUNs can have different policies or settings) | Medium (tunable for 20-30 VM's at a time) | Low (many VMs share the same policies or settings) |
(*) File creation in VMFS grabs a SCSI lock on the LUN. Since each LUN has a limited number of SCSI locks, excessive concurrent file creation in VMFS can cause lock contention, which can hurt performance. This can be apparent if multiple users are concurrently creating VM’s (and therefore VMFS files), or when a VCB-based backup process is concurrently backing up multiple VM’s (and is therefore concurrently creating multiple VMFS REDO files)
(**) e.g. RAID-level, array caches, queue depths, path selection/path dedication
(***) e.g. Backup, other data protection features such as replication, mirroring, etc., capacity optimization features such as de-dupe, thin-provisioning, etc., security and encryption features
I’ve noticed many of the hits on my blog are related to ESXi. One of the most asked questions is how can I SSH to an ESXi hosts? Looking at my wordpress stats, this is also one of the top searches.
By default this isn’t possible. But there’s a way to get this working, just do the following:
Done!
What’s going on… the amount of views / hits have almost doubled over the last couple of weeks. I received more hits in August already than I did in July. Here are some stats that the hosting company provided me, unique hits:
dec - 1225
jan - 7700
feb - 13100
maart - 13300
april - 19233
mei - 19034
jun - 20783
jul - 27564
Aug - 36159
Last month I had 9017.39 MB of traffic. This month I already have 3039.94 MB.
And I just checked this in Wordpress and see the same numbers coming up. Normally I would hit around 1200 a day on the Wordpress Stats. In July this went up to 1800 and now I’ve reached 2700….
Any other bloggers noticing the same trend? Just talking with my colleague about it and could this be related to the ESXi hype… hmmm,
For those that are still using 3.0.2 and are not looking to upgrade to 3.5 U2 yet, VMware just released 3.0.3! Don’t expect new features to popup. Version 3.0.3 is mainly about compatability and patches:
Processors
Newly supported processors are:
Drivers
Upgraded drivers are:
Guest Operating Systems
Newly supported guest operating systems are:
Enhanced Patch and Upgrade Management
Significant improvements to the following patch management utilities:
Service Console
Update for service console from RHEL3 U8 to RHEL3 U9.
Management Agents
Support for new versions of Management Agents:
Backup Software
Support for additional backup software from third-party vendors:
Quality Improvements
A significant number of issues have been fixed. Some of these issues have been described in Resolved Issues in this Release.
Get it while it’s stil hot and read the complete release notes here!