vArchitect Newsletter 001

Veeam Instant Restores Failing after applying latest ESXi 6.0 U2 patch

https://nolabnoparty.com/en/veeam-9-patch-esxi600-201605001-breaks-instant-vm-recovery/

https://forums.veeam.com/vmware-vsphere-f24/esxi600-201605001-patch-release-breaks-instant-vm-recovery-t35499.html

This is something I ran across very recently. It seems that the latest ESXi patch for 6.0 Update 2 will break any Instant Restore capabilities you might use. Check the links above, and try not to install it until VMware has a workaround. If you’ve already installed it, you have a couple options:

  1. a.) Revert the patch in ESXi; you can do this by pressing Shift + R when ESXi boots and it’ll use the previous version stored in its alt boot bank
  2. b.) Get a patch from Veeam support. You’ll have to open a SR, because as of now, it isn’t available to the public, but they have a workaround.

Also, check out the VMware KB on the same topic here.

Keep in mind that although these articles only state “Instant VM Recovery,” it will also affect your SureBackups and anything that uses the vPower technology. Hopefully Veeam will have a public patch that can be installed over v9 Update 1. Stay tuned and we’ll let you know when that happens. For now, you might to get that patch, and also recreate your Virtual Labs on a cluster that DOESN’T have that patch installed. It only has an issue on the actual routing lab, so starting it up on another host should do just fine.

ESXi host BSOD fixed after applying latest ESXi 6.0 U2 patch

What they giveth with one hand, they remove with the other. The same patch which breaks Veeam’s Instant Restore (above) also does manage to fix a rather serious issue that I know I’ve relayed to some of you. There was a major issue with hosts crashing when VMs with the VMXNET3 adapter were under high load. So for those using virtual proxies on 10 GbE connectivity, you no longer have to be concerned with high network activity BSODing your ESXi 6U2 boxes. Do remember, however, that if you’ve performed the workaround as stated the in VMware KB, to re-enable LRO on your ESXi host: esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0

Microsoft Convenience Update breaks VMs with VMXNET3 adapters

This is one to watch out for if you routinely install and/or approve updates from WSUS or SolarWinds that aren’t just of the critical variety. Microsoft has flubbed and released an update labeled a “convenience update” that is anything but convenient. In so applying, the update will essentially remove the networking configuration from your NIC, very likely leaving the affected VM without network connectivity. VMware blogged about this recently to warn customers not to apply the update. Here’s one thing you can do if the update has been mistakenly applied: On your Server 2008 R2 boxes, you can see the old adapter by going to the Device Manager, go to View -> Show Hidden Devices. Expand the Network adapters, and find the one greyed out. This’ll allow you to remove the old one and reconfigure the new one. This is a good use case for your SureBackups. Configure and test those VMs on a periodic basis, and boot them up to recover the networking information if it becomes lost. Just keep in mind the ESXi 6U2 express patch in the first article that will make that a headache.

Veeam vCenter Migration Utility

This one flew very low under the radar, but is an extremely cool utility you can use and something I know I’ve wished for in the past as a Veeam and VMware admin. Veeam has provided a migration tool in KB 2136 that is designed to allow you to continue where your Veeam backup jobs left off even when a VM is migrated to another vCenter. This is pretty slick, and it avoids you having to factor in extended backup windows just after a migration. Read the Challenge section in the KB for a brief explanation of how Veeam tracks VMs in its database. It essentially updates the table where the MoRef is stored so Veeam will not attempt to take a full of that VM. This also means that it won’t give you duplicates in your On-Disk backup chains when you look at them in the console. You’ll need to register an account to get the tool (if you haven’t already), but make sure before you run it that you BACKUP YOUR VEEAM CONFIG. As a reminder how you do that, go to the menu in the upper-left-hand corner of your Veeam console and choose “Configuration Backup”.

Choose the “Backup now” button to have it send that to the repository you’ve configured. And remember, if you choose not to encrypt that config backup, your account credentials used in your AAIP jobs will not be retained if you have to restore! So encrypt the backup to avoid that happening.