vArchitect Newsletter 002

vCenter 6 vSphere-client and dataservice constant warning alarm

Here’s an extremely annoying issue that has been around since 6.0 went GA almost a year-and-a-half ago as of this writing and has irked us on just about every VMware-related engagement we’ve had. It involves constantly seeing the vsphere-client and vmware-dataservice-sca services go up and down in the vSphere web client. While there has been a KB on this issue for some time now, it was only recently updated with a workaround.

If you’re seeing this in your vCenter 6 environment—and chances are you are—then the workaround in this article might be good to try. In our tests in the lab, setting the heap size to 1024 cures the issue entirely. The KB doesn’t specify that value, however the default in a tiny (8 GB) configuration is 768 MB allocated, and without increasing the memory on the appliance, 1024 should be safe. Note that after setting this value and running cloudvm-ram-size –l vsphere-client the amount returned will be 1280. Restart the web client service to have the new heap size take effect.

How to migrate a VSAN cluster from vSS to vDS

I am sure there are some of you that are currently using VSAN in some shape or form, either in a PoC, development, or production. It provides a cost-effective solution that is great for remote offices or even management clusters and can be implemented and managed very easily, but as the saying goes, nothing ever comes easy and you have to work for it. The same goes here and there are a lot of prerequisites for a VSAN environment that are crucial for implementing a healthy system that performs to its full potential. I will not go into much detail here and feel free to contact us if any services are required.

One of the recommendations for VSAN is to use a vDS, and your VSAN license actually includes the ability to use vDS which allows you to take advantage of simplified network management regardless of the underlying vSphere edition.

If you upgrade from vSS to vDS, the steps are a bit different from your normal migration. I recommend you put the host into maintenance mode with the “ensure accessibility” option. Verify the uplink used for the VSAN vmkernel port and use the “manage physical network adapters” to remove the vmnic from vSS and to add it to the vDS. Now, migrate the vmkernel to the VDS. If you review the VSAN health, the network will show with result failed.

To verify multicast network traffic is flowing from your host, use the following command on the ESXi host using bash shell:

tcpdump-uw -i vmk2 -n -s0 -t -c 20 udp port 23451 or udp port 12345

To review your multicast network settings:

esxcli vsan network list

Ruby vSphere Console (RVC) is also a great tool to have in your arsenal for managing VSAN and the following command can be used to review the VSAN state:


To re-establish the network connection you can use the following command:


Re-run the VSAN health test and verify the network shows passed. Now that the VSAN network is up and running, you can migrate the rest of the vmkernel ports over to the vDS.

Critical patch released for ESXi 6 fixes CBT issue

A new patch was released for ESXi 6 on August 5th which fixes several outstanding issues, one of them being the CBT bug that we mentioned in the previous newsletter. The KB detailing all the changes is found here, but these are some of the more notable changes.

1.)Path Selection Policy as Round Robin (PSP_RR) with iops=1 set as default for XtremIO, model XtremApp Array.
2.) Creation of application-consistent quiesced snapshot on VMs with Windows 2008 or higher GOS might result in incorrect (all modified) sectors to be returned by the CBT. The issue causes the time for backup and data to be backed up during every incremental backup cycle to be equivalent to a full backup. Thereby incremental backups are effectively same as full backup. No data is lost or corrupted.
3.) This patch updates the tools-light VIB to include the VMware Tools version 10.0.9.
4.) This patch updates the esx-ui VIB to introduce a new version of VMware Host Client.
5.) ESXi 6.0 hosts that use mpt2sas or mptsas drivers might fail with a purple diagnostic screen
6.) Hybrid VSAN performance (KB 2146267).

It should be available from VMware’s online depot, so deploy with VUM and be sure to update those VMs with VMware Tools included in this version.

Intel NUCs for your home VMware lab

Having a home lab that runs vSphere at a minimum or perhaps more of the VMware software solutions that include SDDC components is becoming the norm for most folks who work with virtualization. Even if you have a playground at your place of business, having a sandbox at home is still hugely beneficial.

One of the more popular choices for hosts in a home lab are the Intel NUC computers. If you’re not familiar, these are small, cube-like PCs that come with Core i3, i5, and i7 processor variants and can accept up to 32 GB of memory at the current moment. They’re all virtualization ready, consume very little power, and produce small amounts of heat. Naturally, they are a good fit for a home lab. is a blog site you might want to bookmark if running a homelab or potentially doing it on the NUC platform interests you.

In fact, there was a recent post which gives a good rundown of the models and how to choose one for your needs. Some are even running Virtual SAN on these little guys. The only downside at the moment is there is but a single 1 GbE NIC on them, and that’s tough to squeeze three to five vmkernel ports’ worth of traffic down at a single time.

William Lam has worked on providing a working VIB driver for USB NICs, but it isn’t officially supported and currently has egress issues. Still, a NUC with one of those USB NICs may work for you, so do give it some thought if you’re looking for home lab hardware.

Veeam Backup & Replication 9 Update 2 available

If you’re currently running Veeam B&R v9, you’ll want to grab the latest Update 2 which was published on August 5th. It contains a number of fixes and improvements, some of which have been back-ported from the pending v9.5. Here are some of the notable enhancements/improvements:

  • Support for Data Domain OS 5.7
  • VSAN 6.2 support
  • NetApp ONTAP 8.3.2 support
  • Support for Planned Failover with cloud replicas (for Cloud Connect users)

Veeam Best Practices doc updated for v9

Veeam has updated their most excellent Best Practices guide for version 9 and has lots of new content over previous version 8. This is a great resource to check and make sure what you’re doing conforms to best practices. Most of the issues we see with Veeam are issues not with or caused by Veeam itself, but because of infrastructure components Veeam leverages. Doing things in Veeam and elsewhere according to best practices can help avoid some of the more common mistakes.

Automatic import of vCenter 6 root certificate

If you’re using vCenter Server 6 and have self-signed certificates, there is a cool script which will automate the process of downloading the root cert and installing that in your trusted certificate store on your local machine to fix those pesky security warning messages seen in your browser.

Online expansion of drives in vCSA 6

We at Sovereign always adopt an appliance-first approach when it comes to vCenter. VMware have made it clear the Windows version of vCenter is not long for this Earth, and their strategy is to use Linux-based appliances for everything. One of the complaints in the past has been vCenter Server Appliance (vCSA) management of disks and partitions. Some may not know that in vCenter 6, the appliance is based on Linux LVM. This coupled with the fact that each main service/function has been broken out into its own virtual disk means they can be expanded online without any downtime to either the vCenter service itself, or the appliance as a whole! One more reason why I love the appliance. Check out this article if you need to extend, say, your log drive, or your netdump partition if you have a great number of hosts. The process is fairly simple: Find the drive in question, extend via the vSphere client, and run a vpxd_servicecfg storage lvm autogrow command to have it detect and extend the file system across the new sectors. Take a look at KB 2126276 if you need help in mapping virtual disks to appliance function. Once caveat here we’ve found is that this does not work on the first and fourth disk because these are partitions and not LVM. If, for some strange reason, you need to extend that, you’ll have to use another method like GParted.

Log Insight 3.6 released

For those using Log Insight–and we think that should be everyone because it’s included with any vCenter license–version 3.6 was released fairly recently. Steve Flanders has a pretty good summary of what’s new, fixed, and changed on his page here. Of note is even better integration with vROps, more dashboard widgets, and syslog enhancements. There are a couple of tech previews also available in this release, and a really nice one is the auto-update for Log Insight agents. This is just a checkbox on the Agents page in Administration, so very easy to configure. Check the official blog post for the release announcement.

NSX 6.2.3 pulled due to severe problems

VMware identified an issue with VMware NSX for vSphere 6.2.3, which affects both new installations as well as upgrades from previous versions. As a result, VMware removed this version from distribution but for some customers it was a little too late. If you have already upgraded, then please review the following:

VMworld 2016

‘Tis that time of year again. Time for another VMworld, but this time in Las Vegas. All three of us (Will, Johann, and Chip) will be in attendance, so feel free to give us a shout if you want to connect and chat about anything. We’ll be talking with several vendors and some of you all, so chances are you will see us on the floor or in a session.

There will be lots to see and do this year, but the main topics we think are going to be important are:

  • VMware and containers
  • Cloud automation
  • Features in upcoming vSphere version
  • NSX

If you haven’t already signed up for sessions in these areas, you might want to do that as soon as possible although many are full already. Don’t forget that many of these sessions will be recorded and available on the VMworld website soon after conclusion of the show.

By the way, for any of you attending VMworld, check out the sites here and here for a list of extra activities including vendor parties. If you have plans on going to the main VMworld party on Wednesday night at the Las Vegas Motor Speedway, you must login to your VMworld account and agree to a liability waiver. Go to this link and login to your account. Under “User Profile” click the Edit button and scroll down until you see this section.


Check the “I agree” box and sign your name in the field beneath, then scroll down and click “Continue.” Your agreement will get encoded into your badge when you check-in at registration.

That’s all for now, and thanks for reading this far. If we don’t see you at VMworld, we’ll be back with another newsletter in September!