Rescanning ESXi storage in a parallel way

Lately, I’ve been doing a lot of work provisioning and decommissioning LUNs on some Fibre Channel arrays. As you all know, this process can be somewhat tedious when provisioning a new LUN on vSphere or removing said LUN.

In the past, I would always use this PowerCLI command

Get-VMhost -Cluster "Cluster1" | Get-VMHostStorage -Refresh

This would first get all the ESXi hosts in the cluster “Cluster1” and then launch the refresh cmdlet on each host. The problem with this is that it can take a really long time for the command to complete because the refresh is executed host by host. Before starting the next refresh, the command waits for the previous one to finish. While this is fine for smaller environments, the time really does add up when you have a decent sized cluster.

Using PowerShell jobs

In order to reduce the time this process takes, I decided this would be a great use case for PowerShell jobs. A job is essentially another PowerShell instance that runs in the background. This means that your main script or PowerShell window doesn’t wait for it to finish before going to the next step. Using jobs will allow us to start the rescan on each host as a job and then start the next job. Meaning the rescans will happen in parallel.

Code

The script is provided as is and can be found on GitHub


ERROR: The host returns esxupdate error code:15

Today I was preparing some updates with VUM on Lenovo servers running vSphere 6.0. Things did not go as expected. While staging the patches I was greeted with this error.

ERROR: The host returns esxupdate error code:15. The package manager transactions is not successful. Check the update Manager log files and esxupdate.log files for more details.

Looking into the esxupdate.log file I could find the following entries:

2018-10-08T08:23:18Z esxupdate: 31744883: BootBankInstaller.pyc: ERROR: The pending transaction requires 242 MB free space, however the maximum supported size is 239 MB. 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR: An esxupdate error exception was caught: 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR: Traceback (most recent call last): 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 238, in main 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:     cmd.Run() 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esx5update/Cmdline.py", line 148, in Run 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 250, in InstallVibsFromSources 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 356, in _installVibs 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Transaction.py", line 399, in _validateAndInstallProfile 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/HostImage.py", line 600, in Stage 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Installer/BootBankInstaller.py", line 719, in PreInstCheck 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR:   File "/build/mts/release/bora-5572656/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/esximage/Installer/BootBankInstaller.py", line 694, in CheckInstallationSize 
2018-10-08T08:23:18Z esxupdate: 31744883: esxupdate: ERROR: InstallationError: ('', 'The pending transaction requires 242 MB free space, however the maximum supported size is 239 MB.')

This seems to be a known issue for Lenovo servers. It turns out they cram a whole lot of drivers in there, more than other vendors. The only known workaround is to remove VIBs from the host that are not needed. This is also described inĀ https://kb.vmware.com/s/article/2144200.

To find out what VIBs are installed, log onto the affected host using SSH and run this command:

esxcli software vib list

Once you have identified a driver that you want to remove, put the host in maintenance mode and this command:

esxcli software vib remove -n qlogic-nx2-provider

In my case I wasn’t using the qlogic nx2 driver so I removed it. Now you should be able to update the host using VUM,


Install VIB on VMware ESXi

Today I got my shiny new host for the homelab. After installing ESXi, I noticed the 10 Gbit NICs weren’t being detected. After looking around a bit I found the drivers in the form of a VIB.

Start by uploading the VIB to a datastore that is accessible to the host. Turn on the SSH service in the security section and fire up a session.

Once connected run

esxcli software vib install -v /vmfs/volumes/NAS1-iSCSI/net-ixgbe_4.4.1-1OEM.600.0.0.2159203.vib

If everything goes well you should get an output like this

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: INT_bootbank_net-ixgbe_4.4.1-1OEM.600.0.0.2159203
   VIBs Removed: VMware_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.600.0.0.2494585
   VIBs Skipped:

Once all is done, reboot the host and you’ll see the installed device pop up, in my case the 10 Gbit NICs