Update ESXi fails with dependency error vmkapi_2_0_0_0

In the past few weeks, I’ve been in the process of updating several standalone ESXi hosts to 6.5. As you would expect, everything went smooth up to a certain point. Several hosts started failing and throwing this error:

The entire error is shown below.

At first, I thought it was something with the image I was using. That’s why I also tried with the vanilla VMware image. Sadly, the same error popped up. Trying different zip files also did not solve the issue. I began looking into how I could make vmkapi 2.0 available on the host. Turns out that there’s not a whole lot written about this.

I started looking at the command again when I remembered there’s also another way I could install the update. When using esxcli to update, you can also use esxcli software profile install command. The update command updates existing VIBs with the ones that are in the specified profile, all the other VIBs aren’t touched. While the esxcli software profile install command installs all the VIBs present in the image profile, it will also remove any other VIBs installed on the server.

When I ran the install command everything proceeded without any issues. Here’s the command I used;

You can also add the –dry-run switch to get a list of VIBs that will be removed, before you actually change anything.


VMworld recap

The saying goes “time flies when you’re having fun” and boy was that true at VMworld! I had heard a lot of stories about VMworld and have been wanting to go for ages. Sadly, I never got the opportunity to actually go until this year! As always, I entered to win a blogger pass and this year it actually worked out 🙂 Thanks again to Corey Romero and the team for giving me this opportunity.

Keynotes

Day 1

Because of the blogger pass, myself and the other bloggers got preferred seating during the keynote. This meant we had a great view of the stage and the speakers, this really helped increase the experience. The day 1 keynote was all about VMware’s vision for the future of IT and I do have to say, some of the things they touched on really hit home for me. It’s clear that the days of just deploying VMs and vSphere are behind us and the landscape is changing. Of course, this won’t happen overnight but the repeated messages really make it clear that it is time to start getting to know all these new technologies before it’s too late.

This has also been made clear with the recent acquisitions VMware made, most of which were already announced in San Francisco but some have been closed by now and you’re starting to see the impact of them already. The new possibilities that are there after the Carbon Black acquisition are really amazing and help the “Intrinsic security on all layers” message become a reality.

There was a lot more to gather from the day 1 keynote but others, like fellow blogger Fabian Lenzker (link), have already done an excellent job writing this down. You can always watch it yourself on YouTube.

Day 2

The day 2 keynote was a lot of fun to watch, it was a continuation of the messaging of day 1 but now the speakers were showing how this would actually come together in a couple of demos. The story that Ray O’ Farrell was bringing was clearly depicted in all the demos that were given, each highlighting a different part of the stack and harping on a different part of the overall vision.

After the tech part was done, Bear Grylls came on stage and gave a talk that really hit home for me. He told the story about the 4 F’s, Failure, Fear, Fire and Faith. These 4 pillars have helped him get through all of the rough times he’s had during his lifetime and how he finds the strength to keep on going, even though there may be constant setbacks. I have to say that I felt really inspired after hearing him talk…

Sessions

As you would expect, a good part of my time was spent in sessions. At times it was really hard to pick a session because there were so many great choices, thankfully all the sessions were recorded!

Monday started with a VCDX workshop, led by Joe Silvagi. I’m not planning on going for the VCDX in the short-term but I do want to start gathering more and more information so this was a great opportunity. For everyone considering going for the VCDX, I highly recommend attending one of these workshops. It was really an eye-opener for me! The other VCDXs that joined the session also gave a lot of great insights.

There’s a lot of sessions that I have already rewatched the recording of and some sessions will need a couple more viewings to really get all the nuggets that are in there! William Lam’s already made a summary page that has all the session recordings, check it out here!

VM Village

The first thing you see when you come in, after the registration, is the VM Village. I was a bit amazed at how big it was, I expected it to be big but seeing it in real life is still something else entirely! One of the larger stands there is the community booth, this is where all the vBrownBag and VMware Code presentations were held. But also where the blogger table was. Hanging out there means you get to meet a lot of other bloggers and people that you’ve heard of in the community. Whenever I had some spare time, I would go there to hang out. Something I can only recommend!

One of my personal highlights took place after VMworld on Tuesday. I went to the vExpert party together with a couple of other Belgian vExperts. When we got there we noticed there was a small crowd gathering in one part of the room. That’s when Corey told us Pat Gelsinger was there to say hi! This was a great opportunity to shake his hand and have a quick chat. We also got a nice picture out of it!

I had a great time at VMworld and can’t wait to go back !


How to reset iLO password from ESXi

Everyone’s been here before, you know the iLO password should be set to something, only to find out that on just this one server the password is different.

There are several ways to reset the iLO password; you could reboot the server and reset it that way. But who likes planning downtime on a server for a password reset? Luckily you find that the server is an ESXi host that was installed using the HPE customized image. This is good news as you can now use the HPONCFG tool to reset the password!

Checking the configuration

Before you start resetting the administrator password, you can always check the current configuration. This will show you the entire configuration done on the iLO, including any additional users that were created.

To start using the HPONCFG tool, first enable SSH on the ESXi host in question and log on. Once logged on, go to /opt/tools . This is where you will find the HPONCFG tool.

To export the current config, run this command to write everything to a file

You can then look at the file using your editor of choice. In the file you can see the IP address assigned, DNS name of the iLO and, most importantly, the users created. You can see here that there’s 3 users

Resetting the password

Now that we know which users exist on the iLO, we can start doing the actual password reset on the user we want. To do this, we will create an XML file on the ESXi host that contains the new password for the user.

Create the new file on the host, using your editor of choice. I like to use vi, so that’s what you will see below.

Now you can press the i key to insert, copy the code below and right click in your ssh window to paste. You can replace the [email protected]! with the password you want to set on the user

Afterwards, hit the escape button and type :wq to save the file. Now we can start the actual reset. Run the command below to perform the reset

If the reset is successful, you should see the output like this

Cleaning up

Now that the password has been reset, login just to make sure everything is working as expected. As a security precaution, make sure you delete the pwreset.xml file, once you have verified you can log in! We don’t want to store passwords in clean text anywhere 😉

The above command will delete the file, if you’re still in the /opt/tools folder.


VMUGBE

Next week, the 14th of June, it’s time for one of my favorite times of the year, the yearly VMUGBE is upon us! It’s an amazing day filled with great sessions, good conversations, all set in a lovely venue.

The Belgian VMUG was the first VMUG I ever attended, 4 years ago already. It’s where I met a lot of people I know in the community and, on a personal level, where I learned to just go up to random people and start talking to them. Something I’m still not very good at but there’s plenty of VMUGs to come!

Why you should attend

VMUGBE+ is a free event that gets top-notch speakers every year with topics ranging from more traditional topics to newer topics like CNA, VMware cloud on AWS and so on.

Coming to the VMUG is also a great opportunity to meet new people that are a part of the same #vCommunity we are all part of. Undoubtedly, everyone has found a solution to their problem on the blog of someone from the community. This is your chance to talk to them and perhaps even thank them!

Since 2013 VMUGBE has become VMUGBE+, which means that the organization will donate an amount of the earnings to a charity in Belgium. This ties in nicely to the whole idea of giving back to the community. You can read more about it here.

Sessions I’m looking forward to

As I said before, VMUGBE has a lot of great sessions but there are a few that I’m really looking forward to attending!

Opening keynote – Joe Baguley – 4 years ago the keynote was also given by Joe Baguley, I still remember it very well. He is the CTO for EMEA and will be talking about how architectures are changing and some new areas of R&D within VMware

VDI by day, compute by night – Johan Van Amersfoort – I have missed this session at the NLVMUG and Empower in Lisbon, there’s no way I’m missing it now!

Kubernetes for the vSphere admin – Eric De Witte – This is a topic that I don’t know a lot about. Looking forward to getting a better understanding of Kubernetes in this session. I also heard it involves rockets!

vExpert Community Session – Stijn Depril – One of the sessions I’m looking forward to the most! Stijn is one of Belgium’s vExpert Pro and a big advocate for the community. The goal of this session is to educate people on what a vExpert is and to get people excited about becoming a vExpert!

Presenting

This VMUG will be special for me, as I will be presenting for the first time! VVOLs is something that’s been talked about since 2011 but to many people it’s still shrouded in mystery. The goal of my session is to dispel some of this mystery and get people excited about what VVOLs can do for your environment.

As you can see, there’s a lot to look forward to. So what are you waiting for? It’s not too late to register! https://www.vmug.be/#register

Image result for arnold do it now

I look forward to seeing you all there!


VCAP6.5-DCV Design Exam Experience

This year, I got the opportunity to go to VMware Empower 2019 in Lisbon. The ticket includes a free exam voucher so I used it to take my VCAP6.5-DCV Design exam. I’m glad to say that I passed! The exam was a lot harder than I expected though.

Having just done the VCP exams, I did not expect the VCAP to be that much harder. In this post, I will try to share some tips that I found useful during preparation and how I experienced the exam

VCAP

Let’s start by clarifying what a VCAP actually is. VCAP stands for VMware Certified Advanced Professional. VMware qualifies the “minimally qualified candidate” as follows in the exam blueprint

A minimally qualified candidate (MQC) achieving the VMware Certified Advanced Professional 6.5 in Data Center Virtualization Design is capable of developing a conceptual design given a set of customer requirements, determining the functional requirements needed to create a logical design, and architecting a physical design using these elements.

This quote, together with the exam title, give a clear picture of where the focus for this exam is. You are also expected to know all the topics that were handled in the VCP exam and in much greater detail. After all, this is an advanced certification exam.

Preparation

During my preparation, I read a lot of whitepapers and articles. Below you can find a list of the ones I used. Don’t limit yourself to just this list, there’s a ton of content out there created by fellow bloggers and VMware itself. Also, make sure to read other posts like this. Most people also link to content that helped them during their preparation.

Official resources

Community resources

  • Graham Barker’s VCAP prep guide is a must read for everyone in my opinion. It’s written for the VCAP6 exam but the design parts still apply to 6.5
  • David Stamen’s tips helped me a lot during my exam, definitely recommend reading them!
  • the vMusketeers created a VCAP6-DCV Design quiz that you should try for sure. I found that test to be harder than the exam itself. It really tests your knowledge on the core design concepts.
  • Daniel Paluszek created an extensive blog post with a lot of material in it. I used it a lot.
  • Jeffrey Kusters’ blog on designs, requirements, etc. really helped me understand these concepts

Exam experience

For me, this was the hardest exam that I’ve done so far. Questions were multiple choice style and also drag and drop style. Obviously, I can’t go talk about the questions but the things I can tell are that you should really be familiar with these concepts

  • RCAR (Requirements, Constraints, Assumptions, and Risks)
  • AMPRS (Availability, Manageability, Performance, Recoverability, Security)
  • RTO (Recovery Time Objective)
  • RPO (Recovery Point Objective)
  • All vSphere features. design and architecture of all components should be known
  • all vSAN features, how to design it and how the architecture works
  • Replication, SRM, …
  • Conceptual, logical and physical design

Now that I passed the exam, my focus will shift to preparing my upcoming VMUG session and afterwards the VCAP6.5 deploy exam. Exciting times ahead!

I hope that this post gives you a good idea of just how much preparation is required to pass the exam. If you have questions or additions, hit me up!


Rescanning ESXi storage in a parallel way

Lately, I’ve been doing a lot of work provisioning and decommissioning LUNs on some Fibre Channel arrays. As you all know, this process can be somewhat tedious when provisioning a new LUN on vSphere or removing said LUN.

In the past, I would always use this PowerCLI command

This would first get all the ESXi hosts in the cluster “Cluster1” and then launch the refresh cmdlet on each host. The problem with this is that it can take a really long time for the command to complete because the refresh is executed host by host. Before starting the next refresh, the command waits for the previous one to finish. While this is fine for smaller environments, the time really does add up when you have a decent sized cluster.

Using PowerShell jobs

In order to reduce the time this process takes, I decided this would be a great use case for PowerShell jobs. A job is essentially another PowerShell instance that runs in the background. This means that your main script or PowerShell window doesn’t wait for it to finish before going to the next step. Using jobs will allow us to start the rescan on each host as a job and then start the next job. Meaning the rescans will happen in parallel.

Code

The script is provided as is and can be found on GitHub


ERROR: The host returns esxupdate error code:15

Today I was preparing some updates with VUM on Lenovo servers running vSphere 6.0. Things did not go as expected. While staging the patches I was greeted with this error.

ERROR: The host returns esxupdate error code:15. The package manager transactions is not successful. Check the update Manager log files and esxupdate.log files for more details.

Looking into the esxupdate.log file I could find the following entries:

This seems to be a known issue for Lenovo servers. It turns out they cram a whole lot of drivers in there, more than other vendors. The only known workaround is to remove VIBs from the host that are not needed. This is also described in https://kb.vmware.com/s/article/2144200.

To find out what VIBs are installed, log onto the affected host using SSH and run this command:

Once you have identified a driver that you want to remove, put the host in maintenance mode and this command:

In my case I wasn’t using the qlogic nx2 driver so I removed it. Now you should be able to update the host using VUM,


Event ID 1000 rundll32.exe_aepdu.dll

While doing my morning check of our monitoring system I came across a strange issue…Two of our servers had been unavailable during the night, this had also happened the night before. Time to dig a bit deeper!

Both servers are domain controllers running Server 2012 R2, no updates were installed in the last few weeks and nothing had been changed either. Pulling up the event log of both servers, I found the exact same event logged on both machines at the time they were unavailable to our monitoring system.

After I quick search, I found out that aeinv.dll is part of the Microsoft Customer Experience Improvement Program (CEIP). If you’re signed up to the CEIP, which you are by default when you install Server 2012 R2, three scheduled tasks will run each night to upload your anonymized date to Microsoft.

  • AitAgent
  • Microsoft compatibility Appraiser
  • ProgramDataUpdater

We had the unavailability after the first and third task ran.

Luckily you can easily opt out of the CEIP by opening Server Manager, going to Local Server and clicking on the link behind Customer Experience Improvement Program.

There you will see that the radio button, Yes, I want to participate in the CEIP is selected. Click on No, I don’t want to participate and click OK.

You’re now opted out and the scheduled tasks will no longer run.

UPDATE 11/07: Turns out that just opting out of the CEIP does not actually prevent the scheduled tasks from running. In order to fully disable them, you need to disable them through the task scheduler or, even better, disable them through PowerShell


Quick post: Bug in Veeam Quick Migration

Last week I was moving some VMs for a client using Veeam Quick Migration. During the migration, I happened to stumble upon some strange behavior.

The source VMs make up an Oracle RAC cluster, using shared VMDKs. One of the requirements for setting up shared VMDKs, using the multi-writer option, is that they are thick eager zeroed disks. When the VMs got to the other side, they wouldn’t boot. I figured something went wrong during the migration so I tried it again. Once the second run completed, the VMs still wouldn’t boot so I started digging around some more.

The error message I was getting was rather vague; “Incompatible device backing specified for device ‘0’.”. After verifying the config of both nodes I eventually decided to look at the disk type on the destination side. That’s when I noticed the disk type was thick provisioned lazy zeroed. Ahah, that’s why they didn’t want to boot! After manually inflating the disks, they were up and running again. I’m starting to suspect that this is a bug.

Running some tests

I started building some more test VMs just to prove that this was, in fact, a bug. One of the options you can set during the Quick Migration wizard, is the disk type. You can explicitly select each of the types, or you can have Veeam use the same format as on the source side. Explicitly selecting thick provisioned eager zeroed or using the same as source also produced a VM with lazy zeroed disks. Time to submit a ticket!

As usual, Veeam support was very helpful and investigated the issue. A couple days later they came back to me and confirmed this was indeed a bug that will be fixed in an upcoming version.

Workaround

This bug is a minor inconvenience since there is an easy workaround. You can login to an ESXi server using SSH and convert the VMDK using the command

More info on how to convert a VMDK can be found in this VMware KB.


Publicly share Office 365 room calendar

A customer asked me if it was possible to have a room mailbox automatically accept meeting requests from external parties. They would also like to publish the calendar of that specific room publicly.

Accept meetings from external parties

Let’s start with the first question. By default, resource mailboxes only accept requests from internal senders. As you might guess, you can’t change this behavior through the GUI, Powershell to the rescue!

Since I didn’t know the cmdlet that would let me change this behavior, the first thing I did was look for all “Calendar cmdlets”. After connecting to the Office 365 PowerShell, I ran this command

Seems like there are a few cmdlets concerning calendars, good info for the second question! The Get-CalendarProcessing cmdlet looks promising, let’s try it out!

As you can see on the highlighted line, this is exactly the property we were looking for. Let’s change it so we get the desired behavior. In the get-command output, I saw a cmdlet Set-CalendarProcessing, this seems like the right one.

This change will only affect new meeting requests, requests that have already been refused won’t be automatically accepted.

Publish calendar publicly

In the cmdlets we got earlier, there wasn’t really one that stood out as a “possible match” so let’s look at the attributes of the calendar itself. In essence, the calendar is just a folder inside of a mailbox object. Let’s query that folder directly.

That’s everything we need and more! As you can see, we can set the PublishEnabled attribute to true but we can do so much more. You can choose the detail level and even set how far back and forth the published calendar needs to go.

Let’s publish the calendar and run the Get-MailboxCalendarFolder cmdlet again to get the URL.

All done! Now you can browse to the URL and verify everything is being displayed as you’d expect.