vCenter update fails: vCenter server is non-operational

Today, while I was updating vCenter in my lab, I ran into a strange issue. The update I was installing failed, when I wanted to try again, this ominous error message popped up.

vCenter being non-operational left me a bit of a doom and gloom feeling but, thankfully, the fix is rather easy.

Fixing the issue

Open up an SSH session to your vCenter server and run the command below to remove the software_update_state.conf file.

You can check the contents of the file by opening it with vi, an example is shown below.

 

Turns out that the state “INSTALL_FAILED” is checked by the python script that performs the installation. The script can be found in /usr/lib/applmgmt/update/py/vmware/appliance/update/update_state.py

Removing the file will make this check pass and the installation will continue.


VCAP-DCV Deploy Exam Experience

After postponing this exam for too long, I finally took the time to study for and take the VCAP-DCV deploy exam. To be honest, I was kind of looking forward to this exam. I love playing in the lab and getting my hands dirty in an environment. So to be able to take an actual lab exam was pretty exciting to me.

In this post, I will try to give some pointers and tips that might help you pass the exam. Please note that I took the exam for the 6.5 version but the same logic applies to other versions.

Preparation

Unlike the design exam, the deploy exam is a lab exam. This means you need to know how to do the stuff you’ve been reading and talking about. But, there are still some similarities. For starters, know the blueprint. This comes back for every exam, but read the blueprint upfront and know what topics are going to be covered on the exam. You should be intimately familiar with these blueprints by now.

Documentation

Although it’s a lab exam, you do have access to all the documentation VMware has on vSphere 6.5 in a folder on your desktop. Unlike what I read in some other posts, you do NOT have access to Adobe Reader in the lab. All PDFs will be opened in the browser, which means you can’t do a search in the entire folder anymore. Keep this in mind, read the documentation, and know where you need to look to find certain commands or info on a topic.

Community resources

As you might have guessed, there are already some great resources available in the community and I’m probably going to be linking the same ones that everyone does but that just shows how good they are.

  • Kyle Jenner’s VCAP6-DCV deployment study guide is the resource that I used the most. Kyle has put a lot of time and effort into explaining every topic covered in the blueprint. Make sure to go through every topic thoroughly.
  • Graham Barker’s VCAP6-DCV exam simulator is great to get a feel for what kind of questions will be asked and what depth they go to. Although the HoLs listed are no longer available, you can still perform these tasks in another HoL or your own lab.

Lab

But the most important resource I used while preparing was my lab. The VCAP exam covers nearly every vSphere feature there is. Like me, you’re probably not familiar with every feature there is. Make sure you lab these things more than once so you know how to do them. If you’ve got a lab that’s set up perfectly, try to get one of your colleagues or friends to break things in the lab. This will come in handy when you’re doing the exam. If you don’t have a lab, just spin up one of the vSphere 6.7 labs and start playing around.

You could also rent a server for a month or 2 like I did, and start building a nested lab. But that’s a topic for another post.

Besides that, there’s no replacement for real-world experience. I would not recommend taking this exam until you’ve got about 1 – 2 years of daily hands-on work done with vSphere. This will make the exam a lot easier.

Exam experience

The exam itself is presented to you in a HoL like fashion, it’s the same UI. If you’re not familiar with how HoLs work, be sure to start up a few so you’re familiar with the interface and how you can change and use the interface. Also, try booking the exam in a center where they have big screens. My test center has 24″ screens, which helped A LOT.

I found the lab to be reasonably performant and had no issues with connection whatsoever. The only minor annoyance was letters appearing more than once while typing but this is probably due to latency. Just be sure to read what you typed if a command fails.

Time management is crucial for this exam, that’s what a lot of other people told me at least. With that in mind, I tried to get the questions done as fast as I could without rushing through them. If I was stumped on one part of the question, I would write it down on the piece of paper I got, and move on to the next question. After I got through all questions, I started going back to the ones I didn’t complete. This will also prevent you from getting frustrated/stuck on 1 question, taking a break will give you a fresh look at the question.

I finished the exam with 42 minutes to spare so I never really felt that I was in a hurry to get everything done but your mileage may vary.

When doing the exam, read the questions carefully. It happened several times that I quickly read a question, started doing things, and afterward re-reading the question to find out I had not done several things.

Make sure you’re familiar with the CLI and PowerCLI, these things can come in handy for doing certain things faster. Also, try to open up the flash client again before taking the 6.5 exam. During the 6.5 days, the H5 client wasn’t yet fully-featured so you may not be able to use it for all questions.

Results

I took the exam on a Friday at 10 AM, so I was expecting to get the results on Monday or Tuesday after that. Around 8 PM I got an e-mail saying that I passed! This was a very pleasant surprise and big kudos to the VMware education team for providing the results so quickly.

I hope this post will help you prepare for the exam, good luck!


VCSA upgrade – deployment sizes missing

Everyone has done VCSA upgrades dozens of times, but every now and then you come across something that you haven’t seen before. Today, this was the case while I was doing an upgrade.

When you get to the deployment size selection, in the first stage of the upgrade, I noticed I was unable to select anything smaller than “Medium” size.

Changing the storage size also didn’t make the Tiny or small sizes available. While I was doing this, I happened to be talking to Jens Herremans (Check out his blog, it’s awesome!). He told me to check the size of the logs partition on the source VM.

Checking log sizes

After opening an SSH session to the source VM and changing to the /var/log/vmware folder, I ran the following command to get a list of all files and their sizes.

I write the output to a file to make it easier to review, also this avoids having to scroll up and having parts capped off. Next, review the file with your favorite editor. You’ll see a list of all the folders, their total size and the individual files with their size. For me the big directories are vpxd, sca, sso, vdcs and vsphere-client.

Every file will also have a timestamp that makes it easy to verify if you still need the logs or not. If you wish to keep the, you can easily move them to a datastore or copy them locally. In this particular environment, we had no need for any of the older log files. I could just go in and rm -rf all the archived logs. For example in the /var/log/vmware/sca folder, the following command will remove all the archived logs but keep the one currently in use

After the cleanup, I ran the upgrade assistant again and was able to select the tiny deployment size.


Update ESXi fails with dependency error vmkapi_2_0_0_0

In the past few weeks, I’ve been in the process of updating several standalone ESXi hosts to 6.5. As you would expect, everything went smooth up to a certain point. Several hosts started failing and throwing this error:

The entire error is shown below.

At first, I thought it was something with the image I was using. That’s why I also tried with the vanilla VMware image. Sadly, the same error popped up. Trying different zip files also did not solve the issue. I began looking into how I could make vmkapi 2.0 available on the host. Turns out that there’s not a whole lot written about this.

I started looking at the command again when I remembered there’s also another way I could install the update. When using esxcli to update, you can also use esxcli software profile install command. The update command updates existing VIBs with the ones that are in the specified profile, all the other VIBs aren’t touched. While the esxcli software profile install command installs all the VIBs present in the image profile, it will also remove any other VIBs installed on the server.

When I ran the install command everything proceeded without any issues. Here’s the command I used;

You can also add the –dry-run switch to get a list of VIBs that will be removed, before you actually change anything.


VMworld recap

The saying goes “time flies when you’re having fun” and boy was that true at VMworld! I had heard a lot of stories about VMworld and have been wanting to go for ages. Sadly, I never got the opportunity to actually go until this year! As always, I entered to win a blogger pass and this year it actually worked out 🙂 Thanks again to Corey Romero and the team for giving me this opportunity.

Keynotes

Day 1

Because of the blogger pass, myself and the other bloggers got preferred seating during the keynote. This meant we had a great view of the stage and the speakers, this really helped increase the experience. The day 1 keynote was all about VMware’s vision for the future of IT and I do have to say, some of the things they touched on really hit home for me. It’s clear that the days of just deploying VMs and vSphere are behind us and the landscape is changing. Of course, this won’t happen overnight but the repeated messages really make it clear that it is time to start getting to know all these new technologies before it’s too late.

This has also been made clear with the recent acquisitions VMware made, most of which were already announced in San Francisco but some have been closed by now and you’re starting to see the impact of them already. The new possibilities that are there after the Carbon Black acquisition are really amazing and help the “Intrinsic security on all layers” message become a reality.

There was a lot more to gather from the day 1 keynote but others, like fellow blogger Fabian Lenzker (link), have already done an excellent job writing this down. You can always watch it yourself on YouTube.

Day 2

The day 2 keynote was a lot of fun to watch, it was a continuation of the messaging of day 1 but now the speakers were showing how this would actually come together in a couple of demos. The story that Ray O’ Farrell was bringing was clearly depicted in all the demos that were given, each highlighting a different part of the stack and harping on a different part of the overall vision.

After the tech part was done, Bear Grylls came on stage and gave a talk that really hit home for me. He told the story about the 4 F’s, Failure, Fear, Fire and Faith. These 4 pillars have helped him get through all of the rough times he’s had during his lifetime and how he finds the strength to keep on going, even though there may be constant setbacks. I have to say that I felt really inspired after hearing him talk…

Sessions

As you would expect, a good part of my time was spent in sessions. At times it was really hard to pick a session because there were so many great choices, thankfully all the sessions were recorded!

Monday started with a VCDX workshop, led by Joe Silvagi. I’m not planning on going for the VCDX in the short-term but I do want to start gathering more and more information so this was a great opportunity. For everyone considering going for the VCDX, I highly recommend attending one of these workshops. It was really an eye-opener for me! The other VCDXs that joined the session also gave a lot of great insights.

There’s a lot of sessions that I have already rewatched the recording of and some sessions will need a couple more viewings to really get all the nuggets that are in there! William Lam’s already made a summary page that has all the session recordings, check it out here!

VM Village

The first thing you see when you come in, after the registration, is the VM Village. I was a bit amazed at how big it was, I expected it to be big but seeing it in real life is still something else entirely! One of the larger stands there is the community booth, this is where all the vBrownBag and VMware Code presentations were held. But also where the blogger table was. Hanging out there means you get to meet a lot of other bloggers and people that you’ve heard of in the community. Whenever I had some spare time, I would go there to hang out. Something I can only recommend!

One of my personal highlights took place after VMworld on Tuesday. I went to the vExpert party together with a couple of other Belgian vExperts. When we got there we noticed there was a small crowd gathering in one part of the room. That’s when Corey told us Pat Gelsinger was there to say hi! This was a great opportunity to shake his hand and have a quick chat. We also got a nice picture out of it!

I had a great time at VMworld and can’t wait to go back !


How to reset iLO password from ESXi

Everyone’s been here before, you know the iLO password should be set to something, only to find out that on just this one server the password is different.

There are several ways to reset the iLO password; you could reboot the server and reset it that way. But who likes planning downtime on a server for a password reset? Luckily you find that the server is an ESXi host that was installed using the HPE customized image. This is good news as you can now use the HPONCFG tool to reset the password!

Checking the configuration

Before you start resetting the administrator password, you can always check the current configuration. This will show you the entire configuration done on the iLO, including any additional users that were created.

To start using the HPONCFG tool, first enable SSH on the ESXi host in question and log on. Once logged on, go to /opt/tools . This is where you will find the HPONCFG tool.

To export the current config, run this command to write everything to a file

You can then look at the file using your editor of choice. In the file you can see the IP address assigned, DNS name of the iLO and, most importantly, the users created. You can see here that there’s 3 users

Resetting the password

Now that we know which users exist on the iLO, we can start doing the actual password reset on the user we want. To do this, we will create an XML file on the ESXi host that contains the new password for the user.

Create the new file on the host, using your editor of choice. I like to use vi, so that’s what you will see below.

Now you can press the i key to insert, copy the code below and right click in your ssh window to paste. You can replace the [email protected]! with the password you want to set on the user

Afterwards, hit the escape button and type :wq to save the file. Now we can start the actual reset. Run the command below to perform the reset

If the reset is successful, you should see the output like this

Cleaning up

Now that the password has been reset, login just to make sure everything is working as expected. As a security precaution, make sure you delete the pwreset.xml file, once you have verified you can log in! We don’t want to store passwords in clean text anywhere 😉

The above command will delete the file, if you’re still in the /opt/tools folder.


VMUGBE

Next week, the 14th of June, it’s time for one of my favorite times of the year, the yearly VMUGBE is upon us! It’s an amazing day filled with great sessions, good conversations, all set in a lovely venue.

The Belgian VMUG was the first VMUG I ever attended, 4 years ago already. It’s where I met a lot of people I know in the community and, on a personal level, where I learned to just go up to random people and start talking to them. Something I’m still not very good at but there’s plenty of VMUGs to come!

Why you should attend

VMUGBE+ is a free event that gets top-notch speakers every year with topics ranging from more traditional topics to newer topics like CNA, VMware cloud on AWS and so on.

Coming to the VMUG is also a great opportunity to meet new people that are a part of the same #vCommunity we are all part of. Undoubtedly, everyone has found a solution to their problem on the blog of someone from the community. This is your chance to talk to them and perhaps even thank them!

Since 2013 VMUGBE has become VMUGBE+, which means that the organization will donate an amount of the earnings to a charity in Belgium. This ties in nicely to the whole idea of giving back to the community. You can read more about it here.

Sessions I’m looking forward to

As I said before, VMUGBE has a lot of great sessions but there are a few that I’m really looking forward to attending!

Opening keynote – Joe Baguley – 4 years ago the keynote was also given by Joe Baguley, I still remember it very well. He is the CTO for EMEA and will be talking about how architectures are changing and some new areas of R&D within VMware

VDI by day, compute by night – Johan Van Amersfoort – I have missed this session at the NLVMUG and Empower in Lisbon, there’s no way I’m missing it now!

Kubernetes for the vSphere admin – Eric De Witte – This is a topic that I don’t know a lot about. Looking forward to getting a better understanding of Kubernetes in this session. I also heard it involves rockets!

vExpert Community Session – Stijn Depril – One of the sessions I’m looking forward to the most! Stijn is one of Belgium’s vExpert Pro and a big advocate for the community. The goal of this session is to educate people on what a vExpert is and to get people excited about becoming a vExpert!

Presenting

This VMUG will be special for me, as I will be presenting for the first time! VVOLs is something that’s been talked about since 2011 but to many people it’s still shrouded in mystery. The goal of my session is to dispel some of this mystery and get people excited about what VVOLs can do for your environment.

As you can see, there’s a lot to look forward to. So what are you waiting for? It’s not too late to register! https://www.vmug.be/#register

Image result for arnold do it now

I look forward to seeing you all there!


VCAP6.5-DCV Design Exam Experience

This year, I got the opportunity to go to VMware Empower 2019 in Lisbon. The ticket includes a free exam voucher so I used it to take my VCAP6.5-DCV Design exam. I’m glad to say that I passed! The exam was a lot harder than I expected though.

Having just done the VCP exams, I did not expect the VCAP to be that much harder. In this post, I will try to share some tips that I found useful during preparation and how I experienced the exam

VCAP

Let’s start by clarifying what a VCAP actually is. VCAP stands for VMware Certified Advanced Professional. VMware qualifies the “minimally qualified candidate” as follows in the exam blueprint

A minimally qualified candidate (MQC) achieving the VMware Certified Advanced Professional 6.5 in Data Center Virtualization Design is capable of developing a conceptual design given a set of customer requirements, determining the functional requirements needed to create a logical design, and architecting a physical design using these elements.

This quote, together with the exam title, give a clear picture of where the focus for this exam is. You are also expected to know all the topics that were handled in the VCP exam and in much greater detail. After all, this is an advanced certification exam.

Preparation

During my preparation, I read a lot of whitepapers and articles. Below you can find a list of the ones I used. Don’t limit yourself to just this list, there’s a ton of content out there created by fellow bloggers and VMware itself. Also, make sure to read other posts like this. Most people also link to content that helped them during their preparation.

Official resources

Community resources

  • Graham Barker’s VCAP prep guide is a must read for everyone in my opinion. It’s written for the VCAP6 exam but the design parts still apply to 6.5
  • David Stamen’s tips helped me a lot during my exam, definitely recommend reading them!
  • the vMusketeers created a VCAP6-DCV Design quiz that you should try for sure. I found that test to be harder than the exam itself. It really tests your knowledge on the core design concepts.
  • Daniel Paluszek created an extensive blog post with a lot of material in it. I used it a lot.
  • Jeffrey Kusters’ blog on designs, requirements, etc. really helped me understand these concepts

Exam experience

For me, this was the hardest exam that I’ve done so far. Questions were multiple choice style and also drag and drop style. Obviously, I can’t go talk about the questions but the things I can tell are that you should really be familiar with these concepts

  • RCAR (Requirements, Constraints, Assumptions, and Risks)
  • AMPRS (Availability, Manageability, Performance, Recoverability, Security)
  • RTO (Recovery Time Objective)
  • RPO (Recovery Point Objective)
  • All vSphere features. design and architecture of all components should be known
  • all vSAN features, how to design it and how the architecture works
  • Replication, SRM, …
  • Conceptual, logical and physical design

Now that I passed the exam, my focus will shift to preparing my upcoming VMUG session and afterwards the VCAP6.5 deploy exam. Exciting times ahead!

I hope that this post gives you a good idea of just how much preparation is required to pass the exam. If you have questions or additions, hit me up!


Rescanning ESXi storage in a parallel way

Lately, I’ve been doing a lot of work provisioning and decommissioning LUNs on some Fibre Channel arrays. As you all know, this process can be somewhat tedious when provisioning a new LUN on vSphere or removing said LUN.

In the past, I would always use this PowerCLI command

This would first get all the ESXi hosts in the cluster “Cluster1” and then launch the refresh cmdlet on each host. The problem with this is that it can take a really long time for the command to complete because the refresh is executed host by host. Before starting the next refresh, the command waits for the previous one to finish. While this is fine for smaller environments, the time really does add up when you have a decent sized cluster.

Using PowerShell jobs

In order to reduce the time this process takes, I decided this would be a great use case for PowerShell jobs. A job is essentially another PowerShell instance that runs in the background. This means that your main script or PowerShell window doesn’t wait for it to finish before going to the next step. Using jobs will allow us to start the rescan on each host as a job and then start the next job. Meaning the rescans will happen in parallel.

Code

The script is provided as is and can be found on GitHub


ERROR: The host returns esxupdate error code:15

Today I was preparing some updates with VUM on Lenovo servers running vSphere 6.0. Things did not go as expected. While staging the patches I was greeted with this error.

ERROR: The host returns esxupdate error code:15. The package manager transactions is not successful. Check the update Manager log files and esxupdate.log files for more details.

Looking into the esxupdate.log file I could find the following entries:

This seems to be a known issue for Lenovo servers. It turns out they cram a whole lot of drivers in there, more than other vendors. The only known workaround is to remove VIBs from the host that are not needed. This is also described in https://kb.vmware.com/s/article/2144200.

To find out what VIBs are installed, log onto the affected host using SSH and run this command:

Once you have identified a driver that you want to remove, put the host in maintenance mode and this command:

In my case I wasn’t using the qlogic nx2 driver so I removed it. Now you should be able to update the host using VUM,