Why YOU should become a vExpert

The end of the year is one of my favourite times of the year. Not just because it’s my birthday or the holiday season but also because it’s time to fill in your vExpert application. It’s a good time to reflect on the past year and what you accomplished.

So what is the vExpert program all about

Unlike what some people may think, the vExpert program is not a certification that you get when you pass an exam. Nor is it necessarily a reflection of your skill as a VMware engineer. At its core, the vExpert program is a marketing and advocacy program, designed to help you, as an individual contributor in getting your messaging about VMware and its products out the door.

What’s more important though is that its a tight knit community of likeminded individuals that all have a shared passion.

Okay that sounds cool but what’s in it for me?

If you’re awarded the vExpert title, you also get a lot of benefits. One of the best things is that you get access to pretty much all the software that VMware has to offer with licenses. This will make it easier to try out new stuff in the lab and get new content out the door.

You also get access to NDA content and briefings by VMware and their partners.

But by far the best resource you get access to, is the vExpert slack channel. There’s a lot of great conversations about VMware topics but also about certification, homelabs, career and just random chit chat. It gives you access to people in the community that can help you out if you’re troubleshooting an issue in the lab or if you want another opinion on something you’re proposing, or whatever.

I’ve been in the program since 2016 and I can honestly say that I’ve made a lot of new connections this way and even built out new friendships. You can usually find a lot of the vExperts in the blogger zone at VMworld, which is great for getting those real world connections.

Speaking of VMworld, as a vExpert, you get an invite to the vExpert party at VMworld US & EU. When I went to the vExpert party in Barcelona in 2019, Pat Gelsinger popped in. A memorable moment for sure!

Belgian vExperts with Pat Gelsinger

That’s great but how do I become a vExpert

In order to become a vExpert, you need to put yourself out there. There’s really a lot you can do:

  • Write blogs
  • Speak at VMUG/events
  • Build a training course
  • Organise a community event
  • Respond to posts on VMTN
  • Write cloud
  • Publish videos

As you can see there’s a lot of things you can do with a very low barrier to entry. By far the easiest way to get started is by starting to blog. Did you come across an issue that you spent hours figuring out, and you couldn’t find any good resolution online? Well, write about the problem you faced, what things you tried to get it resolved, why it failed and what resolved the issue in the end.

Or just start writing about things you do at your job or in your homelab. Use your blog as your personal documentation of how you deployed a certain product. This helps yourself but also others that might be wanting to do the same thing.

I want to apply but I need some help

Filling out your application for the first time can be daunting. But don’t worry, the vExpert program has your back. A few years ago, the vExpert PRO was introduced. A vExpert PRO is someone in your local community that is actively trying to build out the community. They can also help you with filling out your application and giving you pointers on how to improve it.

I hope this gives you some good information about what the vExpert program is and how you can apply to get in. Still have more questions or need help filling in your application? Don’t hesitate to contact me!


Deploying vIDM through vRealize Suite Lifecycle Manager

One of the reasons I got my homelab was to test out stuff that I don’t necessarily have access to at work. Recently, vIDM has peaked my intrest so I decided to deploy it. I had already deployed vRealize Suite Lifecycle Manager so why not use that for the deploy!

Getting everything ready

Before we can actually start installing vIDM, we need to get the binaries onto vRealize Suite Lifecycle Manager (vRSLCM). This can be done in three ways; connect your MyVMware account and download everything onto the appliance, copy the binaries to the appliance yourself through WinSCP, or connect vRSLCM to an NFS share. We’ll be using the last option here.

When you first login to vRSLCM, click on the Lifecycle Operations card which will bring you to your environments, datacenters, etc.

Click on the setting link on the left hand side.

Then, near the bottom, click on the Binary Mapping card. On the net screen, just hit the add binaries button.

Hit the NFS radio button and enter the path to your NFS share. I did run into some issues here because I didn’t realize my Synology also includes the volume name in the NFS share path. As soon as you hit the discover button, you’ll get a list of all the binaries that vRSLCM detects. Import the ones you need, in our case, it’s the OVA file for identity manager.

Creating a new environment

Now that we have the vIDM binary imported, it’s time to prepare for the actual deployment. On the left hand side, click the Create Environment link. On the next screen, click the slide button. This will tell vRSLCM that we’re going to install vIDM. You’ll notice that the environment name changes to globalEnvironment.

Select a default password from your locker, or create a new one and select it. This password will be set on the default admin user in vIDM. Select the datacenter where you want to deploy vIDM to or add a vCenter connection if you haven’t already done so.

On the next page, you can select what product is going to be installed. Notice that you can only select vIDM here. Select the correct version and deployment type. The only 2 deployment types you can choose are Standard and Cluster. For my lab, I’m selecting standard install.

You’ll be presented with the EULA on the next page which you’ll read through entirely before accepting of course… 🙂 Now it’s time to select a certificate, if you’ve got one imported already you can just select that one. I don’t have one available yet so I will generate a new one for the lab and replace it later, once my PKI has been set up.

The next page is all about the infrastructure itself, select your vCenter, datastore, network and all the usual stuff for your environment and hit next.

Enter the network settings for your environment and hit the Edit server selection button.

By default the same DNS servers, that are configured on the appliance will be shown. If you want to add additional ones, you can do so by clicking add new server in previous screen. Select the DNS servers you want and configure them in the correct order

The final page ties everything together and lets you set the final bits of configuration. Review that the certificate and password are the correct ones. Additionally you can select the node size here as well. I selected the medium size here, which deploys the VM with 8 CPUs and 16 GB RAM. I also entered an admin email and default username.

At the bottom of the page, I entered the VM name, FQDN and IP address.

Review and deploy vIDM

Now that all the details have been entered, you can review everything and run through the prechecks. Several checks will be done here, whether the DNS records exists or not, if the IP address is actually free, etc. Correct any errors you get and read through the warnings to make sure you can continue.

As soon as you hit deploy, you will be taken to the request details where you can follow along with every step of the way. I actually quite like looking at this as it looks great.

Once everything is done, you can go to the environments again. You’ll see you have a completed globalenvironment with vIDM in it.

You can just open up a new browser window and go to the FQDN you entered during setup, you should be see the following screen and login with the credentials you set during setup.

That concludes this guide, I hope you found it useful! If you have any more questions or comments just hit me up on twitter or in the comments below!


How to change iLO IP from ESXi

I ran into an issue at work, where the iLO lost all its settings after a hardware intervention. The host was running fine and the iLO NIC was connected, but the IP configuration was missing. Because this is a remote site, it would be easy if the configuration could be set from within ESXi. Luckily, this is a possibility!

Installing the tools

The first thing we need to do is download the VIB from the HPE website. You can find the tools if you search for “HPE Utilities Offline Bundle”. Download the latest zip file, for your specific version of ESXi, and upload it somewhere on your ESXi host.

Next, install the bundle using this command;

esxcli software vib install -d /tmp/HPE-Utility-Component_6.5.0.10.7.1-8.zip

The result will look something like this

Installation result

Reboot the host after the installation. Once the host is back, you can verify that it is installed by running the following command. The output should look like the picture below.

esxcli software vib list | grep hpon*
Verify if software is installed

Listing iLO configuration

Now that we have the tool available, we can check the current configuration of the iLO. First, change the directory to /opt/tools, which is where the tool is installed to. Next, run the following command to export the active config to a txt file.

./hponcfg -w /tmp/ilocfg.txt

Looking at the exported configuration, you can immediately see what’s wrong, in our case the iLO configuration is completely empty.

iLO configuration

Changing the iLO configuration

To change the configuration, we can just make the required changes to the exported configuration file. Just use vi to make the required changes.

Once you’re satisfied with the new config, you can apply it by running this command

./hponcfg -f /tmp/ilocfg.txt

You should be able to reach the iLO on the configured IP address. There’s just one more step to do, and that is to reset the administrator password.

To do this, we’ll create a new xml file with the config to reset the admin password. Just create a new file and give it a name that you want, here we’re using admin.xml. The contents of the file look like this

<ribcl VERSION="2.0">
    <login USER_LOGIN="Administrator" PASSWORD="SuperS3curePass!>
        <user_INFO MODE="write">
            <mod_USER USER_LOGIN="Administrator">
                <password value="SuperS3curePass!" />
            </mod_USER>
        </user_INFO>
    </login>
</ribcl> 

Now we just apply the config file, like we did before;

./hponcfg -f /tmp/admin.xml

Once the configuration is successfully applied, verify that you can access iLO with the new password. If it works, make sure you delete the admin.xml file as this contains the password in clear text!


Change NSX-T password with API

Last night, I logged into the NSX-T manager in my lab and was greeted with the following message.

In the past, I would SSH into the edge nodes and change the password like that. But since NSX-T is suggesting to use the API, I figured I would try it. This would be a lot easier to do for the 3 users on both my edge nodes than having to type out all the commands.

Connecting to the API

Whenever I want to explore an API, I fire up Postman to see what I can learn. To connect to NSX-T, you can just enter the URL in the address bar. In this case, I’m going to explore https://s-bi-nsxmgr1/api/v1/node/users . Where s-bi-nsxmgr1 is the hostname of one of my NSX-T managers, and /api/v1/node/users is the API endpoint I’m querying.

Before we can hit the send button, we need to provide some credentials in the Authorization tab. Here in my lab, I’m just using basic auth and the admin user.

Getting the users

Once you’ve specified credentials to authorize onto NSX-T, you can hit send and get a list of all users that exist in your NSX-T environment. Here in my lab, only 5 users exist, 2 of which are not activated. This will probably be the same in many labs, I only have the root, admin, and audit user configured.

For each user, you also get some additional attributes like the last time the password was changed and the required password change frequency.

Changing the password

So let’s go ahead and change the password of the audit user. Note down the userid of the audit user from the previous API call. In my case, it’s 10002. Now that we’ve got the userid, we can verify if it’s actually the user we want. Type the userid in the URL so it looks like this; https://s-bi-nsxmgr1/api/v1/node/users/10002 and hit enter again. You’ll see all the details of the audit user.

Now to actually change the password, we need to send a body along with our API call. In the body, we need to define 2 attributes; password and old_password. Where password is the new password and old_password is, well it’s the old password of course! 🙂

In postman, go to the Body tab and make sure the radio button is set to raw and the type to JSON. Also change the call type from GET to PUT, since we’re putting a new password.

In the body you can set the following, change the values accordingly;

{
    "password": "UltraHighS3cur!ty123!",
    "old_password": "SuperSecurePassword2"
}

If all goes well you should get the following output

HTTP code 200 indicates a success and you can see the last_password_change attribute has been reset to 0, indicating it was just changed.

So you can see, changing the password via the API is not something to be scared of, in fact it’s dead simple and can save you a lot of time. Just make sure to clean up the postman tabs you used, so no-one can get to the new password.


Clear DNS cache VCSA/PhotonOS

The last few weeks I’ve had to do a couple of IP changes on ESXi hosts. This always goes without a lot of issues but it can be annoying when you have to wait for the new IP address to be updated in DNS and then for it to be visible in vCenter. The quickest way to get the new DNS records, is to clear the DNS cache in VCSA. Since VCSA is based on PhotonOS, this will also work on other PhotonOS VMs.

PhotonOS uses dnsmasq as a local DNS cache/proxy. So all we have to do is restart that service to clear the cache.

First, open up an SSH session to VCSA and enter the bash shell.

Run the following systemctl command to restart the service.

systemctl restart dnsmasq.service

Now, we just have to check that the service is up and running again. We can use systemctl for that as well.

systemctl status dnsmasq.service

If all is well you should see the output like it’s shown above. Run a quick ping to check that the DNS record is resolving to the correct IP and you’re done!

This was a quick post but I kept having to google it. Hope this helps !


vCenter update fails: vCenter server is non-operational

Today, while I was updating vCenter in my lab, I ran into a strange issue. The update I was installing failed, when I wanted to try again, this ominous error message popped up.

vCenter being non-operational left me a bit of a doom and gloom feeling but, thankfully, the fix is rather easy.

Fixing the issue

Open up an SSH session to your vCenter server and run the command below to remove the software_update_state.conf file.

rm /etc/applmgmt/appliance/software_update_state.conf

You can check the contents of the file by opening it with vi, an example is shown below.

 

Turns out that the state “INSTALL_FAILED” is checked by the python script that performs the installation. The script can be found in /usr/lib/applmgmt/update/py/vmware/appliance/update/update_state.py

Removing the file will make this check pass and the installation will continue.


VCAP-DCV Deploy Exam Experience

After postponing this exam for too long, I finally took the time to study for and take the VCAP-DCV deploy exam. To be honest, I was kind of looking forward to this exam. I love playing in the lab and getting my hands dirty in an environment. So to be able to take an actual lab exam was pretty exciting to me.

In this post, I will try to give some pointers and tips that might help you pass the exam. Please note that I took the exam for the 6.5 version but the same logic applies to other versions.

Preparation

Unlike the design exam, the deploy exam is a lab exam. This means you need to know how to do the stuff you’ve been reading and talking about. But, there are still some similarities. For starters, know the blueprint. This comes back for every exam, but read the blueprint upfront and know what topics are going to be covered on the exam. You should be intimately familiar with these blueprints by now.

Documentation

Although it’s a lab exam, you do have access to all the documentation VMware has on vSphere 6.5 in a folder on your desktop. Unlike what I read in some other posts, you do NOT have access to Adobe Reader in the lab. All PDFs will be opened in the browser, which means you can’t do a search in the entire folder anymore. Keep this in mind, read the documentation, and know where you need to look to find certain commands or info on a topic.

Community resources

As you might have guessed, there are already some great resources available in the community and I’m probably going to be linking the same ones that everyone does but that just shows how good they are.

  • Kyle Jenner’s VCAP6-DCV deployment study guide is the resource that I used the most. Kyle has put a lot of time and effort into explaining every topic covered in the blueprint. Make sure to go through every topic thoroughly.
  • Graham Barker’s VCAP6-DCV exam simulator is great to get a feel for what kind of questions will be asked and what depth they go to. Although the HoLs listed are no longer available, you can still perform these tasks in another HoL or your own lab.

Lab

But the most important resource I used while preparing was my lab. The VCAP exam covers nearly every vSphere feature there is. Like me, you’re probably not familiar with every feature there is. Make sure you lab these things more than once so you know how to do them. If you’ve got a lab that’s set up perfectly, try to get one of your colleagues or friends to break things in the lab. This will come in handy when you’re doing the exam. If you don’t have a lab, just spin up one of the vSphere 6.7 labs and start playing around.

You could also rent a server for a month or 2 like I did, and start building a nested lab. But that’s a topic for another post.

Besides that, there’s no replacement for real-world experience. I would not recommend taking this exam until you’ve got about 1 – 2 years of daily hands-on work done with vSphere. This will make the exam a lot easier.

Exam experience

The exam itself is presented to you in a HoL like fashion, it’s the same UI. If you’re not familiar with how HoLs work, be sure to start up a few so you’re familiar with the interface and how you can change and use the interface. Also, try booking the exam in a center where they have big screens. My test center has 24″ screens, which helped A LOT.

I found the lab to be reasonably performant and had no issues with connection whatsoever. The only minor annoyance was letters appearing more than once while typing but this is probably due to latency. Just be sure to read what you typed if a command fails.

Time management is crucial for this exam, that’s what a lot of other people told me at least. With that in mind, I tried to get the questions done as fast as I could without rushing through them. If I was stumped on one part of the question, I would write it down on the piece of paper I got, and move on to the next question. After I got through all questions, I started going back to the ones I didn’t complete. This will also prevent you from getting frustrated/stuck on 1 question, taking a break will give you a fresh look at the question.

I finished the exam with 42 minutes to spare so I never really felt that I was in a hurry to get everything done but your mileage may vary.

When doing the exam, read the questions carefully. It happened several times that I quickly read a question, started doing things, and afterward re-reading the question to find out I had not done several things.

Make sure you’re familiar with the CLI and PowerCLI, these things can come in handy for doing certain things faster. Also, try to open up the flash client again before taking the 6.5 exam. During the 6.5 days, the H5 client wasn’t yet fully-featured so you may not be able to use it for all questions.

Results

I took the exam on a Friday at 10 AM, so I was expecting to get the results on Monday or Tuesday after that. Around 8 PM I got an e-mail saying that I passed! This was a very pleasant surprise and big kudos to the VMware education team for providing the results so quickly.

I hope this post will help you prepare for the exam, good luck!


VCSA upgrade – deployment sizes missing

Everyone has done VCSA upgrades dozens of times, but every now and then you come across something that you haven’t seen before. Today, this was the case while I was doing an upgrade.

When you get to the deployment size selection, in the first stage of the upgrade, I noticed I was unable to select anything smaller than “Medium” size.

Changing the storage size also didn’t make the Tiny or small sizes available. While I was doing this, I happened to be talking to Jens Herremans (Check out his blog, it’s awesome!). He told me to check the size of the logs partition on the source VM.

Checking log sizes

After opening an SSH session to the source VM and changing to the /var/log/vmware folder, I ran the following command to get a list of all files and their sizes.

ls -lahR >> size.txt

I write the output to a file to make it easier to review, also this avoids having to scroll up and having parts capped off. Next, review the file with your favorite editor. You’ll see a list of all the folders, their total size and the individual files with their size. For me the big directories are vpxd, sca, sso, vdcs and vsphere-client.

Every file will also have a timestamp that makes it easy to verify if you still need the logs or not. If you wish to keep the, you can easily move them to a datastore or copy them locally. In this particular environment, we had no need for any of the older log files. I could just go in and rm -rf all the archived logs. For example in the /var/log/vmware/sca folder, the following command will remove all the archived logs but keep the one currently in use

rm -rf sca.log.*

After the cleanup, I ran the upgrade assistant again and was able to select the tiny deployment size.


Update ESXi fails with dependency error vmkapi_2_0_0_0

In the past few weeks, I’ve been in the process of updating several standalone ESXi hosts to 6.5. As you would expect, everything went smooth up to a certain point. Several hosts started failing and throwing this error:

VIB LSI_bootbank_scsi-megaraid-perc9_6.901.55.00-1OEM.500.0.0.472560 requires vmkapi_2_0_0_0, but the requirement cannot be satisfied within the ImageProfile.
VIB LSI_bootbank_scsi-mpt3sas_04.00.00.00.1vmw-1OEM.500.0.0.472560 requires com.vmware.driverAPI-9.2.0.0, but the requirement cannot be satisfied within the ImageProfile.

The entire error is shown below.

At first, I thought it was something with the image I was using. That’s why I also tried with the vanilla VMware image. Sadly, the same error popped up. Trying different zip files also did not solve the issue. I began looking into how I could make vmkapi 2.0 available on the host. Turns out that there’s not a whole lot written about this.

I started looking at the command again when I remembered there’s also another way I could install the update. When using esxcli to update, you can also use esxcli software profile install command. The update command updates existing VIBs with the ones that are in the specified profile, all the other VIBs aren’t touched. While the esxcli software profile install command installs all the VIBs present in the image profile, it will also remove any other VIBs installed on the server.

When I ran the install command everything proceeded without any issues. Here’s the command I used;

esxcli software profile install -p DellEMC-ESXi-6.5U3-14320405-A03 -d /vmfs/volumes/datastore/ISO/VMware-VMvisor-Installer-6.5.0.update03-14320405.x86_64-DellEMC_Customized-A03.zip --ok-to-remove

You can also add the –dry-run switch to get a list of VIBs that will be removed, before you actually change anything.


VMworld recap

The saying goes “time flies when you’re having fun” and boy was that true at VMworld! I had heard a lot of stories about VMworld and have been wanting to go for ages. Sadly, I never got the opportunity to actually go until this year! As always, I entered to win a blogger pass and this year it actually worked out 🙂 Thanks again to Corey Romero and the team for giving me this opportunity.

Keynotes

Day 1

Because of the blogger pass, myself and the other bloggers got preferred seating during the keynote. This meant we had a great view of the stage and the speakers, this really helped increase the experience. The day 1 keynote was all about VMware’s vision for the future of IT and I do have to say, some of the things they touched on really hit home for me. It’s clear that the days of just deploying VMs and vSphere are behind us and the landscape is changing. Of course, this won’t happen overnight but the repeated messages really make it clear that it is time to start getting to know all these new technologies before it’s too late.

This has also been made clear with the recent acquisitions VMware made, most of which were already announced in San Francisco but some have been closed by now and you’re starting to see the impact of them already. The new possibilities that are there after the Carbon Black acquisition are really amazing and help the “Intrinsic security on all layers” message become a reality.

There was a lot more to gather from the day 1 keynote but others, like fellow blogger Fabian Lenzker (link), have already done an excellent job writing this down. You can always watch it yourself on YouTube.

Day 2

The day 2 keynote was a lot of fun to watch, it was a continuation of the messaging of day 1 but now the speakers were showing how this would actually come together in a couple of demos. The story that Ray O’ Farrell was bringing was clearly depicted in all the demos that were given, each highlighting a different part of the stack and harping on a different part of the overall vision.

After the tech part was done, Bear Grylls came on stage and gave a talk that really hit home for me. He told the story about the 4 F’s, Failure, Fear, Fire and Faith. These 4 pillars have helped him get through all of the rough times he’s had during his lifetime and how he finds the strength to keep on going, even though there may be constant setbacks. I have to say that I felt really inspired after hearing him talk…

Sessions

As you would expect, a good part of my time was spent in sessions. At times it was really hard to pick a session because there were so many great choices, thankfully all the sessions were recorded!

Monday started with a VCDX workshop, led by Joe Silvagi. I’m not planning on going for the VCDX in the short-term but I do want to start gathering more and more information so this was a great opportunity. For everyone considering going for the VCDX, I highly recommend attending one of these workshops. It was really an eye-opener for me! The other VCDXs that joined the session also gave a lot of great insights.

There’s a lot of sessions that I have already rewatched the recording of and some sessions will need a couple more viewings to really get all the nuggets that are in there! William Lam’s already made a summary page that has all the session recordings, check it out here!

VM Village

The first thing you see when you come in, after the registration, is the VM Village. I was a bit amazed at how big it was, I expected it to be big but seeing it in real life is still something else entirely! One of the larger stands there is the community booth, this is where all the vBrownBag and VMware Code presentations were held. But also where the blogger table was. Hanging out there means you get to meet a lot of other bloggers and people that you’ve heard of in the community. Whenever I had some spare time, I would go there to hang out. Something I can only recommend!

One of my personal highlights took place after VMworld on Tuesday. I went to the vExpert party together with a couple of other Belgian vExperts. When we got there we noticed there was a small crowd gathering in one part of the room. That’s when Corey told us Pat Gelsinger was there to say hi! This was a great opportunity to shake his hand and have a quick chat. We also got a nice picture out of it!

I had a great time at VMworld and can’t wait to go back !