vSphere / ESXi 7.0 installed on your older hardware (unsupported)

Let me preface this post by saying, this is not a supported way to get vSphere / ESXi 7.0+ running on legacy hardware. Please only perform these actions if you are comfortable with the steps below, and know that you will not be entitled to support for VMware.

For my example in this article, I am using an IBM M3 X3550 Series server. While this approach might also work for other vendor hardware, you may need to make some slight modifications to the process, but it should still work.

TL;DR – You basically need to do two things, 1. Install ESXi onto a USB drive, instead of local hard drives, and 2. Implement the allowLegacyCPU=true mode in the boot.cfg.

Here are the list of things you need:

1. The VMware vSphere ISO to use for the installation. I used this one I downloaded from VMware “VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso”
2. A 16Gb or larger USB flash drive. I would recommend you use something decent here. In my example. I was using a TDK Silver key I had spare at home, but this was a fairly reliable key, and also not used often.
3. Common ssh and shell command familiarity, editing files with vi, General understanding of VMware ESXi installation processes, etc.

Sorry for the photos instead of screenshots. I will try and update these with better screenshots in future.

First thing is first, I went into the IMM bios for this server and changed the boot order as per the photo.
CD/DVD first, then USB second, and Hard Disks later. While my server has 2x 15K 300GB disks in a Raid 1 mirror, these can’t be used as the VMware Hypervisor install location, as the Raid card is no longer supported in ESXi 7.0 HCL.

After this is completed, you can then commence the installation, as there is no Physical DVD/CDRom on the server, I am mounting the ISO via my Windows 10 Virtual Machine Desktop. Historically for these legacy IBM servers, the best solution is to use Windows and use Internet Explorer as the browser for connecting to the IMM IP address, make sure you add the IP address to the Trusted Sites list in IE, and run a version of Java like 1.7.0_60. I also like to add the IP address for the IMM to the Java Exception Site List.

It is a good time to insert your USB key into the server now. It will be formatted by the ESXi installation, so no need to prep it ahead of time. Just make sure it doesn’t have any important files first.

OK, but if you have made is this far you should be ready to boot the machine, and boot from the ISO or Physical CD of VMware Hypervisor v7.0.x, but you will need to stop the booting of the ESXi Installer via the “SHIFT and O” thats o for oh, not a zero. You should see the prompt for this, and you need to run it, else the installer will halt and error out anyway, so if you missed the “SHIFT + O” prompt, reboot the server.

Once you get to the prompt, you will need to manually add the text allowLegacyCPU=true to the command, after the runweasel text as below. It’s also very important to type it exactly, as this is case-sensitive. Many thanks to William Lam for this key ingredient. He shared this post here: and it is well worth a read.

Now the installation should commence…

install proceeding

When prompted as to the location for the installation, you should see the option to install to your USB flash drive. If you don’t see it under Local: then you may need to try a different USB flash drive.

usb format

You will likely be warned about incompatible hardware in the server. Hit Enter to continue, and please note again, this work around should never be used for critical systems!

warnings

Now we are prompted to confirm the installation to the USB flash drive, and the partitions will be lost. Hit F11!

confirming install

Now, you can complete the installation and set a management IP address, and any other settings as you normally would.

completed install

On the next boot up after the install, you will likely need to insert the “allowLegacyCPU=true” option one more time upon boot, using the “SHIFT + O” keys to modify the prompt.

shift-o

Now ESXi should boot up and you all good, but wait theres one more step to ensuring you don’t need to modify the boot command prompt every time you restart the host. Read on !

So we need to edit the boot.cfg file to achieve this. So you need to start the TSM and TSM-SSH services to be able to connect via ssh to your ESXi server. A default install, will have these services set to stopped. So login to your ESXi host via the Management IP address you set and select the Manage Host option on the left menu, then Services tab, and start the services. Here is the before.

stopped services

Here is the after.

services started

Now you can ssh in as root, using the password you set during the installation.

In my following screenshots I looked for these files, and then edited them manually. The best way is to run a shell command find / -name "boot.cfg" and then modify both boot.cfg files with appending the allowLegacyCPU=true to the kernelopt= line. See my screenshots below. The first one, I found one file and edited it, then checked it with cat boot.cfg to make sure it was good.

boot edit

Then I used the find command to see where the other boot.cfg file was. Now these files, should actually just be in the /bootbank and /altbootbank symlinked directories. I wanted to be sure, so used find instead. It might not be also required to edit both bootbanks.

second edit

Lastly, now check both files are updated, and give the ESXi server a reboot from the ESXi UI, and test that the server comes back up after a reboot without any user input.

last check

And here is the money shot. ESXi 7.0 running on a Legacy server, ready for some testing.

esx7 on a legacy m3

But again, this is unsupported by VMware. So please do not bother them if it does not work for you. You will not get any support when using this procedure.

Cheers.

I can’t remember of a technology platform that provided better APIs than VMware’s VADP API framework. While it has had annoying bugs periodically, overall the APIs made it extremely simple, easy, and efficient for VMware VM backups.

It’s no wonder, there are a huge amount of backup product vendors that all claim features like application consistent and incremental forever backup (using VMware Change Block Tracking (CBT)) and instant recovery of VMs. There really is very little competitive differences now between these products! All 25+ vendors are all calling the same library for CBT capture.

But what really matters in terms of costs and RTO is where and how you store that backup data and metadata. In this cloud era, it’s hard not to consider the cloud as a target to store backups and reduce data centre footprint and costs for backup and DR.

Here are 5 simple questions I ask all of my current and prospective customers to think about when thinking about their VMware backup strategy.

1. Why have local on-premises backup copies? Why not back it up directly to the Cloud?

Not all data is born equal.

Multiple studies have shown that it’s important to tier your VMs and then apply backup and retention policies. So for all the Tier-2 VMs, which typically constitute anywhere between 40% to 70%, what if you could eliminate the local copy and backup directly to cloud object storage like AWS S3, S3IA, Azure Blob, Google Nearline, IBM COS?

They all offer 11 x 9s of durability in three availability zones. It costs less. There is no capacity management as you don’t have to scramble for storage when you add the next 100 Tier-2 VMs for backups. There is zero operational burden with this approach.

Obviously, this is not an ‘all or none’ approach. For your Tier-1 VMs, you might still have a requirement or bandwidth constraint so you can choose to have a local cache/backup copy in your data center, and also have a second backup copy in such a cloud object storage.

2. How long are your restores taking today?

Is that acceptable as your data grows…

Obviously, if you like the approach of leveraging cloud object storage, your next thought would be “What about the recovery time objective (RTO)?”

Most backup products, unfortunately, take a long time to recover from their deduplicated backups stored in cloud object storage. The catch cry for some reason is still around the “backup industry”, but I’ve been calling out that the priority is wrong, it should be called a “recovery industry”. We backup so we can recover! That is what a business is really after when it invests in a backup solution, and more often they can’t afford to wait days and hours to get their critical data back from a dedup engine or tape.

Object storage can really help solve two pain points there, it’s infinite in scale, and yet very quick to mount the data back.  Couple it with a next-generation backup product that writes the data in its native application format, and you’re starting to fix a lot of the legacy issues from a backup mindset, versus a recovery mindset! Recovering that 10TB VM or SQL/Oracle Database is just a few minutes away now, It’s a game changer people… seriously!

3. If you are restoring VMs from the cloud, are you concerned about the egress costs?

Optimise every bit that moves. One of the concerns enterprises express is the egress charges from the cloud back to on-premises. Let’s explore, with an example, of how much would it cost you on a monthly basis.

Let’s assume you have 1000 VMs that are being protected. Let’s assume, on an average 20 restore jobs for files/folders are performed per week, i.e. 80 restore jobs a month. Assume that on an average 100 MB of files are restored in each job. This translates to 80 x 100 MB = 8GB of total data restored from the cloud.

Assuming you use AWS S3 IAS (Infrequent Access Storage), it charges $0.01 per GB. The data retrieval charges = $0.08 per month. AWS also charges for the data that leaves AWS cloud at a rate of $0.09 per GB. This translates to $0.72 per month. Thus total costs = $0.08+$0.72 = $0.80 per month, which obviously is very low.

Now let’s look at a scenario where 20 VMs are recovered from cloud object storage back to on-premises. Assume average VM size = 200GB. Thus total data transferred = 20 x 200GB = 4,000 GB. Thus total data transfer charges = ($0.01+$0.09)*4000GB = $400 for the entire month.

The good news is that even this small monetary amount can be reduced further. Consider a next-generation approach, where not all data needs to be recovered if it’s already on-premises. Features like “delta block differencing” technology will lookup its metadata to compare which blocks already exist in the local backup cache on-premises and transfer only those blocks from the cloud which don’t already exist on-premises. So in the above example, if you assume 40% of the blocks already exist on-premises, only 2,400 GB will be copied from the cloud, thus reducing the data transfer costs to $240.

4. Do you require any data immutability capability at the software and cloud storage layer?

Was it a fat-finger or a rogue internal user?

One of the legit concerns enterprises have is that of a rogue or malicious user who could potentially delete backups.

What if at the software layer, an admin can apply a data immutability lock on backups of specific VMs. Once this is applied, even an admin can not expire or purge the backups for those specific VMs.

You can still manage the TCO for disk by setting an expiration date for the backup data, as per the original required policy, or elect to never expire it, yet not be concerned about a rogue admin or a fat finger.

5. Do you like buying hardware appliances? Why not software only, or a SaaS platform?

“The world we have created is a process of our thinking. It can not be changed without changing our thinking.” — Einstein

Many enterprises are getting used to “as-a-Service” consumption model with exposure to GSuite, Office 365, Salesforce, Github, AWS RDS & Redshift and the likes of VMware Cloud in AWS.

So they are also questioning the idea of purchasing hardware appliances for everything, not only backup appliances but also minimising their production compute platforms too. Why not consider a VMware Backup SaaS platform? Why not consume VMware backup and recovery to cloud in a simple per VM subscription pricing model?

In my earlier days at Actifio, we only sold a hardware solution, and many customers pushed back, asking for a software appliance version. We responded to those customers and launched Sky into the marketplace, and it’s been very successful to say the least. But we’re now going one step further and offering a SaaS version for VMware backup to AWS/Azure/Google/IBM cloud. Check out more details here at ActifioGo.com. The only consideration to factor in is some good bandwidth to your local cloud and you can be off and running in a few minutes.

Every few years, technology advances enable new capabilities which enable us to question some of the existing practices/assumptions and adjust appropriately. Examples like API-driven IaaS and PaaS were deemed too complex and resource intensive to automate. But the IT world has shifted, and enterprises are all looking at the benefits of cost optimisation, automation, and orchestration across their entire stack. I’d highly recommend you consider asking yourself the 5 questions above, and see how you can benefit from the likes of a next-generation platform combined with object storage, with 11 x 9s of durability, almost infinite capacity, and pay-as-you-grow models, which simplifies your life and lets you focus on other important tasks.

Check out this short 3-minute video by my good mate Chandra Reddy which compares architectures to leverage cloud object storage effectively. It’s well worth the time.

Feel free to reach out to me, if you want to discuss or dispute any of what I have covered above.

vExpert 2016 Award Announcement

vExpert 2016 Award Announcement

First we would like to say thank you to everyone who applied for the 2016 vExpert program. I’m pleased to announce the list 2016 vExperts. Each of these vExperts have demonstrated significant contributions to the community and a willingness to share their expertise with others. Contributing is not always blogging or Twitter as there are many public speakers, book authors, script writers, VMUG leaders, VMTN community moderators and internal champions among this group.


VMware Advocacy