Updated Post – vSphere ESXi6.0 CBT (VADP) bug that affects incremental backups / snapshots.

VMware recently posted a new KB article 2136854 to advertise the issue. Which is great that this has finally been accepted and advertised to customers and partners. It’s important to know this is not the same one as posted recently also for ESXi 6.0 (KB 2114076) – now fixed in a re-issued build of ESXi 6.0 (Build 2715440) But it is very similar to KB 2090639 from a historical perspective. The Issue If you are leveraging a product that uses VMware’s VADP for backup, then chances are you are leveraging this for not just initial fulls, but regular incremental snapshots (for backup purposes). There are numerous products on the market that leverage this API, it’s virtually the industry standard to use this feature as it results in faster backups. When the incremental changes are being requested through the API (QueryDiskChangedAreas) the API is requested changed blocks, but unfortunately some of the changed blocks aren’t being correctly reported in the first place, so backup data is essentially missing. And backups based on this can be inconsistent when recovered and result in all sorts of problems. The Challenge Currently there is no resolution or hotfix to the issue from VMware. I hope that we will see something in the coming days due to the wide ranging impact to customers and partner products affected. The Workarounds The workarounds in the KB suggests: Do a full backup for each backup, and that will certainly work, but it’s not really a viable fix for most customers Downgrade to ESX 5.5 and virtual hardware back to 10 (ouch !) Shutdown the VM before doing an incremental From the testing we have done at Actifio, option 3 doesn’t actually provide a workaround either, and options 1 & 2 aren’t really ideal either. The Discovery When Actifio Engineers discovered the issue, we contacted VMware and proved the problem leveraging just API calls to demonstrate where the problem was. How did we discover the issue I hear you ask ? Well we managed to discover the issue via our patented fingerprinting feature that occurs post every backup job. This technique (feature) essentially has learnt to not trust the data we receive (history has proven this feature to be useful many times) but to go and verify it against our copy and the original source copy. If we receive a variance in anyway, we trigger an immediate full read compare against the source and update our copy. This works like a Full Backup job, but doesn’t write out a complete copy again, it just updates our copy to line up with the source again (as we like to save disk where we can!). We’ve seen this occur from time to time with our many different capture techniques (not just VADP), so it’s a worthy bit of code to say the least that our customers benefit from. Let’s hope theres a hotfix on the near horizon, so the many many VADP / CBT vendor products that rely on it, can get back to doing what we do best and that’s protecting critical data for our customers that can be recovered without question. Cheers Patch Update – 24th November 2015 Our team have received a pre-release version of the patch, and it looks good from our initial testing. We expect this patch/hot-fix to be released to the public on or around the 27th November, which is good news for all those with ESXi 6.0 deployed or close to deploying. Patch Released – 25th November 2015 VMware have released the patch – it is available here : http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2137545       Click to share article:ShareLinkedInTwitter

VMware Hardware Version upgrades – The manual way.

The below process is somewhat manual in the process, but it’s been validated a few hundred times by myself personally. I like to think of VMware’s Virtual Hardware as a virtual motherboard, I remember this explanation was included in the VMware JumpStart materials I used to deliver back in the VI 3.x days. And the Virtual Hardware comes with certain capabilities and restrictions. Newer motherboards are capable of more processors, more memory, more efficient, etc. And commonly, what you normally need to add/remove or replace on a motherboard is the same as Virtual Hardware, in that it often requires a power down to replace or upgrade such items. Yes Hot-Add has changed this somewhat, and definitely for the better, but it’s sometimes an afterthought in many peoples VM templates. Hardware version 4 was available from VMware ESX 3.x and above (pre vSphere branding). With vSphere 4, VMware released hardware version 7 (going from 4 to 7 in one release!). However VMs running hardware version 4 are also backwards compatible with vSphere 4. To update a VM to hardware version 7, the customer must be running vSphere 4.x or above. The conversion isn’t too hard. but there is about a 5% (my estimate) chance the VM might not boot after the upgrade. It’s irreversible, unless you perform a VM snapshot prior. My usual practise when doing this is: 1. Upgrade VMTools to latest version on the VM. This will normally require a reboot. Even if not prompted, if this is a key VM, do a reboot anyway, and login to the console after to check for any left over driver upgrades that might kick in. 2. Shutdown the VM (The VM has to be shutdown to upgrade the virtual hardware version — again think of it like a motherboard replacement) 3. Create a VM snapshot from vCenter (for purposes of rollback), don’t use Actifio CDS for this step as it will likely be a full ingest or low-splash job taking plenty of time, and it will only exist temporarily. 4. Convert the VM to HW v7 (or 8,9, or 10). Right click on VM, select upgrade hardware version. Confirm you want to proceed. 5. Power on VM & login to a console session, and check IP address settings are fine. For Windows machines the WINS addresses are generally dropped if the customer is using static WINS Server entries, so re-enter these if they are required. This is one of the reasons doing the HW upgrade can be quite manual, though I would argue that customers should have removed WINS sometime ago, but sometimes you just can’t control the customer environment, so I noted the step here as it can be the difference between a successful upgrade or rollback. 6. Restart the VM (this step is important as the hardware upgrade is the same process as a motherboard upgrade) and upon first boot new hardware is often found and plug and play kicks in. Having the latest VMware Tools installed prior to this step is key. 7. Once the VM is restarted, login to the VM console, check all services are started upon boot, and IP addressing is still in place, DNS Search suffixes, WINs, etc. 8. If everything looks OK, you can then delete the VM snapshot from vCenter, which can be done online without requiring the VM to be shutdown. You’re VM is then running hardware version 7/8/9/10 or whatever is the highest version available. There are other methods to do this, and some people skip the step of taking the snapshot, and it can be done quicker. But the above method offers rollback, but unfortunately 3 reboots which is a pain. Even if the customer is on vSphere 4.x and hasn’t upgraded the virtual hardware version, there are performance benefits for doing this on top of changed block tracking benefits. Below are the hardware versions which came out with the release mentioned. HW v4 = ESX/ESXi 3.5 and above HW v7 = ESX/ESXi 4.0 and above HW v8 = ESXi 5.0 and above HW v9 = ESXi 5.1 and above. HW v10 = ESXi 5.5 and above. Your customer is hopefully not confused that HW v4 means vSphere 4, as this is not the case. And also remember these upgrades are irreversible once the snapshot is deleted, and the more recent versions need to be all done via the vSphere Web GUI. Click to share article:ShareLinkedInTwitter