Updated Post – vSphere ESXi6.0 CBT (VADP) bug that affects incremental backups / snapshots.

VMware recently posted a new KB article 2136854 to advertise the issue. Which is great that this has finally been accepted and advertised to customers and partners. It’s important to know this is not the same one as posted recently also for ESXi 6.0 (KB 2114076) – now fixed in a re-issued build of ESXi 6.0 (Build 2715440) But it is very similar to KB 2090639 from a historical perspective. The Issue If you are leveraging a product that uses VMware’s VADP for backup, then chances are you are leveraging this for not just initial fulls, but regular incremental snapshots (for backup purposes). There are numerous products on the market that leverage this API, it’s virtually the industry standard to use this feature as it results in faster backups. When the incremental changes are being requested through the API (QueryDiskChangedAreas) the API is requested changed blocks, but unfortunately some of the changed blocks aren’t being correctly reported in the first place, so backup data is essentially missing. And backups based on this can be inconsistent when recovered and result in all sorts of problems. The Challenge Currently there is no resolution or hotfix to the issue from VMware. I hope that we will see something in the coming days due to the wide ranging impact to customers and partner products affected. The Workarounds The workarounds in the KB suggests: Do a full backup for each backup, and that will certainly work, but it’s not really a viable fix for most customers Downgrade to ESX 5.5 and virtual hardware back to 10 (ouch !) Shutdown the VM before doing an incremental From the testing we have done at Actifio, option 3 doesn’t actually provide a workaround either, and options 1 & 2 aren’t really ideal either. The Discovery When Actifio Engineers discovered the issue, we contacted VMware and proved the problem leveraging just API calls to demonstrate where the problem was. How did we discover the issue I hear you ask ? Well we managed to discover the issue via our patented fingerprinting feature that occurs post every backup job. This technique (feature) essentially has learnt to not trust the data we receive (history has proven this feature to be useful many times) but to go and verify it against our copy and the original source copy. If we receive a variance in anyway, we trigger an immediate full read compare against the source and update our copy. This works like a Full Backup job, but doesn’t write out a complete copy again, it just updates our copy to line up with the source again (as we like to save disk where we can!). We’ve seen this occur from time to time with our many different capture techniques (not just VADP), so it’s a worthy bit of code to say the least that our customers benefit from. Let’s hope theres a hotfix on the near horizon, so the many many VADP / CBT vendor products that rely on it, can get back to doing what we do best and that’s protecting critical data for our customers that can be recovered without question. Cheers Patch Update – 24th November 2015 Our team have received a pre-release version of the patch, and it looks good from our initial testing. We expect this patch/hot-fix to be released to the public on or around the 27th November, which is good news for all those with ESXi 6.0 deployed or close to deploying. Patch Released – 25th November 2015 VMware have released the patch – it is available here : http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2137545       Click to share article:ShareLinkedInTwitter

Deploying the Actifio vCenter Plugin to VMware VCSA 6.0

This is just a short tip to help get the Actifio vCenter Plugin uploaded to your VCSA appliance, so you can start the installation process. By default if you try to scp the install file you will find an error such as the following: Unknown command: 'scp' 1. Login via SSH to the vCenter Server Appliance on port 22, I would normally use the root account here: $ ssh [email protected] VMware vCenter Server Appliance 6.0.0 Type: vCenter Server with an embedded Platform Services Controller Password: Last login: Mon Jun 22 00:15:14 UTC 2015 from 10.0.0.51 on ssh Last failed login: Mon Jun 22 02:38:16 UTC 2015 from 10.0.0.51 on ssh:notty There was 1 failed login attempt since the last successful login. Last login: Mon Jun 22 02:38:17 2015 from 10.0.0.51 Connected to service     * List APIs: "help api list"     * List Plugins: "help pi list"     * Enable BASH access: "shell.set --enabled True"     * Launch BASH: "shell" Command> 2. As per the motd, you need to start the shell so run the command shell.set --enable True Then run the following command to get into the shell shell Next, run the following command to change the default shell chsh -s "/bin/bash" root Now you can scp the Actifio VCP file to the vCenter Server Appliance. FileZilla or command line scp are your friend here (example below): scp -P 22 ActifioVCPInstaller_unix_6_1_2.sh [email protected]:/tmp/. Now go back to the VCSA shell and change the default shell back chsh -s /bin/appliancesh root Now you can run the installer with the following command sh /tmp/ActifioVCPInstaller_unix_6_1_2.sh And now follow the bouncing ball to get your Actifio vCenter Plugin installed. Click to share article:ShareLinkedInTwitter

vExpert 2015 Honour

I am duly honoured to have become a vExpert for 2015. My first time round too. Back when the program was first launched I worked at VMware, and employees weren’t eligible to become one. I figured that meant my chances would be slim to none of getting the nod in the future. However, I was really surprised  last week to learn that I was to receive the honour this year! There are truly many more people that contribute to the virtualizsation community than I, and I am energised by the award, and hope to get involved even more this year. To all those that received a nod for this year, congrats ! It’s a very nice feeling to be amongst such an esteemed fellowship. I am inspired and thankful to the community who almost daily produce great articles that assist me, educate me or provoke thoughtful discussion. But in reality I just like reading the comments and vendor declarations trying to spin an article to suit their flashy widget of the day. Anyways, I hope to further contribute something decent this year, instead of my usual tripe :) Beers, Jeff Click to share article:ShareLinkedInTwitter