5 Questions I Ask Every Customer about their VMware Backup Strategy

I can’t remember of a technology platform that provided better APIs than VMware’s VADP API framework. While it has had annoying bugs periodically, overall the APIs made it extremely simple, easy, and efficient for VMware VM backups. It’s no wonder, there are a huge amount of backup product vendors that all claim features like application consistent and incremental forever backup (using VMware Change Block Tracking (CBT)) and instant recovery of VMs. There really is very little competitive differences now between these products! All 25+ vendors are all calling the same library for CBT capture. But what really matters in terms of costs and RTO is where and how you store that backup data and metadata. In this cloud era, it’s hard not to consider the cloud as a target to store backups and reduce data centre footprint and costs for backup and DR. Here are 5 simple questions I ask all of my current and prospective customers to think about when thinking about their VMware backup strategy. 1. Why have local on-premises backup copies? Why not back it up directly to the Cloud? Not all data is born equal. Multiple studies have shown that it’s important to tier your VMs and then apply backup and retention policies. So for all the Tier-2 VMs, which typically constitute anywhere between 40% to 70%, what if you could eliminate the local copy and backup directly to cloud object storage like AWS S3, S3IA, Azure Blob, Google Nearline, IBM COS? They all offer 11 x 9s of durability in three availability zones. It costs less. There is no capacity management as you don’t have to scramble for storage when you add the next 100 Tier-2 VMs for backups. There is zero operational burden with this approach. Obviously, this is not an ‘all or none’ approach. For your Tier-1 VMs, you might still have a requirement or bandwidth constraint so you can choose to have a local cache/backup copy in your data center, and also have a second backup copy in such a cloud object storage. 2. How long are your restores taking today? Is that acceptable as your data grows… Obviously, if you like the approach of leveraging cloud object storage, your next thought would be “What about the recovery time objective (RTO)?” Most backup products, unfortunately, take a long time to recover from their deduplicated backups stored in cloud object storage. The catch cry for some reason is still around the “backup industry”, but I’ve been calling out that the priority is wrong, it should be called a “recovery industry”. We backup so we can recover! That is what a business is really after when it invests in a backup solution, and more often they can’t afford to wait days and hours to get their critical data back from a dedup engine or tape. Object storage can really help solve two pain points there, it’s infinite in scale, and yet very quick to mount the data back.  Couple it with a next-generation backup product that writes the data in its native application format, and you’re starting to fix a lot of the legacy issues from a backup mindset, versus a recovery mindset! Recovering that 10TB VM or SQL/Oracle Database is just a few minutes away now, It’s a game changer people… seriously! 3. If you are restoring VMs from the cloud, are you concerned about the egress costs? Optimise every bit that moves. One of the concerns enterprises express is the egress charges from the cloud back to on-premises. Let’s explore, with an example, of how much would it cost you on a monthly basis. Let’s assume you have 1000 VMs that are being protected. Let’s assume, on an average 20 restore jobs for files/folders are performed per week, i.e. 80 restore jobs a month. Assume that on an average 100 MB of files are restored in each job. This translates to 80 x 100 MB = 8GB of total data restored from the cloud. Assuming you use AWS S3 IAS (Infrequent Access Storage), it charges $0.01 per GB. The data retrieval charges = $0.08 per month. AWS also charges for the data that leaves AWS cloud at a rate of $0.09 per GB. This translates to $0.72 per month. Thus total costs = $0.08+$0.72 = $0.80 per month, which obviously is very low. Now let’s look at a scenario where 20 VMs are recovered from cloud object storage back to on-premises. Assume average VM size = 200GB. Thus total data transferred = 20 x 200GB = 4,000 GB. Thus total data transfer charges = ($0.01+$0.09)*4000GB = $400 for the entire month. The good news is that even this small monetary amount can be reduced further. Consider a next-generation approach, where not all data needs to be recovered if it’s already on-premises. Features like “delta block differencing” technology will lookup its metadata to compare which blocks already exist in the local backup cache on-premises and transfer only those blocks from the cloud which don’t already exist on-premises. So in the above example, if you assume 40% of the blocks already exist on-premises, only 2,400 GB will be copied from the cloud, thus reducing the data transfer costs to $240. 4. Do you require any data immutability capability at the software and cloud storage layer? Was it a fat-finger or a rogue internal user? One of the legit concerns enterprises have is that of a rogue or malicious user who could potentially delete backups. What if at the software layer, an admin can apply a data immutability lock on backups of specific VMs. Once this is applied, even an admin can not expire or purge the backups for those specific VMs. You can still manage the TCO for disk by setting an expiration date for the backup data, as per the original required policy, or elect to never expire it, yet not be concerned about a rogue admin or a fat finger. 5. Do you like buying hardware appliances? Why not software only, or a SaaS platform? “The world we have created is a process of our thinking. It can not be changed without changing our thinking.” — Einstein Many enterprises are getting used to “as-a-Service” consumption model with exposure to GSuite, Office 365, Salesforce, Github, AWS RDS & Redshift and the likes of VMware Cloud in AWS. So they are also questioning the idea of purchasing hardware appliances for everything, not only backup appliances but also minimising their production compute platforms too. Why not consider a VMware Backup SaaS platform? Why not consume VMware backup and recovery to cloud in a simple per VM subscription pricing model?

Deploying the Actifio vCenter Plugin to VMware VCSA 6.0

This is just a short tip to help get the Actifio vCenter Plugin uploaded to your VCSA appliance, so you can start the installation process. By default if you try to scp the install file you will find an error such as the following: Unknown command: 'scp' 1. Login via SSH to the vCenter Server Appliance on port 22, I would normally use the root account here: $ ssh root@10.0.0.10 VMware vCenter Server Appliance 6.0.0 Type: vCenter Server with an embedded Platform Services Controller Password: Last login: Mon Jun 22 00:15:14 UTC 2015 from 10.0.0.51 on ssh Last failed login: Mon Jun 22 02:38:16 UTC 2015 from 10.0.0.51 on ssh:notty There was 1 failed login attempt since the last successful login. Last login: Mon Jun 22 02:38:17 2015 from 10.0.0.51 Connected to service     * List APIs: "help api list"     * List Plugins: "help pi list"     * Enable BASH access: "shell.set --enabled True"     * Launch BASH: "shell" Command> 2. As per the motd, you need to start the shell so run the command shell.set --enable True Then run the following command to get into the shell shell Next, run the following command to change the default shell chsh -s "/bin/bash" root Now you can scp the Actifio VCP file to the vCenter Server Appliance. FileZilla or command line scp are your friend here (example below): scp -P 22 ActifioVCPInstaller_unix_6_1_2.sh root@10.0.0.10:/tmp/. Now go back to the VCSA shell and change the default shell back chsh -s /bin/appliancesh root Now you can run the installer with the following command sh /tmp/ActifioVCPInstaller_unix_6_1_2.sh And now follow the bouncing ball to get your Actifio vCenter Plugin installed.

Windows System State Backups with Actifio

Say Whaaaaa ? Yes you heard correctly. Due to numerous customer requests I researched this capability, while it’s not efficient entirely if you include it as part of a daily backup, but we can do it. Read on if you want more. I said it’s not efficient, as the process is essentially creating a new compressed file everyday (10GB+ is not uncommon). What this will do is cause a lot of new unique data to enter into the dedup pool. But here is the thing, we have a few ways to provide the capability without the storage consumption penalty. I will continue to research further to see if we can reduce the file size somehow with Microsoft, but here is what’s possible today. This procedure should work for Windows Server 2008+ and is fairly simple. 1. Local command line backup facilities must be enabled. If they are not these can be added via the GUI, or else run in an administrator command prompt the following command: C:\Windows\>servermanagercmd -install Backup-Tools 2. Run a pre script via VMware Tools, or the Actifio Connector script, with the command in it. wbadmin start -systemstate -backuptarget:D:\ -quiet In the above example the D:\ must exist as a target, but this could easily be changed for the Actifio Staging disk (or mount point) as the target. In future, I hope to further refine the testing to only include the last 1-3 days of data in the local system for recovery and 2, exclude the backup of all files in the C:\Windows\ folder and items beneath it. I would strongly suggest customers don’t do this on every server, and run it daily, but it might form as an additional policy to be run weekly on a domain controller (PDCe) as an example if required on a regular basis. Actifio also supports Bare Metal Recovery, which includes System State Backups as part of the licensing. So this article itself is not part of that solution, as it is built in. Now another approach to solving the idea, without storage penalty you could do the following and it applies only for Virtual Machines. 1. Ensure the machine that you want a system state backup for, has an SLA policy applied that includes the snapshot option. While this is not strictly mandatory, it will help with the speed of recovery for any system state image, as the machine can be mounted back instantly. 2. When you need a system state item, you can do an instant mount of the VM from the point in time you need to recover from. Mount the VM back to a hypervisor and as usual the VM will not be attached to the network. 3. Power on the VM and connect to the console session. Then login using local admin or cached credentials. 4. Run the command as above: wbadmin start -systemstate -backuptarget:D:\ -quiet This will create a system state backup from the VM at the time the snapshot backup was initiated. 5. Then Power down the Virtual Machine. 6. Attach the D:\ VM Disk (or disk where you stored the system state backup image) to any other running VM, normally best to attach to the same VM as where the backup was done, and attach the drive to the running VM. 7. You can then proceed with the System State restore process that you would normally follow, as documented by a Microsoft KB article or TechNet article. 8. Once completed you can Unmount and delete the volume from the running VM, and as a best practise it’s probably wise to initiate a manual backup of the VM/server where the recovery was performed. The benefit to the VM Instant mount approach, is that you still have access to the data, even though it’s not explicitly stored in the dedup pool as a new unique blocks every day. This will save storage in your long term retention pool, and only add an additional few steps in the recovery of system state restores. The instant mount feature Actifio provides greatly reduces the time to recover system state data, along with any files, databases, VM Disks, or entire VMs in seconds.