In May, the company I work for made the dive into a virtual environment. I must say, I love the capabilities of the VMware environment. Restarting a server went from a 5-10 minute process to a 1 minute process, 2 servers replaced 10, and the ability to move Guest OS's from one host to other is absolutely amazing. Of course our implementation wasn't without fault, but it went fairly well.
Fairly well... Once our consultants split, we discovered that our backup solution wasn't cutting it. We had purchased Symantec's Backup Exec 2010 with the VMware APIs that allow vCenter to take snapshots and back those up. In case of failure, we just restore the entire VM from the snapshot. Easy, right? Yeah, if we could ever get a backup. Our data transfer rates were somewhere around the 200MB/min mark. When you're moving over 2TB of data...well, that takes several days. That's unacceptable.
I won't go through all the steps of what I did to troubleshoot the problem, but it went something like this. VLAN'd off the iSCSI traffic, created a new subnet for iSCSI traffic, moved iSCSI traffic to a dedicated switch, enabled jumbo frames and updated every single driver and firmware available. At this point we would reach 600MB/min if we were lucky. I spent weeks on the phone with IBM, VMware and Symantec, trying every little solution they offered. Again, no help.
Over a month later, the problem is solved and you'll never believe what it was... well maybe you will since you've seen the title of this post. The Backup Exec/VMware API doesn't work as expected. Nope, not at all. I would love to call Symantec and inquire about the issue, but I've spent 4 hours on hold before and never got issues resolved with them.
Here is our workaround. Hopefully this will save someone some trouble one day. I had to install the Backup Exec agent on every Guest OS, just as if they were physical servers. Now we get speeds closer to 2,000MB/min which is a lot more acceptable. I would much rather use the VMware API, but for now, this is the only way we can get reasonable speeds. Maybe Symantec can squash some bugs soon and I can use the product as advertised.
Disclaimer: This post is meant to be informative for others troubleshooting slow data transfer speeds using Backup Exec 2010 and the VMware API.