Engineering case study

From 70+ Hours to ~20: Fixing a Slow 2 TB Restore

When a Proxmox restore across our own site-to-site VPN was estimated to take nearly three days, our engineers measured, found the real bottleneck and fixed it.

~4×
throughput improvement
70+ → ~20 h
estimated 2 TB restore time
~70%
reduction in recovery time

Why This Matters to Clients

NetOz uses multiple locations for infrastructure testing, offsite backups and expansion. Every site is connected by VPN with BGP running on each side, using a mix of OpenVPN, WireGuard and IPsec. This work happened on that internal network, while one of our engineers was migrating a 2 TB test VM and did not want to wait days for it.

A backup is only as good as the time it takes to bring it back. Recovery time is often limited by the network path between sites rather than storage or compute, and the same troubleshooting discipline applies to the VPNs, hybrid cloud links and restorations we run for clients.

The Environment

Workload

A Proxmox Backup Server restore of a VM with a disk of about 2 TB. PBS reported 0% zeroes, so the whole disk had to be transferred.

Network

A virtualised pfSense firewall with an OpenVPN (UDP, AES-NI enabled) site-to-site tunnel, routed with BGP via FRR.

WAN

Roughly 1 Gbps down and 500 Mbps up, testing at about 947 Mbps and 473 Mbps.

The Problem

Despite reasonable bandwidth, the restore was crawling at about 7 to 8 MB/sec sustained, with an estimated completion time of more than 70 hours. The link was behaving like a 50 to 60 Mbps connection.

Measuring the Tunnel

Rather than guess, the team ran iperf3 across the OpenVPN tunnel:

  • TCP, 4 streams: about 26 Mbps with 447 retransmits.
  • UDP at 200 Mbps, forward: 135 Mbps received with 0% loss.
  • UDP at 200 Mbps, reverse: 129 Mbps received with 35% packet loss.

The tunnel was collapsing under load on the receiving side. The root cause was architectural: not bandwidth, not storage and not Proxmox, but the OpenVPN tunnel itself, even with AES-NI enabled.

The Fix: Move to WireGuard

The team replaced the OpenVPN tunnel with WireGuard on a new transit network, assigned the WireGuard interface in pfSense, re-established BGP over the tunnel and adjusted the firewall rules. OpenVPN could probably have been tuned further, but after a short investigation the team chose to move on.

After the migration, the same tests showed:

  • UDP at 200 Mbps: the full 200 Mbps, 0% packet loss and about 0.04 ms jitter.
  • TCP, 4 streams: about 440 Mbps sustained.
  • Real PBS restore: about 29.3 MB/sec (roughly 234 Mbps), measured over a 20 GiB sample in 700 seconds.

At around 30 MB/sec, the full 2 TB restore works out to about 19 to 20 hours.

Before and After

MetricOpenVPNWireGuard
TCP throughput~26 Mbps~440 Mbps
UDP stabilityLoss under load0% packet loss
Restore speed~7 to 8 MB/sec~29 to 30 MB/sec sustained
Effective bandwidth~60 Mbps~230 to 240+ Mbps
2 TB restore time~70+ hours~20 hours
Packet lossSevere under loadNone observed

What We Took Away

We still appreciate OpenVPN, particularly for its ease of use for client VPN connections and its desktop support. For direct site-to-site links where we want to saturate the available bandwidth, WireGuard has performed better for us.

It is consistent with our recommendation of WireGuard for most new deployments of our managed business VPN. It is also a reminder to test the network path, not just the storage, when planning backup and disaster recovery for a hybrid cloud environment.

The full write-up, including the OpenVPN configuration, iperf3 commands and restore calculations, is on our blog: How replacing OpenVPN with WireGuard reduced a 2TB Proxmox restore from 70 hours to 20.

Slow Links or Slow Restores?

If your VPN or backup recovery is not performing the way it should, talk to the team that did this work.

Talk to an Engineer

Or call us on 1800 134 656