Our OPNsense routers were VMs. Their disks lived on shared, network-defined storage. When the storage cluster stopped, OPNsense stopped. When OPNsense stopped, we lost the routes we needed to fix the storage cluster.
We spent an afternoon digging out of the loop. This post is the two lessons we took away from it, written down so we do not relearn them.
TL;DR
- OPNsense is your firewall, your router, and the thing standing between every other service and the rest of the network. It belongs on bare metal. For anything critical, put it on two bare metal boxes and let CARP handle failover at the application layer.
- If you must run it as a VM, its disk goes on local storage. Never on any network-defined or distributed storage. The firewall must not depend on a system whose recovery depends on the firewall.
Everything below is the story of why we believe this now.
The Original Design
Our two OPNsense gateways ran as VMs on a virtualization cluster. Their disks sat on the cluster's shared, network-defined storage pool. We configured CARP between the two instances for active/passive failover. On paper, we had HA.
It looked clean. It let us live-migrate firewalls between hosts. Snapshots and backups came for free from the hypervisor layer. And for months, nothing pushed back on the design.
What Broke
A broadcast loop formed on the VLAN carrying the storage cluster's internal traffic. The trigger was mundane. A switch port profile did not survive a reboot cleanly. The consequences were not mundane.
The loop starved traffic between the storage cluster's control nodes. Enough of them became unreachable, and the cluster lost quorum. No quorum meant no cluster. No cluster meant every VM disk hosted on the storage went dark.
This included the active OPNsense VM. It failed to boot. Without OPNsense, the default gateway disappeared from every VLAN. And now the tooling we used to reach the switch, push configuration management changes, or even reach the switch controller ran through a firewall unable to start.
We drew the circular dependency on the whiteboard:
Fix the network → need OPNsense running
Start OPNsense → need its storage cluster readable
Storage cluster healthy → need the storage VLAN clean
Clean the VLAN → need to reach the switch
Reach the switch → need the network working
Every link in the chain had its own independent justification when we built it. Put together, they formed a deadlock no amount of remote access was going to solve.
We got out by restoring the firewall VM from a backup snapshot onto a hypervisor's local disk, booting it there, and letting OPNsense's routing come back independently of the storage cluster. Once packets were flowing again, the cluster re-established quorum and the rest of the stack followed.
The incident was recoverable. The design behind it is not something we want to ship twice.
Lesson 1: Critical Infrastructure Belongs on Bare Metal
The right home for OPNsense, or any firewall, router, or DNS authority the rest of your platform depends on, is a dedicated physical box. Two of them, for anything production.
This is not a dig at virtualization. It is a statement about dependency layers. A firewall running as a VM inherits every failure mode of the hypervisor, the storage layer, the cluster networking, and the management plane. Each of those is a system you sometimes need to fix. Each of those fixes is easier if the firewall is independent of them.
Two bare metal boxes, not one. A single bare metal firewall is a single point of failure. Two bare metal firewalls running CARP give you:
- Independent hardware, independent power, independent NICs.
- Active/passive failover with no dependency on hypervisor live migration or shared storage.
- The ability to reboot one box for upgrades without announcing an outage window.
- No assumption about any other system in the rack being alive when the firewall needs to fail over.
Forget HA if you are thinking about backing this with network-defined storage. Shared storage makes live migration simple. It does not make your firewall more available. It gives your firewall a new way to die. A dead firewall then blocks every recovery you will ever do.
Application-layer HA (CARP between two OPNsense instances) is the right primitive here. It does the one thing you need: fail traffic over when the active node is gone. It does not care which hypervisor, which disk, or which cluster anything lives on.
Lesson 2: If It Must Be a VM, Use Local Storage
There are legitimate reasons to run OPNsense as a VM. Perhaps you are bootstrapping and the hardware budget has not arrived. Perhaps it is a lab or a secondary site where the blast radius is small. Perhaps your operational model is "everything is a VM" and you are not ready to break it.
Fine. Put the disk on local storage anyway.
The rule we now encode in our automation:
Bootstrap and recovery services run off local storage. Replication happens at the application layer, not the storage layer.
This means:
- Each OPNsense VM lives on its own hypervisor's local disk. Two VMs, two hosts, two disks, zero shared fate.
- The switch controller, which holds switch configuration, on local storage too. Any note saying "migrate to shared storage when healthy" is now the opposite: stay on local, always.
- Any DNS resolver, DHCP server, or bootstrap service you would need during a recovery follows the same rule.
What you give up is live migration. You lose one-click hypervisor evacuation. You have to plan a maintenance window and fail CARP over first.
This is a feature. It forces you to practice the failover you would otherwise only meet in an incident.
What Changed
After the outage we rewired the design to match what we now believe:
| Before | After |
|---|---|
| Firewall VMs on network-defined storage | Firewall VMs on host-local storage |
| HA via hypervisor shared storage | HA via CARP at the application layer |
| Switch controller on shared storage | Switch controller on local storage |
| Live-migrate firewalls between hosts | Fail CARP over, reboot host, fail back |
| Backups to a central backup target | Same backups, plus tested local restore |
The backup was what saved us. If you take nothing else from this post: restore your firewall backup to a local disk on a hypervisor it has never seen, and confirm it boots.
We have put the restore drill on the calendar quarterly. A backup you have not tested is not a backup. It is an assumption.
What About Bare Metal OPNsense?
For the designs we draw now, two small dedicated appliances run OPNsense on the metal. They share nothing but a link for CARP sync and an uplink to the switch. No hypervisor. No shared storage. No cluster filesystem. The firewall is the firewall.
Hardware cost is modest. The mental cost of knowing the firewall is independent of every other system is substantial.
If we had done this from day one, the incident above would have been a twenty-minute switch problem instead of an afternoon-long cluster recovery.
When Any of This Is Negotiable
A virtualized firewall on shared storage is defensible in one scenario: a lab, dev environment, or short-lived bootstrap phase. In those settings, a storage failure is a learning opportunity, not a customer incident.
Even then, treat it as temporary. Have a written plan to get off it. Do not let the convenience of live migration turn into a production commitment.
The Principle Underneath
Every outage like this one teaches the same thing: recovery services must not depend on the systems they recover.
Your firewall is a recovery service. So is your DNS resolver, your DHCP server, your bastion host, your SSH jump node, and the controller holding your switch config. Everything in the list needs to come up in a universe where nothing else works yet. If any of them depend on shared storage, a cluster filesystem, or a hypervisor needing its own network to boot, you have built a deadlock. It will fire eventually.
Bare metal where it matters. Local disks where it must be a VM. Application-layer HA, not storage-layer HA, for the things the rest of the platform leans on.
The next outage will still surprise us. It will not surprise us in this specific way again.
ZSoftly helps Canadian businesses design production network and private cloud infrastructure. If you want a second set of eyes on your dependency graph, talk to us.
