Network design. I started my homelab / selfhost journey about a year ago. Network design was the topic that scared me most. To challenge myself, and to learn about it, I bought myself a decent firewall box with 4 x 2.5G NICs. I installed OPNsense on it, following various guides. I setup my 3 LAN ports as a network bridge to connect my PC, NAS and server. I set the filtering to be applied between these different NICs, as to learn more about the behavior of the different services. If I want to access anything on my server from my PC, there needs to be a rule allowing it. All other trafic is blocked. This setup works great so far an I’m really happy with it.
Here is where I ran into problems. I installed Proxmox on my server and am in the process of migrating all my services from my NAS over there. I thought that all trafic from a VM in Proxmox would go this route: first VM --> OPNsense --> other VM. Then, I could apply the appropriate firewall rules. This however, doesnt seem to be the case. From what I’ve learned, VMs in Proxmox can communicate freely with each other by default. I don’t want this.
From my research, I found different ideas and opposing solutions. This is where I could use some guidance.
- Use VLANs to segregate the VMs from each other. Each VLAN gets a different subnet.
- Use the Proxmox firewall to prevent communication between VMs. I’d rather avoid this, so I don’t have to apply firewall rules twice. I could also install another OPNsense VM and use that, but same thing.
- Give up on filtering traffic between my PC, NAS and server. I trust all those devices, so it wouldn’t be the end of the world. I just wanted the most secure setup I could do with my current knowledge.
Is there any way to just force the VM traffic through my OPNsense firewall? I thought this would be easy, but couldn’t find anything or just very confusing ideas.
I also have a second question. I followed TechnoTim to setup Treafik and use my local DNS and wildcard certificates. Now, I can reach my services using service.local.example.com, which I think is neat. However, in order to do this, it was suggested to use one docker network called proxy. Each service would be assigned this network and Traefik uses lables to setup the routes. ’
Would’t this allow all those services to communciate freely? Normally, each container has it’s own network and docker uses iptables to isolate them from each other.
Is this still the way to go? I’m a bit overwhelmed by all those options.
Is my setup overkill? I’d love to hear what you guys think! Thank you so much!
VLANs all the way. I have several VLANs, including:
EDIT: An alternative would be to replace or supplement Proxmox with Docker/Podman on the bare metal of the server. The container networking would be isolated by default. If you can replace your VM needs with containers, that may get you what you want.
Thanks for your input. Am I understanding right, that all devices in one VLAN can communicate with each other without going through a firewall? Is that best practice? I’ve read so many different opinions that it’s hard to see.
You’re correct about vlan.
Think of vlan is a regular switch. Connect stuff, they communicate. Make two vlans in a switch, think of turning your physical switch into two separate switches.
Connect you switches to a router. Don’t want to waste two cables from your switch that’s cut in half? Do trunk port, with vlan tagged.
Lots of videos will explain better.
Best practice is to separate things of different trust levels into different vlans. You can filter and control the traffic between those vlans with your router.
As previously mentioned, in the Enterprise and business world, best practice is to separate and management from VMS and applications. We call this data plane and control plane. You would restrict access to your proxmox for other hypervisor interface from the VMS and applications. For small home setups and funsies, this gets a little complicated, but if it’s your career choice or interest, it’s a good thing to explore.
Key terms you can research: data plane, control plane, out of band management, air gapped.
I’ve read about those two destinctions but I am simply lacking the number of ports on my little firewall box. I still only allow access to management from my PC, nothing else - so I feel good enough here. This all is more a little project for me to tinker on, nothing serious.
You’re explanation with trust makes sense. I will simply keep my current setup but put different VMs on different VLANs. Then I can seperate my local services from my public services, as well as isolate any testing VMs.
I’ve read that one should use one proxy instance for local access and one for public services with internet access. Is it enough to just isolate that public proxy or must I also put the services behind that proxy into the DMZ?
Thank you for your good explantion.