The upgrade process went through without issues (it is always a good idea to backup your firewall configuration though, use the gui to export it as xml). The real trouble came after pfSense rebooted. We use the internal squid3 package as a reverse proxy to be able to map various services hosted on internal LAN servers for our external HTTPS port. This is a nice feature because you can set up your DNS entries to the same (wan) ip address, and then have squid do the mapping based on domain names, eg https://x.site.com -> machineA:443, https://y.site.com -> machineB:443, https://z.site:com -> machineB:9000, for example. All good.
Except that with this latest pfSense release, this squid mapping is completely broken. The reverse proxy was refusing to do its job, and our infrastrucutre dev servers were not accessible anymore from outside. Squid log in pf first showed that it could not bind to port 443 due to permission issues. We knew that this usually happens when
- service uid does not have privileges to bind
- another service already running on this port
- network stack is configured to disallow binding ports in a particular range
It was the third one, but it took us some digging until we ruled out the first two. First, when squid is stopped, the .pid file in
/var/run/ remains there, causing issues when you want to start it again. See this bug report about the issue.
We simply deleted the .pid file, and tried to start squid again, and this time we had a different error message:
Okay, no problem. This is easy, we just need to run sysctl command inside the BSD to set this value to 0 (wondering what was this value before the upgrade). Except that if you run this command, nothing happens. The operating system allows you only to specify port numbers above 1024, and we have no idea why (yes, we are root).
After digging some more internet pages, we ended up with this discussion, and quoting now the key comment:
Use old workaround instead until we find a way to fix it again. Listen squid on a high port and nat it from 80/443 to configured port.
So, it seems port binding below 1024 is broken, and the suggested solution is to configure your squid server to listen on port > 1024, (we used 4444) and use a NAT rule to forward from 443 to 4444. It is important to note, that you need to enter your external wan ip address as the NAT ip in this NAT rule, not the internal interface address. Phew! Service routing is working again!