Today we upgraded our pfSense firewall to 2.2 as we want to use L2TP/IPSec VPN to access our internal machines from outside. The release notes states that this feature is finally here.

The upgrade process went through without issues (it is always a good idea to backup your firewall configuration though, use the gui to export it as xml). The real trouble came after pfSense rebooted. We use the internal squid3 package as a reverse proxy to be able to map various services hosted on internal LAN servers for our external HTTPS port. This is a nice feature because you can set up your DNS entries to the same (wan) ip address, and then have squid do the mapping based on domain names, eg https://x.site.com -> machineA:443, https://y.site.com -> machineB:443, https://z.site:com -> machineB:9000, for example. All good.

Except that with this latest pfSense release, this squid mapping is completely broken. The reverse proxy was refusing to do its job, and our infrastrucutre dev servers were not accessible anymore from outside. Squid log in pf first showed that it could not bind to port 443 due to permission issues. We knew that this usually happens when

  • service uid does not have privileges to bind
  • another service already running on this port
  • network stack is configured to disallow binding ports in a particular range

It was the third one, but it took us some digging until we ruled out the first two. First, when squid is stopped, the .pid file in /var/run/ remains there, causing issues when you want to start it again. See this bug report about the issue.

We simply deleted the .pid file, and tried to start squid again, and this time we had a different error message:

The following input errors were detected:

The field 'reverse HTTPS port' must contain a port number higher than net.inet.ip.portrange.first sysctl value(1024).
To listen on low ports, change portrange.first sysctl value to 0 on system tunable options and restart squid daemon.1

Okay, no problem. This is easy, we just need to run sysctl command inside the BSD to set this value to 0 (wondering what was this value before the upgrade). Except that if you run this command, nothing happens. The operating system allows you only to specify port numbers above 1024, and we have no idea why (yes, we are root).

$ sysctl net.inet.ip.portrange.first=0
net.inet.ip.portrange.first: 1024 -> 1024

$ sysctl net.inet.ip.portrange.first=1200
net.inet.ip.portrange.first: 1024-> 1200

$ sysctl net.inet.ip.portrange.first=0
net.inet.ip.portrange.first: 1200 -> 1024

After digging some more internet pages, we ended up with this discussion, and quoting now the key comment:

Use old workaround instead until we find a way to fix it again. Listen squid on a high port and nat it from 80/443 to configured port.

So, it seems port binding below 1024 is broken, and the suggested solution is to configure your squid server to listen on port > 1024, (we used 4444) and use a NAT rule to forward from 443 to 4444. It is important to note, that you need to enter your external wan ip address as the NAT ip in this NAT rule, not the internal interface address. Phew! Service routing is working again!