pfSense 2.5.x + LetsEncrypt + haProxy – Proper mitigation of expiring LE-Intermediate-CA

Assuming you read the title of this article, it is very likely you are running the same setup I am: pfsense with haproxy as reverse-proxy for various webservices hosted from a single shared (probably domestic) IP, together with globaly trusted LE-Certs to do ssl-offloading for some sites (that was a long one).

If you do so, you might have encountered the same problem as I do: The old intermediate CA (the one with R3 in the name) of LetsEncrypt is expiring, and pfSense (note that this currently only applies to 2.5.x, 2.4.5 did just not notify you about the expiry) will send you mails (if properly configured) and notifications one month prior to expiry:

If you search this problem, the general advice is „just delete the old CA“, but I wouldn’t, because you might experience a bad wake-up, at least if still certs are bound to the old intermediate…

My pfSense noticed me yesterday, and i did as noticed above. The result was, that haproxy won’t start anymore, because it  was missing some trusted intermediate-certs. So i gladly backuped the old intermediate before deleting it, so I imported it again (in the screenshot noted as „le-old“):As you can see, it has certificates attached to it (3 in this case), which is the Problem here. If you have updated your pfSense, you might as well have another Intermediate with the title „Acmecert: O=Let’s Encrypt, CN=R3, C=US“, but I stayed on 2.5.0 (as of writing this, 2.5.2 is out), because the handling of wireguard changed in this versions, and i did not have migrated yet. If you updated away from 2.5.0, this problem might not affect you.

If you are still on 2.5.0 like me, you can update the acme-package in pfSense manually. While this is in general not be considered as a good idea, it does not have a lot of impact here. It has php as dependency, which will likely update to a newer minor version also, but – as far as i can tell – updating acme to a newer version did not had any negative impact so far.

Edit: Meanwhile I updated pfSense to 2.5.2, and there have been no problems resulting from updating acme prior to this manually.

Your update procedure should be as follows:

Update the acme-plugin. You should now have two R3-Intermediate certs, from which one is about to expire. For conveniance i renamed this one to „le-old“:As you might notice, there are still certs attached to it, so go to your amce-config and re-issue one site. After this, come back to System/Certificate Manager/CAs, and verify, that now at least one cert is attached to the new intermediate, under „certificates“ you can determine the exact relationship. If (and only if) this succeeded for one cert, you now should restart haproxy manually, so that the webservice behind that cert now delivers it correctly. You should check if the webservice has the new Cert, and if the new Cert is using the new intermediate.

If this procedure has worked for all certificates, the old intermediate should have now zero certificated issued. Anyways, you should restart haproxy (verifing that it will still run), backup the old intermediate now, delete it from the Web-UI and then again restart haproxy. Note, that haproxy won’t throw you an error-Message on failure, you have to examine the system log, and pay attention to the buttons. If you see these circle-shaped arrow, that means the service is running, and you can restart it. a „Play“-button indicated, that the service is stopped, and can be run by clicking the button. Always make sure haproxy is running before proceeding th the next step (having the haproxy-statistics-site open in another tab might be useful)!

You really should restart haproxy after every step. On one hand, you can verify, that it is still running, on the other hand, it caches certs, and you need to restart it when reissueing a new one.

Lessons learned: if I would have migrated some month ago, and all issued certs were already attached to the new intermediate, this would most likely did not affect me, and „just delete it“ would have been the answer also for me.