I was always a fan of CACert but unfortunately, Gentoo recently decided to no longer trust CACert by default which caused our overlay to become unavailable for a few weeks without us noticing it (thanks for not notifying users about that unexpected move…). Gentoo thus forced us to switch to Let’s Encrypt immediately (or pay for a commercial certificate) if we wanted to keep our repository easily accessible. While we could have issued a LE certficate manually, that method is completely impractical as certificates issued by LE expire after just 90 days, while CACert’s certificates lasted 6 months or even 2 years if you passed the assurance tests.
Choosing an ACME client, aka “bot”
The implied requirement for automated renewal means you have to install a client for the so-called ACME protocol and allow it to generate a key & CSR, submit it to LE, somehow provide a domain-based verification, finally reload your server application and repeat that whole process periodically on its own. As we want to install certificates into an Apache web server but there’s no module to handle LE certificate issuance directly from within Apache yet, we have to use some tool to perform the necessary tasks for us. The first client published by LE themselves (“Certbot” or app-crypt/acme on Gentoo) had a terrible reputation as it was heavily modifying the system it was being installed on. Soon after the initial start last year, multiple alternate implementations became available, more or less system-specific and more or less disruptive in the way they hook into the system.
I wanted to avoid that. The client I would choose for our server should not manipulate any configs on its own and it shouldn’t install extra dependencies outside the regular package management provided by the system (portage on Gentoo). It should just handle certificate creation/registration and renewal, nothing else. And it should have a comprehensive documentation.
Judging from just the documentation, my personal favourites so far have been acme.sh and acmetool. While acme.sh would have had the big advantage that it requires basically no (unusual) dependencies as it’s “just a shell script”, that was also the reason why I decided against deploying it on my server. While there’s no rational reason against that script, I, as well as the friend I’m sharing the server with, had a gut instinct that didn’t allow us to trust an externally developed shell script to perform some periodic task in processing external data (i.e. interaction with LE’s ACME server). I therefore decided to try acmetool instead, which has been developed in Go.
This article thus details the installation of acmetool on Gentoo but most steps should apply to other clients as well.
Installing the acmetool ebuild
Unfortunately, acmetool doesn’t have an ebuild in the official portage tree yet. Luckily, jmesmon already wrote an ebuild we can use instead. If you haven’t used layman before, then please install it now as we have to add two overlays:
layman -a jmesmon layman -a go-overlay
Check what’s the latest stable release available as an ebuild (e.g. check with eix by running eix-update and eix acmetool). At the time of writing it was app-crypt/acmetool-0.0.58. As overlays generally only contain “unstable” ebuilds, you may need to add a keyword to /etc/portage/package.keywords:
~app-crypt/acmetool-0.0.58
Then, simply install it via emerge app-crypt/acmetool.
Configuring acmetool
Installation via portage has only given us a binary so far:
# equery files acmetool * Searching for acmetool ... * Contents of app-crypt/acmetool-0.0.58: /usr /usr/bin /usr/bin/acmetool /usr/share /usr/share/doc /usr/share/doc/acmetool-0.0.58 /usr/share/doc/acmetool-0.0.58/README.md.bz2
Now we need to perform a few more steps following the official user guide before we can start issueing certificates. We don’t trust acmetool in automated setup and execution in the same way we don’t trust most other server applications, so we are going for the (experimental) non-root setup instead of just running quickstart setup right away as root. If you’re more adventurous, you may skip this part but will end up using different paths and maybe run into some unexpected operational errors as I’ve not tested quickstart setup as root.
Preparing directories and setting up a system user
We need a directory for acmetool to keep its state and store certificates in. We will choose /var/lib/acme/ following the user guide. We then add a system user and group, both called acme and prepare a directory /usr/libexec/acme/hooks which will contain scripts/binaries acmetool will call throughout its various stages of operation. We set it to group-writable only for the duration of initial setup.
mkdir /var/lib/acme/ groupadd acme useradd --system -d /var/lib/acme/ -s /sbin/nologin -g acme acme chown acme:acme /var/lib/acme/ mkdir -p /usr/libexec/acme/hooks chown root:acme /usr/libexec/acme/hooks chmod g+w /usr/libexec/acme/hooks
The ACME protocol defines a few different methods to verify domain ownership, which is an essential (and for domain-based certificates the only) task in establishing trust to have a CA issue certificates for a claimed domain. Currently provided methods include running a dedicated server application, (temporarily) replacing any server running on ports 80 or 443, adding a token to the domain’s nameserver records or using some temporary certificate. Some methods may require adding custom challenge hooks. That sounds pretty cumbersome but as we want to get certificates for a web server anyway we choose the least sophisticated method of all which just requires us to serve a simple plain file at a location requested by the CA which has been standardized as /.well-known/acme-challenge/. The only important thing to note is that we need that directory accessible from all webspaces we want to request certificates for; also keep in mind that the verification process may have to be repeated automatically, so you may want to use one global directory instead of using many webspace/VHost-specific directories. We will configure Apache in a minute; for now, just create a directory to contain these files somewhere on your system where it makes sense – I chose to simply add it under /var/lib/acme again:
mkdir -p /var/lib/acme/.well-known/acme-challenge chown acme:acme /var/lib/acme/.well-known/acme-challenge
Permissions should be fine as they are by default (755) – our acme user needs permission to write to that directory and it should also be readable by our web server. It won’t contain any sensitive information so we can simply keep it globally readable.
Running quickstart setup
Finally, we can run the quickstart setup. Remember we took the effort to avoid having to run it with root privileges so we switch to our unprivileged acme user before we run the command:
su acme -s /bin/bash acmetool quickstart
The quickstart setup asks you a few questions which I’ve answered as follows:
- “Select ACME Server”: Choose “Let’s Encrypt (live)”
- “Select Challenge Conveyance Method”: Choose “WEBROOT”
- “Enter Webroot Path”: Set to
/var/lib/acme/.well-known/acme-challenge(or whatever directory you chose above) - if you deviated from acmetool’s default path (which we just did), a confirmation dialog will popup; check the path and confirm it (however, you could change it later by editing
/var/lib/acme/conf/target) - accept Let’s Encrypt’s Terms of Service
- if you want, enter an administrative email address in case there is any need for communication
- “Install auto-renewal cronjob?”: Choose “No” unless you want to install a user-specific crontab (I hate these… we are going to set up renewal manually)
Verifying hooks, securing the directory and allowing privileged access
After the wizard has finished, you should have some files in your hook directory (/usr/libexec/acme/hooks/). Check them and if you don’t like what you see, feel free to remove or replace them. The reload hook may be essential to have your server make use of auto-renewed certificates unless it auto-detects a change of certificates and activates them automatically (Apache doesn’t). The script I got with acmetool 0.0.58 looked fine (but don’t take my word for granted) and is configurable by defining a SERVICES variable in /etc/default/acme-reload but you will want to check yourself that it does what you want. Alternatively, you can write your own hooks.
Unfortunately, reloading server applications usually requires root privileges…
So, finally, make sure the hooks are not globally executable and (unless you have any reason to do otherwise) not writable by unprivileged users; basically we just want to enable our ACME client to run those scripts and no unprivileged user should be able to modify them:
chown -R root:acme /usr/libexec/acme/hooks/ chmod -R g-w,o-rwx /usr/libexec/acme/hooks/
According to the documentation, you should be able to add a sudo rule but it also says that acmetool would require an additional argument which I don’t see… So we are doing it the stupid way and enable the setuid bit; e.g.:
# run only if you understand the implications, read below #chmod u+s /usr/libexec/acme/hooks/reload
Enabling the setuid bit causes the flagged file to be executed as the user who owns the file, in this case root. This allows an unprivileged user who just needs permission to execute the file to launch it in the context of the file’s owner, i.e. with root privileges. Doing so may lead to serious security issues and is generally discouraged but, unfortunately, sometimes necessary. Double-check that the file you are enabling the setuid bit on is clean, not writable by unprivileged users, only executable by the minimum of users who require that privileged access and don’t enable it on anything you don’t understand. Remember that any other command run by that script will be executed with the same elevated privileges, so you need to check those as well.
Make sure you understand all implications before setting that flag and take a moment to think about possible alternatives.
The same precautions and warnings apply to defining a sudo rule but at least sudo can check a few more conditions.
If you decided to take that possible security risk, verify permissions and you should see permissions like that:
# ls -al /usr/libexec/acme/hooks/ total 20 drwxr-x--- 2 root acme 4096 Oct 30 14:50 . drwxr-xr-x 3 root root 4096 Oct 30 14:37 .. -rwxr-x--- 1 root acme 4316 Oct 30 14:50 haproxy -rwsr-x--- 1 root acme 1730 Oct 30 14:50 reload
Configuring Apache
We want to make /var/lib/acme/.well-known/acme-challenge/ be accessible as /.well-known/acme-challenge/ across all virtual hosts. Fortunately, there’s an easy way to accomplish that in Apache, you just need to decide where to add the configuration. For example, you could simply add it to /etc/apache2/httpd.conf but then you would have to merge your changes each time an update changes the original config file. Personally, I found it more convenient to add any custom global configuration in an additional file added to /etc/apache2/vhosts.d/ (although it’s not defining any VHosts at all) as that’s guaranteed to survive any world/Apache update. This may require that your VHosts use prefixes (such as 10_, 20_, …) to establish a specific order when Apache reads those files, so that our new directives come first.
Regardless of where you are going to put these lines, an easy way to serve our globally shared directory is to define a generic Alias and set permissions so that access to that directory by-passes any authentication you may have configured:
Alias /.well-known/acme-challenge/ /var/lib/acme/.well-known/acme-challenge/
<Directory /var/lib/acme/.well-known/acme-challenge>
Options None
AllowOverride None
Require all granted
</Directory>
You may have some reverse proxies configured or you may be using rewrite rules. In that case, you have to exclude the directory from being served from uninvolved backend servers and/or exclude it from rewrites. Depending on what you’re doing you may find the following directives useful which could go into the global config file; however, adding them to just the virtual hosts that need it seems more appropriate:
# if you're using ProxyPass, add before all other ProxyPass directives ProxyPass /.well-known/acme-challenge/ ! # if you're using RewriteRule, add before all other RewriteRule directives RewriteRule ^/?\.well-known/acme-challenge/ - [L]
Finally, test your config (/etc/init.d/apache2 configtest) and reload or restart Apache if it’s ok.
You will want to verify that your configuration actually works, so create some file in that directory. For example:
echo test >/var/lib/acme/.well-known/acme-challenge/test
Then check some of the domains you intend to create certificates for by accessing that same filename from a browser, e.g. http://your.domain/.well-known/acme-challenge/test
You need to get this working before you can register certificates using the “webhook” method.
Register a certificate
Guess what? We’re nearly done! Yay!!! :D (That was easy to accomplish, wasn’t it? No?! I know… :( No clue what they were thinking when they decided to restrict certificates to 90 day cycles, I guess it was a compromise they had to make in order to get a contract with a commercial, assured CA to effectively ruin the PKI business model and provide free certs for everyone…)
After all, we can finally request a certificate. If all goes well it’s just a matter of running a single command and wait for a few seconds while acmetool talks to the CA. Remember to run acmetool as user acme such as:
su acme -s /bin/bash -c '/usr/bin/acmetool want your.domain www.your.domain'
If you get any errors or just want to monitor what exactly is going on, add --xlog.severity=debug to get a verbose output. Also remember to check your web server logs (access & error logs) to see if either acmetool with its self-test or Let’s Encrypt has any issues retrieving the requested file. Otherwise acmetool should hopefully terminate without any output whatsoever.
Since this is our first attempt to register a certificate, check that /var/lib/acme/live/your.domain/ now contains at least cert, chain and privkey, which should be a symlink to some file in /var/lib/acme/keys/. Check that your private key is not readable by unwanted users (owner should be acme, permissions 600).
You can now simply add the certificate to your Apache configuration for those domains as you would normally do for any other CA:
SSLCertificateFile /var/lib/acme/live/your.domain/cert SSLCertificateKeyFile /var/lib/acme/live/your.domain/privkey SSLCertificateChainFile /var/lib/acme/live/your.domain/chain
Make sure to use the privkey symlink since the target may change on renewal. Reload your config and access the website with your browser. You may also want to check that you didn’t miss any hard to detect issues using the excellent SSL Server Test provided by Qualys, just make sure you don’t accidentally leave “Do not show the results on the boards” unchecked.
Setting up a cronjob for automated renewal
According to the user guide, we should only have to call acmetool --batch, so we simply configure a cronjob to run that command once per day or once every x hours… I thought it would be a good idea to run it every 6 hours, so our /etc/crontab now has a line like that:
54 */6 * * * acme /usr/bin/acmetool --batch
(Note: Since I just issued our first certificate today, we won’t know until the end of January if renewal really works. I will be getting back next year and update this post if I find anything was missing.)
And that’s it, we’re finally done…