It’s very easy to make a custom CA and issue certs. Here’s a good tutorial.

Unfortunately in practice It depends greatly on what’s on the other side (the client app). Some examples:

• DAVx5 on Android works perfectly fine and uses the client cert from the system store. 10/10, this is how all apps should work.
• Ntfy on Android works perfectly fine but wants the client cert file loaded in the app, it doesn’t use the one loaded in the system store. This sucks because instead of loading a cert into the system store once and then deleting it you have to keep the cert file around for this kind of apps, in Android shared storage, which is accessible to all apps.
• Same for Immich, wants the cert loaded in the app. Also, it will randomly lose it (on both iOS and Android). Yes, you heard that right. So it’s basically useless and I had to resort instead to a key in a custom HTTP header; which isn’t exactly the same as mTLS, but helps secure the service at reverse proxy level so it’s better than nothing.
• Firefox on Android will use the cert from the system store, and then it will crash. Again, useless.

It’s also not exactly straightforward to use mTLS with reverse proxies.

Let’s take for example Caddy and say you want unconditional mTLS for all reverse proxy hosts. Easy enough:

tls /path/to/domain-cert/fullchain.pem /path/to/domain-cert/privkey.pem {
  client_auth {
    mode required
    trust_pool file /path/to/custom/ca.pem
  }
}

But suppose you don’t want unconditional mTLS, you’d like to let clients in if they have mTLS or a custom header, or do different things depending if the client has valid mTLS or not. Does Caddy offer a built-in conditional to act on mTLS status? Nope!

As a workaround I’m setting the client_auth mode to verify_if_given and then using a DIY conditional that checks if the variable http.request.tls.client.certificate_der_base64 is empty or not. But it’s undocumented so who knows if it may break at any point.

For reference, how you handle both custom headers and mTLS at once (after setting the mode as I’ve mentioned):

@immich host "whatever.example.com"
handle @immich {
  @not_authorized {
    not header X-Custom-Pass "LONGRANDOMKEY01" # jim
    not header X-Custom-Pass "LONGRANDOMKEY02" # bob
    vars_regexp {http.request.tls.client.certificate_der_base64} ^$
  }
  error @not_authorized 403
  reverse_proxy http://immich.lan:port
}

The nested “not not” is required because Caddy can only do logical AND in group conditionals, so to do logical OR you basically have to do NOT (NOT a AND NOT b).