I have a similar setup.

Getting the DNS to return the right addresses is easy enough: you just set your records for subdomain * instead a specific subdomain, and then any subdomain that’s not explicitly configured will default to using the records for *.

Assuming you want to use Let’s Encrypt (or another ACME CA) you’ll probably want to make sure you use an ACME client that supports your DNS provider’s API (or switch DNS provider to one that has an API your client supports). That way you can get wildcard TLS certificates (so individual subdomains won’t still leak via Certificate Transparency logs). Configure your ACME client to use the Let’s Encrypt staging server until you see a wildcard certificate on your domains.

Some other stuff you’ll probably want:

• A reverse proxy to handle requests for those subdomains. I use Caddy, but basically any reverse proxy will do. The reason I like Caddy is that it has a built-in ACME client as well as a bunch of plugins for DNS providers including my preferred one. It’s a bit tricky to set this up with wildcard certificates (by default it likes to request individual subdomain certificates), but I got it working and it’s been running very smoothly since.
• To put a login screen before each service I’ve configured Caddy to only let visitors through to the real pages (or the error page, for unconfigured domains) if Authelia agrees.