You can absolutely use it without a reverse proxy. A proxy is just another fancy HTTP client that contacts the server on the original client’s behalf and forwards the response back to it, usually wrapped in HTTPS.

All you have to do is expose the desired port(s) to all addresses:

# ...
  - ports:
    - 8080:8080

…and obviously to set the URL environment variables to localhost or whatever address the server uses.