Trouble moving a container and its data to another server

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨Natal@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

Hello again, I hope it’s okay that I make several posts in a rather short time, I’m stumped.

I run a series of containers on old computer A for the recipe manager Tandoor Recipes.

I want to move it to another computer B so I initially thought I would: -copy the env and docker-compose -dump the source database

Move everything to the new computer, compose everything and fill the database from the dump.

I got 500 server errors so I went on Discord and asked what was the proper way of doing this. I’ve been told in theory I could shut everything down, pack my Tandoor folder in a zip, paste it on the target computer B and boot everything and voilà.

None of this works properly.

I do manage to get an instance of Tandoor running on my new computer, it displays every recipe I had originally, but it has an issue when trying to create a new recipe. I get a white page with “Server Error (500)”. It does not happen on the original Tandoor, despite being all the same files in theory.

I noticed that on my source computer, the postgres DB directory changes permission when I start the container, as well as the directory containing the recipe pictures. So I’m wondering if wrong permissions might be corrupting data while copying stuff?

Thanks

source

Comments

Sort:hotnew top

OneCardboardBox@lemmy.sdf.org ⁨1⁩ ⁨year⁩ ago
Agree with the other user that using tar and then zipping the tarball will preserve permissions.

Alternatively, you could open an ssh connection and rsync the files between them.

Last option would be to export your recipes from the old host, build a new container from scratch on the new one, and import them back again: docs.tandoor.dev/features/import_export/

source
- Natal@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I thought so, too. I played around with the tar idea and it led me to discover the permissions are wrong on the original computer. I have added new info on the main post about it but basically the owner is “user:70” and I have no idea where that comes from. I tried using CHOWN to reset everything to my own user, which is the same on both computers (1000:1000 uidguid), but whenever I restart the container, it locks again.
  
  source
  - OneCardboardBox@lemmy.sdf.org ⁨1⁩ ⁨year⁩ ago
    The user and group id inside the container doesn’t have to match any user on your host machine. It’s possible that user:70 is configured as the user to launch inside the container, in which case you should set the ownership of the directory to match what the container expects.
    
    Eg: The container for my torrent client runs as user 700 group 700. My host machine does not have either of those IDs defined. My torrent directory must be chown’d by 700:700 or else the container can’t read/write torrents.
    
    source
- NullPointerException@lemm.ee ⁨1⁩ ⁨year⁩ ago
  agreed, good find with that import/export feature, didn’t think to check for that since I don’t use tandoor. personally I’d just use the native import/export feature though since it feels less hacky than copying the whole DB imo. but if OP still wants to copy the files over, it might be easier to either just tar or tar + gzip rather than tar + zip so they can do it all in one command.
  
  source
m@social.tthi.as ⁨1⁩ ⁨year⁩ ago
I've done a couple of host migrations since using Docker for all my services.
I don't even bother with database dumps or anything like that, I copy my compose files and mapped directories, being sure to preserve permissions, and all my services come back up without any issues.
source
- outcide@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Just make sure that the container is shutdown before you copy the database or you can getcorruption.
  
  source
- Natal@lemmy.world ⁨1⁩ ⁨year⁩ ago
  That’s basically what the author of the app told me to do. I’m having wrong permission issues I believe and I haven’t found yet who/what assigns the wrong permission when I start the container.
  
  source
NullPointerException@lemm.ee ⁨1⁩ ⁨year⁩ ago
you’re probably right about it being a permissions error but also including some relevant logs would be useful (or just stick all of them on pastebin or something and give us a link). could also be a docker misconfiguration considering it’s a different system and I assume they’re not identically set up with Ansible or something similar

source
- Natal@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I tried using tar as you said, and it didn’t work. Which led me to investigate and realize the owner of the postgres folder is unknown to me. Changed it back with CHOWN but it reverts back to the weird owner when I restart the container so I’m missing some knowledge and know-how. I’m trying to figure out who sets that permission in the first place.
  
  source
  - NullPointerException@lemm.ee ⁨1⁩ ⁨year⁩ ago
    I checked out your edit and what’s going on is pretty weird. my recommendation (if you don’t wanna just use the native import/export feature and give up on DB dumps) is to (on the new server) create a minimal tandoor docker-compose.yml, run it, do minimal setup, see if it works, and if the fresh container works, check the permissions, then just replicate those perms for your other postgres directory that contains data you actually care about. good luck!
    
    source
CumBroth@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
As others have already mentioned, you are probably correct that it’s a permission error. You could follow the already posted advice to use tools that maintain permissions like rsync, but fixing this botched backup manually could help you learn how to deal with permissions and that’s a rather fundamental concept that anyone selfhosting would benefit from understanding.

If you decide to do this, I would recommend reading up on the concept of user and group permissions on linux and the commands that allow you to inspect ownership and permissions of directories and files as well as the UID and GID of users. Next step would be to understand how Docker handles permissions for mapped directories. You can get a few pointers from this short explanation by LSIO: docs.linuxserver.io/…/understanding-puid-and-pgid. Bear in mind that this is not a Docker standard, but something specific to LSIO Docker images. See also docs.docker.com/compose/…/05-services/#long-synta…. This can also be set when using docker run by using the –user flag.

Logs can also help pinpoint the source of the issue. The default docker compose setup in Tandoor’s docs sets up several containers, one of which acts as a database (db_recipes based on postgres:15-alpine). Inspect that in real time using docker logs -f db_recipes to see the exact errors.

source
- Natal@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Thanks a lot! I certainly need to learn about permissions and docker mapped directories in general. This is still very unclear in my head and it prevents me from troubleshooting my own stuff which is frustrating. You’re all very cool but I’d like to not post a lemmy every time an app has wrong permissions haha. I’ll have a read.
  
  source
  - CumBroth@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
    In response to your update: Try specifying the user that’s supposed to own the mapped directories in the docker compose file. Then make sure the UID and GID you use match an existing user on the new system you are testing the backup on.
    
    First you need to get the id of the user you want to run the container as. For a user called foo, run id foo. Note down the UID and GID.
    
    Then in your compose file, modify the db_recipes service definition and set the UID and GID of the user that should own the mapped volumes:
    
    db_recipes: restart: always image: postgres:15-alpine user: "1000:1000" # Replace this with the correct UID and GID matching your user volumes: - ./postgresql:/var/lib/postgresql/data env_file: - ./.env
    
    Recreate the container using docker compose up -d (don’t just restart it; you need to load the new config from the docker compose file). Then inspect the postgresql directory using ls -l to check whether it’s actually owned by user with UID 1000 and group with GID 1000. This should solve the issue you are having with that backup program you’re using. It’s probably unable to copy that particular directory because it’s owned by root:root and you’re not running it as root (don’t do that; it would circumvent the real problem rather than help you address it).
    
    Now, when it comes to copying this to another machine, as already mentioned you could use something that preserves permissions like rsync, but for learning purposes I’d intentionally do it manually as you did before to potentially mess things up. On the new machine, repeat this process. First find the UID and GID of the current non-root user (or whatever user you want to run your containers as). Then make sure that UID and GID are set in the compose files. Then inspect the directories to make sure they have the correct ownership. If the compose file isn’t honoring the user flag or if the ownership doesn’t match the UID and GID you set for whatever reason, you can also use chown -R ./postgresql to change ownership.
    
    source
Edo78@feddit.it ⁨1⁩ ⁨year⁩ ago
If you are able to see every recipe I’d rule out data corruption. If the issue arise only when try to create a new recipe (or editing an old one) the usual thing to check are permissions. Don’t zip but use some tools to maintain permissions (tar, rsync, etc) and try again

source
- Natal@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Good idea! I tried sync with my other computer and the software complained about not having permissions on the postgres folder. I investigated and found out postgres directory is owned by user:70. I have yet to find where that comes from. I changed it with a sudo chown but it reverts to the wrong owner when I restart the container.
  
  source