Coder.com•5w ago

Docker in workspaces and volumes

Hello, I'm running container with Sysbox in Kubernetes based workspaces. I have persisted with Kubernetes PV the workspace /home/coder and /var/lib/docker directory. Developers run container inside their workspace with --rm option (docker run --rm) so while restarting the coder workspace, the nested developers container should be deleted. This working well except for anonymous volumes that are created with the developers container. While i'm restarting the workspace, and checking the /var/lib/docker/volumes the anonymous still there. These lead to full fill the Kubernetes PV. It's look like a problem of gracefull shutdown of the workspace that running docker container. Is anyone as encounter the same problem ?

10 Replies

Codercord•5w ago

<#1275743565162741806>

Category

Bug report

Product

Coder OSS (v2)

Platform

Linux

Logs

Please post any relevant logs/error messages.

Phorcys•5w ago

yes, this makes sense as docker doesn't delete the volumes by default you should add a docker volume prune -f line to the coder_agent's shutdown_script if you want dangling volumes gone

Ottobus•4w ago

Thanks, Good idea for the shutdown_script. But docker is supposed to deleted anonymous volume when --rm option is specified as mentionned here in the doc:

Anonymous volumes are given a random name that's guaranteed to be unique within a given Docker host. Just like named volumes, anonymous volumes persist even if you remove the container that uses them, except if you use the --rm flag when creating the container, in which case the anonymous volume is destroyed

(https://docs.docker.com/engine/storage/#volumes).

Docker Documentation

Manage data in Docker

Overview of persisting data in containers

Phorcys•4w ago

do the volumes get deleted if you gracefully shutdown the container that was started via docker run?

Ottobus•4w ago

yes, here is an example while stopping the container by myself:

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

> docker run --rm --detach registry.internal/xxx/xxx:mariadb-10.5.23
c035e8cde666aa23267cabfecd8ce9ebb5ae18570b80334753f8181c9664d0b0

> docker container ls --all                                                                                                 
CONTAINER ID   IMAGE                                                              COMMAND                  CREATED          STATUS         PORTS      NAMES
c035e8cde666   registry.internal/xxx/xxx:mariadb-10.5.23   "docker-entrypoint.s…"   11 seconds ago   Up 9 seconds   3306/tcp   relaxed_pare

> docker volume ls                                                                                                          
DRIVER    VOLUME NAME
local     016b2500068df0e58ea1bfb4aa4e4b74e18c2a14b215bcf501e87201cfc5439b

> docker stop c035e8cde666
c035e8cde666

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

> docker run --rm --detach registry.internal/xxx/xxx:mariadb-10.5.23
c035e8cde666aa23267cabfecd8ce9ebb5ae18570b80334753f8181c9664d0b0

> docker container ls --all                                                                                                 
CONTAINER ID   IMAGE                                                              COMMAND                  CREATED          STATUS         PORTS      NAMES
c035e8cde666   registry.internal/xxx/xxx:mariadb-10.5.23   "docker-entrypoint.s…"   11 seconds ago   Up 9 seconds   3306/tcp   relaxed_pare

> docker volume ls                                                                                                          
DRIVER    VOLUME NAME
local     016b2500068df0e58ea1bfb4aa4e4b74e18c2a14b215bcf501e87201cfc5439b

> docker stop c035e8cde666
c035e8cde666

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

And here is with restarting the workspace:

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

> docker run --rm --detach --env registry.internal/xxx/xxx:mariadb-10.5.23
56d836cac3938cb7f080d459aff2e7f9076f89155020cc2dab4e9e45919c7bf9

> docker container ls --all                                                                                                 
CONTAINER ID   IMAGE                                                              COMMAND                  CREATED          STATUS          PORTS      NAMES
56d836cac393   registry.internal/xxx/xxx:mariadb-10.5.23   "docker-entrypoint.s…"   11 seconds ago   Up 10 seconds   3306/tcp   intelligent_leavitt

> RESTARTING WORKSPACE
> docker volume ls                                                                                                          
DRIVER    VOLUME NAME
local     d76dbe298f4dd9461b9f0a9f58d45bb9b4240cd50210123b6bbd38061ab18387

> docker container ls --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

> docker volume ls         
DRIVER    VOLUME NAME

> docker run --rm --detach --env registry.internal/xxx/xxx:mariadb-10.5.23
56d836cac3938cb7f080d459aff2e7f9076f89155020cc2dab4e9e45919c7bf9

> docker container ls --all                                                                                                 
CONTAINER ID   IMAGE                                                              COMMAND                  CREATED          STATUS          PORTS      NAMES
56d836cac393   registry.internal/xxx/xxx:mariadb-10.5.23   "docker-entrypoint.s…"   11 seconds ago   Up 10 seconds   3306/tcp   intelligent_leavitt

> RESTARTING WORKSPACE
> docker volume ls                                                                                                          
DRIVER    VOLUME NAME
local     d76dbe298f4dd9461b9f0a9f58d45bb9b4240cd50210123b6bbd38061ab18387

Phorcys•4w ago

hey, could you try adding something like docker stop $(docker ps -a -q) (to stop all containers) in your shutdown_script to see if you get the intended behavior?

Ottobus•4w ago

It's working well, I have add xagrs to manage case when there si no container. Thanks

Phorcys•4w ago

alright could you open an issue on GitHub to ask if there isn't something Coder should be doing to gracefully stop Docker?

Ottobus•4w ago

Ok: https://github.com/coder/coder/issues/14465

GitHub

Gracefully stop Docker · Issue #14465 · coder/coder

Problem An issue occurred while stopping/starting or restarting workspaces that run docker container. It's look like, docker container aren't gracefully shutdown because anonymous volumes a...

Phorcys•4w ago

thanks a lot!

Gaming

Programming

Docker in workspaces and volumes