Docker in workspaces and volumes

Hello, I'm running container with Sysbox in Kubernetes based workspaces. I have persisted with Kubernetes PV the workspace /home/coder and /var/lib/docker directory. Developers run container inside their workspace with --rm option (docker run --rm) so while restarting the coder workspace, the nested developers container should be deleted. This working well except for anonymous volumes that are created with the developers container. While i'm restarting the workspace, and checking the /var/lib/docker/volumes the anonymous still there. These lead to full fill the Kubernetes PV. It's look like a problem of gracefull shutdown of the workspace that running docker container. Is anyone as encounter the same problem ?
10 Replies
Codercord
Codercord3mo ago
<#1275743565162741806>
Category
Bug report
Product
Coder OSS (v2)
Platform
Linux
Logs
Please post any relevant logs/error messages.
Phorcys
Phorcys3mo ago
yes, this makes sense as docker doesn't delete the volumes by default you should add a docker volume prune -f line to the coder_agent's shutdown_script if you want dangling volumes gone
Ottobus
OttobusOP3mo ago
Thanks, Good idea for the shutdown_script. But docker is supposed to deleted anonymous volume when --rm option is specified as mentionned here in the doc: Anonymous volumes are given a random name that's guaranteed to be unique within a given Docker host. Just like named volumes, anonymous volumes persist even if you remove the container that uses them, except if you use the --rm flag when creating the container, in which case the anonymous volume is destroyed (https://docs.docker.com/engine/storage/#volumes).
Docker Documentation
Manage data in Docker
Overview of persisting data in containers
Phorcys
Phorcys3mo ago
do the volumes get deleted if you gracefully shutdown the container that was started via docker run?
Ottobus
OttobusOP3mo ago
yes, here is an example while stopping the container by myself:
> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME

> docker run --rm --detach registry.internal/xxx/xxx:mariadb-10.5.23
c035e8cde666aa23267cabfecd8ce9ebb5ae18570b80334753f8181c9664d0b0

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c035e8cde666 registry.internal/xxx/xxx:mariadb-10.5.23 "docker-entrypoint.s…" 11 seconds ago Up 9 seconds 3306/tcp relaxed_pare

> docker volume ls
DRIVER VOLUME NAME
local 016b2500068df0e58ea1bfb4aa4e4b74e18c2a14b215bcf501e87201cfc5439b

> docker stop c035e8cde666
c035e8cde666

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME
> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME

> docker run --rm --detach registry.internal/xxx/xxx:mariadb-10.5.23
c035e8cde666aa23267cabfecd8ce9ebb5ae18570b80334753f8181c9664d0b0

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c035e8cde666 registry.internal/xxx/xxx:mariadb-10.5.23 "docker-entrypoint.s…" 11 seconds ago Up 9 seconds 3306/tcp relaxed_pare

> docker volume ls
DRIVER VOLUME NAME
local 016b2500068df0e58ea1bfb4aa4e4b74e18c2a14b215bcf501e87201cfc5439b

> docker stop c035e8cde666
c035e8cde666

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME
And here is with restarting the workspace:
> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME

> docker run --rm --detach --env registry.internal/xxx/xxx:mariadb-10.5.23
56d836cac3938cb7f080d459aff2e7f9076f89155020cc2dab4e9e45919c7bf9

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
56d836cac393 registry.internal/xxx/xxx:mariadb-10.5.23 "docker-entrypoint.s…" 11 seconds ago Up 10 seconds 3306/tcp intelligent_leavitt

> RESTARTING WORKSPACE
> docker volume ls
DRIVER VOLUME NAME
local d76dbe298f4dd9461b9f0a9f58d45bb9b4240cd50210123b6bbd38061ab18387
> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

> docker volume ls
DRIVER VOLUME NAME

> docker run --rm --detach --env registry.internal/xxx/xxx:mariadb-10.5.23
56d836cac3938cb7f080d459aff2e7f9076f89155020cc2dab4e9e45919c7bf9

> docker container ls --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
56d836cac393 registry.internal/xxx/xxx:mariadb-10.5.23 "docker-entrypoint.s…" 11 seconds ago Up 10 seconds 3306/tcp intelligent_leavitt

> RESTARTING WORKSPACE
> docker volume ls
DRIVER VOLUME NAME
local d76dbe298f4dd9461b9f0a9f58d45bb9b4240cd50210123b6bbd38061ab18387
Phorcys
Phorcys3mo ago
hey, could you try adding something like docker stop $(docker ps -a -q) (to stop all containers) in your shutdown_script to see if you get the intended behavior?
Ottobus
OttobusOP3mo ago
It's working well, I have add xagrs to manage case when there si no container. Thanks
Phorcys
Phorcys3mo ago
alright could you open an issue on GitHub to ask if there isn't something Coder should be doing to gracefully stop Docker?
Ottobus
OttobusOP3mo ago
GitHub
Gracefully stop Docker · Issue #14465 · coder/coder
Problem An issue occurred while stopping/starting or restarting workspaces that run docker container. It's look like, docker container aren't gracefully shutdown because anonymous volumes a...
Phorcys
Phorcys3mo ago
thanks a lot!
Want results from more Discord servers?
Add your server