Coder.com•7mo ago

Recovering from disaster

There has been a disaster with our hypervisor that runs our Coder-OSS server. We installed Coder on Ubuntu and used it with its default settings, including the built-in PostgreSQL database. The disaster has caused issues with the Coder server. Although we can still run coder server, attempting to access the dashboard yields the following error:

Internal error fetching workspace resources.
get workspace builds: pq: missing chunk number 0 for toast value 124531 in pg_toast_16691

Internal error fetching workspace resources.
get workspace builds: pq: missing chunk number 0 for toast value 124531 in pg_toast_16691

Is there a way to fix this? If not, how can we back up the workspaces and import them into a new Coder server?

Solution:

I was able to successfully restore the SQL dump to the Coder database that is built-in. The process was completed using:

PGPASSWORD=$(cat /home/coder/.config/coderv2/postgres/password) psql -h localhost -p $(cat /home/coder/.config/coderv2/postgres/port) -U coder -d coder -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" -f coder.sql

...

Jump to solution

8 Replies

Codercord•7mo ago

<#1235445393920229420>

Category

Other

Product

Coder OSS (v2)

Platform

Linux

Logs

Please post any relevant logs/error messages.

Atif•7mo ago

Do you have any previous backups of the postgress database?

kerxkerbyOP•7mo ago

We made a mistake as we forgot to back up the database. The login credentials for the accounts that created the workspaces are still valid (as it seems since it can still log in). Can we list all docker containers (workspaces) and back up their data? Hi @Atif, Suppose we have access to the original built-in PostgreSQL, what should we back up? Should we only back up the "coder" database? Also, what is the proper way to restore it? I tried to restore it by running coder server and connecting to the SQL server, but the response was not successful. How do we run /home/coder/.config/coderv2/postgres so the client can be connected and be able to fix the issues.

Atif•7mo ago

https://coder.com/docs/v2/latest/admin/configure#migrating-from-the-built-in-database-to-an-external-database

Configuration - Coder v2 Docs

Learn how to configure Coder

Atif•7mo ago

See if this can help

Solution

kerxkerby•7mo ago

I was able to successfully restore the SQL dump to the Coder database that is built-in. The process was completed using:

PGPASSWORD=$(cat /home/coder/.config/coderv2/postgres/password) psql -h localhost -p $(cat /home/coder/.config/coderv2/postgres/port) -U coder -d coder -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public;" -f coder.sql

kerxkerbyOP•7mo ago

There are still some issues that need to be addressed, one of which is that Docker is currently unable to run.

May 03 01:21:17 coder systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 4546 and the job result is failed.

May 03 01:21:17 coder systemd[1]: Failed to start Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit docker.service has finished with a failure.
░░
░░ The job identifier is 4546 and the job result is failed.

Btw, I found a copy of our Coder-OSS server from a SATA drive. I'm curious if there's a way to backup the Docker workspaces to the new Coder-OSS server. I'm planning to execute: sudo rsync -avhP --delete -e "ssh -l coder" /var/lib/docker/ 192.168.193.61:/var/lib/docker/ Are there any important details I may have missed?

Atif•7mo ago

I think if all your workspaces are in docker that should do it Given you already have backed up and restored the coder postgress DB

Gaming

Programming

Recovering from disaster