Coder.com•9mo ago

Goroutine dump with Coder V2 on Fly.io

Hey all! I'm deploying Coder to Fly.io, with an external database, following this (https://coder.com/blog/remote-developer-environments-on-fly-io) guide. I'm getting a very long Goroutine dump, and it goes on for around ~40-50 lines. I will try and get a sample after dinner, and sanitise it. However, I was wondering if anyone else using Fly had a similar issue? Thanks!

Coder

Remote developer environments on Fly.io

Coder moves developer workspaces off local machines and into your cloud infrastructure, centralizing their management and safeguarding your code and data.

18 Replies

Codercord•9mo ago

<#1217203906979954709>

Category

Bug report

Product

Coder OSS (v2)

Platform

Linux

Logs

Please post any relevant logs/error messages.

Dom R (shymega)OP•9mo ago

Got some logs. I'm not really sure where the cause is here:

/opt/hostedtoolcache/go/1.21.5/x64/src/os/signal/signal.go:287 +0x165
  goroutine 22 [select]:
  runtime.gopark(0xc000074f88?, 0x2?, 0x0?, 0x0?, 0xc000074f84?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc000074e30 sp=0xc000074e10 pc=0x43e46e
  runtime.selectgo(0xc000074f88, 0xc000074f80, 0x4088b7?, 0x0, 0x0?, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/select.go:327 +0x725 fp=0xc000074f50 sp=0xc000074e30 pc=0x44e9e5
  database/sql.(*DB).connectionOpener(0xc00057b380, {0x95b6d40, 0xc00042a370})
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:1218 +0x87 fp=0xc000074fb8 sp=0xc000074f50 pc=0xb1f0e7
  database/sql.OpenDB.func1()
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:791 +0x28 fp=0xc000074fe0 sp=0xc000074fb8 pc=0xb1d508
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x471e21
  created by database/sql.OpenDB in goroutine 1
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:791 +0x165
  goroutine 44 [select]:
  runtime.gopark(0xc000647e68?, 0x2?, 0x80?, 0xb3?, 0xc000647dcc?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc0005b4bf0 sp=0xc0005b4bd0 pc=0x43e46e
  runtime.selectgo(0xc0005b4e68, 0xc000647dc8, 0xf?, 0x0, 0x0?, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/select.go:327 +0x725 fp=0xc0005b4d10 sp=0xc0005b4bf0 pc=0x44e9e5
  github.com/coder/coder/v2/coderd/database/pubsub.(*Watchdog).publishLoop(0xc000584420)
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:91 +0x27a fp=0xc0005b4fc8 sp=0xc0005b4d10 pc=0x21ca5ba
  github.com/coder/coder/v2/coderd/database/pubsub.NewWatchdogWithClock.func1()
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:54 +0x25 fp=0xc0005b4fe0 sp=0xc0005b4fc8 pc=0x21ca2a5
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005b4fe8 sp=0xc0005b4fe0 pc=0x471e21
  created by github.com/coder/coder/v2/coderd/database/pubsub.NewWatchdogWithClock in goroutine 1
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:54 +0x1f6
  goroutine 37 [chan receive]:
  runtime.gopark(0xc00042c8a0?, 0xc00042c840?, 0xe0?, 0x5c?, 0xc0005dfed0?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc0005b5e80 sp=0xc0005b5e60 pc=0x43e46e
  runtime.chanrecv(0xc00042ce40, 0xc0005dff98, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/chan.go:583 +0x3cd fp=0xc0005b5ef8 sp=0xc0005b5e80 pc=0x4099cd
  runtime.chanrecv2(0xc1744aff8b3d4f2d?, 0x4720a06c?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/chan.go:447 +0x12 fp=0xc0005b5f20 sp=0xc0005b5ef8 pc=0x4095f2
  github.com/lib/pq.(*Listener).listenerConnLoop(0xc000455ce0)
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:837 +0x26e fp=0xc0005b5fb0 sp=0xc0005b5f20 pc=0x174ab0e
  github.com/lib/pq.(*Listener).listenerMain(0xc000455ce0)
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:856 +0x18 fp=0xc0005b5fc8 sp=0xc0005b5fb0 pc=0x174ab58
  github.com/lib/pq.NewDialListener.func1()
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:525 +0x25 fp=0xc0005b5fe0 sp=0xc0005b5fc8 pc=0x1749425
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005b5fe8 sp=0xc0005b5fe0 pc=0x471e21
  created by github.com/lib/pq.NewDialListener in goroutine 1
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:525 +0x187
  goroutine 41 [IO wait]:
  runtime.gopark(0x191?, 0xb?, 0x0?, 0x0?, 0x8?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc000078c20 sp=0xc000078c00 pc=0x43e46e
  runtime.netpollblock(0x488578?, 0x407e06?, 0x0?)

/opt/hostedtoolcache/go/1.21.5/x64/src/os/signal/signal.go:287 +0x165
  goroutine 22 [select]:
  runtime.gopark(0xc000074f88?, 0x2?, 0x0?, 0x0?, 0xc000074f84?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc000074e30 sp=0xc000074e10 pc=0x43e46e
  runtime.selectgo(0xc000074f88, 0xc000074f80, 0x4088b7?, 0x0, 0x0?, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/select.go:327 +0x725 fp=0xc000074f50 sp=0xc000074e30 pc=0x44e9e5
  database/sql.(*DB).connectionOpener(0xc00057b380, {0x95b6d40, 0xc00042a370})
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:1218 +0x87 fp=0xc000074fb8 sp=0xc000074f50 pc=0xb1f0e7
  database/sql.OpenDB.func1()
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:791 +0x28 fp=0xc000074fe0 sp=0xc000074fb8 pc=0xb1d508
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x471e21
  created by database/sql.OpenDB in goroutine 1
    /opt/hostedtoolcache/go/1.21.5/x64/src/database/sql/sql.go:791 +0x165
  goroutine 44 [select]:
  runtime.gopark(0xc000647e68?, 0x2?, 0x80?, 0xb3?, 0xc000647dcc?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc0005b4bf0 sp=0xc0005b4bd0 pc=0x43e46e
  runtime.selectgo(0xc0005b4e68, 0xc000647dc8, 0xf?, 0x0, 0x0?, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/select.go:327 +0x725 fp=0xc0005b4d10 sp=0xc0005b4bf0 pc=0x44e9e5
  github.com/coder/coder/v2/coderd/database/pubsub.(*Watchdog).publishLoop(0xc000584420)
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:91 +0x27a fp=0xc0005b4fc8 sp=0xc0005b4d10 pc=0x21ca5ba
  github.com/coder/coder/v2/coderd/database/pubsub.NewWatchdogWithClock.func1()
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:54 +0x25 fp=0xc0005b4fe0 sp=0xc0005b4fc8 pc=0x21ca2a5
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005b4fe8 sp=0xc0005b4fe0 pc=0x471e21
  created by github.com/coder/coder/v2/coderd/database/pubsub.NewWatchdogWithClock in goroutine 1
    /home/runner/actions-runner/_work/coder/coder/coderd/database/pubsub/watchdog.go:54 +0x1f6
  goroutine 37 [chan receive]:
  runtime.gopark(0xc00042c8a0?, 0xc00042c840?, 0xe0?, 0x5c?, 0xc0005dfed0?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc0005b5e80 sp=0xc0005b5e60 pc=0x43e46e
  runtime.chanrecv(0xc00042ce40, 0xc0005dff98, 0x1)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/chan.go:583 +0x3cd fp=0xc0005b5ef8 sp=0xc0005b5e80 pc=0x4099cd
  runtime.chanrecv2(0xc1744aff8b3d4f2d?, 0x4720a06c?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/chan.go:447 +0x12 fp=0xc0005b5f20 sp=0xc0005b5ef8 pc=0x4095f2
  github.com/lib/pq.(*Listener).listenerConnLoop(0xc000455ce0)
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:837 +0x26e fp=0xc0005b5fb0 sp=0xc0005b5f20 pc=0x174ab0e
  github.com/lib/pq.(*Listener).listenerMain(0xc000455ce0)
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:856 +0x18 fp=0xc0005b5fc8 sp=0xc0005b5fb0 pc=0x174ab58
  github.com/lib/pq.NewDialListener.func1()
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:525 +0x25 fp=0xc0005b5fe0 sp=0xc0005b5fc8 pc=0x1749425
  runtime.goexit()
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0005b5fe8 sp=0xc0005b5fe0 pc=0x471e21
  created by github.com/lib/pq.NewDialListener in goroutine 1
    /home/runner/go/pkg/mod/github.com/lib/[email protected]/notify.go:525 +0x187
  goroutine 41 [IO wait]:
  runtime.gopark(0x191?, 0xb?, 0x0?, 0x0?, 0x8?)
    /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:398 +0xce fp=0xc000078c20 sp=0xc000078c00 pc=0x43e46e
  runtime.netpollblock(0x488578?, 0x407e06?, 0x0?)

Apologies for the huge block, it goes on for some length.

Phorcys•9mo ago

where do these logs come from ? the workspace build logs ?

Dom R (shymega)OP•9mo ago

I'm deploying to Fly.io via that post I linked to. These logs come from the container startup of the base Coder instance. There's a lot of lines, and I'm not sure where the dump starts from. I could probably do a tee invocation to log it all. (I'm using Coder V2)

Phorcys•9mo ago

oh okay, I understood that you were deploying your workspaces on fly.io, not Coder on fly.io could you try getting the full logs please ?

Dom R (shymega)OP•9mo ago

I've just tried again - I get a bit further this time. My guess is that when Coder can't connect to Postgres (pubsub?), all comes crashing down. I'm now on the login screen, but my original attempt was to use 6PN to access Coder. I don't want it public. I can't get all of the logs though. The buffer is full.

Phorcys•9mo ago

is there some way you could get it to log to a file or something like that ? hmm, that's weird, it definitely shouldn't crash like that

Dom R (shymega)OP•9mo ago

OK. So I've figured out that if I set the base URL to http://app_name.internal, it crashes. That also happens with HTTPS. If you use .fly.dev, it works. I've attached as much logs I've got here:

Dom R (shymega)OP•9mo ago

coder.txt

Dom R (shymega)OP•9mo ago

.internal should work fine if it uses the Fly DNS, because it'll just resolve to the 6PN IPv6 address. But I'm not sure the way this goroutine is handled is quite right though...

Phorcys•9mo ago

yeah and it really shouldn't crash like that could you create an issue over @ coder/coder to explore the root cause of this bug ?

Atif•9mo ago

Yes please create an issue and we can see how we can improve the experience in this case.

Phorcys•8mo ago

hey @Dom R (shymega), did you get the time to create an issue for this ? that way we can reference it here

Dom R (shymega)OP•3mo ago

I haven't yet had the time. Problem is, I don't think Coder officially supports Fly.io anymore (not helped by Fly discontinuing their TF provider), and I'm not sure if this issue will have much traction. I've got it on my todo list though. Hey, so I tried to replicate the bug - at last! - and so far so good. I've managed to create a user. I don't know if Coder does still support Fly, but we'll see how I get on.

Phorcys•3mo ago

hey! well technically there isn't "official support" or not (except for integrations with partners) so long as Terraform supports Fly, then Coder should support it too! oh, well, you are right

Phorcys•3mo ago

the Terraform provider for Fly.io was dropped by Fly https://community.fly.io/t/fresh-produce-volumes-endpoints-in-machines-api/14408/20

Fly.io

[Fresh Produce] Volumes endpoints in Machines API

Hi @charsleysa, I think the best answer I can give for this is that the community is welcome to fork the fly-apps Terraform provider. I can appreciate that this is not the most satisfying answer, given that we also don’t promise never to pick this up again. Ultimately it’s code that handles users’ API keys. There’s an expectation that we’d ensu...

Phorcys•3mo ago

I will close this thread since there is nothing we can do about it, though let's continue discussing this through your GH issue so that we can avoid misunderstandings in the future :)

GitHub

Add warning to documentation for Fly.io · Issue #14599 · coder/coder

I am under the impression that Fly.io is no longer supported by Coder because of the unmaintained Terraform provider. This issue is to track the progress of adding a deprecation warning to the docu...

Codercord•3mo ago

@Phorcys closed the thread.

Gaming

Programming

Goroutine dump with Coder V2 on Fly.io