Set env var in dockerfile during build

I've got the following in my dockerfile

# This var simply prevents sqlx from connecting to the database at compile time
ARG SQLX_OFFLINE=0
# Copy the cargo configs from the depcache stage. We're building a fully static binary
RUN cargo build --release --target x86_64-unknown-linux-musl --bin backend

# This var simply prevents sqlx from connecting to the database at compile time
ARG SQLX_OFFLINE=0
# Copy the cargo configs from the depcache stage. We're building a fully static binary
RUN cargo build --release --target x86_64-unknown-linux-musl --bin backend

My goal here is to have the build read SQLX_OFFLINE from my variables on railway during build and set the SQLX_OFFLINE environment var for the docker build. However I only want this to be set during builds on railway, and not when running docker locally. Any guidance will be greatly appreciated!

92 Replies

Percy•2y ago

Project ID: d7c966b3-07dd-4a31-9cb9-82de5002e54f

Percy•2y ago

You might find these helpful: - Start command variable passing failing to pass vars - Invalid environment variables unless ENV included in Dockerfile - environment variables are not picked up in railway build but works locally

⚠️ experimental feature

ShahOP•2y ago

d7c966b3-07dd-4a31-9cb9-82de5002e54f

milo•2y ago

ARG SQLX_OFFLINE will read variables and default to 0

ShahOP•2y ago

Yeah I tried that and the build is still trying to perform compile time checks I did follow the last item in Percy's list though, apparently docker vars must be at the top of the file. But that didn't work either

milo•2y ago

i think the sqlx checking comes in another command - not cargo build

ShahOP•2y ago

It comes during compile unless the SQLX_OFFLINE var is set to "true" During build

milo•2y ago

oh. so i guess you need to default to true, not 0/1

ShahOP•2y ago

Yeah I'm defaulting to "false" actually, and have the variable in my service set to "true". The idea was that when my team builds locally they'll be building against a local postgres instance which will always be up to date with migrations, therefore checks will all pass. But in Railway the migrations for some reason always happen last, so the compile checks always fail on PR environments

milo•2y ago

cargo sqlx migrate? u might have better luck using nixpacks tbh

ShahOP•2y ago

I'm still having trouble getting it to work 😦 No idea what's wrong with the nixpacks build

milo•2y ago

got any logs

ShahOP•2y ago

No but I can have some generated. Give me a few moments

milo•2y ago

sure

ShahOP•2y ago

Interestingly enough the nixpacks build actually made it to the point where it failed due to compile time checks Not sure why it even tried to check, aren't variables set at build time as well?

milo•2y ago

variables are in every stage. can we see where it is failing with some logs?

ShahOP•2y ago

#13 121.1    Compiling backend v0.1.0 (/app)
#13 121.2 error: failed to find data for query INSERT INTO tokens (id, user_id, user_device, token_type)
#13 121.2                    SELECT id, user_id, user_device, token_type
#13 121.2                    FROM (
#13 121.2                        VALUES ($1::uuid, $2::uuid, $3, $4::int), ($5::uuid, $6::uuid, $7, $8::int)
#13 121.2                    ) AS t (id, user_id, user_device, token_type)
#13 121.2                    WHERE EXISTS (
#13 121.2                        SELECT FROM users WHERE id = $2::uuid
#13 121.2                    )
#13 121.2                    ON CONFLICT (user_id, user_device, token_type)
#13 121.2                    DO UPDATE SET id = EXCLUDED.id
#13 121.2
#13 121.2   --> src/services/token.rs:43:15
#13 121.2    |
#13 121.2 43 |           match sqlx::query!(
#13 121.2    |  _______________^
#13 121.2 44 | |             "INSERT INTO tokens (id, user_id, user_device, token_type)
#13 121.2 45 | |             SELECT id, user_id, user_device, token_type
#13 121.2 46 | |             FROM (
#13 121.2 ...  |
#13 121.2 62 | |             TokenType::Refresh as i32,
#13 121.2 63 | |         )
#13 121.2    | |_________^
#13 121.2    |
#13 121.2    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
#13 121.2
 
#13 121.2 error: failed to find data for query SELECT id FROM tokens WHERE id = $1 AND user_id = $2 AND user_device = $3 AND token_type = $4
#13 121.2   --> src/services/token.rs:82:15
#13 121.2    |
#13 121.2 82 |           match sqlx::query!(
#13 121.2    |  _______________^
#13 121.2 83 | |             "SELECT id FROM tokens WHERE id = $1 AND user_id = $2 AND user_device = $3 AND token_type = $4",
#13 121.2 84 | |             token_id,
#13 121.2 85 | |             user_id,
#13 121.2 86 | |             user_device,
#13 121.2 87 | |             token_type as i32,
#13 121.2 88 | |         )
#13 121.2    | |_________^
#13 121.2    |
#13 121.2    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
#13 121.2
 
#13 121.5 error: could not compile `backend` due to 2 previous errors
 
#13 ERROR: executor failed running [/bin/bash -ol pipefail -c cargo build --release]: exit code: 101
-----
> [stage-0  6/10] RUN --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-/root/cargo/git,target=/root/.cargo/git --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-/root/cargo/registry,target=/root/.cargo/registry --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-target,target=/app/target cargo build --release:
-----
executor failed running [/bin/bash -ol pipefail -c cargo build --release]: exit code: 101

#13 121.1    Compiling backend v0.1.0 (/app)
#13 121.2 error: failed to find data for query INSERT INTO tokens (id, user_id, user_device, token_type)
#13 121.2                    SELECT id, user_id, user_device, token_type
#13 121.2                    FROM (
#13 121.2                        VALUES ($1::uuid, $2::uuid, $3, $4::int), ($5::uuid, $6::uuid, $7, $8::int)
#13 121.2                    ) AS t (id, user_id, user_device, token_type)
#13 121.2                    WHERE EXISTS (
#13 121.2                        SELECT FROM users WHERE id = $2::uuid
#13 121.2                    )
#13 121.2                    ON CONFLICT (user_id, user_device, token_type)
#13 121.2                    DO UPDATE SET id = EXCLUDED.id
#13 121.2
#13 121.2   --> src/services/token.rs:43:15
#13 121.2    |
#13 121.2 43 |           match sqlx::query!(
#13 121.2    |  _______________^
#13 121.2 44 | |             "INSERT INTO tokens (id, user_id, user_device, token_type)
#13 121.2 45 | |             SELECT id, user_id, user_device, token_type
#13 121.2 46 | |             FROM (
#13 121.2 ...  |
#13 121.2 62 | |             TokenType::Refresh as i32,
#13 121.2 63 | |         )
#13 121.2    | |_________^
#13 121.2    |
#13 121.2    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
#13 121.2
 
#13 121.2 error: failed to find data for query SELECT id FROM tokens WHERE id = $1 AND user_id = $2 AND user_device = $3 AND token_type = $4
#13 121.2   --> src/services/token.rs:82:15
#13 121.2    |
#13 121.2 82 |           match sqlx::query!(
#13 121.2    |  _______________^
#13 121.2 83 | |             "SELECT id FROM tokens WHERE id = $1 AND user_id = $2 AND user_device = $3 AND token_type = $4",
#13 121.2 84 | |             token_id,
#13 121.2 85 | |             user_id,
#13 121.2 86 | |             user_device,
#13 121.2 87 | |             token_type as i32,
#13 121.2 88 | |         )
#13 121.2    | |_________^
#13 121.2    |
#13 121.2    = note: this error originates in the macro `$crate::sqlx_macros::expand_query` which comes from the expansion of the macro `sqlx::query` (in Nightly builds, run with -Z macro-backtrace for more info)
#13 121.2
 
#13 121.5 error: could not compile `backend` due to 2 previous errors
 
#13 ERROR: executor failed running [/bin/bash -ol pipefail -c cargo build --release]: exit code: 101
-----
> [stage-0  6/10] RUN --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-/root/cargo/git,target=/root/.cargo/git --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-/root/cargo/registry,target=/root/.cargo/registry --mount=type=cache,id=s/e544dd74-1db0-4228-8b65-63cb3e2d86f7-target,target=/app/target cargo build --release:
-----
executor failed running [/bin/bash -ol pipefail -c cargo build --release]: exit code: 101

milo•2y ago

is DATABASE_URL being provided to this service?

ShahOP•2y ago

Yes

milo•2y ago

are they in the same project? or

ShahOP•2y ago

Yep same project

milo•2y ago

okay, weird. one sec send the table at the top of the build logs

ShahOP•2y ago

You mean the table structure or?

milo•2y ago

yea just a sccrfeenshot of it

ShahOP•2y ago

error: error returned from database: relation "tokens" does not exist Oh

milo•2y ago

need to the see the contents

ShahOP•2y ago

Sure one sec

milo•2y ago

like the table nono not the db table the table tha was logged at the top of anguish

table at the top of build logs in ur deployment that is being logged by nixpacks

ShahOP•2y ago

milo•2y ago

can u send ur nixpacks.toml

ShahOP•2y ago

I don't have one

milo•2y ago

env variables? anything to do with nixpacks?

ShahOP•2y ago

ShahOP•2y ago

That's all I use

milo•2y ago

?????? something is wrong here u must have a nixpacks.toml somewhere

ShahOP•2y ago

Nope, I'm just letting the deployment decide what to do I never needed the nixpacks file because I was using docker

milo•2y ago

ok time to figujre out why it thinks musl iS good rust-toolchain.toml?

ShahOP•2y ago

I think railway builds to musl by default @Percy cargo musl builds?

milo•2y ago

it doesnjt

Percy•2y ago

You might find these helpful: - Nixpacks doesn't include directory when building - Nixpacks build fail - nixos not building

⚠️ experimental feature

milo•2y ago

ye it doersnt nvm, it kinda does have u got a rust-toolchain file

ShahOP•2y ago

Nope

milo•2y ago

ok cool

ShahOP•2y ago

Avoiding all of this is why we used Docker We wanna keep the railway builds identical to local builds with the exception of env vars

milo•2y ago

well it has decided to do something we're not telling it to do we have never done anytthing to make rust:build stages someone on ur team has

ShahOP•2y ago

Hm Where would that be configured?

milo•2y ago

any nixpacks.toml / nixpacks.json do u have a railway config file

ShahOP•2y ago

Yes But i've ignored it for the duration of trying nixpacks by just renaming it

milo•2y ago

ShahOP•2y ago

milo•2y ago

ok go to service settings

milo•2y ago

milo•2y ago

find what builderis what is the file called?

ShahOP•2y ago

milo•2y ago

it's still using ur dockerfile

ShahOP•2y ago

Oh right my bad I just tried switching back One sec

milo•2y ago

rename the dockerfile to somethjign else then trigger deploy then send a csreenshot of that

ShahOP•2y ago

milo•2y ago

get rid of build command save then redeploy it shoulda uto redeloy

ShahOP•2y ago

milo•2y ago

ok can u send ur file structure from the root

ShahOP•2y ago

milo•2y ago

oh. in service settings set root path to backend

ShahOP•2y ago

That's already set

milo•2y ago

is there a railway.toml/nixpacks.json at root?

ShahOP•2y ago

Nope

milo•2y ago

can u send a screenshot of sertvice settings under builder

ShahOP•2y ago

milo•2y ago

rename dockerfile to _dockerfile and railway.toml to _railway.toml

ShahOP•2y ago

Okay No changes on that window

ShahOP•2y ago

milo•2y ago

no env variables?

ShahOP•2y ago

I've got them set in the ui

milo•2y ago

this is acc weird ugh]

ShahOP•2y ago

Yeah this is why we use docker Everyone here hates the railway build steps lol

milo•2y ago

well, as we are railway, we'll try fix the issue with nixpacks everyone here as in who

ShahOP•2y ago

My team

milo•2y ago

cc @coffeecup nixpacks isn't detecting a rust app d7c966b3-07dd-4a31-9cb9-82de5002e54f

ShahOP•2y ago

d7c966b3-07dd-4a31-9cb9-82de5002e54f In the meanwhile is it safe for me to continue trying to get the arg passed into docker or should I branch off here for future troubleshooting?

milo•2y ago

u can keep trying, sure

ShahOP•2y ago

milo•2y ago

but our focus now is prob fixing nixpacks lol

ShahOP•2y ago

No worries I'll just make another branch from this and make a dummy PR to get the environment spun up Update: I got nixpacks to run, I just had to set the language provider to rust. However I still get the same error as a few weeks ago post-build

failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount358826668/app/bin: lstat /var/lib/docker/tmp/buildkit-mount358826668/app/bin: no such file or directory

milo•2y ago

wtfff

ShahOP•2y ago

#13 20.60     Finished release [optimized] target(s) in 20.18s
#13 DONE 20.7s
 
 
#6 [stage-1 2/5] WORKDIR /app/
#6 sha256:11d15f4c04ce8945912fdbd9d315d21f8f7e39ae03941de47e6c9297d7b2f6a3
#6 CACHED
 
#14 [stage-1 3/5] COPY --from=0 /etc/ssl/certs /etc/ssl/certs
#14 sha256:5a83e54bfde5c8ce7c8e04595b02cc007626de694961b62cdb6de80fca92a2ba
#14 CACHED
 
#15 [stage-1 4/5] RUN true
#15 sha256:658583ceb4bf4eb056e58352c9b5ef0b29a74af04feb5d8ce2d7d412b1d3fc4a
#15 CACHED
 
#16 [stage-1 5/5] COPY --from=0 /app/bin/backend /app/bin/backend
#16 sha256:796684f699e68ea9bc16d8bcb134d888282e032df5460436a21036b83fda80b4
#16 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount358826668/app/bin: lstat /var/lib/docker/tmp/buildkit-mount358826668/app/bin: no such file or directory
-----
> [stage-1 5/5] COPY --from=0 /app/bin/backend /app/bin/backend:
-----

#13 20.60     Finished release [optimized] target(s) in 20.18s
#13 DONE 20.7s
 
 
#6 [stage-1 2/5] WORKDIR /app/
#6 sha256:11d15f4c04ce8945912fdbd9d315d21f8f7e39ae03941de47e6c9297d7b2f6a3
#6 CACHED
 
#14 [stage-1 3/5] COPY --from=0 /etc/ssl/certs /etc/ssl/certs
#14 sha256:5a83e54bfde5c8ce7c8e04595b02cc007626de694961b62cdb6de80fca92a2ba
#14 CACHED
 
#15 [stage-1 4/5] RUN true
#15 sha256:658583ceb4bf4eb056e58352c9b5ef0b29a74af04feb5d8ce2d7d412b1d3fc4a
#15 CACHED
 
#16 [stage-1 5/5] COPY --from=0 /app/bin/backend /app/bin/backend
#16 sha256:796684f699e68ea9bc16d8bcb134d888282e032df5460436a21036b83fda80b4
#16 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount358826668/app/bin: lstat /var/lib/docker/tmp/buildkit-mount358826668/app/bin: no such file or directory
-----
> [stage-1 5/5] COPY --from=0 /app/bin/backend /app/bin/backend:
-----

I did get the docker file to work though, turns out everything was set correctly all this time and the sqlx-data.json file was out of sync with the actual db state

milo•2y ago

lol

ShahOP•2y ago

Yeah totally my bad there But I will still hold off on nixpacks due to that above thing though, I remember you saying it's a bug a while back

milo•2y ago

yeah, unfortunately we should try and fix this tho

ShahOP•2y ago

Let me know if I can help in any way

jr•2y ago

Very strange. I can take a look today.

milo•2y ago

can u let me know how/if u fix it

ShahOP•2y ago

+1 to that, I'm happy to help if I can

Gaming

Programming

Set env var in dockerfile during build