Railway•8mo ago

rust deploy uses 10x more ram than local dev

hey guys, i have a rust api that is using 4-8gb of ram suddenly, now for rust that's unheard of considering it used to chill at 200mb +/- when running cargo run --release --package x locally i get out to a max of 1.5gb ram (this includes compiling) yet on railway it ticks out to 8gb at peak (this includes compiling) takes roughly 26 minutes to compile, build, deploy on railway There's a total of 950 +/- crate dependencies projectId = 7c6a1d9f-d3d2-4808-96b1-10c8df175503 No deploy that happened in that weird spike, nor am i able to reproduce the low memory usage anymore

132 Replies

Percy•8mo ago

Project ID: 7c6a1d9f-d3d2-4808-96b1-10c8df175503

Brody•8mo ago

dockerfile or nixpacks?

FlorisOP•8mo ago

nixpacks

// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = 'cargo run --package xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = 'cargo run --package xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

Brody•8mo ago

you should not do cargo run as the start command, you should be building a release and then running the resulting binary

FlorisOP•8mo ago

i had a cargo run --release but i just changed it a few min ago

Brody•8mo ago

let's see your new nixpacks.toml file please additionally, @milo / salvage your input here would be greatly appreciated

FlorisOP•8mo ago

// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --release --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = './target/release/xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --release --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = './target/release/xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

Brody•8mo ago

yeah that looks appropriate now

FlorisOP•8mo ago

yeah well i used to do it like this in the start but had tremendous amount of issues as you might remember it would just brick and never deploy unless done via the cli

Brody•8mo ago

I do remember, did you also have those variables set? I don't see any purpose to them and can see them causing issues

FlorisOP•8mo ago

yes no i hadn't set them yet this is just me trying to get it working in a normal way

Brody•8mo ago

for good measure, remove them please

FlorisOP•8mo ago

the nixpacks_cache ?

Brody•8mo ago

and then let me know if you run into further issues yes, both those variables you have there

FlorisOP•8mo ago

okay, gotcha 2 min the target could be up to 20gb in some weird c folder hence i deferred it to the cache folder

Brody•8mo ago

I'm assuming the default cache stuff nixpacks does is sufficient

FlorisOP•8mo ago

compiling rn same issue as before on railway and on nixpacks locally

FlorisOP•8mo ago

local

Brody•8mo ago

just freezes there?

FlorisOP•8mo ago

yes

milo•8mo ago

god

FlorisOP•8mo ago

i love railway

milo•8mo ago

the rust provider in nixpacks honestly is shit

FlorisOP•8mo ago

but rust has been very painful to work with on railway

milo•8mo ago

yeah i've moved to cargo-chef in a dockerfile

FlorisOP•8mo ago

i was reading about that yesterday

Brody•8mo ago

haha milo beat me to it

milo•8mo ago

much better then nixpacks

Brody•8mo ago

agree with Milo here

milo•8mo ago

also this issue looks unrelated to nixpacks

FlorisOP•8mo ago

right okay, so you advise me to do the same im not sure, it stalls local and on railway

milo•8mo ago

nixpacks or

FlorisOP•8mo ago

yes

milo•8mo ago

okay nevermind this is a nixpacks issue 😂

FlorisOP•8mo ago

milo•8mo ago

im not sure why it is making it stall that bad though

Brody•8mo ago

yes, I'm not a rust dev but have heard great things about cargo chef and when nixpacks is painful (almost always) dockerfiles are the way to go

FlorisOP•8mo ago

we still chilling here

milo•8mo ago

from yours truly

FlorisOP•8mo ago

yeah it was the exact same issue months ago somehow the cli fixed it

milo•8mo ago

just use cargo-chef, see if that helps

FlorisOP•8mo ago

omw

Brody•8mo ago

yep cargo chef in a dockerfile for sure

milo•8mo ago

ping me if this happens again (or get brody to lol)

FlorisOP•8mo ago

yeah as funnily enough we went to rust cus of low resource usage whahaha

milo•8mo ago

no you dont need to install it

milo•8mo ago

please read up how to use it:https://github.com/LukeMathWalker/cargo-chef

GitHub

GitHub - LukeMathWalker/cargo-chef: A cargo-subcommand to speed up ...

A cargo-subcommand to speed up Rust Docker builds using Docker layer caching. - LukeMathWalker/cargo-chef

FlorisOP•8mo ago

you just make the recipe.json ? i am

milo•8mo ago

you do all that in the dockerfile

FlorisOP•8mo ago

ah i see

milo•8mo ago

jsut look at the example

milo•8mo ago

this bit

FlorisOP•8mo ago

should be able to copy most of that

milo•8mo ago

yup

FlorisOP•8mo ago

milo•8mo ago

delete ur nixpacks.toml make sure the Dockerfile is named Dockerfile

FlorisOP•8mo ago

yes wait im showing u something

FlorisOP•8mo ago

milo•8mo ago

thats just nixpacks doing extra stuff dont worry about that please delete ur nixpacks,toml

Brody•8mo ago

you somehow needed up with two rust providers but yeah, Dockerfile time

FlorisOP•8mo ago

well i think it's a bug in NixPacks when cargo workspaces are enabled as xylex_wss is not the main bin it starts building a random workspace and the core bin

milo•8mo ago

nixpacks and rust dont mix unfortunately

FlorisOP•8mo ago

going to docker now ig

milo•8mo ago

yup

FlorisOP•8mo ago

fucking hate docker time consuming

milo•8mo ago

not really this will be easy

Brody•8mo ago

yeah you won't need any kind of complex Dockerfile

FlorisOP•8mo ago

okay so i changed these from app to xylex

FlorisOP•8mo ago

anything besides that ? my cargo is called xylex

milo•8mo ago

what is the binary u wanna okay cool should be good

FlorisOP•8mo ago

railway autodetects this yes

milo•8mo ago

yes

FlorisOP•8mo ago

milo•8mo ago

let us know how it goes

Brody•8mo ago

and please cancel those old deployments

FlorisOP•8mo ago

whahah yes i will

FlorisOP•8mo ago

i am guessing one of these should've been app still

Brody•8mo ago

I assume that means you need to install openssl

FlorisOP•8mo ago

bros too experienced

milo•8mo ago

RUN apt update -y && apt install openssl libssl-dev -y put that in the line before the ENTRYPOINT instruction

Brody•8mo ago

perfect

FlorisOP•8mo ago

man idk where milo came from but they needa give him a raise bro's got shortcuts to paste solutions

Brody•8mo ago

Milo is the rust professional around here

FlorisOP•8mo ago

what a king

Brody•8mo ago

milo•8mo ago

just from my personal rust app dockerfile haha

FlorisOP•8mo ago

are you in the railway dev team milo

Brody•8mo ago

he's not, just a community mod

FlorisOP•8mo ago

i see, so it is personal projects/companies that u used rust in

milo•8mo ago

yeah, some enterprise things as well rust is really good and scalable in my experience

FlorisOP•8mo ago

yes we switched in january it is amazing

milo•8mo ago

haha

FlorisOP•8mo ago

i am unsure if the ssl install was the right call certificate verify failed"

milo•8mo ago

:anguish: what dependencies are u using

Brody•8mo ago

might need ca-certificates too? just a guess

FlorisOP•8mo ago

shouldn't matter it deploys fine on ubuntu server

milo•8mo ago

can you send ur cargo.toml

FlorisOP•8mo ago

yes

Brody•8mo ago

I mean this is not the same at all though

milo•8mo ago

are you calling reqwest directly

FlorisOP•8mo ago

alright brace for impact guys

// cargo.toml 
[package]
name = "xylex"
version = "0.1.0"
edition = "2021"


[dependencies]
polars = {version = "0.35.0", features = ["lazy", "rolling_window"] }
reqwest = { version = "0.11.23", features = ["json", "blocking"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
supabase_rs = "0.2.2"
thiserror = "1"
rocket = "0.5.0"
serde_json = "1.0.111"
dotenv = "0.15.0"
mail-send = "0.4.6"
mail-builder = "0.3.1"
rsi = "0.1.2"
csv = "1.1"
arrow = "*"
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
clokwerk = "0.4.0"
image = "0.24.8"
rusttype = "0.9.3"
rand = "0.8.5"
qrcode = "0.13"
async-trait = "0.1.77"
futures = "0.3.3"
imageproc = "0.23.0"
tokio-tungstenite = { version = "0.21.0", features = ["native-tls"] }
serde_yaml = "0.9.31"
lazy_static = "1.4.0"
trading_sessions = "0.1.2"
md-5 = "0.10.6"
hex-literal = "0.4.1"
fibonacci_retracement = "0.1.0"
futures-util = "0.3.30"
futures-channel = "0.3.30"
url = "2.5.0"
sha256 = "1.5.0"
strum = "0.26"
strum_macros = "0.26"
sled = "0.34.7"

xylex_spread_tracker = { path = "./xylex_spread_tracker" }
xylex_card_generator = { path = "./xylex_card_generator" }
xylex_hash_router = { path = "./xylex_hash_router" }
xylex_scheduler = { path = "./xylex_scheduler" }
xylex_discord = { path = "./xylex_discord"}

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true


[profile.release]
opt-level = 3

[workspace]
members = [
    "xylex_wss",
    "xylex_card_generator",
    "xylex_spread_tracker",
    "xylex_scheduler",
    "xylex_hash_router",
    "xylex_discord"
]

// cargo.toml 
[package]
name = "xylex"
version = "0.1.0"
edition = "2021"


[dependencies]
polars = {version = "0.35.0", features = ["lazy", "rolling_window"] }
reqwest = { version = "0.11.23", features = ["json", "blocking"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
supabase_rs = "0.2.2"
thiserror = "1"
rocket = "0.5.0"
serde_json = "1.0.111"
dotenv = "0.15.0"
mail-send = "0.4.6"
mail-builder = "0.3.1"
rsi = "0.1.2"
csv = "1.1"
arrow = "*"
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
clokwerk = "0.4.0"
image = "0.24.8"
rusttype = "0.9.3"
rand = "0.8.5"
qrcode = "0.13"
async-trait = "0.1.77"
futures = "0.3.3"
imageproc = "0.23.0"
tokio-tungstenite = { version = "0.21.0", features = ["native-tls"] }
serde_yaml = "0.9.31"
lazy_static = "1.4.0"
trading_sessions = "0.1.2"
md-5 = "0.10.6"
hex-literal = "0.4.1"
fibonacci_retracement = "0.1.0"
futures-util = "0.3.30"
futures-channel = "0.3.30"
url = "2.5.0"
sha256 = "1.5.0"
strum = "0.26"
strum_macros = "0.26"
sled = "0.34.7"

xylex_spread_tracker = { path = "./xylex_spread_tracker" }
xylex_card_generator = { path = "./xylex_card_generator" }
xylex_hash_router = { path = "./xylex_hash_router" }
xylex_scheduler = { path = "./xylex_scheduler" }
xylex_discord = { path = "./xylex_discord"}

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true


[profile.release]
opt-level = 3

[workspace]
members = [
    "xylex_wss",
    "xylex_card_generator",
    "xylex_spread_tracker",
    "xylex_scheduler",
    "xylex_hash_router",
    "xylex_discord"
]

[package]
name = "xylex_card_generator"
version = "0.1.0"
edition = "2021"

[dependencies]
image = "0.25.0"
imageproc = "0.24.0"
reqwest = "0.12.2"
rusttype = "0.9.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"

[package]
name = "xylex_card_generator"
version = "0.1.0"
edition = "2021"

[dependencies]
image = "0.25.0"
imageproc = "0.24.0"
reqwest = "0.12.2"
rusttype = "0.9.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"

[package]
name = "xylex_discord"
version = "0.1.0"
edition = "2021"

[dependencies]
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
thiserror = "1"
serde_json = "1.0.111"
dotenv = "0.15.0"
serde_yaml = "0.9.31"
reqwest = "0.12.2"

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true

[profile.release]
opt-level = 3

[package]
name = "xylex_discord"
version = "0.1.0"
edition = "2021"

[dependencies]
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
thiserror = "1"
serde_json = "1.0.111"
dotenv = "0.15.0"
serde_yaml = "0.9.31"
reqwest = "0.12.2"

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true

[profile.release]
opt-level = 3

[package]
name = "xylex_hash_router"
version = "0.1.0"
edition = "2021"

[dependencies]
dotenv = "0.15.0"
serde_json = "1.0.115"
supabase_rs = "0.2.2"

[package]
name = "xylex_hash_router"
version = "0.1.0"
edition = "2021"

[dependencies]
dotenv = "0.15.0"
serde_json = "1.0.115"
supabase_rs = "0.2.2"

[package]
name = "xylex_spread_tracker"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = "1.10.4"
reqwest = "0.12.2"
serde = "1.0.197"
serde_derive = "1.0.197"
serde_json = "1.0.115"
serde_yaml = "0.9.34"
supabase_rs = "0.2.2"
tokio = "1.36.0"

[package]
name = "xylex_spread_tracker"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = "1.10.4"
reqwest = "0.12.2"
serde = "1.0.197"
serde_derive = "1.0.197"
serde_json = "1.0.115"
serde_yaml = "0.9.34"
supabase_rs = "0.2.2"
tokio = "1.36.0"

[package]
name = "xylex_wss"
version = "0.1.0"
edition = "2021"

[dependencies]
amqprs = "1.5.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"
tokio =  "1.36.0"

[package]
name = "xylex_wss"
version = "0.1.0"
edition = "2021"

[dependencies]
amqprs = "1.5.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"
tokio =  "1.36.0"

Brody•8mo ago

you can attach files lmao

FlorisOP•8mo ago

oh well this works too

milo•8mo ago

add rustls-tls as a feature of reqwest also, half these depenendices are obsolete :anguish: and can be replaced by something better, not needed at all, or stdlib

FlorisOP•8mo ago

i mean bro im sure but i'm not sure if refactoring 155k lines of rust is something i wanna do tonight

milo•8mo ago

i mean, you don't need strum-macros if you have strum, lazy-static is just stupid these days, dotenv should use dotenvy and thats just to name a few anyways do this please

FlorisOP•8mo ago

yes its deploying

milo•8mo ago

okay cool

Brody•8mo ago

damn Milo knows his stuff

FlorisOP•8mo ago

yeah but what's the difference between dotenv and dotenvy

milo•8mo ago

dotenvy is maintained and isn't.. bad

milo•8mo ago

https://rustsec.org/advisories/RUSTSEC-2021-0141.html

RUSTSEC-2021-0141: dotenv: dotenv is Unmaintained › RustSec Advisor...

Security advisory database for Rust crates published through https://crates.io

FlorisOP•8mo ago

i see ill make a linear ticket ig

milo•8mo ago

same api, literally just changing crate name anyways!

FlorisOP•8mo ago

same issue lmfao

milo•8mo ago

screenshot of error

FlorisOP•8mo ago

"certificate verify failed", file

milo•8mo ago

well then i dont know why show me the line of code that is causing this erorr

FlorisOP•8mo ago

every single one that does an api request

Brody•8mo ago

maybe?

milo•8mo ago

show me please

FlorisOP•8mo ago

pub async fn discord_logs(
    log_message: &str
) -> Result<(), Box<dyn Error>> {
    let client: Client = Client::new();
    let webhook_url: String = env::var("DISCORD_WEBHOOK_URL").unwrap();
    let current_time: String = Utc::now().to_string();

    client.post(&webhook_url)
        .body(json!({
            "content": format!(
                "**xylex-daemon @ {} | **{}",
                current_time, log_message
            )

        }).to_string())
        .header(
            "Content-Type",
             "application/json"
        )
        .send()
        .await?;

    Ok(())
}

pub async fn discord_logs(
    log_message: &str
) -> Result<(), Box<dyn Error>> {
    let client: Client = Client::new();
    let webhook_url: String = env::var("DISCORD_WEBHOOK_URL").unwrap();
    let current_time: String = Utc::now().to_string();

    client.post(&webhook_url)
        .body(json!({
            "content": format!(
                "**xylex-daemon @ {} | **{}",
                current_time, log_message
            )

        }).to_string())
        .header(
            "Content-Type",
             "application/json"
        )
        .send()
        .await?;

    Ok(())
}

all this shouldn't matter as it works fine on any other deploy it's just railway can't build it unless it's 4-8gb ram

milo•8mo ago

well thats norrmal about high ram amounts yeah no clue, you aren't directly telling it to use openssl and the features make it use rustls so im lost sorry

FlorisOP•8mo ago

ig so but only 10-20k lines r being deployed on railway and idle usage used to be 200mb or so

milo•8mo ago

sorry idk

FlorisOP•8mo ago

thanks for the help though guys, i appreciate your time have a good night

Gaming

Programming

rust deploy uses 10x more ram than local dev