Something went wrong with a deployed template
After I deployed this template https://railway.app/template/CnHq4I I got a popup saying that "Something went wrong" and none of the services are deploying. Could someone please check what might be the issue? ๐
101 Replies
Project ID:
502c4bb1-0e01-42d2-808c-94e4e2af686b502c4bb1-0e01-42d2-808c-94e4e2af686b
Btw this is my template.. i'm trying to create a template so someone from the Temporal team can debug the issues i'm having ๐
Corrected a few things.. but yeah still doing this
Private networks are automatically created if I have the "Unexposed" option right?
Names are based on the service name?
Hmm yeah idk went over everything a 2nd time.. everything looks fine
Also I noticed that after you create the template and then go back to update it, it doesn't have a spot to edit the description (only the name)
you want to click update info, not edit
Where is "Update Info"?
while your template isnt setup perfectly, the issue above is not something thats your fault
I only have an Edit button
My page looks like this
Oh
It's cause it's not published yet
I'm clicking edit on a Personal Template
correct
Also regarding the template I'm kinda just going off this template as a reference https://railway.app/template/AhDDtZ But i'm adding ElasticSearch (which is having issues so I'm creating this template to send off to the Temporal team to look into)
will need to tag in @fp for this one, the repos involved are not being cloned into the users gh account
yeah it should be public dunno ๐not the issue
this is also not the issue, and i know you where just copying from the original template, but
TEMPORAL_CORS_ORIGINS should be https://${{RAILWAY_PUBLIC_DOMAIN}}
TEMPORAL_ADDRESS should be ${{Temporal.RAILWAY_PRIVATE_DOMAIN}}:${{Temporal.PORT}}
ES_SEEDS should be ${{Elasticsearch.RAILWAY_PRIVATE_DOMAIN}}
POSTGRES_SEEDS should be ${{Postgres.RAILWAY_PRIVATE_DOMAIN}}
DB_PORT should be 5432Ahhh ty.. honestly when I was copying stuff over ... I was like there has got to be a variable name for these rather than hard coding them ๐ Thank you! I will change them now ๐
angelo let the hardcoded variables slide, for context the original temporal template was a bounty
Ohhhh ok gotcha that's dope! I mean it's super awesome .. cause i've been debating whether to deploy on Kubernetes or not haha.. but then last minute found this template searching.. it's been great so far.. well minus the Elasticsearch stuff.. but hoping the Temporal team can help debug the issue!
whats your issues with Elasticsearch?
Uggh so many things. Sometimes i'll try to deploy and Temporal will tell me "Waiting for ElasticSearch to start up", and then other times i'll deploy and it will say something related to like "Bad Shards" in ES (when viewing the UI). Idk.. I have an entire thread in the Temporal Slack trying to debug. But I told the guy.. let me just create a template on Railway and you guys can debug with it ... much easier ๐
But I can see logs on Elasticsearch saying that Temporal added indexes.. so it's like not a connectivity issue.. I don't think...
I have no idea 
i do
temporalio/auto-setup is an alpine based image
https://docs.railway.app/reference/private-networking#workaround-for-alpine-based-images๐ฎ
Well.. DNS resolution COULD still work tho right.. cause it has worked a few times
after restarting the instance like 5 times
but yeah other than that it's been a pain
it would be flakey as best and ive never been able to get internal dns resolution working without that variable on an alpine image
Interesting
Let me add it in a project I have rn ๐ฎ
What about the elasticsearch service I have?
The bitnami/elasticsearch
Would it need to be set there too?
nope
does elasticsearch listen on
:: ?Uhhhhh
since the private network is ipv6 only, the services need to bind to
:: instead of 0.0.0.0 to communicate with each other on the private networkSo elastic has 2 things
So it has this -> [2023-11-22T21:26:58,062][INFO ][o.e.t.TransportService ] [railway] publish_address {172.17.0.172:9300}, bound_addresses {[::]:9300}
And this -> [INFO ][o.e.h.AbstractHttpServerTransport] [railway] publish_address {172.17.0.172:9200}, bound_addresses {[::]:9200}
looks good to me
Should I add it then.. or no?
we went off track, add what to what exactly
ENABLE_ALPINE_PRIVATE_NETWORKING=true
to the elasticsearch service
nope, only needed on
Temporalkk, i'm gonna redeploy ES, then temporal
Oh I should prob mention.. wiping a volume always throws an error for me
dunno why
I mean I think it get's wiped.. but just says there was an error
Created a new one to make sure there's not old data tho
It started up just fine @Brody ๐ฎ
BUTTTTT
I'm not convinced.. I need to duplicate it lol
Sooo that may have fixed it but... i'll need to wait until the template is sorted and test it again ๐
yeah we gotta wait for fp, it's likely something broken with the gh integration
Appreciate your help thus far ๐
I am seeing alot less errors in the Temporal service tho forrrr sure
Is there a way to cap memory usage for a service? I'm worried that this is gonna keep growing and growing until it's default configured limit (not sure what that is)
Oh nvm I think the Bitnami ES image caps at 1g per docs
er wait
it's at 1.8GB now ๐ง
there's no way to cap the memory of a service, if there was you would just end up with the app crashing
Right.. it's gotta be app configured instead
heap does not mean total memory allowed
Hmm how the heck do I stop it from growing to like 32gb then? ๐
it can't, it would crash before that since your on hobby
well yeah I know that..
I just mean like
if there's a memory leak with elastic search there's not much you can do
๐
set to 512m and see what happens
I see
[railway] heap size [512mb] in the logs.. so we'll see what happens ๐
Interesting.. on my temporal UI.. I just got this error
Is this the same Alpine issue?yes, did you set the variable?
Yeah that was on the temporal instance.. this is on my temporal-ui service
is that based on a docker image too?
It only happens every now and again tho.. I refresh and it's fine
uhhh yeah
link me please
temporalio/ui:2.21.3
err
yep thats an alpine image too
set the variable
How do you check btw?
lots of way, this was the fastest right now https://hub.docker.com/layers/temporalio/ui/latest/images/sha256-d184b667ace6a4357c2cdedd3a259b4b67e2b4dea258538922b5cd496ffc3d1d?context=explore
layer 3 mentions
apk add apk is alpine's package manager, therefor its an image based on alpineahhh got it! Thanks ๐
.
There's only a single repo, right? If I understand correctly, the repo doesn't get cloned when trying to deploy the template?
Hmm, looks like some branch weirdness.
yeah single repo for multiple services with different root dirs set
For some reason, it's trying to clone the
railway-temporal branch.
Which obviously doesn't exist.๐ค
I might need a coffee for this. I shall return. 
Solution
Definitely needed the coffee.
would have never guessed that
edge cases!! wooo
Should be good now!
Lmaoooo it was cause of the โtreeโ in my name? Thatโs awesome lmaoooo. Thanks for the fix ๐
Your username is now a test-case in our codebase. ๐
Ahahaha sweet ๐
Hey Brody, how would I add a custom build command to this service, in the template? I see it has a custom start command, but don't see anything for the build step
in a railway.json file
that lives in that services root dir
ohhh ok.. i'll have to look up how to format that ๐ got it thanks
I have 2 services that rely on the root dir tho actually
1 for the worker and 1 for the client
use the schema https://docs.railway.app/deploy/config-as-code#json-schema
Is there a way to specify which service uses the railway.json file? Cause I have 2 services that use the same
/hello-world/ folder/code
I want to do a custom build step on only 1 of the services.. not both
There's a way to set a Railway Config File Path in the settings of the service.. but I don't see an option to set it for the service (in the template builder)ah gotcha, set the build command in a nixpacks.toml file instead, you can make multiple of them with different names like
hello-world.nixpacks.toml and then set a service variable NIXPACKS_CONFIG_FILE to the name of the file nameah sweet! lemme try that 
I had another question.. hope you don't mind.. just trying to get a lay of the land of sorts still haha. I was looking at my billing.. and my Egress charges seem kinda high for my services.. esp on my Temporal service. I think I used the proxy on accident for the Postgres URL. Would I be charged in that case if I used the proxy URL instead of the private URL? I'm assuming data passed via the private network (service to service) is free, yes?
I had another question.. hope you don't mind.. just trying to get a lay of the land of sorts still haha. I was looking at my billing.. and my Egress charges seem kinda high for my services.. esp on my Temporal service. I think I used the proxy on accident for the Postgres URL. Would I be charged in that case if I used the proxy URL instead of the private URL? I'm assuming data passed via the private network (service to service) is free, yes?Also I had another question.. hope you don't mindi dont! thats that im here for!
my Egress charges seem kinda high for my services ... I think I used the proxy on accident ... Would I be charged in that case if I used the proxy URL instead of the private URL?that would indeed cause egrees fees when using the databases tcp proxy
I'm assuming data passed via the private network (service to service) is free, yes?correct, you definitely want to use the private domains, this is me trying to do that for you https://discord.com/channels/713503345364697088/1177012474122354778/1177025004035129434
Yep gotcha.. yeah I was using the old template (had the old proxy URL's I think). I'm redeploying my project on the new template which uses all private URL's and such ๐ Thanks!
no prob
Hmm ok so I setup the environment variables like you said, and I added a file called
temporal-worker.nixpacks.toml inside the /hello-world/ dir like you said. That file has the following in it
Does this look correct to you? The custom build command isn't set tho after I deploy the template unfortunately ๐ง
Or would it just look like this
one sec
first option was right
maybe the variable wants a
/ prefixedoh hmm
this is interesting
I see the custom build command in the logs
But it's not set here
it wouldnt be
Ah ok
nixpacks.toml is a build config, railway.json is a service config
Ah gotcha! Appreciate the help 
Adding Prometheus and Grafana to the Temporal template
very nice
Brody, this should calculate a secret in a Template correct?
I have it set as a variable in my template, but when the template deployed.. the value didn't get replaced for some reason
show me a screenshot please?
sec lemme redeploy
When I go to Variables after deploying.. it shows this
hmmm thats not ideal, ill ask jr when hes active
(jr is already in this thread)
Awesome thanks