trying to connect to postgres w/ next.js
As it says on the tin. The error message is:
I am totally new to Railway and deploying websites more generally and I have no idea where to even start, although StackOverflow suggests I could try changing my Railway settings to add the custom build command
CI=false npm run build
but this, too, did not work.351 Replies
Project ID:
551bf6a9-effd-4096-953e-ef1e8bd61cb5
My project ID is
551bf6a9-effd-4096-953e-ef1e8bd61cb5
.
I presume the issue is maybe in package.json
or package-lock.json
but I'm unsure where to start.Hey! Are those two included in the repo?
They should be!
Can you show your package.json?
could you send your package.json?
Yes, they are.
🧠 ---🧠
you need to specify a start command
Right. Right now that's, uh.
oh I see
oh I see so build command and start command are a bit different
so what should I specify as the start command
Yes!
is it an API? JS or TS?
It's all JS/HTML.
Uh...I don't think it's an API really.
It's--here lemme show you the site as it exists
What do u use to run it locally?
https://eliotweb-production.up.railway.app/index.html
I use a Python mini-server.
The text browser works fine, but I'm now working on an admin site to update JSONs (or maybe the database, which I have yet to implement, but which is why I installed mysql) when editing text files.
(The website is for viewing and comparing the texts of two editions of a 17th-century Bible translation in a language that isn't spoken anymore, so typos when copying are inevitable, so I need to be able to update the JSONs/(database entries) it's pulling from when I edit the text files...)
(so the idea behind the admin page is: select a book--e.g. suppose I just proofread Exodus--and it re-processes the updated text into the relevant JSON entries like the concordance and the verse JSON that contains the compared verses and stuff like that)
(Previously I had been struggling to figure out how the hell to get it to
require('fs')
from node_modules, which it really didn't want to do. I get that it's trying to firewall itself from the user's system but I want to be able to change server-side files!
(I still cannot get fs
working but that's a different problem
brb--showerWould it be possible for you to share your repo? Sounds like you have an unconventional setup
Railway’s Nixpacks builder is fantastic at automatically detecting languages and assigning build commands, but only if the project is structured similar to a standard project
back
(sorry, it's very cold in my apartment right now so I ran the hot water high for a good long while)
GitHub
GitHub - cdnilsen/eliotweb
Contribute to cdnilsen/eliotweb development by creating an account on GitHub.
It's like that it isn't structured similarly because I'm very much an amateur.
I’m having a hard time parsing through it, where are the python files?
It deployed fine 'till I tried...
Also, what start command do you run locally?
till I tried installing express and mysql
py -m http.server
oh
there are two minor and mostly-irrelevant python files in texts and gentium
texts/proofedEditions.py
which only contains a list of proofed editions
that's from back when this was a local flask app
gentium is the font that the stylesheet calls, so I don't mess with it
Right now the relevant stuff is
--node_modules which can probably be cleaned up and streamlined, but that's another learning curve right now
--texts which has the plain text files
--textJSON which has JSON files of each book. these are old-ish JSON files which I need to write a new program in JS, they're just copied from an old repository where I used Python to make themnode_modules can be entirely gitignored, can you do that now?
uhhh lessee
pff
okay
just gitignored node_modules (I think!)
Did you commit it? I’m not seeing it
did so just now
should be
node_modules/*
not just node_modules/right-oh
okay
deployment still failing
Yep that was just a step to clean up the repo, getting on my pc now to troubleshoot more
Alright, your repo is very strangely formatted. Is there a reason you stopped using flask? I think that would be the way to go, even though you only have one python file in the entire repo (?)
Uh, the reason I stopped using Flask is that I wanted more experience doing this in more-or-less pure JS.
If I were to use Flask, how would I have it boot the Flask server up when it opens the webpage?
Totally makes sense, you should be using a JS framework then rather than just running a python http server
There's a lot of work to do there if you want to use a framework, or we can just get this working as hacky as it is. Which would you like to do?
I'd like to get it working hackily if I can, and stick to JS and some basic JS libraries.
The first thing to fix is obviously just getting it deploying again, then I'd like to figure out how to get FS working
or something else that can talk to and update server-side databases or JSONs
This doesn't need to be crazy scalable; it's for a small handful of academics.
Great sounds good. Please create a file called
nixpacks.toml
with this text:
In the top of the repo?
Yes, at the root
gotcha.
unless you run that command somewhere else in the repo for some reason?
When you start locally
I don't, I'm just making sure I've got my p's and q's crossed and dotted.
Got it
pushed
build failed
but I presume in a more interesting fashion, let's see
i presume I need to change my build command
ah yes
i had set it to
CI=false npm run build
let's kill that
excellent!
it deploys
okayYeah that's the play. When you send logs, can you use this tool? https://bookmarklets.up.railway.app/log-downloader/
It'll get all the lines and puts them all into a file
gotcha
okay so it now deploys again
Great, can you access it?
through the url I mean
Uhh...
no, actually...
GET https://eliotweb-production.up.railway.app/index.html 503 (Service Unavailable)
lessee
oh wait
i'll need to install python server-side, won't I...?
Okay, python's not installed. Add this to the top of your nixpacks.toml
You also may have to change the start command to say python instead of py
aye, i'll try that once it finishes deploying or trying to
all righty
deployed successfully, now let's see if it crashes...
oh wait
Railway Config File Path Location in the repo to a file that contains Railway build and deploy config.that's currently
railway.toml
, does it need to be nixpacks.toml
No, those are two different files. nixpacks.toml controls the build, railway.toml controls railway settings
k it failed to respond, so I changed the start command to python
no logs?
If there are no logs, change the start command to
python http.server $PORT
and add an environment variable called PORT set to 8080uh, oh right
Those build logs look fine, what about the deployment logs?
Those appear to be empty?
"No deployment logs. All output sent to stdout will be displayed here"
That's expected if you're not logging anything when your app starts
Sounds like you're not
right, okay
so like this?
No, remove the PORT line from here
sorry :(
this is all incredibly new to me
Set environment variables from the environment variables tab on Railway. You can find it on the service page before you click into a deployment
no problem, all good
all righty
building now...
so like this
Yup you got it
says it deployed fine but is still not responding
do I need to give it a few?
No logs?
ah, aha
python: can't open file '/app/http.server': [Errno 2] No such file or directory
do I need to create an empty 'app' folderYou don't even have an http.server file...
look
like I said
i'm new to this
No I know, I'm just wondering how it worked before lol
I just ran
py -m http.server
from terminal to work with it locally
which seemed to work fineYou removed the -m from the start command, that's the issue here
Please add that back
aha
hmm
still not responding...
deploy logs?
aha!
there it is
hurrah
okay so
...now I should set up a database, perhaps?
so JS isn't gonna be very happy working with databases, right
or like
I should do a Python backend in Flask to do anything involving editing JSONs
First thing I would do is use a JS framework like react
Once you've done that, then you can add a database and store your strings there rather than in text files
Well I'll want the text files around anyways because those are what I edit and proofread
I...I know people really like frameworks but I'm trying to write this in bare-metal JS to the extent possible
oh, uh
can I change gitignore back to not ignoring node modules or is that a bad idea
but yeah, my main concern right now is just
setting up an sql database on the thing
and then writing programs that can read from and write to it
so i suppose what I'm asking is just
what's the easiest way to do that
preferably without using any frameworks
You can kick off a database on Railway by deploying a template from the + icon on the project dashboard
Add a postgres or mariadb database from there
(why not mysql?)
Then Google how to connect to the database and run queries and integrate that into your setup
mysql is good too
It just uses more RAM, so is more expensive
mariadb is an optimized version of mysql
got it
...will I need to remove node_modules from gitignore
i presume I will?
No, node_modules generally shouldn't be committed
got it, okay... (I think)
lessee mariadb doesn't show up
There isn't a reason to commit it since when you build your project it will reinstall all of those modules
definitely not mongodb or redis?
(mariadb isn't listed)
Did you click template before searching?
yep
these are the--
oh
template
You can choose whatever you want. In my experience mariadb has been the cheapest option
...it needs to be in a separate github repo?
seems like I should just create an sqlite3 (or whatever) and setup file locally and commit (unless there's a strong reason to do otherwise, such as that I am talking out of my backside
Railway's storage is ephemeral. When you kick off a new deploy the filesystem is wiped and reset from the github repo. Since sqlite is an on-disk database system, it will be wiped each time you deploy
There are ways to make it not do that, but it's more setup than just deploying a template instead
And yes, generally databases should be in a separate service/github repo. Having all your code and processes in one can be a recipe for disaster
Got it.
Ah, I see...
okay, I've got an sqlite3 db set up now in /eliotdb
So you've decided to do the extra setup? I don't recommend it
Oh, sorry :(
okay so I should kill that and then do mariadb instead
sorry, I misunderstood
No prob, yes
got it. okay, it seems to be...woops okay lessee
it automatically mounted to 'data', I had intended to take a screenshot and ask about that
but
data will be a volume, that's correct
volumes are a way to avoid the ephemeral storage. They persist between deploys
but within the repository, as the server sees it.
gotcha
The template should be fine as is, don't make any changes to it
so from the perspective of the pages as deployed, it's within the same folder, but it doesn't live on github
okay, because I didn't give it its own repository by accident
Services don't see each other's filesystems, you'll have to communicate with the db through the url
I'm not sure what you mean by this
well
I told it to mount to /data
i think to /data within the repository...lessee okay so how do I talk to it
You'll have to Google that for yourself, I'm not too familiar with JS
gotcha
so it looks like this
I'm confused... Why is it called sqlite3 data?
Which service is this data attached to?
er...
it's attached to eliotweb
So you DO want to do the complicated setup
okay, deleted that
no
i don't, i made a mistake
Like I've said, sqlite3 is not the way to go
okay i deleted the volume
Great. Did you make the mariadb service?
one minute, redeploying
works, good, okay
now--
okay
is there anything I want to change here
Nope, you're good to go
cool.
MariaDB KnowledgeBase
Getting Started With the Node.js Connector
Available through the Node.js repositories; install using npm.
looks like I will need to temporarily de-gitignore node_modules so I can install this, maybe?
no.
sorry.
:(
sorry I'm nearly there and will stop being obnoxious soon
No, you don't. Node_modules is just a cache for your local computer.
Okay.
But I will want to install it locally and push so that I can use it?
It's not even properly gitignored right now, I was wrong in my message about that. It should just be
You would add it to your package.json, then push
Like I said I'm not super familiar with JS so I may be wrong about how to install packages. I just know that node_modules should not be committed
awrighty. I've now got
and...pushed...
okay
oh!
anyways-- @Adam thanks very much for your help
No prob, happy to help
all right so
Firebase doesn't appear to be the right choice so back to the drawing board with uh
I chose the 'expressjs with postgres' template
I did say postgres or marian
you shouldn’t need to deploy a full stack template, just the single database service
aka, just “mariadb” or “postgres”
okay
so yeah now I have a postgres database
aaaand
(how do I set a table column as an array here?)
word should not be primary key, just fyi
All words are unique, though.
really?
This is a concordance database.
none are repeated?
gotcha
For this one, yeah, that's why I need verses and versecounts to be arrays. It stores every single verse in the texts where that word occurs, along with how many times in that verse it shows up.
adding tables is best done through a program like dbgate imo
I can walk you through getting that up and running
That'd be great, yeah.
okay
just ran
npm install -g dbgate-serve
in terminal
a handful of deprecated packages...that’ll be the dbgate package, that’s not what we’re looking for
DbGate | Open Source SQL+noSQL Database Client
Powerful cross-platform SQL client and database manager for MySQL, SQL Server, PostgreSQL, SQLite, MongoDB, Redis, Oracle. Fast, modern and 100% free.
okay, uninstalled
download this, connect to your new db using the credentials in the environment variables
it's not possible to do this from npm?
no, npm is a js package installer. this is an app
right, downloading now
actually you’re right, it is possible… I didn’t know that was a thing people did
if you want to do that, follow this page
npm
dbgate-serve
Opensource database administration tool - web interface. Latest version: 5.2.7, last published: 3 months ago. Start using dbgate-serve in your project by running
npm i dbgate-serve
. There are no other projects in the npm registry using dbgate-serve.can't hurt to have both
okay so what to fill into 'default database' here
the url?
hmm no that didn't quite work...
click
use database url
, then use the database urlhmm
oh maybe that was private
use the variable that isn't private
woot
okay
okay
(varchar (250) will have no trouble with unicode, will it...?)
no
I store unicode in varchar
great
okay so I now have a dummy table with only one column, which is each word
and a 'total count' column'
all right, so
excellent
okay, so it showed up
so now
i need to figure out how to write data to it from the website
which i know is going to need front-end / back-end stuff
and
that is where things get messy
chatgpt can help you with the code, it’s pretty good with simple stuff like this
hold up okay so
it says to run
node server.js
locally
so should I add like
currently that's really db.js
, but should I add like
node db.js
to the start instructions on the railway appright now you’re using python http.server as your start command. As I mentioned yesterday, changing your start command will likely be needed along with potentially reworking your code to fit within a framework
so yes, replace python http.server with node db.js
to prevent issues going forward I highly recommend spending the time now to standardize your app
hmm.
well, so I moved
db.js
into an app
folder
ah
does package.json
need to be in that folder toohonestly, no clue
I don’t know js
if you’re using chatgpt you could give it your current project structure and ask how you can reformat it
yep, this is really useful
thanks!
i may be back if there's anything big chatgpt can't help w/
sounds good, good luck
oh, yeah
okay so my main issue is that the thing seems unable to deploy :/
er
What are the errors?
lemme try restarting the container
Mind making your repo public again or just inviting me to it?
i mean it's right there
and db.js--
yeah no prob
one minute
GitHub
GitHub - cdnilsen/eliotweb
Contribute to cdnilsen/eliotweb development by creating an account on GitHub.
try now
i'll be back in a sec, I need to take a shower and do some laundry
Could be that db.js isn't a module? Or something like that
Just because it's a file doesn't mean it's a module
That's a code issue, not a Railway issue. I won't be able to help with that
wat the heck is going on in this repo (no offense)
amateur coding :(
brb
back
well don't you declare or not declare something to be a module when it's, like. called
oh oh oh hmm lemme try
node app/db.js
, silly me
hmm.
nope.please take adam's advice here
i do know javascript, yet i still dont know what you have going on
gotcha
(i think)
okay maybe I should start over by just having react-app create stuff in a new folder
If I’m being honest, I would scrap your whole project structure as is and start from scratch. Not deleting code, but reformatting it and getting everything done properly
Yeah, that's what I'm thinking of doing.
if you don’t clean it up, you’re going to look back on it in a few months and not have any idea what’s going on
glad to hear it
what are you trying to achieve? a backend and a frontend?
Well, I'll need both to talk to the server, yes?
the server would be the backend
the frontend talks to the backend which talks to the db
or is there another server you’re communicating with?
Uh, there's a Postgres db I've set up, as I mentioned.
I'm currently unfucking a casing issue which is quite common with React stuff and then I'll start rebuilding the thing.
pffff no that's also a total mess
okay
starting again from scratch...
Yep, I included that. Just wondering if there's another server you're communicating with, but I'm assuming that's not the case
Nope.
you would likely want a structure like this
(the period denotes the root of the repo)
two apps, a frontend and a backend, this means you will have two railway services for each app, along with the postgres database, totalling 3 services in your railway project canvas
Right, gotcha.
Okay, so, lessee...
just use this as a starting point for the frontend app https://github.com/brody192/create-react-app-starter
its ready to run on railway
okay. so /src/ is backend and /public/ is frontend?
everything in the repo i linked gets put into the frontend folder
I think a key point of confusion here is right now you don't have a backend and frontend
They're mashed together
they do be pretty mashed
Ideally, you separate them out into the frontend and backend. You could clean up your code and keep them both in the frontend, then just have that directly communicate with the db
but best case would be to separate them out
right
well i just cleaned out the repo (the old stuff is still there so I don't need to rewrite it)
so now there's a frontend and backend folder
pinned your repo so I can find it easily
thx
I think you misunderstood what brody said
^
public and src are both part of the frontend
This is all frontend
including node-modules?
you dont want node_modules in the repo anywhere, please remove it
okay
node_modules is in the gitignore...
That's weird
well, that's what I was suggested to do a while back
Yeah, but it's not ignoring the folder
okay
well
i have now gotten rid of it so
this also isnt the frontend code from the repo i linked
right just pushed
that's better
still not the frontend code from the repo i linked though lol
rightoh
done
okay, added that frontend code
there will also be no need for that nixpacks.toml file, you may remove it
the one in the root?
yeah I got rid of that
cool looks good
show me a screenshot of your railway project's canvas please
and then uh
what should the custom start command be here
do I tell it to use frontend for the root directory?
/bin/bash: node: command not found
remove the start command (and a build command if you have one)
then set the root directory to
/frontend
woot!
okay so that works
okay so
most immediately
the postgres databsae consists of uh
a table
words_diacritics
with one row and two columns: "word":"kâh" (primary key), "total_count":1
and so what I would like to do is a proof-of-concept page that can display that, and also add another word with a countwell then you need to deploy the backend.
frontend makes a request to the backend, the backend makes a query on the database, formats the resulting data, returns the formatted data back to the frontend, the frontend displays the data to the user
to deploy the backend you need a
/backend
folder with your backend code in the repo located within that folder, then you need to create another railway service in the same railway project, and set its root directory to /backend
ahhhh okay...
okay
I think I am going to go try and find some lunch
and then work on my own
thanks so much!
no problem!
So--
So, React is frontend only right?
How do I have it talk to backend?
hmm okay so I can use express in the backend...
yeah write an express app
the express code will live in a
/backend
folder within your repoRight, right.
With your frontend React app, how does it know in the first place that--I don't see
index.js
being called from index.html
?React is smart, it knows that files with the same name should be used together
OK. So what do I need to put in the backend?
server.js
and then...
oh, should I like
set up an express app from npm while terminal is in the backend folder
that would be sensible...so like
i should
cd backend
, then do npm install express
?
okay but now there are node modules in the backend.
that's still kosher?its fine locally, just dont let that folder make it to the repo, make sure its being gitignored
could I delete it with no ill effects?
or, I suppose I'll need to have it around to mess w/ it locally
lessee, uh
you need it locally, make sure it get gitignored
i can have two separate gitignore files in frontend and backend right
yes
all righty
so that should just be
node_modules
with no slashes right
okay, so, lessee
all right
and then if I added index.js
, I should have node index.js
as the start command in the backend service on Railwayheres a good resource to have on hand https://www.toptal.com/developers/gitignore/
Thanks!
OK, so
DEV Community
Connect a Railway database(postgreSQL) with node-postgres in Express
There are two options for connecting to Railway's postgres database. Using a connection string or a...
bro slow down
sorry
get a hello world demo running on railway first
an express app that just returns a json message when you visit
right
well
right now, index.js is still uh
now do I need to change the port to the port on the postgres database?
the express server has nothing to do with postgres, but that does look like it will work
right, okay
well then ignoring that port anyways...
And then
res.send
is the piece that talks to the frontend?no, that sends a response to the http request
i see...
lets just get this running on railway first
read here please
Right, I've got that. There's eliot-backend and eliot-frontend on railway and they're deploying from the backend and frontend folders respectively.
gotta stop doing this https://github.com/cdnilsen/eliotweb/blob/main/backend/db.js#L5
use environment variables please
sorry :/
okay, uh
...environment variables don't seem to be listed on the postgres service?
only service variables
lessee what google says
Service variables are environment variables
they're environment variables specific to that service
yep and you need to create a new railway service in your project to run the backend
Wait, a second service to run the backend?
of course
so I need more than just eliot-frontend and eliot-backend?
Yes, the frontend and backend should be run separately
afaik they are?
Did you name your database backend?
No, it's called...it's just called Postgres
I could change the name
Nope, it should just be called postgres
pictures are worth 1000 words, so please show us a screenshot of your railway project
You've called your db backend before, just making sure
looks good
Looks good. Your backend and frontend are in different services
is the backend's root directory set to
/backend
?Yep.
generate a domain for it
all righty
it's now at
eliotweb-backend-production.up.railway.app
(frontend is at https://eliotweb-production.up.railway.app/
)thats going to take a bit to come alive since you just created it
sure
well
its live now
whaddya know
yep
okay, so how do I un-expose the postgres URI
now the rest is up to you
okay.
great! (I think)
since unexposing the db uri is almost certainly the thing to do most immediately
(should I nuke the current postgres and start a new one for that reason?)
(it has basically nothing on it that I can't redo easily from dbgate)
yeah you leaked the username and password, delete it, create a new one, once done:
in railway on your backend service set a service variable
DATABASE_URL=${{Postgres.DATABASE_URL}}
then in code use process.env.DATABASE_URL
instead of the url stringokay. so instead of this...
i want uh
?
please read my message again
ah, right--
awright
perfect
righty, okay. uh it isn't console.logging anything like it should, but
show me your db.js code
and then index.js is:
close enough
it should be logging the port to console, though, no?
check the deploy logs
Server running on port 7935oh, so it logs it to the railway console rather than browser?
its a web server, console logs go to the place the server is running on
gotcha.
okay, well, I think I have annoyed you enough for now, and I think I do owe you a coffee, so let me do that
now to get backend talking to frontend
other way around
see this message for the flow
sorry just one last thing: The port for the postgres database is 14224, but the backend says it is running on 7935.
express has nothing to do with postgres
the ports being different is how it's supposed to be, it's of no concern
gotcha
oh just saw the trains, thank you very much!!!
no problem!
and then for the uh
if I have a
wordeditor.js
file in the backend
to look at the words_diacritics table in the database
(which I'd cribbed off an online example)
it has uh
and I assume I want to use likeyou have already created a pool in your db.js file, use that pool
port = process.env.PORT
oh, okay
please remember that the port express runs on has nothing to do with postgres
right.
so do I
import pool
into wordeditor.js
?sounds about right
lesseeee
so now I have
now...let's say I want to
getWords
somewhere else
oh, hmm
in my frontend, could I just do: declare index.js
to be a module, import getWords
from ../backend/wordeditor.js
?your backend and frontend code is separate, please dont start mixing them again
first, your backend would need to return these words via a json api
so dont touch the frontend until your backend can do that
rightoh. is there a good guide to doing that?
watch some youtube tutorials? ask chat gpt?
yeah, i've found a couple of articles
hmm hmrr
perfect
For what the example has as
do I replace localhost:3000 with process.env.DATABASE_URL?
again, express itself has nothing to do with postgres
please please remember that
right, right...
but use the cors middleware from express
It also didn't like when I tried uh
so db.js is
and then in
wordeditor.js
I have
and the deploy logs said that import wasn't valid because it wasn't a module, or something
(but wordeditor exports its functions, so)common nodejs issue that people run into, theres lots of information on it online
(also got cors installed)
right
like there's only one package.json file in backend right
yeah thats normal
so, added
"type": "module",
to that
...giving me the same thing
maybe i need to use const...require
instead of import
aha!still an issue, but a better issue
check your deploy logs
okay so this is one of those, I have called the postgres port when I needed to call the local port or vice versa
yes?
show me your db.js file
show me your service variables on the backend
is this even the right syntax lol
quite possibly not...?
have a read of pg's docs
hmmm
looks like the correct syntax is
const pool = new Pool(dbURL);
hmm
okay, that gives me new and interesting errors
thanks!
i think i'm gonna call it for the evening
alright!
morning
on the rec of a friend, I have switched to next.js
i ran
npm i -g npm-check-updates
, then ncu -u
, then npm -i
, and it still squawks about this...hmm still got this after switching to Yarn
but far fewer warnings about dependencies
chatgpt it?
chatgpt is good at solving most code issues and interpreting logs
switching from express to next, switching from npm to yarn.
are you trying to make this project harder on yourself lol
okay, friend made a basic version to talk to the server
er
but he put some TS in /src/, which compiles to /dist/ if I run
tsx -w
--but then it ends up in dist
per the tsconfig
so do I need to do like
can I run ../src/ tsx -w
as a build command in the backend and then set it to be calling its code from /dist/
i mean, i can run npx tsc
on each of the TS files in there
but still
but doing so doesn't work for index.ts
hmm
okay killed one of my services and now have only postgres and a single service
which gets its stuff from the top directory, and runs npm start
as its start command
oh right uh
i can add a connection of sorts as a service variable
in the service...
okay soawright lessee
added a bunch of extra service variables to try and call the pool
oh
i bet the issue is that I was using the database port in
index.ts
finally got the damn thing working
thanks alljust woke up, but glad you got it working!!