Budi Posts - Answer Overflow

Budi

Explore posts from servers

•Created by Budi on 10/28/2024 in #✋｜help

How to sync state of production database to PR or staging environment?

I locally develop using a Postgres instance run via Docker. My production environment in Railway has a Postgres service. For each PR Railway creates a staging environment, with a Postgres db service. Is there any way to get the state of database in the PR environment (or maybe a dedicated staging environment) to be the same as the production environment? It'd be great to be able to test my database migrations in a staging or PR environment where my db state is identical to production, to avoid making mistakes. Thanks as always!

37 replies

DTDrizzle Team

•Created by Budi on 10/27/2024 in #help

How to keep dev and production DBs in sync?

I'm using Drizzle in a SvelteKit project. My local database is Postgres run via Docker. My production and staging databases are Postgres on Railway, also via a Docker instance. I'm wondering how to keep these in sync? Local development with drizzle-kit generate and drizzle-kit migrate is working well, however I'm running into an issue where a PR I'm merging has so many schema changes that that the migration files generated for my local config aren't successfully being applied to my production database. I get errors like these:

{

  severity_local: 'NOTICE',

  severity: 'NOTICE',

  code: '42P06',

  message: 'schema "drizzle" already exists, skipping',

  file: 'schemacmds.c',

  line: '132',

  routine: 'CreateSchemaCommand'

}

{

  severity_local: 'NOTICE',

  severity: 'NOTICE',

  code: '42P07',

  message: 'relation "__drizzle_migrations" already exists, skipping',

  file: 'parse_utilcmd.c',

  line: '207',

  routine: 'transformCreateStmt'

}

Migration failed 🚨: PostgresError: syntax error at or near "q"

    at ErrorResponse (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:788:26)

    at handle (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:474:6)

    at TLSSocket.data (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:315:9)

    at TLSSocket.emit (node:events:519:28)

    at addChunk (node:internal/streams/readable:559:12)

    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)

    at TLSSocket.Readable.push (node:internal/streams/readable:390:5)

    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23) {

  severity_local: 'ERROR',

  severity: 'ERROR',

  code: '42601',

  position: '1',

  file: 'scan.l',

  line: '1241',

  routine: 'scanner_yyerror'

}

{

  severity_local: 'NOTICE',

  severity: 'NOTICE',

  code: '42P06',

  message: 'schema "drizzle" already exists, skipping',

  file: 'schemacmds.c',

  line: '132',

  routine: 'CreateSchemaCommand'

}

{

  severity_local: 'NOTICE',

  severity: 'NOTICE',

  code: '42P07',

  message: 'relation "__drizzle_migrations" already exists, skipping',

  file: 'parse_utilcmd.c',

  line: '207',

  routine: 'transformCreateStmt'

}

Migration failed 🚨: PostgresError: syntax error at or near "q"

    at ErrorResponse (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:788:26)

    at handle (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:474:6)

    at TLSSocket.data (file:///app/node_modules/.pnpm/[email protected]/node_modules/postgres/src/connection.js:315:9)

    at TLSSocket.emit (node:events:519:28)

    at addChunk (node:internal/streams/readable:559:12)

    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)

    at TLSSocket.Readable.push (node:internal/streams/readable:390:5)

    at TLSWrap.onStreamRead (node:internal/stream_base_commons:191:23) {

  severity_local: 'ERROR',

  severity: 'ERROR',

  code: '42601',

  position: '1',

  file: 'scan.l',

  line: '1241',

  routine: 'scanner_yyerror'

}

I think that I'm missing a reliable process that I can use for local > staging > production development. Thanks!

8 replies

RRailway

•Created by Budi on 10/4/2024 in #✋｜help

Got Railway credits, unsure why

Was curious to learn about how I acquired these credits. Obviously happy I have them, but would love to understand what I did that resulted in getting them.

10 replies

RRailway

•Created by Budi on 9/13/2024 in #✋｜help

Possible to get Railway variables in Github Action?

I have postgres next to my app service in Railway. Automated Github environments are enabled in Railway. I wanted to run Drizzle's database management tool, Drizzle Studio on each Railway deploy. To do that, I wanted to use a Github Action, so that when I push to my branch it'll run the studio. However I run into the problem that I don't know how to dynamically insert the DATABASE_URL service variable that Railway generates. I could hardcode it for my main or staging environments, but was wondering if there's a way to dynamically insert these? If not, any suggestions on how I could run a database admin tool that allows me to edit the postgres db spun up for each new Railway environment? Thx

21 replies

XXata

•Created by Budi on 9/10/2024 in #help

Plans to support Railway for PR based workflows?

Would be great to have env variables injected on automatic Github PR deployments that Railway creates. https://docs.railway.app/guides/github-autodeploys Is this on the roadmap?

3 replies

RRailway

•Created by Budi on 9/3/2024 in #✋｜help

Add caching to Node Dockerfile

I want to ask for feedback on my Dockerfile. This is the first time I've set up caching. This Dockerfile works. My first deploy of this file was 143 secs, the second 124 secs. But that's still really slow IMO for that second try. Have I set this up properly for Node and PNPM as package manager? Is there anything else I can do to speed these builds up? I also noticed it's required to hardcode your session-id. I've now bound this to the service-id of my main production environment in Railway. What are the implication of this when a different environment is created in Railway for your project, for example when you create a new PR on Github? Thanks!

# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

59 replies

RRailway

•Created by Budi on 9/2/2024 in #✋｜help

Help with Github automatic deployments

I've configured Github Automatic deployments per the docs. My production service is linked to my /main branch in the repo. I further have PR environments enabled and give them the env vars of main. 1. New Railway environments are created successfully whenever we create a new PR, however they don't appear to close after we merge and close the PRs. 2. Also, the environments don't start with any services and therefore don't automatically deploy anything. The only workaround I've found is to manually enter the PR env in the Railway GUI and to sync the services with the ones running in the production environment. 3. The Railway bot comments on our PRs but always says the environment's deleted. See screenshot. How can I fix these issues? Thanks.

10 replies

RRailway

•Created by Budi on 8/31/2024 in #✋｜help

Private networking in SvelteKit app

I've deployed a SvelteKit application on Railway using a Dockerfile. Within that project I have a separate service for cron jobs, also deployed with a Dockerfile. The cron service calls endpoints on my SvelteKit application. This works fine using the public domain of the app service using HTTPS, but when attempting to use private networking it doesn't work. In my app service, I've configured the server to listen on 0.0.0.0 with Railway's injected port. This builds well and logs Listening on 0.0.0.0:8080, so I know it's working.

vite.config.ts:

import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vite';

export default defineConfig({
    plugins: [sveltekit()],
    optimizeDeps: {
        include: ['src/lib/components/**/*.svelte', 'src/routes/**/*.svelte']
    },

    /* Railway requires http:// to be used for private networking, as well as a port to specified. This is injected automatically during run time.

    https://docs.railway.app/guides/public-networking#port-variable
    https://docs.railway.app/guides/private-networking
    
    */
    server: {
        port: Number(process.env.PORT),
        host: process.env.HOST || '0.0.0.0'
    }
});

vite.config.ts:

import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vite';

export default defineConfig({
    plugins: [sveltekit()],
    optimizeDeps: {
        include: ['src/lib/components/**/*.svelte', 'src/routes/**/*.svelte']
    },

    /* Railway requires http:// to be used for private networking, as well as a port to specified. This is injected automatically during run time.

    https://docs.railway.app/guides/public-networking#port-variable
    https://docs.railway.app/guides/private-networking
    
    */
    server: {
        port: Number(process.env.PORT),
        host: process.env.HOST || '0.0.0.0'
    }
});

37 replies

DTDrizzle Team

•Created by Budi on 3/31/2024 in #help

What's the best way to query my db?

I'm running Supabase Postgres locally with Drizzle. In production, I'd use the connection pooler which would ensure I don't have too many concurrent connections. I can't locally however and Supabase often gives this error: PostgresError: remaining connection slots are reserved for non-replication superuser connections. I'm initiating the db as such:

import { drizzle } from "drizzle-orm/postgres-js"
import postgres from "postgres"
import * as schema from "../schemas/drizzleSchema"

export default function db() {
    const connectionString = process.env.DATABASE_URL

    if (!connectionString) {
        throw new Error("LOCAL_DATABASE_URL is undefined")
    }

    const client = postgres(connectionString)

    return drizzle(client, { schema })
}

import { drizzle } from "drizzle-orm/postgres-js"
import postgres from "postgres"
import * as schema from "../schemas/drizzleSchema"

export default function db() {
    const connectionString = process.env.DATABASE_URL

    if (!connectionString) {
        throw new Error("LOCAL_DATABASE_URL is undefined")
    }

    const client = postgres(connectionString)

    return drizzle(client, { schema })
}

And then in the necessary functions I import db and call it, like so: const database = db(). I have many helper functions which each create a db connection for the query, for example:

export async function getPostById(postId: number) {
    try {
        const database = db()
        return await database.query.posts.findFirst({ where: eq(posts.id, postId) })
    } catch (err) {
        error(400, "Couldn't get post")
    }
}

export async function getPostById(postId: number) {
    try {
        const database = db()
        return await database.query.posts.findFirst({ where: eq(posts.id, postId) })
    } catch (err) {
        error(400, "Couldn't get post")
    }
}

Is this the right way of doing things? Or should I pass the database to the helper functions, instead of initiating a connection there? I want to ensure I'm not creating unnecessary connections to the DB. Are there any best practices I should be aware of? Thanks!

8 replies

DTDrizzle Team

•Created by Budi on 3/30/2024 in #help

Studio not enough info to infer relation

I'm getting this error:

  throw new Error(
        ^

Error: There is not enough information to infer relation "__public__.posts.tags"

  throw new Error(
        ^

Error: There is not enough information to infer relation "__public__.posts.tags"

Here's my schema: https://github.com/budivoogt/bv_markdown_blog/blob/a15ec5062337460647693dd070e504e9eca54e14/src/lib/schemas/drizzleSchema.ts I believe I followed the docs carefully and ensured every table has its relations defined. The app works fine but studio doesn't launch. Any suggestions?

19 replies

DTDrizzle Team

•Created by Budi on 3/15/2024 in #help

Can't access innerJoin result in page data using Svelte

In src/routes/+layout.server.ts I have this load function, which returns postTags via a Drizzle query to my Supabase Postgres DB. It's successfully returning a result which I'm able to log to my console. However, I am unable to access the property via export let data or $page.data on child pages. Strangely, the other properties returned from my load functions are made accessible fine. I don't have any load functions that override the returned postTags in this server load function. Why could this be? I can't figure it out and since the other properties are returning fine, I doubt it's a Svelte issue. I'm not getting any typescript errors either. So I suspect it may be something related to the ORM. Thanks! Relevant code: https://github.com/brucey0x/bv_markdown_blog/blob/ebe102f5a4713443d5b2e8e1c1a77cad6b88403c/src/routes/%2Blayout.server.ts https://github.com/brucey0x/bv_markdown_blog/blob/ebe102f5a4713443d5b2e8e1c1a77cad6b88403c/src/app.d.ts

4 replies

DTDrizzle Team

•Created by Budi on 12/29/2023 in #help

Drizzle Studio not working with bun

I had a basic project running with pnpm, then switched to bun and removed the dotenv package and its imports in /drizzle.config.ts. Now when I run bun drizzle-kit studio I get this error: /drizzle/drizzle.config.json file does not exist Previously it didn't look in the /drizzle folder for the config file, and automatically detected that my config was in root and ended with a .ts extension. If I run bun drizzle-kit studio --config ../drizzle.config.ts it picks up the config file, but I get this error:

 Invalid input  "url" is a required option for driver "turso". You can read more about drizzle.config: https://orm.drizzle.team/kit-docs/config-reference
 Invalid input  "authToken" is a required option for driver "turso". You can read more about drizzle.config: https://orm.drizzle.team/kit-docs/config-reference
error: "drizzle-kit" exited with code 1

 Invalid input  "url" is a required option for driver "turso". You can read more about drizzle.config: https://orm.drizzle.team/kit-docs/config-reference
 Invalid input  "authToken" is a required option for driver "turso". You can read more about drizzle.config: https://orm.drizzle.team/kit-docs/config-reference
error: "drizzle-kit" exited with code 1

Here's my drizzle.config.ts:

import type { Config } from "drizzle-kit"

export default {
    schema: "./drizzle/schema.ts",
    out: "./drizzle/migrations",
    driver: "turso",
    dbCredentials: {
        url: process.env.LOCAL_DATABASE_URL as string,
        authToken: process.env.LOCAL_DATABASE_AUTH_TOKEN as string
    },
    // Print all statements
    verbose: true,
    // Always ask for confirmation, even they aren't breaking
    strict: true,
    // Execute every migration statement individually
    breakpoints: true
} satisfies Config

import type { Config } from "drizzle-kit"

export default {
    schema: "./drizzle/schema.ts",
    out: "./drizzle/migrations",
    driver: "turso",
    dbCredentials: {
        url: process.env.LOCAL_DATABASE_URL as string,
        authToken: process.env.LOCAL_DATABASE_AUTH_TOKEN as string
    },
    // Print all statements
    verbose: true,
    // Always ask for confirmation, even they aren't breaking
    strict: true,
    // Execute every migration statement individually
    breakpoints: true
} satisfies Config

The one change I've made to my config file is that I removed the dotenv package that I previously used (when still using pnpm/node). It seems that drizzle studio isn't picking up the env variables via bun. Is this intentional behaviour? How should I resolve this? Thanks!

2 replies

Gaming

Programming