Phatso Posts - Answer Overflow

Phatso

Explore posts from servers

•Created by Phatso on 8/12/2024 in #help

Importing local files (streaming large JSON files)

is there an easy way to load a massive json file into memory? i found: https://deno.land/x/[email protected] but it's fetch on a local file example doesn't seem to work for me:

error: Uncaught (in promise) TypeError: Invalid URL:

error: Uncaught (in promise) TypeError: Invalid URL:

am i doing something wrong or is there a new way to load local files

6 replies

DDeno

•Created by Phatso on 5/10/2024 in #help

Websockets API with async processing (is there a better way?)

My Deno app has a websockets layer that receives "requests" from the websocket clients. The clients ask it to update data in the database, retrieve data, etc. similar to a REST api There will be 5-10 clients who will be sending a very large volume of requests. In my initial setup, I created this websockets layer that would offload the actual work for each request to a pool of Workers, who would then send the response back through the Worker pool. I'm hitting some kind of bottleneck. Despite my best efforts, I don't know how to identify it. So I have a few questions: 1. Do you know of an obvious bottleneck with this setup that I'm not realizing? 2. Is there a better way to go about this? I would really like to keep this app simple, and it doesn't feel simple right now A couple of bits of context: 1. I simply need to use Websockets. In the ecosystem I'm developing for, HTTP calls are not a good option 2. My main goal with the Worker Pool was to make sure my Websockets layer wasn't being blocked or delayed while each request is processed, but I'm starting to second guess this design Some code, in case it helps paint a better picture: The websockets layer:

import { WebSocketClient } from "websocket/mod.ts"

import type { WebSocketServer } from "websocket/mod.ts"
import type { WorkerPool } from "./worker_pool.ts"

export const handleSocket = (wss: WebSocketServer, workerPool: WorkerPool) => {
    wss.on("connection", function (ws: WebSocketClient) {
        const returnMessage = (message: string) => {
            ws.send(message)
        }

        ws.on("message", function (message: string) {
            const command = JSON.parse(message)
            const { type } = command

            if (type === "request") {
                workerPool.handleMessage(command, returnMessage)
            }
        })
    })
}

import { WebSocketClient } from "websocket/mod.ts"

import type { WebSocketServer } from "websocket/mod.ts"
import type { WorkerPool } from "./worker_pool.ts"

export const handleSocket = (wss: WebSocketServer, workerPool: WorkerPool) => {
    wss.on("connection", function (ws: WebSocketClient) {
        const returnMessage = (message: string) => {
            ws.send(message)
        }

        ws.on("message", function (message: string) {
            const command = JSON.parse(message)
            const { type } = command

            if (type === "request") {
                workerPool.handleMessage(command, returnMessage)
            }
        })
    })
}

The Worker Pool is attached as a file to this message. I'm feeling worried about my decisions here, I need to come up with a good solution soon. Any help at all is greatly appreciated 🙏

12 replies

CDCloudflare Developers

•Created by Phatso on 9/2/2023 in #general-help

"Leave cloudflared running to download the token automatically"

I use cloudflare SSH tunnels to access some of my remote servers. My SSH config and everything is configured well - it does work:

Match host <my.example.domain> exec "cloudflared access ssh-gen --hostname %h"
  ProxyCommand cloudflared access ssh --hostname %h
  IdentityFile ~/.cloudflared/%h-cf_key
  CertificateFile ~/.cloudflared/%h-cf_key-cert.pub

Match host <my.example.domain> exec "cloudflared access ssh-gen --hostname %h"
  ProxyCommand cloudflared access ssh --hostname %h
  IdentityFile ~/.cloudflared/%h-cf_key
  CertificateFile ~/.cloudflared/%h-cf_key-cert.pub

However, when I try to ssh into the server I always have to manually click the URL it generates, i.e.:

Please open the following URL and log in with your Cloudflare account:

https://<my.example.domain>/cdn-cgi/access/cli?<blah>

Please open the following URL and log in with your Cloudflare account:

https://<my.example.domain>/cdn-cgi/access/cli?<blah>

and then wait for the callback to complete, and then finally I'm connected. After that minorly annoying process, I'm teased by cloudflared with this message:

Leave cloudflared running to download the token automatically 🙂

That smug cli is telling me my life could be way easier if I just leave it running - super! Except.. can I? Any time I try to install the cloudflared service worker: sudo cloudflared service install, it complains:

Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]

Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]

Okay fine, so I make an empty config file in /etc/cloudflared:

sudo touch /etc/cloudflared/config.yaml

sudo touch /etc/cloudflared/config.yaml

And then sudo cloudflared service install:

2023-09-02T23:55:52Z ERR Configuration file /etc/cloudflared/config.yaml was empty
2023-09-02T23:55:52Z ERR Configuration file /etc/cloudflared/config.yaml was empty
Configuration file  must contain entries for the tunnel to run and its associated credentials:
tunnel: TUNNEL-UUID
credentials-file: CREDENTIALS-FILE

2023-09-02T23:55:52Z ERR Configuration file /etc/cloudflared/config.yaml was empty
2023-09-02T23:55:52Z ERR Configuration file /etc/cloudflared/config.yaml was empty
Configuration file  must contain entries for the tunnel to run and its associated credentials:
tunnel: TUNNEL-UUID
credentials-file: CREDENTIALS-FILE

But.. I don't want to make a tunnel? I'm trying to connect to a tunnel. I have to give it information about a tunnel that I do not want to start? Am I missing something? Can I just give it dummy data or something? Do I have to configure it with the specific tunnel that I'm trying to connect to?

7 replies

DDeno

•Created by Phatso on 12/6/2022 in #help

Import from variable path?

Howdy - I'm trying to make my Cloudflare Workers app self-hostable in a Docker Container. To do this, I'm going to run it with Deno when self-hosting. Ideally I would like to change none of my actual application code and defer any environment-specific setup to different files. i.e.:

src/
  - index.js
  - selfhost.js
  - cloudflare.js

src/
  - index.js
  - selfhost.js
  - cloudflare.js

And then:

// index.js
import { app, serve } from (isSelfHost ? "./selfhost.js" : "./cloudflare.js")

// index.js
import { app, serve } from (isSelfHost ? "./selfhost.js" : "./cloudflare.js")

I know this doesn't work, so my question is how do I achieve something like this? Ultimately the problem is I can't be requiring Deno packages in cloudflare workers or vice versa.

3 replies

Gaming

Programming