fr3fou Posts - Answer Overflow

fr3fou

Explore posts from servers

RRailway WWindmill MDCMara's Developper Corner

•Created by fr3fou on 11/3/2024 in #help

Relative imported module doesn't respect optional/non-imported Python dependencies

I have the following file in f/main/requirements.txt :

duckdb==1.1.2
polars[pyarrow]==1.12.0

duckdb==1.1.2
polars[pyarrow]==1.12.0

I also have a script called f/main/get_db, which only imports duckdb:

from typing import TypedDict
import duckdb
import textwrap
...

from typing import TypedDict
import duckdb
import textwrap
...

Running wmill script generate-metadata produces a script.lockfile for the get_db script correctly:

duckdb==1.1.2
polars==1.12.0
pyarrow==18.0.0

duckdb==1.1.2
polars==1.12.0
pyarrow==18.0.0

(the pyarrow & polars dependencies are required if you want to convert duckdb results to a polars dataframe) In my flow, f/main/my_flow, inside an inline script, I import the get_db module:

from typing import TypedDict
from f.main.get_db import get_db

from typing import TypedDict
from f.main.get_db import get_db

The generated lockfile, however, doesn't include the optional polars & pyarrow dependencies:

duckdb==1.1.2

duckdb==1.1.2

I've tried running wmill script generate-metadata and wmill flow generate-locks, but none of them seem to produce the desired output. A workaround is for me to include the pyarrow / polars dependencies in a comment in my inline script (under a extra_requirements section), but this means that I have to pin 2 versions separately which is undesired.

2 replies

WWindmill

•Created by fr3fou on 10/31/2024 in #help

What's the recommended way to run dbt transformations?

Currently migrating from a Prefect pipeline which uses https://github.com/datarootsio/prefect-dbt-flow and was wondering how I can run my dbt models in Windmill. This library automatically makes a new Task/Node in the DAG for each dbt model. I'm not really sure what the best way is to pull your dbt files (.sql models & yaml configurations) into Windmill, as well as running them using the dbt CLI. One way I thought of is to package my dbt models (and dbt CLI tool) in a Docker image, which I would then pull from a private registry inside Windmill. This, however, won't handle the breaking-down-into-multiple-tasks (scripts in Windmill) part. Any ideas? I'm also considering trying sqlmesh but I've never used it, but I think it has a similar "deployment" model.

2 replies

WWindmill

•Created by fr3fou on 10/30/2024 in #help

Dev/Prod envs using 2 instances (Community Edition)

Would I encounter any issues if I have 2 GitHub branches: - dev - master and use wmill sync push in a GitHub Action [with 2 different domains / tokens]?

16 replies

MDCMara's Developper Corner

•Created by fr3fou on 10/1/2024 in #🐕get-help

Images are pushed every time

9 replies

MDCMara's Developper Corner

•Created by fr3fou on 9/20/2024 in #🐕get-help

Plugin publishes unshared notes

I have a note which links to another note which doesn't have a share: true property. For some reason, however, the plugin keeps pushing the linked notes (at the root of the repo). Here's my config

{
  "github": {
    "branch": "master",
    "automaticallyMergePR": false,
    "dryRun": {
      "enable": false,
      "folderName": "github-publisher"
    },
    "tokenPath": "%configDir%/plugins/%pluginID%/env",
    "api": {
      "tiersForApi": "Github Free/Pro/Team (default)",
      "hostname": ""
    },
    "workflow": {
      "commitMessage": "[PUBLISHER] Merge",
      "name": ""
    },
    "verifiedRepo": true
  },
  "upload": {
    "behavior": "yaml",
    "defaultName": "src/content/blog",
    "rootFolder": "src/content",
    "yamlFolderKey": "type",
    "frontmatterTitle": {
      "enable": false,
      "key": "slug"
    },
    "replaceTitle": [
      {
        "regex": "/\\s+/g",
        "replacement": "-",
        "type": "title"
      }
    ],
    "replacePath": [],
    "autoclean": {
      "includeAttachments": true,
      "enable": true,
      "excluded": []
    },
    "folderNote": {
      "enable": false,
      "rename": "index.md",
      "addTitle": {
        "enable": false,
        "key": "title"
      }
    },
    "metadataExtractorPath": ""
  },
  "conversion": {
    "hardbreak": false,
    "dataview": true,
    "censorText": [
      {
        "entry": "/!\\[\\[(.*\\.((mov)|(mp4)|(webm)))\\|?(.*?)?]]/gi",
        "replace": "![$6](/videos/$1)",
        "flags": "",
        "after": false
      },
      {
        "entry": "/banner: \"\\[\\[(.*)\\]\\]\"/gi",
        "replace": "banner: ../../assets/$1",
        "flags": "",
        "after": false
      },
      {
        "entry": "/cardCover: \"\\[\\[(.*)\\]\\]\"/gi",
        "replace": "cardCover: ../../assets/$1",
        "flags": "",
        "after": false
      }
    ],
    "tags": {
      "inline": false,
      "exclude": [],
      "fields": []
    },
    "links": {
      "internal": true,
      "unshared": false,
      "wiki": true,
      "slugify": "strict"
    }
  },
  "embed": {
    "attachments": true,
    "overrideAttachments": [
      {
        "path": "/.*(mov|webm|mp4)$/",
        "destination": "/public/videos/{{name}}",
        "forcePush": false
      }
    ],
    "keySendFile": [
      "banner",
      "cardCover"
    ],
    "notes": false,
    "folder": "src/assets",
    "convertEmbedToLinks": "keep",
    "charConvert": "->",
    "unHandledObsidianExt": [],
    "sendSimpleLinks": true,
    "useObsidianFolder": false
  },
  "plugin": {
    "shareKey": "share",
    "excludedFolder": [
      "Journals"
    ],
    "copyLink": {
      "enable": false,
      "links": "https://simo.sh",
      "removePart": [],
      "transform": {
        "toUri": true,
        "slugify": "strict",
        "applyRegex": []
      }
    },
    "setFrontmatterKey": "Set",
    "shareAll": {
      "enable": false,
      "excludedFileName": "DRAFT"
    }
  },
  "tabsId": "plugin-settings"
}

{
  "github": {
    "branch": "master",
    "automaticallyMergePR": false,
    "dryRun": {
      "enable": false,
      "folderName": "github-publisher"
    },
    "tokenPath": "%configDir%/plugins/%pluginID%/env",
    "api": {
      "tiersForApi": "Github Free/Pro/Team (default)",
      "hostname": ""
    },
    "workflow": {
      "commitMessage": "[PUBLISHER] Merge",
      "name": ""
    },
    "verifiedRepo": true
  },
  "upload": {
    "behavior": "yaml",
    "defaultName": "src/content/blog",
    "rootFolder": "src/content",
    "yamlFolderKey": "type",
    "frontmatterTitle": {
      "enable": false,
      "key": "slug"
    },
    "replaceTitle": [
      {
        "regex": "/\\s+/g",
        "replacement": "-",
        "type": "title"
      }
    ],
    "replacePath": [],
    "autoclean": {
      "includeAttachments": true,
      "enable": true,
      "excluded": []
    },
    "folderNote": {
      "enable": false,
      "rename": "index.md",
      "addTitle": {
        "enable": false,
        "key": "title"
      }
    },
    "metadataExtractorPath": ""
  },
  "conversion": {
    "hardbreak": false,
    "dataview": true,
    "censorText": [
      {
        "entry": "/!\\[\\[(.*\\.((mov)|(mp4)|(webm)))\\|?(.*?)?]]/gi",
        "replace": "![$6](/videos/$1)",
        "flags": "",
        "after": false
      },
      {
        "entry": "/banner: \"\\[\\[(.*)\\]\\]\"/gi",
        "replace": "banner: ../../assets/$1",
        "flags": "",
        "after": false
      },
      {
        "entry": "/cardCover: \"\\[\\[(.*)\\]\\]\"/gi",
        "replace": "cardCover: ../../assets/$1",
        "flags": "",
        "after": false
      }
    ],
    "tags": {
      "inline": false,
      "exclude": [],
      "fields": []
    },
    "links": {
      "internal": true,
      "unshared": false,
      "wiki": true,
      "slugify": "strict"
    }
  },
  "embed": {
    "attachments": true,
    "overrideAttachments": [
      {
        "path": "/.*(mov|webm|mp4)$/",
        "destination": "/public/videos/{{name}}",
        "forcePush": false
      }
    ],
    "keySendFile": [
      "banner",
      "cardCover"
    ],
    "notes": false,
    "folder": "src/assets",
    "convertEmbedToLinks": "keep",
    "charConvert": "->",
    "unHandledObsidianExt": [],
    "sendSimpleLinks": true,
    "useObsidianFolder": false
  },
  "plugin": {
    "shareKey": "share",
    "excludedFolder": [
      "Journals"
    ],
    "copyLink": {
      "enable": false,
      "links": "https://simo.sh",
      "removePart": [],
      "transform": {
        "toUri": true,
        "slugify": "strict",
        "applyRegex": []
      }
    },
    "setFrontmatterKey": "Set",
    "shareAll": {
      "enable": false,
      "excludedFileName": "DRAFT"
    }
  },
  "tabsId": "plugin-settings"
}

96 replies

WWindmill

•Created by fr3fou on 7/22/2024 in #help

`ApiError: Internal Server Error` when writing a 2MB csv file

Using the wmill.writeS3File function to write a small csv file from ~20k lines and it seems to give this error.

ApiError: Internal Server Error
ApiError: Internal Server Error
    at catchErrorCodes (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:224:15)
    at file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:269:45
    at Generator.next (<anonymous>)
    at fulfilled (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:5:58)
    at eventLoopTick (ext:core/01_core.js:168:7)

ApiError: Internal Server Error
ApiError: Internal Server Error
    at catchErrorCodes (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:224:15)
    at file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:269:45
    at Generator.next (<anonymous>)
    at fulfilled (file:///tmp/windmill/cache/deno/npm/registry.npmjs.org/windmill-client/1.365.0/dist/core/request.js:5:58)
    at eventLoopTick (ext:core/01_core.js:168:7)

If I try the same script with less data (thus making the file smaller), e.g. 5k csv lines it works fine. Upping it to 10k (it's ~1MB then), it still crashes. I'm also confident that it's not the serialization that's failing and it is indeed the call to writeS3File. Running CE v1.365.0, self hosted, Deno TypeScript, Self Hosted Minio

3 replies

MDCMara's Developper Corner

•Created by fr3fou on 6/2/2024 in #🐕get-help

Is it possible to somehow publish assets without a note?

I want to make an image gallery with all the images stored in Obsidian in different folders, e.g. Nature/ Urban/ Interior/ etc. I want to simply push all the assets into the respective folder into my GitHub repo without a note referencing them.

3 replies

MDCMara's Developper Corner

•Created by fr3fou on 5/27/2024 in #🐕get-help

Refresh all published notes doesn't seem to work