League of Extraordinary FoundryVTT Developers•10mo ago

Advice for compendium data in a custom system

Hello, i'm new here! I was wondering if I could get a little direction for storing data about my games items/actors. I looked at how it is done in pf2e and dnd5e. They both seem to have gone different directions. DnD5e uses .db files for their packs data https://github.com/foundryvtt/dnd5e/tree/master/packs while PF2E uses .json files https://github.com/foundryvtt/pf2e/tree/master/packs Should I be using .db files or .json files for my packs? and how should I prepare these files so Foundry can see the data in the packs? One thing I tried to do was to: - create an empty world - set up a compendium pack by hand using my systems sheets - used the foundry cli tool to 'unpack' my worlds pack data - moved those json files into my systems /packs folder.
- updated my manifest so it would point to the correct /packs, and restarted my other test world to see if it could use the packs provided by my system. The pack folders are there, but non of the items are inside. In the future, I would like to be able to curate the data into a .db file / .json files in an automated way to avoid a bunch of manual data entry through my sheets.

6 Replies

Eranziel•10mo ago

My recommendation is to use .json files, with a build script that uses the Foundry CLI to pack them into LevelDB databases for Foundry. Foundry won't read the .json files directly, it needs them in the databases. .db files are for NeDB, which is what Foundry used for its database in version 10 and earlier. v11+ uses LevelDB. If Foundry detects .db files and no LevelDBs when launching a world, it will convert those .dbs to the LevelDB format and then ignore them from that point on. Note that the CLI can use some other formats instead of .json if you prefer. I know it supports YAML, for example.

Eclipxs•10mo ago

Thanks!
So you have the .json (or yml) files in /packs for src control, and then during release before zipping the system, you run the cli to pack them into LevelDB? then zip the system?

Eranziel•10mo ago

Yup, that's the general idea

Eclipxs•10mo ago

Sweet! that makes sense now. tyvm

Eranziel•10mo ago

Examples from Lancer: - macros compendium content in yml: https://github.com/Eranziel/foundryvtt-lancer/tree/2.0-rc/src/packs/core_macros - gulpfile calls the CLI to pack those up: https://github.com/Eranziel/foundryvtt-lancer/blob/7473ea2f4491657755cf35cefb8682630d69b958/gulpfile.js#L46 - gulpfile packages the build for distribution: https://github.com/Eranziel/foundryvtt-lancer/blob/7473ea2f4491657755cf35cefb8682630d69b958/gulpfile.js#L169 I wouldn't necessarily copy-paste our gulpfile, it's based on a rather old project template and been updated over time. If you're using one of the more modern project templates they probably have a bit better tooling in place, haha. But definitely take inspiration.

Eclipxs•10mo ago

sounds good, thanks again. i'll take a look and adapt where needed 🙂 got this working with github actions if anyone is interested:

jobs:
  version:
    name: Version
    runs-on: ubuntu-latest
    outputs:
      semVer: ${{ steps.gitversion.outputs.semVer }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Setup .NET Core SDK
        uses: actions/setup-dotnet@v3.2.0
      - name: Install GitVersion
        uses: gittools/actions/gitversion/setup@v0
        with:
          versionSpec: '5.x'
      - name: Determine Version
        id: gitversion
        uses: gittools/actions/gitversion/execute@v0
  release:
    name: Create Release
    runs-on: ubuntu-latest
    needs: [version]
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Setup Node.js environment
        uses: actions/setup-node@v4.0.0
      - name: Pack packs
        run: |
          npm install -g @foundryvtt/foundryvtt-cli
          mkdir -p /home/runner/work/my-system/Data
          fvtt configure set dataPath "/home/runner/work/my-system/Data"
          fvtt package workon "my-system" --type "System"
          fvtt package pack -n "my-items" --in ./packs/my-items/_source --out ./packs
      - name: Zip Files
        run: zip -r "./my-system.zip" ./*
      - name: Release
        id: create_version_release
        uses: ncipollo/release-action@v1
        with:
          name: Release ${{needs.version.outputs.semVer}}
          allowUpdates: false
          draft: false
          prerelease: false
          token: ${{ secrets.GITHUB_TOKEN }}
          artifacts: "./my-system.zip"
          tag: ${{needs.version.outputs.semVer}}

jobs:
  version:
    name: Version
    runs-on: ubuntu-latest
    outputs:
      semVer: ${{ steps.gitversion.outputs.semVer }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Setup .NET Core SDK
        uses: actions/setup-dotnet@v3.2.0
      - name: Install GitVersion
        uses: gittools/actions/gitversion/setup@v0
        with:
          versionSpec: '5.x'
      - name: Determine Version
        id: gitversion
        uses: gittools/actions/gitversion/execute@v0
  release:
    name: Create Release
    runs-on: ubuntu-latest
    needs: [version]
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Setup Node.js environment
        uses: actions/setup-node@v4.0.0
      - name: Pack packs
        run: |
          npm install -g @foundryvtt/foundryvtt-cli
          mkdir -p /home/runner/work/my-system/Data
          fvtt configure set dataPath "/home/runner/work/my-system/Data"
          fvtt package workon "my-system" --type "System"
          fvtt package pack -n "my-items" --in ./packs/my-items/_source --out ./packs
      - name: Zip Files
        run: zip -r "./my-system.zip" ./*
      - name: Release
        id: create_version_release
        uses: ncipollo/release-action@v1
        with:
          name: Release ${{needs.version.outputs.semVer}}
          allowUpdates: false
          draft: false
          prerelease: false
          token: ${{ secrets.GITHUB_TOKEN }}
          artifacts: "./my-system.zip"
          tag: ${{needs.version.outputs.semVer}}

this was just a simple test with a single pack, though it wouldn't be hard to make this a little more generic and handle multiple packs.

Gaming

Programming

Advice for compendium data in a custom system