Development setup for testing?
Starting my Foundry development journey and documenting it as I go along. I'm setting up my development environment. Started by reviewing contents of the foundryvtt.wiki Development notes, the various pages from foundryvtt.com on package development, scanning this Discord server, and the https://foundryvttt-hub.com site. I've found a few blog posts on module development and have been reviewing code repositories for various modules.
I have, so far, seen no automated testing (the
Quench module implies that someone has some testing but I have yet to see any in the wild---and the irony that the quench module itself contains no test is not lost on me).
So, does anyone use automated tests? If so, what and how?
I guess I want to check that I'm not missing something obvious and in danger of 'reinventing the wheel' before I go too far down this path.
Current thinking
For integration/end-end testing I currently my plan is to use docker to run up test instances of FoundryVTT. The 'module under test' being mounted into the modules directory and a modeJS script to use Setup.installPackage to recursively install dependent packages and game system (with options to specify additional installs). Create test world etc. either through code, snapshot, or using UI driver (Playwright). Then run integration/end-to-end test suits (mocha for integration, Playwright for end-to-end).
Unit testing is simpler as it does not require Foundry or dependencies.
This configuration should allow automated testing (especially for CI) and, importantly, allow me to easily test the impact when dependencies update.Foundry Virtual Tabletop | Foundry Virtual Tabletop
The official website and community for Foundry Virtual Tabletop.
7 Replies
@Ulenar Of Mondarth There have only been a dozen or so attempts to create some automated testing setup running with Foundry, but none resulted in any kind of framework someone could plug their package into. Which is to say I'm interested, so please do ping me once you have a setup :excitedsprout:.
Re: Quench, the module does contain tests, just no CI runnable ones. Look at the nonsense/example tests.
Thanks, will ping you if I make progress. Can you point to details of any of the aborted attempts? (If nothing else they will tell me what didn't work out.) Re:
@Ulenar Of Mondarth gave :vote: LeaguePoints™ to @Ethaks (#7 • 387)
I don't think I ever even got to see something like a proper repo for most any of those endeavours. There's an example of a setup decidedly not intended for CI usage for Dice So Nice, so that might be of interest. @LukeAbby also expressed interest in test automation in the past, not sure whether there's anything public though.
Something to keep in mind for automated installs is that Foundry cannot be distributed in any shape or form, and Foundry does not take kindly to (repeated) automated downloads of the software. IIRC that's been a point of contention in the past. Downloading a copy locally with docker for actual play is not really an issue (or all that different from "normal" downloads), but a CI setup hitting the download for every commit is probably a no go, so caching might be necessary on that front.
As Ethaks said, I've automated my manual tests 😄
I only run Playwright on my computer because of the issue highlighted by Ethaks.
It takes screenshots and compare them and create an HTML report that I manually review when some screenshots are using random numbers
Feel free to check my repo and take whatever you need, I've some code in here to open a world and login as a user before triggering macros and code
Yeah, using a local cache for the docker should avoid excessive downloads and since its cached rather than distributed as part of the image it avoids distribution issues. I'll take a stab at setting up a CI workflow with testing and folks can tell me where I screwed up 😉
Thanks. Will take a look.
@Ulenar Of Mondarth gave :vote: LeaguePoints™ to @JDW [UTC+1] (#12 • 274)