I think I figured it out.
I added a middleware, as you suggested to filaments AdminPanelProvider.php.
That middleware checks if the host I'm accessing is present on the central domains. If not, a 403 is shown.
public function handle(Request $request, Closure $next): Response