K
Kord4mo ago
vyfor

[Kotlin/Native] Linking stage fails

I've spent days trying to figure out this issue, and it turns out that the build fails whenever I explicitly specify a Ktor engine. The Kord native build fails on linux (irrelevant of dependencies) with the following error:
The /home/runner/.konan/dependencies/x86_64-unknown-linux-gnu-gcc-8.3.0-glibc-2.19-kernel-4.9-2/x86_64-unknown-linux-gnu/bin/ld.gold command returned non-zero exit code: 1.
output:
/tmp/konan_temp5888808197062752774/kord-test.kexe.o:out:function libcurl_curl_ws_send_wrapper76: error: undefined reference to 'curl_ws_send'
/tmp/konan_temp5888808197062752774/kord-test.kexe.o:out:function libcurl_curl_ws_meta_wrapper77: error: undefined reference to 'curl_ws_meta'
The /home/runner/.konan/dependencies/x86_64-unknown-linux-gnu-gcc-8.3.0-glibc-2.19-kernel-4.9-2/x86_64-unknown-linux-gnu/bin/ld.gold command returned non-zero exit code: 1.
output:
/tmp/konan_temp5888808197062752774/kord-test.kexe.o:out:function libcurl_curl_ws_send_wrapper76: error: undefined reference to 'curl_ws_send'
/tmp/konan_temp5888808197062752774/kord-test.kexe.o:out:function libcurl_curl_ws_meta_wrapper77: error: undefined reference to 'curl_ws_meta'
37 Replies
vyfor
vyforOP4mo ago
windows build succeeds. minimal repro: https://github.com/vyfor/kord-test i think I need to build curl from source with websockets support. i copied a script that does just that, but now the linker is unable to find curl at all
SchlaubiBus
SchlaubiBus4mo ago
GitHub
setup-curl/action.yml at v1 · kordlib/setup-curl
Contribute to kordlib/setup-curl development by creating an account on GitHub.
SchlaubiBus
SchlaubiBus4mo ago
you can also find an example here
GitHub
GitHub - DRSchlaubi/alphabet: Test bot for Kord on Kotlin/JS and Ko...
Test bot for Kord on Kotlin/JS and Kotlin/Native. Contribute to DRSchlaubi/alphabet development by creating an account on GitHub.
vyfor
vyforOP4mo ago
works like a charm, thank you ! hey, i'm new to Docker, and i tried using that Docker template, but i'm not sure how the application is built. i added my build command to the Dockerfile, but it's just skipping it, and in the end, it can't find the compiled application:
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 229B done
#1 DONE 0.0s
#2 [internal] load metadata for ghcr.io/kordlib/docker:main
#2 DONE 0.8s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [internal] load build context
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [2/4] WORKDIR /usr/app
#5 CACHED
#6 [3/4] RUN ./gradlew linuxX64MainBinaries --no-daemon
#6 CACHED
#7 [4/4] COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot
#7 ERROR: failed to calculate checksum of ref ...: "/build/bin/linuxX64/releaseExecutable/Yu.kexe": not found
#8 [1/4] FROM ghcr.io/kordlib/docker:main@sha256:c0ddc1a985f3ec5a8173d1874fdb1b2a462ae5ce7b0da148b212369b410f016e
#8 resolve ghcr.io/kordlib/docker:main@sha256:c0ddc1a985f3ec5a8173d1874fdb1b2a462ae5ce7b0da148b212369b410f016e done
#8 DONE 0.0s
------
> [4/4] COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot:
------
Dockerfile:5
--------------------
3 | WORKDIR /usr/app
4 | RUN ./gradlew linuxX64MainBinaries --no-daemon
5 | >>> COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot
6 |
7 | ENTRYPOINT ["/usr/app/bot"]
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref iqp39297sznk0a0ur6cmttkwd::7x6n37e3cthl222u0dws4c34f: "/build/bin/linuxX64/releaseExecutable/Yu.kexe": not found
exit status 1
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 229B done
#1 DONE 0.0s
#2 [internal] load metadata for ghcr.io/kordlib/docker:main
#2 DONE 0.8s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [internal] load build context
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [2/4] WORKDIR /usr/app
#5 CACHED
#6 [3/4] RUN ./gradlew linuxX64MainBinaries --no-daemon
#6 CACHED
#7 [4/4] COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot
#7 ERROR: failed to calculate checksum of ref ...: "/build/bin/linuxX64/releaseExecutable/Yu.kexe": not found
#8 [1/4] FROM ghcr.io/kordlib/docker:main@sha256:c0ddc1a985f3ec5a8173d1874fdb1b2a462ae5ce7b0da148b212369b410f016e
#8 resolve ghcr.io/kordlib/docker:main@sha256:c0ddc1a985f3ec5a8173d1874fdb1b2a462ae5ce7b0da148b212369b410f016e done
#8 DONE 0.0s
------
> [4/4] COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot:
------
Dockerfile:5
--------------------
3 | WORKDIR /usr/app
4 | RUN ./gradlew linuxX64MainBinaries --no-daemon
5 | >>> COPY build/bin/linuxX64/releaseExecutable/Yu.kexe /usr/app/bot
6 |
7 | ENTRYPOINT ["/usr/app/bot"]
--------------------
error: failed to solve: failed to compute cache key: failed to calculate checksum of ref iqp39297sznk0a0ur6cmttkwd::7x6n37e3cthl222u0dws4c34f: "/build/bin/linuxX64/releaseExecutable/Yu.kexe": not found
exit status 1
gdude
gdude4mo ago
Run the build outside of the container the copy command is for copying files from outside the container into it but you shouldn't build in the container b/c you'll be leaving all the development stuff in the layers alternatively, you can use docker to build, but don't use a dockerfile for that, use the docker command directly and map a volume between a local build directory and the target directory in the container but yeah really the canonical way to do this is to build outside of the container and then copy the result into it
vyfor
vyforOP4mo ago
ohh right, that's how it works i'll give it a go
gdude
gdude4mo ago
there isn't really any good reason to build inside the container you can't run eg an ARM container on x64 anyway so do you want a bit more in-depth info?
vyfor
vyforOP4mo ago
that'd be awesome
gdude
gdude4mo ago
so, containers are structured in a more complex way than you'd expect when you write a dockerfile, you're instructing the docker daemon on how to build the final container image but it actually creates a separate layer for each actionable command what that means is if, for example, you have a step running git, then a step downloading a toolchain, then a step building the project, then a step deleting the git files... each one of those is a layer docker has to download and deploy whenever you use the container, and those layers also have to be uploaded to your docker registry you really want to avoid doing that if you can the containers themselves run in a very light virtualisation layer that shares the kernel with the host OS which means you can't really cross-compile easily and the containers you're building and running will be specific to your machine's architecture tho these days that's really just ARM and x64 (there are others, but commonly)
vyfor
vyforOP4mo ago
oh hmm i never realized that
gdude
gdude4mo ago
docker is a lot simpler in concept than most people expect then you have volumes, which allow you to map a path in a container to a mount of some sort often that'll be a bind mount to a path on the host, but not always the dockerfile won't contain a mapping, but you can still define the volumes you expect to exist but you probably don't need to worry about that right now
gdude
gdude4mo ago
this reference is pretty handy https://docs.docker.com/reference/dockerfile/
Docker Documentation
Dockerfile reference
Find all the available commands you can use in a Dockerfile and learn how to use them, including COPY, ARG, ENTRYPOINT, and more.
gdude
gdude4mo ago
and if you need to generate a dockerfile based on your project's data, I released this today https://docs.kordex.dev/docker-plugin
vyfor
vyforOP4mo ago
gotcha! i've never had to deal with Docker before, i went for it as the hosting wouldn't allow me to do authorized stuff unless i used a container, though i found a workaround just now
gdude
gdude4mo ago
docker is very useful
vyfor
vyforOP4mo ago
i don't understand one thing though
gdude
gdude4mo ago
I'd recommend learning it for deployment work ah?
vyfor
vyforOP4mo ago
you said that i have to run the build command outside of the container, but these two actually depend on each other the linker needs curl setup beforehand
gdude
gdude4mo ago
you'd build against curl statically that way it'd be included in your final artefact I am not sure if k/n can actually do that, though
vyfor
vyforOP4mo ago
i think the dockerfile builds libcurl, but expects the application to be already compiled
gdude
gdude4mo ago
if you're building an application then you should have everything in the same environment libcurl is a very common library if you need to custom-compile it, then your build script should be doing that you can technically still do that in docker, even in a dockerfile, but you don't want to upload that container once you've built it so what you could do is build the container, then run it locally by tag using a bind mount volume so you can get the output generated in some local directory then you've got libcurl and you can put it wherever it needs to be for the project to build and then create a new container using another dockerfile though really if you're building in docker you should be using a one-shot single use container there's no reason you can't compile libcurl on the host if you're compiling this project as well
vyfor
vyforOP4mo ago
got it, either ways im forced to use a container the build keeps failing though
gdude
gdude4mo ago
The github actions yaml explains exactly how to compile libcurl with websocket support Without a container
vyfor
vyforOP4mo ago
nevermind, i got the build working thank you guys ! though i'd appreciate it if kordx.emoji (feature/mpp) could be republished
SchlaubiBus
SchlaubiBus4mo ago
I would say that building in a seperate container (multi-stage build) is the canonical way these days however building outside is better for caching
SchlaubiBus
SchlaubiBus4mo ago
https://github.com/vyfor/kord-native-docker That's not how you're supposed to do it
GitHub
GitHub - vyfor/kord-native-docker: Docker base images for Kord nati...
Docker base images for Kord native applications. Contribute to vyfor/kord-native-docker development by creating an account on GitHub.
vyfor
vyforOP4mo ago
that's not how i'm doing it
SchlaubiBus
SchlaubiBus4mo ago
You can check my Dockerfile and workflow Is the other repo public
vyfor
vyforOP4mo ago
FROM gradle:8.9.0-jdk11 as builder
WORKDIR /app
COPY . .

RUN apt-get update && apt-get install -y \
build-essential \
nghttp2 \
libnghttp2-dev \
libssl-dev \
make \
g++ \
curl \
libpsl-dev \
libcurl4-openssl-dev \
libssl3 \
ca-certificates \
psl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

RUN curl -o curl.tgz https://curl.se/download/curl-8.9.1.tar.gz
RUN tar xzvf curl.tgz && \
cd curl-*/ && \
./configure --prefix=/usr/local --with-openssl --enable-websockets --with-nghttp2 && \
make -j4 && make install && ldconfig

RUN cp -r /usr/local/lib/libcurl* /usr/lib/ && \
cp -r /usr/local/lib/libcurl* /usr/lib64 && \
cp -r /usr/local/include/curl/ /usr/include && \
cp -r /usr/local/lib/libcurl.a /usr/lib/libcurl.a && \
cp -r /usr/local/lib/libcurl.la /usr/lib/libcurl.la && \
cp -r /usr/local/lib/libcurl.so /usr/lib/libcurl.so && \
cp -r /usr/local/lib/libcurl.so.4 /usr/lib/libcurl.so.4 && \
cp -r /usr/local/lib/libcurl.so.4.8.0 /usr/lib/libcurl.so.4.8.0
ENV LD_LIBRARY_PATH=/usr/local/lib:/usr/lib:$LD_LIBRARY_PATH

RUN ./gradlew linkReleaseExecutableLinuxX64
RUN mv ./build/bin/linuxX64/releaseExecutable/Yu.kexe ./Yu && \
chmod +x ./Yu

CMD ["./Yu"]
FROM gradle:8.9.0-jdk11 as builder
WORKDIR /app
COPY . .

RUN apt-get update && apt-get install -y \
build-essential \
nghttp2 \
libnghttp2-dev \
libssl-dev \
make \
g++ \
curl \
libpsl-dev \
libcurl4-openssl-dev \
libssl3 \
ca-certificates \
psl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

RUN curl -o curl.tgz https://curl.se/download/curl-8.9.1.tar.gz
RUN tar xzvf curl.tgz && \
cd curl-*/ && \
./configure --prefix=/usr/local --with-openssl --enable-websockets --with-nghttp2 && \
make -j4 && make install && ldconfig

RUN cp -r /usr/local/lib/libcurl* /usr/lib/ && \
cp -r /usr/local/lib/libcurl* /usr/lib64 && \
cp -r /usr/local/include/curl/ /usr/include && \
cp -r /usr/local/lib/libcurl.a /usr/lib/libcurl.a && \
cp -r /usr/local/lib/libcurl.la /usr/lib/libcurl.la && \
cp -r /usr/local/lib/libcurl.so /usr/lib/libcurl.so && \
cp -r /usr/local/lib/libcurl.so.4 /usr/lib/libcurl.so.4 && \
cp -r /usr/local/lib/libcurl.so.4.8.0 /usr/lib/libcurl.so.4.8.0
ENV LD_LIBRARY_PATH=/usr/local/lib:/usr/lib:$LD_LIBRARY_PATH

RUN ./gradlew linkReleaseExecutableLinuxX64
RUN mv ./build/bin/linuxX64/releaseExecutable/Yu.kexe ./Yu && \
chmod +x ./Yu

CMD ["./Yu"]
i just copied the gh action
SchlaubiBus
SchlaubiBus4mo ago
yeah that's also not what you are supposed to do
vyfor
vyforOP4mo ago
hmm
SchlaubiBus
SchlaubiBus4mo ago
ghcr.io/kordlib/docker:main is a runtime image, not a build-time image it's there so you don't need to build curl inside your container you install a bunch of things in your container you do not need
vyfor
vyforOP4mo ago
i know that some of them aren't really necessary, but i've been struggling to get this app working for days. i got frustrated and ended up copying things from all kinds of gists until it finally worked lol
SchlaubiBus
SchlaubiBus4mo ago
This way you just bloat your docker image since you have the entire c toolchain
vyfor
vyforOP4mo ago
got it, do you see any other issues with the dockerfile?
SchlaubiBus
SchlaubiBus4mo ago
generially you shouldn't build in the end-product image, also you should not use cp and mv but the COPY and MOVE commands simply copy mine
vyfor
vyforOP4mo ago
alright
Want results from more Discord servers?
Add your server