Failure to build bootstraps
Hi I would like to make my own task-specialized version of the termux app bootstrap archives, I tried to do what this said https://github.com/termux/termux-packages/wiki/For-maintainers#build-bootstrap-archives :
./scripts/run-docker.sh ./scripts/build-bootstraps.sh &> build.log:
build.log:https://termbin.com/ck8u
My host OS is Gentoo GNU/Linux coreutils glibc amd64.GitHub
For maintainers
A package build system for Termux. Contribute to termux/termux-packages development by creating an account on GitHub.
40 Replies
ok the log is huge, that 14 megabytes log so that termbin link is incomplete
here is a different one
PrivateBin
Encrypted note on PrivateBin
Visit this link to see the note. Giving the URL to anyone allows them to access the note, too.
here's the error that is at the very bottom since that page might crash a slow computer
I haven't edited any code yet, i think it makes sense for me to start by trying to build completely default bootstraps to make sure they work, then change what i need to after
my goal is that I want to generate these (from default local git repository sources that i downloaded with
git clone --recursive https://github.com/termux/termux-packages.git like the documentation said build-bootstraps.sh does)
bzip2 is a subpackage of libbz2 maybe that's catching it off guard?I see this in the source code of the script, should I change that to say something else?
Seeing as the last change to the
libbz2 package was 3 months ago, I'm gonna assuming the bootstrap builds just fine usually.
So no.
make that more like 4 years ago... since the build script for that package was touchedI also confirmed the docker container opens and runs to perform this task and inside the container is ubuntu. so I wouldn't guess it is a problem with docker. I checked using this command
docker exec -it 253a545c7708 /bin/bash and then used cat /etc/lsb-release and no other commands inside the container.The docker image should work perfectly.
That's kind of the reason it exists, to mitigate weirdness caused by varying environments.
yes I know, that is why it is also hard for me to find what is wrong when a docker build doesn't work
I have used the containers of a different project Redroid including their builder container, on this system (none of those containers are still running right now) and all of them work fine
i used
git pull and see that i still had a very recent version of the repository and only 2 packages were updated
i will delete this container and then run the build again as a 2nd test
yes i ran it again and it did the same thing
i tried the same thing again on a clean install ubuntu 22.04 host and have the same result
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
i can see that the reason why the bootstraps from github actions don't have this error is because they use
generate-bootstraps.sh not build-bootstraps.sh, maybe build-bootstraps.sh is unnecessary for official Termux builds so goes unmaintained.
however I plan to rely on termux official repositories this way: my build will use a very few specific bootstrap packages customized, then use apt-mark holdon them to prevent my users from accidentally replacing them with updates to official packages, then contain the official termux mirrors at the time my build was created so that non-bootstrap packages can be fetched from them at runtime, and just hope that works for a conveniently long time until such time as apt cannot satisfy the dependencies of non-held packages with the versions of the held packages and my version breaks. I don't want to add additional packages to the bootstrap packages list because A. I want to benefit as much as possible from the rolling-release feature, security and performance updates provided by the packages in Termux official mirrors and B. I want to benefit as much as possible from the default mirror autoselection process Termux currently uses in order to optimize the download process of non-bootstrap packages for as many users as possible
for my priorities i view those reasons as outweighing the stability of fully locking the versions of all packages.
thank you, i was able to build some bootstraps from source after making these changes
however, when I compare them with the bootstraps that are generated by generate-bootstraps.sh, I see a problem with them that I would like to fix in the most optimal way
The bootstraps created by build-bootstraps.sh are extremely large, 5 times the size of the ones that generate-bootstraps.sh creates
I can see that one reason for this is that they are filled with many more files especially files in places like share/doc
How can I make
build-bootstraps.sh reduce the size of the archives to be more similar to those from generate-bootstraps.sh so that my .apk will not be huge?
it's not obvious to me what generate-bootstraps.sh is doing that reduces the size
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I didn't add any packages, the only changes I have made so far are the ones here
which I thought were alternative package names for the same packages which i'm forced to change those to due to
build-bootstraps.sh being broken in its initial state.
thank you for helping though, I might set up a server solely to host packages once for generate-bootstraps.sh to check whether I can get the edited packages I need into the smaller bootstraps of generate-bootstraps.sh that way. However I don't know if there will be an easy way to do that and still have the official Termux mirrors in the bootstrapssources.list
reading through lots of comments by Maxython, he says things like this multiple times
Might as well ask him.
@Max Ivan any ideas?
i get the impression from this that:
-
build-bootstraps.sh tends to include large build dependency packages in the archives that are unnecessary
- The version of build-bootstraps.sh currently in the repository is known to be broken for now and is not recommended and this status is undocumented, the info only being possible to find in the issue threads
i haven't read the entire diff of every single PR modifying build-bootstraps.sh next, so i think that is my next step to check whether the issue is currently in the process of being fixed anywhere,because i've so far avoided considering trying to use
generate-bootstraps.sh for my goal because the doc tells me that it is not compatible with my use-case
which i take to imply that were I to attempt to use it I would run into more problems
Wha?
idk I assumed you know what the issue might be
i'm trying your code right now like this
I need time to understand what's going on here
Ok, now I understand what's going on here. You are now faced with the fact that all packages that are compiled by build-bootstraps.sh are included in bootstrap which is not correct. I was going to solve this problem, but due to the fact that I had a lot of things in my life, I did not have much time to work in Docker (in addition to this, I also had things to do in other Termux projects). At the moment, I'm working on another project (with glibc-packages for Termux) and I can't be sure that this issue will be resolved soon. The only thing I can do now is answer your questions. Do you have them?
I want to create a custom Termux that is exactly identical to normal Termux in every way, including mirrors, except for signing keys and that it includes a Bash payload which is executed every time it launches a new Bash session. The purpose behind this level of control is not malicious, rather it is to provide very inexperienced users of my specialized software, who do not understand how to physically type Bash commands, with a fully automated experience, enabling those who cannot type Bash commands to still run my software. I want to write this by editing the termux-packages and termux-app repositories in as clean and source-controlled a way as possible in order to make proving the absence of malicious code, maintaining the project, and merging upstream Termux changes when necessary as easy as possible. Do you have an idea of the most direct approach I can take to doing this that is both fully source-controlled and minimally changed from default Termux?
This thread documents how I've travelled my first approach. I believe with the exact method explained in this thread, currently I could get exactly what I described I need, with the minor issue that the
.apk I would distribute would be 500 megabytes unnecessarily.
(I was going to edit the bash source package used to generate the bash package contained in the bootstraps such that it includes my payload and have my payload then use apt-mark hold bash at runtime so that my app works for as long as possible)
It's completely fine that you haven't finished rewriting the build-bootstraps.sh . I've postponed this particular project for a long time because I know that I have requirements for such a project that Termux was not intended to fulfull, which could easily necessitate much more work on my part than I am actually willing to put into it.
My potential users who are unable to install my software because they do not have the experience necessary to understand how to type Bash commands have waited a long time already, and they can wait longer.
I am sure there are many who would say that Termux should never be used by anyone who is not experienced enough to understand how to copy and paste Bash commands. From that perspective I don't disagree with those people, but it does not change the fact that there are potential users who want my app, but will never be able to use it unless I make this project.
What I mean by "cleanly source-controlled" is that I don't really want a build script that unpacks some default bootstraps, edits their bashrcs to contain my payload, then repacks them and builds the .apk, even if that is the "easiest" way to make a one-off of what I want and could itself be source-controlled. I think that is messy.
What I really actually want is to edit only this line in the termux-app repository https://github.com/termux/termux-app/blob/master/app/build.gradle#L185 and the 4 hashes further down, to match my own termux-packages repository that contains code that can fully generate bootstraps (somehow) that match the bootstraps in its own releases page that gradle can then download.Well, the only thing that comes to mind how to make it easier for your users without creating special packages and without using "build-bootstraps.sh" is using a graphical shell - https://wiki.termux.com/wiki/Graphical_Environment. The bash shell is not only the main shell in Termux (the base of all Linux distributions), but other packages depend on it as well. Therefore, bash without Termux is nowhere. To create such a bootstrap (with a graphical shell), it may be enough to use generate-bootstraps.sh, but here it is worth considering that it is configured to work with packages only from the main repo, and here you need an x11 repo (so you have to configure it a bit). Also, could you tell me the name of your program?
In cases where user input is needed for the specific task, a GUI is helpful yes, but in this situation, i believe having that would require the termux-x11 apk which is yet another apk users need to install, and the idea for this task is to keep user input to a minimum by default because when user input is unlimited, some users cannot do the steps
this repository and a series of nested git submodules within it contain all the code of my app https://github.com/robertkirkman/sm64ex-coop-android-base
The goal of the code I've been talking about in this thread is a subproject to fully no-compromises automate the bash directions in this specific document in my project in a way that meets arbitrary requirements I decide
https://github.com/robertkirkman/sm64ex-coop/blob/android/README_android.md
The "payload" I've described earlier will look something like this. This is my work-in-progress version of the code I will put there which I haven't tested since I will do that after I compile Termux app apk with the build settings I want.
There will be a focus on dropping to interactive Bash shell at success, any failure, and maybe also at the start with a "press key to stop... 5...4...) construction, which should allow my app to both do everything unprompted for all novice users, and provide identical functionality to normal Termux for all advanced users until the release of the replacement for official Termux 0.118 which would necessitate me merging that to my version and will inevitably cause old releases of my version to stop working, this is a price i choose for not having to preinstall a controlled version of every package
I'm aware that a common practice of many projects is to distribute bash scripts as a single command. example
my philosophy is that for my specific scenario, that practice would be useless because the problem is that many inexperienced users are unable to type that or copy it or paste it, yes no matter how short it is, so that does not solve the problem, so for "automate the steps in this documentation" to actually happen in this context and for the script above to have any point, it needs to be inside an
.apk that seamlessly uses only Android Java pop-ups for all user input and then proceeds with no further input needed as much as possible.
I did this, which successfully built bootstraps of the correct size from local sources after running first ./scripts/run-docker.sh ./scripts/generate-bootstraps.sh then ./scripts/run-docker.sh ./scripts/build-bootstraps.sh
this is a very inefficient shortcut forcing the behavior I want, but this is acceptably programmatic for my needs, now I can proceed with the other parts of my project that depend on this
i haven't tested the resulting bootstraps yet, but i will get to that eventually, at that point i'll find out whether this breaks the bootstraps.
After a couple more changes, primarily this, i can also cause github actions to publish bootstraps that are both the correct size and able to contain packages built with the changes I need, on the github releases page of my repository
i'm just writing what I did here because I believe info like this could help any others trying to use Termux's source code to build a task-specialized copy of Termux but who still want to make it use the official Termux package mirrors for as long as that may continue to work. The first things I did were to read the termux github wiki, and when the steps in the wiki produced errors and code found in the issues and PRs helped them somewhat but not completely, I assembled the info here.
forgot to mention that i've tested the bootstraps fully compiled into my build now and everything seems to work exactly how I want. Thank you both for helping