In this post, I’m going to talk about my current rewrite of
zip-builder, why it’s necessary, and what it will look like.
Why A Rewrite?
Because the existing version sucks.
I didn’t have much idea for organization when I wrote it the first time around, nor did I have much understanding of how Golang works. Some functions are horrendously long and difficult to grok. The existing codebase exists in such a state that I’m afraid to change more than a couple of lines in it, much less add any new features. Even trying to fix bugs is horrendously difficult.
With this rewrite, build recipes (
build.toml files) will be more succint and extensible, with more features and automated downloading of apps from multiple sources.
What Will The Finished Product Look Like?
The design isn’t fully set, but it’ll work a lot like the current version of
zip-builder. Below is a summary of the major differences.
New Download Sources
zip-builder essentially supports two sources:
http(s) URLs and apps from F-Droid. The way it does this is… poorly done, as it requires using the base URL of the F-Droid repository in the same field that the download URL is usually in, then set
What happens if I want to add the ability to download from a site like APKPure or APKMirror? Do I add
is_apkmirror flags? What if multiple flags are set to
http package support “downloading” from the local filesystem? I’ve never even tried…
The new version overhauls this by allowing for specially-defined “protocols” for links. You’ll have the usual
http(s):// protocols, as well as:
file://for local files
f-droid://for apps from F-Droid
And these intended-but-not-promised sources:
termux://for executables from Termux
apkmirror://for apps from one of the APK mirroring sites
zip-builder program will only need to parse the “protocol” and send the contents to the handler defined for it.
On some machines, running
zip-builder on a large build recipe leads to almost 100% chance of crashing due to accessing a
nil map or unlocking an unlocked
Mutex - a Golang structure to help ensure concurrency-friendly access to things.
This is the worst consequence of not knowing what I was doing the first time around. The code used to try to make managing read/write access easier ended up making everything more complex and is even the cause of some crashes.
I’m being careful in this rewrite to make sure that read and write locks get unlocked appropriately and that all map structures are initialized before they get used.
“Testing” in the first
zip-builder was me running the program against the build scripts I used and making sure it didn’t crash. If I was feeling cautious, I might also change a couple values to see what happened. There was no automated testing, though.
This time around, everything has a unit test. They may not cover all possible errors (which are hard to anticipate), but will cover at least common errors and ensure that valid configurations run correctly.
This will avoid issues I recently discovered in the original version because I never tested for them, like the program completely breaking if you tried to have arch-specific apks for an app.
Custom Install Scripting
The application takes care of basic installation things like ensuring permissions are correct and that conflicting files are removed, but it can’t account for everything a user may want to do. The new version of
zip-builder will have the ability to specify custom installation instructions, separated into pre- and post-install. This allows for custom preparation and custom setting up afterward, but a careful user will also be able to use the two fields to add additional conditions that must be met before a file can be installed.
In a similar vein, the new version will also be circumventing the
edify language restriction.
Normally, the flashable zip file contains an
update-binary executable and an
updater-script file that is interpreted by
update-binary. Some zips, annoyed with the limitations of
edify, put an empty file at
updater-script and put the actual installation logic in a script at
Some go a step further and bundle a bunch of executable binaries in the zip, do the above script-switch, and then call a separate
install.sh file from inside
update-binary - looking at you, OpenGApps. I’m sure they have good reason for it, but holy crap.
The rewrite of
zip-builder is probably going to go for the simpler solution of just putting the install script in
update-binary, but who knows? Maybe I’ll figure out why they went for the more complex route.
Right now, I’m working on defining the basic data structures of the new
zip-builder and writing tests for them and their related methods. This is likely going to be the bulk of the work involved, as the rest of the program should be little more that calling those methods and making sure nothing bad happened. It’s not usable for more than downloading files right now, but I’ve also written a lot of tests and functionality. Updates will come later as I get more functionality in place.