In this post, I’m going to talk about my current rewrite of zip-builder, why it’s necessary, and what it will look like.

Why A Rewrite?

Because the existing version sucks.

I didn’t have much idea for organization when I wrote it the first time around, nor did I have much understanding of how Golang works. Some functions are horrendously long and difficult to grok. The existing codebase exists in such a state that I’m afraid to change more than a couple of lines in it, much less add any new features. Even trying to fix bugs is horrendously difficult.

"When you delete a block of code you thought was useless" - The Pink Panther trims a tree and the world falls away underneath him.

With this rewrite, build recipes (build.toml files) will be more succint and extensible, with more features and automated downloading of apps from multiple sources.

What Will The Finished Product Look Like?

The design isn’t fully set, but it’ll work a lot like the current version of zip-builder. Below is a summary of the major differences.

New Download Sources

The existing zip-builder essentially supports two sources: http(s) URLs and apps from F-Droid. The way it does this is… poorly done, as it requires using the base URL of the F-Droid repository in the same field that the download URL is usually in, then set is_fdroid_repo to true.

What happens if I want to add the ability to download from a site like APKPure or APKMirror? Do I add is_apkpure and is_apkmirror flags? What if multiple flags are set to true?

Does Golang’s http package support “downloading” from the local filesystem? I’ve never even tried…

The new version overhauls this by allowing for specially-defined “protocols” for links. You’ll have the usual http(s):// protocols, as well as:

  • file:// for local files
  • f-droid:// for apps from F-Droid

And these intended-but-not-promised sources:

  • termux:// for executables from Termux
  • apkpure:// or apkmirror:// for apps from one of the APK mirroring sites
  • Others?

The main zip-builder program will only need to parse the “protocol” and send the contents to the handler defined for it.

Better Concurrency

On some machines, running zip-builder on a large build recipe leads to almost 100% chance of crashing due to accessing a nil map or unlocking an unlocked Mutex - a Golang structure to help ensure concurrency-friendly access to things.

This is the worst consequence of not knowing what I was doing the first time around. The code used to try to make managing read/write access easier ended up making everything more complex and is even the cause of some crashes.

I’m being careful in this rewrite to make sure that read and write locks get unlocked appropriately and that all map structures are initialized before they get used.

Unit Tests

“Testing” in the first zip-builder was me running the program against the build scripts I used and making sure it didn’t crash. If I was feeling cautious, I might also change a couple values to see what happened. There was no automated testing, though.

This time around, everything has a unit test. They may not cover all possible errors (which are hard to anticipate), but will cover at least common errors and ensure that valid configurations run correctly.

This will avoid issues I recently discovered in the original version because I never tested for them, like the program completely breaking if you tried to have arch-specific apks for an app.

Custom Install Scripting

The application takes care of basic installation things like ensuring permissions are correct and that conflicting files are removed, but it can’t account for everything a user may want to do. The new version of zip-builder will have the ability to specify custom installation instructions, separated into pre- and post-install. This allows for custom preparation and custom setting up afterward, but a careful user will also be able to use the two fields to add additional conditions that must be met before a file can be installed.

In a similar vein, the new version will also be circumventing the edify language restriction.

Normally, the flashable zip file contains an update-binary executable and an updater-script file that is interpreted by update-binary. Some zips, annoyed with the limitations of edify, put an empty file at updater-script and put the actual installation logic in a script at update-binary.

Some go a step further and bundle a bunch of executable binaries in the zip, do the above script-switch, and then call a separate install.sh file from inside update-binary - looking at you, OpenGApps. I’m sure they have good reason for it, but holy crap.

The rewrite of zip-builder is probably going to go for the simpler solution of just putting the install script in update-binary, but who knows? Maybe I’ll figure out why they went for the more complex route.

Current Progress

Right now, I’m working on defining the basic data structures of the new zip-builder and writing tests for them and their related methods. This is likely going to be the bulk of the work involved, as the rest of the program should be little more that calling those methods and making sure nothing bad happened. It’s not usable for more than downloading files right now, but I’ve also written a lot of tests and functionality. Updates will come later as I get more functionality in place.

Comments