Be Careful of Your Dependencies

Starting from a situation I encountered, just when I am writing this blog:

My operating system -- Valecium is currently still in the alpha stage.

It does not provide system headers.

I need to use Linux headers to build the toolchain.

The Kernel Archives broke.

They return 404 for the exact version I need -- 6.12.7.

Leaving me staring at my screen, don't know what to do.

Luckily for me they had already discovered this issue, and I only had to wait 10 hours for this to be fixed.

But what about 10 hours was 10 days? Or the problem was never discovered and fixed because the project is inactive? Of course, my project does not have many dependencies, but there are many that do.

Problems Will Not Stop

We are humans, or AI (if AI is reading this), can never 100% guarantee that everything we do is correct, that is common knowledge.

But the worst part is that machines follow a set of rules, defined by us, so even if the machine executes the rules perfectly, the rules could be wrong, this is where the undefined behaviors come it, and still breaks the things.

So, if we cannot remove the problems from source, then we have to counteract them, and in this blog, I organized a few rules to save your project in case of your dependencies break.

Different Types of "dependency problems"

Not all breakage is the same, and it helps to know which kind you're staring at before you pick a way to resolve that problem, and systematically to reduce the cost when your dependencies breaks..

Version compatibility: The dependency itself did not die; it just moved on without you realizing. If you are still using the old API and they have already deprecated it for a few months. This is the slowest, most survivable kind of problem -- it usually gives you warning, if you're paying attention. This is especially a thing when you are dealing with modern technology like AI (e.g. the /chat/completions API is already replaced with /responses still allowed though).

A bug with a vulnerability: This one is scarier because it does not announce the problem every time you are running it. That dependency looks fine, compiles fine, runs fine -- until someone finds out it's been leaking memory, or worse, that it was compromised on purpose. As the famous XZ incident.

Support for that dependency going down: This is exactly what happened to me, as the start of this blog. If someday your npm or crate.io is down, and you just accidentally deleted your cache to test something, you are done. Which basically what it means, is that in order to make your program work you are relying on a specific level of uncertainty that is always not zero, which also applies to the previous type -- Bugs that cause vulnerabilities.

Hardware compatibility: A famous thing that I encounter is that when you are compiling, you used -march=native or -O3, and just luckily you are super rich to use a supercomputer with a Xeon processor and GCC decided to go with AVX512. And another poor person on an i5 will just get an illegal instruction error because they don't have AVX512.

Always Have a Backup

This is the cheapest and easiest way, and probably the one you should already be doing either because you are told to by your boss or had learnt a lesson from not doing so like what happened to me.

Having backups is common knowledge, and not just in software, in basically anything you do in just normal life actions, with it being the first line of defense.

In the context of writing software, "backup" isn't one thing. It's a few different insurance policies stacked together:

The National Archives (United Kingdom), CC BY 3.0, via Wikimedia Commons

1. Keep a copy of your dependency's source. If the archive 404s, the maintainer is gone, or your companies firewall decided to block the source. You will not be stuck waiting on someone else's infrastructure.

2. Be compatible with multiple dependencies that does the same thing. If one breaks just switch to another one and you instantly get the program back up and running, leaving you with lots of time to fix the problem from source. For example, you can support both Vulkan and OpenGL when making a game.

3. Support more than one version of a dependency. This one mostly answers the version-compatibility problem -- if you're not hard-locked to exactly one version, an upstream change doesn't automatically become your emergency.

Of course, none of this stops the dependency from breaking. It just makes sure you don't break with it.

Also haven't you realized, this simple thing of having backups is already fixing all 4 of the problems that I said in the previous section, with the only downside I can think of being a few more lines of code, not even that much.

Do Not Depend, Copy

The next tier up is deciding you don't want to depend on anyone's uptime at all, you statically link most of what you need, besides the truly unavoidable stuff like LLVM. Another way to say it be self-contained.

This is the strongest answer to the "support went down" problem specifically. If the code is already sitting inside your binary, it doesn't matter if npm, or the Kernel Archives, or any registry is down. Or there are any not compatible versions, you don't care.

A project that I think did this well is Blender, every time you build it, it will download the pre-compiled dependencies only for itself, and creates a self-contained directly on install with all dependencies and statically links most of it.

Although this method ensures full version compatibility and removes the need for downloading, it would be worse for the other 2. If someone found a bug in one of your statically linked dependencies, you don't just swap the system shared library and you are done, you have to push the update to everyone that is using this software, which is annoying.

Nuclear: Do Everything Yourself

If you really want the last word on every single failure mode: version drift, vulnerabilities, hardware quirks, all of it. This is the only option that actually gets you there, because you're no longer depending on anyone else's judgment at all.

Ludovic Courtès, GFDL 1.3, via Wikimedia Commons

But you see, you cannot really not depend on anything, because there is always something below your software, like the OS kernel, or the hardware. Therefore, getting the correct amount that you do yourself is a very tricky thing, and if you do not get that right, the same problems will still emerge.

But "ultimate control" and "ultimate workload" are the same option here. You now have to think of everything yourself, write far more code than the feature actually needed, and since you're human, same as every maintainer whose dependency just broke on you. You will write your own share of bugs into it. You have not removed the risk. You've just moved it from "someone else's mistake" to "your mistake," which is a trade, not a victory.

Have Fun

All of this, the backups, the self-contained builds, the nuclear option. Is really only for when the stakes are real. When a break actually costs you something: users, uptime, a release date, your own mental sanity at 2 o'clock staring at a 404.

If it's just a project for fun, none of this is a rule you owe anyone. Let it break. Fix it when you feel like it, or don't. As most of these dependency problems will get fixed within a few hours/days to fix, and it is never a problem to do a roll back if you use git.

The entire point of the four problems and three answers above was to never always do the maximum. It was knowing what you're risking, so you can choose how much you're willing to pay to avoid it. For a hobby project, the most suitable answer is just go alone with it (the dependencies), and wait patiently if it breaks.

---

私は毎週およそ 6 時間、主な仕事の外で C、Java、システム性能に関する深掘り記事を書いています。もし内容があなたのエンジニアリングの道具箱に価値を加えるなら、プロジェクトを支援する方法は次の 2 つです：

単発サポート

特定の修正やチュートリアルへのお礼として。 コーヒーを奢る（$5）

その他の支援方法

さらに支援方法を知り、ブログコミュニティを維持するために、サポート ページをご覧ください。