1. What and why
2. BOM (Bill of Materials)
3. Printing and initial assembly
4. Cabling decisions
5. The final assembly
6. Final words

What and why

I follow multiple people on Mastodon and YouTube that are part of the home lab community (I’m not much into the Reddit community though). I also follow a lot of hashtags, and watch the eventual algorithm-pushed video (yeah, I know). For whatever reason the 10-inch racks have become more and more common, probably because more people are buying the tiny computers and/or SBC (single-board-computers) and they want to get themselves more organized, without spending too much space.

Not wanting to spend too much space is certainly my case: I already have a standard 9U 19″ rack attached to a wall somewhere in the house that holds a lot of equipment, most importantly my network gear (sorry, can’t post a picture, it would reveal way much more information than I am willing to share). The current 19″ rack has (for now) 4 Raspberry Pi in one printed 1U tray, which I have been slowly phasing out in favor of a new set of Lenovo M720Q Tiny. But it already is very full of other things, so I keep the Lenovos in an printer stand in my home office (the printer is somewhere else), so I wanted to build something compact and visually appealing.

I did a quick pulse check on Mastodon, as well as on one of the Discord groups I hang around. And those discussions gave me lots of good ideas from Alex Kretzschmar, Jan Wildeboer, Rachel and Fishd (plus some other people who never publicly posted their racks).

After lots of consideration, instead of buying a pre-built rack, I decided to print my own. After all I have a very nice 3D printer, and I would have to print all the trays anyway, so why not print the entire rack. In the end I decided to go with the Lab Rax 10″ Server Rack – Bolted Version – 5U, because it looked really nice, easy to print, and has a strong community around it.

And by the way: a shout out to my lovely wife! She was present in every single step of this project, helping me select the parts and keeping my good (bad) taste in check all the time. We had a lot of fun working on this together!

DISCLAIMER: Most of the Amazon links across this post are affiliate links that will give a portion of the purchase to Alex Kretzschmar, who is partially to blame for some of the choices for this rack. I suggest you check out his blog and his podcast, it’s all good quality material. But you can buy whatever you want, wherever you want. These are the ones I got that you can use as an example for your project, since it took me a while to research everything.

BOM (Bill of Materials)

These are the 3D model files I used.

• Lab Rax 10″ Server Rack – Bolted Version – 5U
  • I chose the bolted one for my own convenience, but lots of people recommend the original one with brass inserts.
• 10 Inch Rack 1/2 U Keystone Panels
• 1/2U Ventilation Panel for 10″ Rack
• UniFi Switch US-8-60W 10 – Inch Rackmount
  • What I like about this one the most is that they created a little cut so you can see the blinking lights (which are on the top of this switch).
• V2.1.1 Parametric 10 inch rack shelf
  • There are many shelf models, but I could not find one exactly the way I wanted, so I created my own using this customizable model. Here are the parameters I used.

And this is the list of hardware involved.

• Bambu Lab X1 Carbon
• Bambu Lab Black PETG HF (33102)
  • Purchased from the cool people at Ecovate 3D in Raleigh. If you’re in the region, they’re an awesome source for anything 3D printing related.
• Bambu Lab Carbon Fiber 3D Effect Plate
• M6-1x12mm Button Head Socket Cap Screws – 50Pack
• M6-1 (50 Pack) Hex Head Nuts
• 10Gbps CAT7 Coupler RJ45 Keystone
• HDMI Keystone Jack
• 8K HDMI Cable 1.5ft
• I already had a lot of 1.5ft CAT6 cables in a box, so no link for those.

Printing and initial assembly

I wasn’t in a hurry, so I printed everything over the course of about two weeks. The frame was split in two parts, each one took about 5h30m to print, plus 2h for each of the 4 grates, then each Lenovo tray took about 7h, the UniFi tray about 5 hours, then the smallest parts about 1h each. I guess that counts for about 47h of printing time.

I have not printed any of the top, bottom or side panels yet, because I am a bit concerned about the air flow. I’ll have to keep an eye on the temperature of each device for a while before I make a final decision. And if I do put the panels, most likely I will do some kind of open panel, and put a decent fan on the top one.

On the total time count above I’m ignoring all the time I spent to dial in the PETG HF. Honestly that was the hardest part of the entire project. In the end I had to create a very precise dance of printing, then washing the plate (warm water and soap) and drying the filament for 3 hours before being able to print the next part.

After a lot of sweat and tears I had the first part done.

And before anyone calls me out: yes, the last Lenovo tray is not the same size as the others. I miscalculated and printed it 3 mm longer than it was needed, so I corrected in on the others, but didn’t care to print a new one. There’s just a small silicone drop behind the machine to prevent it from sliding.

Cabling decisions

I’m still not sure if this is the rack’s final form, because the cabling in the back got a little bit messy. Most likely I will need to print some kind of cable organizer to hold them tidy in the back.

But I did have to make some interesting decisions in terms of cabling that might help you understand why I bought some of the parts I listed above.

Since this is my office, there are a few other devices connected to the switch, and that’s why I use all of the 8 ports. There’s a chance I might need more ports, so either I would need to add another 8-port switch, or buy a 16-port. Either way that would force me to review the patch panel situation, or maybe even expand the rack height (I think I can fit up to 8U in the space, and there’s also this cool top patch panel).

I have one single-port GL.iNET Comet KVM, which I use to manage the machines if they require maintenance. It is currently resting behind the switch, on top of the top-most Lenovo. So, instead of having a bunch of cables dangling from the back, or having to deal with the mess of changing ports around, I used the HDMI keystone jacks to plug the KVM input on the first port of the patch panel, then the other ports are connected to each one of the Lenovo. So, if I need to access any one of them, I just need to use an “HDMI patch cord”. One thing I would like to improve is how to manage the keyboard: for now the USB-C cable that comes out of the KVM is dangling in the back, so when I need to connect to one of the machines I just pull the cable and connect to the USB-C port in the front of the Lenovo.

Also, I took this chance to borrow a page out of Alex’s bag of tricks, and I bought myself one Anker Prime Charging Station, along with the necessary USB-to-Lenovo cables for a poor man’s kind of PDU, and to save a lot of space by not having to keep all those Lenovo power bricks. And honestly, that’s better than buying the UniFi monster PDU.

The final assembly

You have already seen the teaser, so I will give you one extra picture of the final assembly, that includes the HDMI patch cable, where you can also see some of the other peripherals attached to the lab, and get an idea about how it sits on my printer stand. Please excuse the cable mess in the back.

Final words

This was no doubt my most fun 3D printing project so far. It took many hours, lots of sweat and tears, but the end result is stunning, if I may say so myself.

I hope this gives you inspiration, and you come around to print (or build) your own rack. And if you do, please share either on a blog or a Mastodon post, and let me know about it. It would make me really happy to know that I have contributed to your project one way or another.

If you have any questions/comments/suggestion/feedback, or just would like to chat, feel free to reach out to me on Mastodon, or post a comment below. I will try my best to reply back to you with something useful.

1. Introduction
2. Choosing where to store the files
3. Initializing the workspace the “clickOps” way
4. Initializing the workspace the “coding” way
5. Initializing the Dev Container
6. Adding the other repos to the Dev Container
7. Make it happen
8. A note about GitHub Copilot
9. Conclusion

Introduction

Let’s get one thing out of the way: one of the many reasons I use all these features is because of GitHub Copilot. The more context you provide to it, the more precise it will be (I added a note about that at the end of the post). There are multiple other reasons why someone could make use of this feature so, in my humble opinion, you should not disregard this blog post just because of this stated fact. I just wanted to put this piece of information out in the open.

And another important thing: this blog was not AI generated. It’s 100% human written.

With all that said, let me give you more context about this post: I heavily use Visual Studio Code both professionally and for my personal projects (for short: VS Code or simply Code). For a long time I have been using Dev Containers for many reasons, but the two main ones are: isolation from my base OS and reproducibility (in case I have to rebuild or jump to a different machine). A couple of months ago I discovered the existence of Multi-root Workspaces, and it was clearly something I have been missing for a long time (even though it’s not a new feature).

Both at work and at home I need to deal with multiple repositories. Either because each repository stores a different piece/aspect of the larger project I am working on, or because I need to cross-reference different repositories to use them to build something else. Using Multi-root Workspaces makes my life easier because I can put multiple repositories in the same place/view, without having to mix them up in the filesystem or removing their “independence”. I have worked with Git submodules before, and it was a real mess (for me).

Choosing where to store the files

Both workspaces and dev containers require configuration files that can be tracked via Git. I find it a good exercise to do so, because it gives me a history of the changes made and, as I said before, makes it easier to move to a different machine. So the first thing you need to do is choose where you’re going to store your configuration files. You have two choices, and I will give my personal take on each one of them:

1. One of the existing repositories – this might be a good choice if you want to make your life simpler, or if all of the other repositories in the same workspace are direct dependencies to the one you’re working at.
2. A brand new repository – I use this one because, for me, this is the cleanest way, and allows me to shift things around. Think of it as a desk: you can change everything on top of the desk based on what you’re working on, but the way you arrange your space remains the same no matter which task you’re performing.

With that in mind, this is how I organized the example directories for this blog post:

~/tmp/blog $ tree
.
├── cloned-repo1
├── cloned-repo2
├── homelab
├── homelab-ansible
├── homelab-kubernetes
├── homelab-terraform
└── personal-project

8 directories, 0 files

I chose “homelab” as my place to store the configuration files. So I went ahead and created the repository on my self-hosted Forgejo, then cloned it locally. All the other directories are examples of cloned repositories which I am already working on (names changed for demonstration purposes).

Initializing the workspace the “clickOps” way

The workspace can be initialized by clicking around on the menus, or manually (like I will explain below). If you want to do this by clicking on the menus:

1. Open your main/configuration directory in VS Code (in my case it’s “homelab“).
2. Go to “File” -> “Add Folder to Workspace...“, select the folder, then click “Add“.
3. As soon as you add the first folder you will see the top of the explorer window will switch to “Untitled (Workspace)“.
4. Repeat adding all the other folders.
5. When done, go to “File” -> “Save Workspace As...“, make sure you are on the first directory/folder you selected, then give it a file name, which will be your new workspace name.

If at this point VS Code asks if you want to trust the workspace, you have to, otherwise the Dev Containers won’t work.

Initializing the workspace the “coding” way

If you want to write code to initialize workspace, this is the way.

Open your main/configuration directory in VS Code (in my case it’s “homelab“). Right-click on an empty space in the explorer window then select “New File...“. Create a new file named “homelab.code-workspace” (the name doesn’t really matter, as long as it ends with “.code-workspace“). Inside that file you will add the following content:

{
  "folders": [
    { "name": "homelab", "path": "." },
    { "name": "cloned-repo1", "path": "../cloned-repo1" },
    { "name": "cloned-repo2", "path": "../cloned-repo2" },
    { "name": "homelab-ansible", "path": "../homelab-ansible" },
    { "name": "homelab-kubernetes", "path": "../homelab-kubernetes" },
    { "name": "homelab-terraform", "path": "../homelab-terraform" },
    { "name": "personal-project", "path": "../personal-project" }
  ]
}

Notice that the paths here are all “../” because in my example everything is under the same parent directory. It doesn’t really matter. You can simply point the path to anywhere else you see fit (the variables “${userHome}” or “${env:HOME}” also work).

Also, the folder names are just how they will appear in your explorer window, so they can be anything you want. You can use, for example “github/cloned-repo1” or “gitlab/cloned-repo2“

As soon as you save the file you will notice a “Open Workspace” button on the bottom right corner. You can click on it. If the button is gone or you can find it, go to the menu “File -> Open Workspace from File...” and select the file you just created.

VS Code will reopen in workspace mode, and you will see the list of all the directories you just told it to map. It will also ask if you trust this workspace. You have to, otherwise the Dev Containers won’t work.

Initializing the Dev Container

1. At this point you should have your workspace open. Make sure you have the workspace file selected in the file explorer. This is important to make sure the Dev Container will be created in the correct place.
2. Now hit “CTRL+SHIFT+P” and search for “Dev Containers: Add Dev Container Configuration Files...“.
3. VS Code will complain that it can’t open your other folders, but that’s not a problem. We will tell it how to open them inside the container later, so just click “Continue“.
4. On the next dialog select “Add Configuration to Workspace“.
5. The next dialog will ask for a template file. I decided to go with “Ubuntu“, for simplicity. I actually use a custom-built container, but that’s a conversation for a different blog post.
6. On the next dialog select “noble” (that will install Ubuntu 24.04).
7. And on the following “Select Features” drop down you can select whatever you want, or nothing. When done, just click “OK“.
8. The last dialog will confirm if you want to include “.github/dependabot.yaml“, which is a good practice if you are hosting in GitHub. Go with it. One more file won’t make any difference. In my case I am using Renovate, so it doesn’t make any difference either. When done, click “OK“.

DO NOT CLICK the small popup on the bottom right corner of your screen saying that VS Code detected a Dev Container, and asking if you want to re-open the workspace using it. The configuration is not ready yet. You may, if you want, click on the little “X” at the corner of the dialog to close it.

Adding the other repos to the Dev Container

This is the “not so fun” part, because it requires some manual configuration. It so happens that, because of the isolation that we so much want, you need to tell the container where the other repositories are. For the purposes of this blog they are all in the same parent directory, but in practice it does not matter at all. They could be anywhere in your filesystem, as long as they are accessible by the container during startup time. So, in the “decontainer.json” file, you need to add the configuration lines below. Anywhere inside the brackets should be fine (just make sure you add the appropriate comma where it’s needed, after all this is just a JSON file).

	"mounts": [
		"source=${localWorkspaceFolder}/../cloned-repo1,target=/workspaces/cloned-repo1,type=bind",
		"source=${localWorkspaceFolder}/../cloned-repo2,target=/workspaces/cloned-repo2,type=bind",
		"source=${localWorkspaceFolder}/../homelab-ansible,target=/workspaces/homelab-ansible,type=bind",
		"source=${localWorkspaceFolder}/../homelab-kubernetes,target=/workspaces/homelab-kubernetes,type=bind",
		"source=${localWorkspaceFolder}/../homelab-terraform,target=/workspaces/homelab-terraform,type=bind",
		"source=${localWorkspaceFolder}/../personal-project,target=/workspaces/personal-project,type=bind"
	]

There are a few things you should notice:

• You obviously don’t need to specify the current repo because that’s the container’s root filesystem, so it will be mounted automatically.
• The special variable “${localWorkspaceFolder}” is a type of “pwd“. It just replaces with the current full path of where the workspace file is saved (more on that later). In this case the “../” points to the fact that all directories are under the same parent. You can simply point the source to anywhere else you see fit (the variables “${userHome}” or “${env:HOME}” also work).
• The target name after “/workspaces” is how the directory will show up inside your container, so it can be anything you want. You can use, for example “github/cloned-repo1” or “gitlab/cloned-repo2”. Just make sure the “target” here actually matches the “path” specified in the workspace definition.

Make it happen

Now hit “CTRL+SHIFT+P” and select “Dev Containers: Rebuild and Reopen in Container...“. You will be presented with a very ominous message complaining that the container might not work because of the folder structure. That’s fine. The container will mount the things that need to be mounted. Just click “Continue“.

At this point it will use Docker (or Podman) to build a new container. Since there isn’t a lot going on, it should be really fast, but if it’s taking a while there is a small “show log” dialog down at the bottom right.

Et voilà! At this point your workspace will be open in a Dev Container, with all your other repos, safely isolated from your main OS.

A note about GitHub Copilot

As I mentioned, one of the reasons why I am doing this is to be able to provide a lot of context to Copilot, so let me tell you a bit more about that.

• You can ask Copilot to search and corelate code across all the repositories.
• If one or more repositories have “copilot-instructions.md” files, it will concatenate all of them and use as part of your prompts.
• If one or more repositories have agents or skills files, it will use them when working with files inside those particular directories.
• If you want agents or skills files that will work across the workspace, just add it to the top-level directory, and make a note on the instructions file to use them when working on the other repositories. They will be all concatenated (or at least that’s how it seems to be working for me).

Conclusion

I don’t know if there are better ways to achieve this, but this is the only way I could figure it out since, like I said, I could not find any blog/video about trying to do the same thing. But so far it has been working really well for me.

I sincerely hope that you can make good use of everything that was demonstrated here!

If you have any questions/comments (criticism?) or just would like to chat, feel free to reach out to me on Mastodon, or post a comment below. I will try my best to reply back to you with something useful.

The point of this diagram is just to demonstrate the wide mix of technologies that I use to run my various tests, as well as my internal productivity workloads. I use different hardware, software and cloud platforms, all managed with a mix of FluxCD, Terraform, Ansible and good old Bash scripts, all with as much as GitOps as possible.

For the time being, I am planning to trim down a bit, because I have a lot of unused capacity.

I hope this gives you some ideas for your next adventures. And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon!

1. Introduction
2. Note about creating the VM
3. The boot problem
4. Configuring the secure boot
5. Conclusion

Introduction

I’ve been running my Talos Linux Kubernetes cluster with 3 nodes (1 Framework Desktop + 2 Lenovo M720Q Tiny), but I’ve been thinking about making some changes to the topology (which I will blog about later), so that required me to spin up a new control-plane node. In order to free up some hardware, I decided to do it as a VM on a Proxmox cluster.

DISCLAIMERS:

• No AI or LLM was used during the research of this problem, or for writing this blog post.
  • I did, however, copy and paste some text from my previous post.
• The content here is provided AS IS, without any warranties that it will work for you or will not damage your computer.
• This is purely a demonstration of MY personal experience, with the expectations that it will help someone else.
• Please, don’t blame me if this bricks your computer or cause any data corruption on your Proxmox installation.

Note about creating the VM

I just wanted to make a few observations about the process of creating the VM. Talos already provides some good documentation, but it seems like this was targeted at Proxmox 8.x, while I am using 9.x. This means some of the options moved to different locations, which is fine, you will probably will figure those out. However I had to make some changes to make this “perfectly right” (or at least the way I wanted):

1. The document is not clear about the disk size. The System Requirements page indicates a minimum of 10GiB for a control-plane is required, but that did not work for me. When bootstrapping Cilium, I ran out of EPHEMERAL space, so I increased the disk to 32GiB as suggested by default (it’s thin-provision anyway), and set the ephemeral disk to 20GiB.
2. The document also mentions that your boot image should contain the QEMU guest agent, but it doesn’t mention the fact that you have to actually click on the “Qemy Agent” box, next to the “Machine” type, when creating the VM.

The boot problem

Similar to the previous problems with the Framework and the Lenovo, as soon as I booted I was presented with this very “user friendly” error message:

BdsDXE: failed to load Boot0002 "UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci (0x1F,0x2)/Sata(0x1,0xFFFF,0x0) : Access Denied

I didn’t worry because I knew exactly what this was about.

Configuring the secure boot

Knowing what to do, I just had to find the correct sequence in Proxmox.

1. Reset the VM.
2. When you see the Proxmox logo hit ESC.
3. You will see the “Please select boot device” menu, from there you select “EFI Firmware Setup”.
4. The you go “Device Manager” -> “Secure Boot Configuration”.
5. Set the “Secure Boot Mode” to “Custom Mode”, then select “Custom Secure Boot Options”.
6. Select “KEK Options” -> “Enroll KEK” -> “Enroll KEK using File” -> select the CDROM device -> “EFI” -> “keys” -> “uki-signing-cert.der” -> “Commit Changes and Exit”.
7. Same thing for “DB Options” -> “Enroll Signature” -> “Enroll Signature Using File” -> select the CDROM device -> “EFI” -> “keys” -> “uki-signing-cert.der” -> “Commit Changes and Exit”.
8. Now hit ESC until you see “Reset”.
9. The VM will reboot straight into the CD/DVD.

At this point you should be able to install Talos as usual, and have a fully functional system.

Conclusion

I still have to contact someone on Sidero Labs, or rummage through their forums, to understand why this problem exists in every single hardware. Clearly they’re not signing their images with a key that is accepted by the BIOS vendors, I just don’t understand why, and if they have plans to fix that or not.

Anyway. I hope this has been useful for you! And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon!

Updated on 2026-04-15 – Fix the BIOS menu sequence missing one step. Also added an extra step to return to “Deployed Mode”.

1. Introduction
2. The boot problem
3. Configuring the secure boot
4. Conclusion

Introduction

Running my Talos Linux Kubernetes cluster with a single node has been working fine so far. It’s good enough for me because the machine is powerful enough to run everything I need, and still has a lot of power for future growth. However I wanted to have better resilience, and the ability to fail-over at least some of the apps during a node maintenance (not all of them require high uptime).

So I decided to get the Lenovo M720Q Tiny for a few reasons: it’s small, and it’s very low power. The icing on the cake is that I bought them refurbished, and they were very cheap. I won’t guarantee anything, but I think the instructions here should work for any other model of the Lenovo Tiny. I am just guessing here, because the BIOS looks very similar to my old Lenovo laptops (I don’t have any to compare right now).

DISCLAIMERS:

• No AI or LLM was used during the research of this problem, or for writing this blog post.
  • I did, however, copy and paste some text from my previous post.
• The content here is provided AS IS, without any warranties that it will work for you or will not damage your computer.
• This is purely a demonstration of MY personal experience, with the expectations that it will help someone else.
• Please, don’t blame me if this bricks your computer or cause any data corruption.

The boot problem

Just like it happened with my Framework Desktop before, as soon as I booted the Lenovo with the Talos ISO on a USB stick, I was greeted with the following message:

Secure Boot Violation - Invalid Signature detected. Check Secure Boot Policy in Setup

Even though I already have an IP KVM, I’m not gonna post any screenshots here. I want this post to be easily searchable.

Anyway, I knew exactly what to do. I just had to figure out the “how”.

Configuring the secure boot

I did not have to fumble around to figure out what I needed, so let’s get to it:

1. Once you power it up and you see the Lenovo logo, start hitting F1 to get into the BIOS.
2. Then go to “Security” -> “Secure Boot”, and select the option “Reset Platform to Setup Mode”.
3. It will ask if you really with to do that, hit the “Yes”.
4. It will come back to the same screen, so hit ESC then go to “Exit” then “Save Changes and Exit”, then “Yes”.
5. When it reboots and you see the Lenovo logo again, start hitting F12 until you see the boot menu, then select the USB with the Talos boot image.
6. In the talos menu you will select “Enroll Secure Boot keys: auto”.
7. Don’t touch anything! Just let it work.
8. Once complete, it will reboot. The next step is optional, but highly desirable for the best security.
   • Go back into the BIOS, then “Security” -> “Secure Boot”, and select the option “Enter Deployed Mode”, then save and exit.
   • Explanation: after enrolling the key the BIOS is put in “User Mode”, however that allows changing the keys from the OS level. Changing to “Deployed Mode” requires any key changes to be done via the boot sequence.
9. On the next boot you can hit F12 again and boot into the Talos install.

At this point you should be able to install Talos as usual, and have a fully functional system.

Conclusion

And there you go! Hopefully your Talos install will complete without any issues at all. You will also be able to easily execute any future version upgrades without having to connect a KVM to your Lenovo.

And for the sake of everyone else in the future, I will contact Talos, the company, and show them this blog post. Hopefully they will add these steps to their documentation, but if not, at least it will be here for anyone else that ever needs it.

I hope this has been useful for you! And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon!

1. Introduction
2. The environment
3. The accelerated video library problem
4. Configuring the hardware and installing Talos
5. Installing the Node Feature Discovery tool
6. Installing the AMD GPU driver (plugin?)
7. Installing ollama
8. Installing llama.cpp
9. Quick benchmarks
10. Conclusion

Introduction

When I decided that I wanted to run Kubernetes in my home lab, I also told myself that I wouldn’t want to be yet another guy who blogs about k8s. However, in recent weeks, I have had three separate conversations with three different people about this one particular topic, which makes me think not a lot of people talk about it, so I’ll give my 2c.

For about two months I’ve been running ollama and llama.cpp on a Talos Linux Kubernetes cluster, on top of a Framework Desktop PC with the AMD Strix Halo CPU (AMD Ryzen AI Max+ 395 GPU). The reasons why I chose this particular combination might be a subject for a different blog post. Suffice to say, getting everything up and running was a bit challenging (I even blogged about my boot problems with the Framework Desktop), but I think I finally got it set up to a point which I can comfortably make use of it with very little maintenance, and I feel like some people might benefit from me sharing the knowledge the I acquired with that experience.

The environment

I’ll list what I have running here, so you can make a decision about whether or not you want to keep reading.

• AI/LLM tooling:
  • ollama – Like it or not this is the de facto tool to run AI/LLM in a home lab or at a small scale. It is also the most supported tool by any other application that require external tooling for AI inference. I use it for various things like Home Assistant Voice and Karakeep. I even maintain my own “ChatGPT” at home (the actual name of the tool is Open WebUI).
  • llama.cpp – Not as widely known, but it’s an AI/LLM tool that moves in a much faster pace than ollama. Not a lot of apps support talking to it specifically, but it has an OpenAI-compatible API server, which allows some of them to talk to it without knowing exactly what it is. I had to run this tool for a while because ollama support for AMD is still lagging behind (more on that later).
• Kubernetes platform
  • IMHO (in my humble opinion) Talos Linux is the easiest way to get a Kubernetes cluster these days. “What about k3s?“, you might ask. Well, k3s/minikube/kind they all require you to install an OS (operating system) first, and you have to maintain it as well. Meanwhile Talos is it’s own operating system, running as an OCI immutable image, so it simplifies your cluster maintenance by a lot. I like to say that Talos allows me to run Kubernetes at home the same way I run it in a public cloud (like AWS EKS, GCP GKS or Azure AKS).
• Hardware
  • You can use whatever you want. Whatever I did here should apply to any piece of hardware that has the same CPU architecture. There are multiple choices in the market like the GMKtec and Minisforum. I don’t have any of those, but most of what I am going to talk about here should apply to them because everything I am using is related to the AMD processor itself, and they all have models with the same CPU/iGPU architecture that I have used here.
  • I have a Framework Desktop. it’s a small-but-powerful desktop running AMD Ryzen AI Max+ 395 (Strix Halo architecture), with 128GB RAM and 2x 1TB NVMe. It has an AMD Radeon 8060S iGPU (integrated GPU) which can dynamically allocate up to 96GB of the RAM to be used as VRAM. I chose this particular piece of hardware specifically because of their engineering choices of CPU and memory, all soldered on the motherboard, and the fact that the entire thing can be easily upgraded, just like a desktop. I’m not gonna lie: there are some challenges with the shared memory, which I’m gonna talk about later, but I the form factor, the size and the power utilized makes it a very worthwhile solution (it uses LESS power than my Razer laptop running an NVIDIA 3070).

The accelerated video library problem

One big characteristic of this particular CPU architecture is the fact that you can dynamically allocate part of the RAM to be used as video RAM (VRAM). However, the combination of any kernels up to 6.16.9 (unsure of the exact version) and ROCm 6.x (AMD accelerated graphics driver, equivalent CUDA for NVIDIA) cause a bug that prevents dynamic VRAM allocation. At the time of this writing I am running Talos Linux version 1.11, which runs on top of the Linux kernel version 6.12.43, so I am (tangentially) affected by this problem.

This particular problem doesn’t exist in newer kernels (which will come with the future release of Talos 1.12) combined with with ROCm 7.x (which is not yet supported by ollama). So, for the time being, I decided that I am going to use the Vulkan library. In theory ROCm should be better, but we’re dealing with limitations of the combination of newer hardware architecture with old software, so we just need to go with the flow for now.

In theory you can use ROCm 6.x (much slower), but to do that you need to set a fixed VRAM size on the hardware level. For the Framework desktop this you can set the fixed VRAM size by going into the BIOS, then the “Advanced” menu, then “iGPU Memory Configuration”, change it to “Custom” and set the “iGPU Memory Size”. The amount here depends on what you wanna do, but 16GB should be sufficient for most cases. You can go all the way up to 96GB. You can also monitor the VRAM usage to decide when to re-allocate (subject for a future blog post). In my case, I started this way until the day ollama started supporting Vulkan, then I switched back to dynamic allocation.

Configuring the hardware and installing Talos

Apart from a problem to get Talos to boot in this particular machine, you should just follow whatever method you want to use to install it. There is one VERY IMPORTANT step though: you need to use an image that contains the AMD system extensions. There are multiple ways you can go about that:

• If you’re starting from scratch, or installing Talos without any tools, you should generate a new image using the Image Factory. Basically just go into the web site, follow the prompts, then add the “amdgpu” and “amd-ucode” extensions. This will generate an image that you can use to boot from a USB drive, or use as install source for an install tool.
• If you are using (or plan to use) a tool, like Talhelper for example, you can just boot the system with a normal boot image, but then you have to modify your configuration to add these extensions: “siderolabs/amdgpu” and “siderolabs/amd-ucode“.

Installing the Node Feature Discovery tool

Now we’re out of the hardware and into the software section of the install process.

Before anything, you need to install the Node Feature Discovery add-on. Basically what it does is discovering the hardware on your node (based on a bunch of pre-defined rules) and exposing it to your Kubernetes cluster as node tags. This is not an exclusive Talos thing, this is a very important tool for any cluster install when you need to expose specialized hardware.

In my case I used the Helm install method, but since this CPU architecture is very new, at the time of this writing I needed to create a custom rule that correctly detects the PCI device as an AMD GPU.

Installing the AMD GPU driver (plugin?)

“I thought we already installed the driver during Talos install“. Okay! You got me! Technically we are installing the Kubernetes plugin that allows workloads to have access to the AMD GPU device. There are two possible ways you can install the plugin:

• The AMD GPU Device Plugin for Kubernetes is by far the easiest way, and the one that will use the least resources in your cluster. I honestly recommend that you try that one first, before going down the same rabbit hole that I did (read the next bullet point). The document I linked talks about installing it using manifests, but there is also a Helm chart for it. Pick whatever you prefer. Sadly I don’t have a values file to share, because the first time I used this plugin I didn’t know about the NFD custom rule I mentioned in the previous topic, so the config was ugly, and eventually I gave up on this method in favor of the next one, so even if I went back into my Git history, the config wouldn’t be valid. In case you need some customization to make it detect the iGPU, you can read what I did for my final install here.
• The second method (which I am using) is with the AMD GPU Operator. This method installs the same device plugin as the previous bullet point, but also adds more features on top of that. One of the additional features is the kernel driver installation, which is very convenient for most cases, but it wouldn’t work for Talos, so it had to be disabled. The other thing it enables is metrics collection and GPU partitioning/slicing, which are advanced topics that most people don’t care about, but I want to experiment with. Both of these things are advanced topics which I am yet to explore (and are the reasons why I chose this method), but I haven’t yet. If you want to go down that route, you might want to know that I installed it using the Helm chart, so you can take a look at my values file. And due to how this video card works, I had to write a custom rule to detect it.

Installing ollama

The easiest way I found to install ollama is by using this 3rd-party Helm chart, but it needs to be tweaked for our purposes. You can see all the tweaks I did in my values file. There are a lot of comments there which will help you decide whether or not you need to tweak anything. The most important things here are:

• “resources” – the “amd.com/gpu: 1” parameter requests a node with the AMD GPU.
• “extraEnv” – you need to enable OLLAMA_VULKAN (otherwise it will fallback to ROCm).
• “volumes” and “volumeMounts” – you need to mount “/dev/dri” and “/dev/kfd“, which are the devices that give direct access to the video card.

Notice that, as I mentioned before, ollama comes pre-compiled and linked against ROCm 6.x, which has the memory allocation bug with AMD Strix Halo iGPUs. Gladly ollama 1.12.11 introduced support for the Vulkan library, which is not as good as the ROCm library, but that is the thing which currently allows this application to be usable in my setup. Once ollama updates to ROCm 7.x, I will try again and see if there is any difference.

Once the install is complete, ollama will be available inside your cluster on port 11434. As of any Kubernetes resource, it can be reachable via the hostname “<deployment>.<namespace>.svc.cluster.local” (or “ollama.ollama.svc.cluster.local” if you’re using the defaults). Optionally, you can expose it outside the cluster using an ingress controller or a Gateway API route.

Installing llama.cpp

Installing llama.cpp is a chore, because I could not find a Helm chart, so I wrote this huge and cumbersome manifest. I started using it as a point of comparison, before ollama had support for Vulkan, since lamma.cpp comes with Vulkan out of the box. Today I use it more for experiments, but my real workload use is on top of ollama, because they are basically performing at the same level. This is also the tool that I will need to play around with GPU partitioning/slicing.

Be warned: only use it if you know what you’re doing and why you’re doing it. If you copy my manifest, be mindful that it is being provided AS IS, and it might not work at all. Don’t blame me if it doesn’t work. Don’t ask for support. Basically, it’s your own adventure, and yours alone.

Quick benchmarks

Just so you can confirm that everything is working, and you can have a baseline for future comparison (when ollama upgrades, or you make configuration changes), you can run a quick benchmark.

Since I wanted to be able to compare with someone else’s install, I decided to use Jeff Geerling’s obench.sh. If you want a quick-and-dirty way of running your own, here’s how I do it in my ollama deployment:

host$ kubectl exec -n ollama -it deployment/ollama -- /bin/bash
ollama# apt update && apt install curl bc -y
ollama# curl  https://raw.githubusercontent.com/geerlingguy/ollama-benchmark/refs/heads/main/obench.sh -o obench.sh && chmod +x obench.sh
ollama# ./obench.sh -c 10 -m llama3.2:3b
(...)
Average Eval Rate: 87.14 tokens/second

Note on the above: the power peak was 100W.

And, in case you install llama.cpp and you want to benchmark that as well, here’s how I do it:

host$ kubectl exec -n llama-cpp -it deployment/llama-cpp -- /bin/bash
llama-cpp$ ./llama-bench -m /data/Llama-3.2-3B-Instruct-Q4_K_M.gguf -n 128 -p 512,4096 -pg 4096,128 -ngl 99 -r 2
(...)
(at the one-before-the-end line, where test reads "tg128", you will find your token/s: 90.47 ± 0.99)

Note on the above: the power peak was 112W

And one last note: I literally copied/pasted the commands out of the repository’s README. You can tweak to use whatever you want.

Conclusion

Installing AI tools on a Kubernetes cluster is complicated, but teaches you a lot about the ins-and-outs of how things work. It really gives you a lot more exposure to the behind-the-scenes than just running a plain Docker container on your laptop. There are a lot of pitfalls, and you need to understand what you’re doing, but once it’s running, there’s no maintenance other than version upgrades.

I really hope you can make good use of everything that was demonstrated here!

If you have any questions/comments (criticism?) or just would like to chat, feel free to reach out to me on Mastodon, or post a comment below. I will try my best to reply back to you with something useful.

1. How Talos manages the certificates
2. How to spot check the certificate expiration on the server side
3. Using monitoring tools to check for the certificate expiration
4. Don’t forget to check the client cert as well
5. Conclusion

How Talos manages the certificates

This is what they explained in their documentation:

Talos Linux automatically manages and rotates all server side certificates for etcd, Kubernetes, and the Talos API. Note however that the kubelet needs to be restarted at least once a year in order for the certificates to be rotated. Any upgrade/reboot of the node will suffice for this effect.

And I can fully confirm that. A few days ago I had an issue that forced me to reboot my single-node cluster. After it came back up the certificates were renewed. Keep reading and I will explain how I figured that out.

How to spot check the certificate expiration on the server side

There are multiple ways of doing that, and the easiest/most portable way is to simply check for the API certificate dates.

$ echo | openssl s_client -connect <api_fqdn_or_ip>:6443 2>/dev/null \
  | openssl x509 -noout -enddate
notAfter=Dec 3 11:55:14 2026 GMT

This should work for the majority of the cases, because all the certificates are renewed at the same time, and that check can be done from a machine that does not have authorization to access the Talos API.

There are two other ways which are more specific to Talos:

$ talosctl config info
Current context:     talos
Nodes:               <node_ip>
Endpoints:           <node_ip>
Roles:               os:admin
Certificate expires: 11 months from now (2026-11-23)

$ talosctl get KubernetesDynamicCerts -o yaml \
  | yq -r '.spec.apiServerKubeletClient.crt' \
  | base64 -d | openssl x509 -text -out - | grep "Not After"
            Not After : Dec  3 11:55:14 2026 GMT

Both of them are pretty easy to use. The first one does basically the same thing as the second, but gives an output that is easier to read. The second one queries for the certificates on the API server, the kubelet service and the front-end proxy. I’m filtering for the kubelet certificate for simplicity.

But, as the documentation says, you shouldn’t worry too much: all the certs are going to be rotated at the same time, so they all have the same expiration date.

And yes, I did find out that my certificates were rotated after a reboot when I was doing the research for this blog post. So I can confirm the documentation is accurate.

Using monitoring tools to check for the certificate expiration

As I mentioned before, you can safely assume that the API certificate expiration will be the same as the internal kubelet, because Talos updates them all together, so you can add a remote check to whatever monitoring service you use.

I personally use Uptime Kuma to monitor various aspects of my home lab which are not integrated into the Kubernetes cluster. In the HTTP(s) monitor there is an option “Certificate Expiry Notification”. I do have a monitor to check if the Talos cluster is up, but since it uses it’s own internal CA, Uptime Kuma does not recognize it, so I had to check the “Ignore TLS/SSL errors for HTTPS websites” option. The problem is that this option also disables the certificate expiration. I even opened a request for enhancement, to separate the CA check from other TLS/SSL errors, because I think there are valid situations (like this one) where the CA will not be recognized. (Edit: the request was rejected. I won’t argue, it’s their right).

But, for completeness sake, I extracted the internal cluster CA that issues the API certificate like this:

$ kubectl get configmap -n kube-system kube-root-ca.crt -o jsonpath='{.data.ca\.crt}' > talos-api-ca.crt

For most people using Uptime Kuma, at this point you can simply copy talos-api-ca.crt inside the container data directory, then pass the NODE_EXTRA_CA_CERTS environment variable and restart the container.

uptime-kuma:
  (...)
  environment:
    NODE_EXTRA_CA_CERTS: /app/data/talos-api-ca.crt
  (...)

In my case, since I run my own private CA, I had to actually concatenate my existing homelab-ca.crt file with the new talos-api-ca-crt into a new file named combine.crt, and load that one instead. That’s because NodeJS can only load one file via the NODE_EXTRA_CA_CERTS env var.

Now you can go into Uptime Kuma, add an HTTP(s) monitor, point at https://<api_fqdn_or_ip>:6443 and check the box “Certificate Expiry Notification”. But note: since you are not using a client certificate, you need to add 401 as a valid “Accepted Status Codes”.

Don’t forget to check the client cert as well

Just like Michael explained in his blog post, I am also the sole admin for this cluster, so I don’t care about configuring complicated authentication methods, and I simply use the standard kubectl client certificate. This one also has an expiration date, so you need to check for it as well. Michael’s solution checks out his kubeconfig from his password store, but if you, like me, just use plain kubeconfig, this is how you can check it:

$ grep client-certificate-data ~/.kube/config | awk '{print $2}' \
  | base64 -d | openssl x509 -text -out - | grep "Not After"
            Not After : Nov 23 01:06:26 2026 GMT
$ grep client-certificate-data ~/.kube/config | awk '{print $2}' \
  | base64 -d | openssl x509 -checkend 2592000 -noout
Certificate will not expire

With that, you can use Michael’s code to check your expiration when you open a shell session, if you want.

Conclusion

It’s really useful to keep an eye in the #homelab hashtag on Mastodon, as well as follow the homelab bot. There’s a lot of great content coming out of that community.

I’m also happy that this situation forced me to understand how Talos handles certificates, before I end up in a bad spot with them.

I hope this has been useful for you! And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon!

1. Introduction
2. The boot problem
3. First, the wrong solution
4. The final/correct solution for the boot problem
5. Conclusion

Introduction

I have been running a single-node Talos Linux Kubernetes cluster for a while. I still have my old and reliable Raspberry Pis, but I wanted to “up my game”, so I’m running this cluster in parallel, until I decide what I really want to do with it. Up until today, I have been running it on a single salvaged HP EliteDesk 800 G4 SFF, which was upgraded from i5 to i7, and 8GB to 64GB RAM. However, I was unable to find a decent GPU that would both fit on this case without modifications, and would be able to run some decent AI model using ollama.

I have been eyeing the Framework laptops for years, but so far I haven’t had the need to buy one. However, it was a great coincidence that they released their Desktop version right around the time when I was trying to find a GPU solution. After beating around the bush a lot, looking at many different possible solutions (vendors, form factors, etc), I decided it was a good option for me, so I got the one with the AMD Ryzen AI Max+ 395 CPU and 128GB of RAM. The main reason is: if it doesn’t do what I need it to do (expose the GPU via Kubernetes, and run ollama [or any other AI engine] decently) then the worst that can happen is that I get a new super spec’d up desktop to replace my current aging laptop.

DISCLAIMERS:

• No AI or LLM was used during the research of this problem, or for writing this blog post.
• The content here is provided AS IS, without any warranties that it will work for you or will not damage your computer.
• This is purely a demonstration of MY personal experience, with the expectations that it will help someone else.
• Please, don’t blame me if this bricks your computer or cause any data corruption.

The boot problem

Right, you don’t want to hear my sob story. You’re here to get Talos booting on your Framework Desktop.

First, let me tell you one thing: this post does not have any screenshots because I don’t have an IP KVM (yet), and any pictures I would take with my phone had a terrible reflection of me and my messy home office. So I wanted to spare you from those.

But this is the problem: I downloaded the official ISO boot image, burned it into a USB drive, plug it in, power on, and then I was presented with this lovely error:

EFI USB Device (Generic Flash Disk) boot failed.

I knew what that problem meant, so I didn’t panic. This is related to the fact that the secure boot trust certificate database does not know about Talos. I’ve been through this when I first attempted to install it on my HP, and I gladly found this blog post explaining how to get it working. So I just needed to borrow the ideas, and figure out how to do that on the Framework.

First, the wrong solution

I call it “wrong” just because it’s a temporary solution, not permanent. And also because I initially did this way because I was not paying attention, and went to the wrong menu options.

NOTE: this section is only for “extra knowledge”. The right/correct solution is in the next section. Keep reading only if you are extra curious about how I wasted my time.

When you power on the Framework Desktop, keep spamming F2 until you see the BIOS menu. Go to the option “Administer Secure Boot”, and follow these steps:

• “Select a UEFI file as trusted for execution”
• If you don’t already have anything installed on this machine, just select the first device. If you do, make sure you select the USB drive.
• Select “<EFI>” -> “<BOOT>” -> “BOOTX64.EFI” (this tells the BIOS to start GRUB)
• You will be asked “Add this hash image to allowed database (db)”, you tell it “Yes”
  • Do you already see where my mistake is?
• Now go to the same “Select a UEFI file” menu.
• Select the USB device again.
• Select “<EFI>” -> “<LINUX>” -> “Talos-<version>.efi” (this tells the BIOS to allow Talos to boot)
• Same as before, say “Yes” to the “Accept” question.
• Now hit F10 to save and reboot.

At this point you should be able to install Talos as usual, but there is one caveat: the installed system will NOT boot properly. After you complete the install, you have to remove the USB device before the system reboots, then spam F2 again. Then you need to follow the same steps as I described above, but this time you will select the files from the NVMe drive.

Why is this the wrong solution? Because it only trusts those specific files. As soon as you upgrade your Talos version, your system will be unbootable, because the files will change, and you will be forced to follow this procedure again. So, yeah, not a good solution. Please don’t do that.

The final/correct solution for the boot problem

As I learned with my journey with my first node, the proper way to fix this problem is to tell the BIOS to trust the Talos UEFI certificates.

NOTE: if you already did the wrong thing, like I did before, it’s better if you reset your secure boot keys to the factory defaults. It’s an easy option to spot in the menus.

Make sure you have the USB device burned with the Talos installer already plugged in (this does not work from a Talos already installed in the NVMe). Power on the Framework Desktop, and keep spamming F2 until you see the BIOS menu. Go to the option “Administer Secure Boot”, and follow these steps:

• Select “PK Options” -> “Enroll PK” -> then select “AUTH” as the signature format.
• Select the USB device.
• Select “<loader>” -> “<keys>” -> “<auto>” -> “PK.auth” -> Enroll? “Yes”.
  • In my case, I got the message “No data added!!”, but I am keeping this here just in case something changes in the future.
• You will get bumped back to the secure boot menu. If not, hit ESC.
• Select “KEK Options” -> “Enroll KEK” -> then select “AUTH” as the signature format.
• Select the USB device.
• Select “<loader>” -> “<keys>” -> “<auto>” -> “KEK.auth” -> Enroll? “Yes”.
• You will get bumped back to the “Enroll KEK” screen, and you should see “[PKCS7] talos.dev”.
• Now hit ESC to go back to the secure boot main menu.
• Follow the exact same steps for “db options”, using the “db.auth” file.
• At the end you will scroll down the screen and also see “[PKCS7] talos.dev” at the bottom of the list.
• IMPORTANT: hit F10 to save. If you just reboot from here, all is going to be lost.
• I repeat: hit F10 to save.

At this point you should be able to install Talos as usual, and have a fully functional system.

Conclusion

And there you go! Hopefully your Talos install will complete without any issues at all. You will also be able to easily execute any future version upgrades without having to connect a KVM to your Framework.

And for the sake of everyone else in the future, I will contact Talos, the company, and show them this blog post. Hopefully they will add these steps to their documentation, but if not, at least it will be here for anyone else that ever needs it.

I hope this has been useful for you! And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon!

Updates:

• 2025-03-05 – Added a note about the RENOVATE_GITHUB_COM_TOKEN variable.
  • This is a “Day 2” type of problem. After using this solution for a few days, I realized that it was not pulling the release notes, and there was an observation on the merge request that I should fix it by following the self-hosted documentation.
  • IMPORTANT: if you are reading this after you have been using this for a while, like me, you need to pull a new version of the “renovate” Docker image, which contains the updated variable. If it’s not working for you, it probably means you are using the old image, so try with the GITHUB_COM_TOKEN variable instead.

Table of Contents:

1. Introduction
2. Configuring Forgejo
3. Creating the Forgejo API Personal Access Token (PAT)
4. Creating the GitHub API Personal Access Token (PAT)
5. Creating the Renovate workflow
6. Finalizing
7. Conclusion

Introduction

For a while I have been meaning to find ways to automate the way I update my self-hosted applications. The first step was to figure out how to run GitHub-style workflows on my self-hosted Forgejo instance. The second step was to find some tool to automatically help me figure out when something needs to be updated. I have looked at many options, but I ended up deciding to use Renovate, simply because I wanted something that was integrated into my infrastructure as code (IaC) workflow.

Just so you know, I also looked at Watchtower, but that tool is very specifically directed at Docker containers, and I wanted something more generic. Renovate, on the other hand, supports a very wide range of “data sources” (called “managers”), so it can be used for various other purposes, like custom developed applications, or other non-docker things (like maintaining Ansible module versions, which I intend to use with my custom Execution Environment, once I figure out how).

Also, in order to make this more clearly useful, most likely I will go back to all my Docker Compose files and make sure I use explicit container versions, rather than “latest”. This way Renovate can pick them up and to it’s magic.

Configuring Forgejo

Renovate works by creating pull requests in your repository, so here you have a choice of simply having to check your web UI for any newly created requests, or you can receive notifications via email. I chose the former, since I don’t want to keep looking at my UI all the time. But this is totally optional, and you can skip this step if you don’t want email.

To get email, you need to have some way to deliver it. Forgejo needs an SMTP server, and how you achieve that is out of scope of this post. However I will leave you with this bonus knowledge nugget: I have configured an email to Pushover notification service. If you need/want that, go on, read the other blog post, then come back here.

So, in my case, I have modified the following configuration options on <forgejo-path>/gitea/conf/app.ini (you can find all the configuration options in the official documentation):

[service]
ENABLE_NOTIFY_EMAIL = true

[mailer]
ENABLED = true
PROTOCOL = smtp
SMTP_ADDR = smtp-pushover
FROM = forgejo@[REDACTED my domain]

Notice that I can’t use “localhost” here, because that would be the Forgejo’s localhost. So I have to use “smtp-pushover“, which is the internal hostname of the SMTP service inside the Docker network.

Creating the Forgejo API Personal Access Token (PAT)

There are two ways you can do this: either use your own API token, or create a new user for the Renovate service. The former is easier, but I prefer to have separate users for specific purposes, so I went with the latter.

If you want to use your own user token:

• On the web UI, click on your user picture on the top right, then click “Settings”, then “Applications”.
• On that screen, give the token a name, then select the permissions as described in this document.
• Once you click “Generate token”, it will show up only one time on your screen. So make sure to copy it into a safe place.

If you want a separate user:

• On the web UI, click on your user picture on the top right, then click “Site Administration”, then “Identity & Access”, then “User accounts”.
• Create a new user account, and make note of the password. Make sure to uncheck the box that says “Require user to change password (recommended)”. I gave mine the username “renovatebot”, and the email “renovatebot@[REDACTED my domain]”
• On the following screen, edit the user and add a “Full name”. This is important. Without that, it won’t be able to commit to Git. In my case I called it “Renovate Bot”.
• Now you have two choices: either you login in to the web UI using that account, and create a token just like the personal token as described before, or you can do it via CLI.
• If you want to do it via CLI, here’s the one-liner:

$ curl -H "Content-Type: application/json" -d '{"name":"renovatebot","scopes":["write:repository","read:user","write:issue","read:organization","read:misc","read:package"]}' -u renovatebot https://forgejo.[REDACTED my domain]/api/v1/users/renovatebot/tokens

• This will ask you for the user password.
• It will output the API key it will show up only one time on your screen. So make sure to copy it into a safe place.
• Now go to the repository where you will run the workflow, click on “Settings”, then “Collaboration”, and add the “renovatebot” user.

No matter which way you created the token, you have to pass it to the Renovate application, which will be executed inside a runner, so it will need to be passed as an action secret.

• On the web UI, click on your user picture on the top right, then click “Settings”, then “Actions”, then “Secrets”.
• Add a secret, calling it RENOVATE_TOKEN, and the value being the token you created in the previous steps.

Creating the GitHub API Personal Access Token (PAT)

This is optional, and only applies if you do have a GitHub account. If you don’t, you can skip this section.

In order to get the release notes for (most of) the updated images, you need a personal access token from GitHub. For that, go to the GitHub web UI, click on your user on the top-right, then click “Settings”, scroll all the way to the end and click “Developer Settings”. Now click on “Personal access tokens” then “Tokens (classic)” and generate a new token.

Back in your Forgejo web UI, click on your user picture on the top right, then click “Settings”, then “Actions”, then “Secrets”. Create a new secret called RENOVATE_GITHUB_COM_TOKEN, and paste the content that you just copied from GitHub.

Creating the Renovate workflow

Now that you have all the foundations ready, you should be good to create your workflow by copying my example on GitHub Gist. Just in case you don’t remember: inside your repository create a directory tree named .forgejo/workflows, and create a YAML file there (with any name) with the contents of the workflow.

I need to call your attention to a few details:

• My example workflow will run at the 15 minute mark of every hour. You can tailor that to any time you want, using the crontab notation.
  • Optionally you can shorten that time to make it run every time you push to the repository. This might be useful while you’re testing your configuration.
• There are two extra Docker volumes which have been added because I use my own private CA.
  • Also there is a line starting with “env-regex” which was also required because I need to pass my private CA to the underlying application (via the NODE_EXTRA_CA_CERTS variable).
• The line that starts with RENOVATE_REPOSITORIES is optional, with a catch. Basically it tells Renovate to run against this repository only.
  • Optionally you can tell it to run against all your repositories by setting the appropriate configuration option. I didn’t want to do that right now. Maybe at a later time, when I create some “maintenance” repository. For the time being, this workflow is exclusive to this repository.
• The RENOVATE_GITHUB_COM_TOKEN is optional (read the section above).

Finalizing

Once you push the workflow, if everything is running as expected, you should see a pull request in your repository with the title “Configure Renovate”. This is simply a way to tell Renovate that the configuration has been completed, and you agree to allow it to run it’s course. Go to that pull request and merge it, as any usual merge.

Now, if everything is working right, and you have outdated configurations, you should see one or more new pull requests suggesting updates to your applications. I my case, I had two, right on the first run.

Conclusion

That’s it! At this point you should be all set to start seeing automatic notifications to update your applications. And if you followed the bonus steps, you will even receive the notifications via email (or push message on your phone).

I hope this has been useful for you. And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon.

1. Introduction
2. Getting an API key
3. Running the smtp-pushover app
4. Testing
5. Conclusion

Introduction

For a while I have been using Pushover to sent notifications from Uptime Kuma to my phone. This is very convenient for me in many ways. Initially I was doing that via Home Assistant, but this got very convoluted, because it started mixing up with automation-related notifications, and not giving me the visibility I wanted. I have used Pushover many many years ago for other abandoned projects, so I decided to go back into it.

At some point I realized that some of my self-hosted applications can only send notifications via email, or their notifications being more useful that way (the biggest one being Forgejo). This presents a challenge, since I do not host my own email, and I would like to avoid creating dummy accounts just for that. Initially I thought about going with Mailgun, which was recommended by a lot of people in the home lab community on Mastodon, but I backed away from it because email is not encrypted, so I would be exposing a lot of my internal data to 3rd parties.

So I found this nice application (smtp-pushover) that accepts email messages and sends them to me as Pushover notifications. While there is still unencrypted email involved, its only local to my servers, and they only exist in memory, until dispatched to the Pushover API. The biggest advantage of using Pushover for delivery is that all the notifications are encrypted, and on top of that, they are very short-lived.

Getting an API key

The first step is to get an API key. If you’re a Pushover user, you probably already know how to do that, but it doesn’t hurt to get a refresher (this is not something we do very often).

• Log into your Pushover account.
• Scroll down to “Your Applications”.
• Click on “Create an Application/API Token”.
• All you need here is a name, and I simply called it “smtp-pushover”.
• Click on the “Create Application” and you will be redirected to the page where you can copy the key.

Running the smtp-pushover app

Before running the app, you need to save the API key somewhere. Since I run the app in Docker Compose, with the definition file stored in Git, it’s better to put the secrets somewhere else. For that, I created a simple file named .smtp-pushover.env on the same directory as my compose file, then added my user key, and the application token.

PORT=25 # optional, defaults to 25
PUSHOVER_USER=...
PUSHOVER_TOKEN=...

Now you can add the following to your compose file:

  smtp-pushover:
    restart: unless-stopped
    container_name: smtp-pushover
    image: ghcr.io/mattbun/smtp-pushover:0.5.31
    ports:
      - "25:25"
    env_file:
      - .smtp-pushover.env

Note that the image version is optional. You can use “latest” if you want. But I started to use explicit version numbers because I will begin using Renovate to update my containers.

Now you can restart your compose.

Testing

The developer suggested a one-liner for testing, that goes like this:

$ echo -e "Subject:Test\n\nHello" | msmtp blah@blah.com

Run this, and you should have a notification on your phone. Great!

But, hang on. I don’t have msmtp installed, and I don’t want to. So here’s a trick about how you can do that manually.

$ telnet localhost 25
Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 a9e93df6e1b5 ESMTP
helo localhost
250 a9e93df6e1b5 Nice to meet you, [172.23.0.1]
mail from: <test@localhost>
250 Accepted
rcpt to: <example@example.com>
250 Accepted
data                  
354 End data with <CR><LF>.<CR><LF>
Subject: pushover test

this should work fine
.
250 {"status":1,"request":"79793830-05db-7009-92e6-7f2ef74a92a5"}
^]
telnet> quit

Oh man. I haven’t done that in ages!

But anyway, I do have a fully working notification service!

Conclusion

Now for any application that you want to send an email, simply point them to use that host as their SMTP server. If the application is running on the same compose, make sure to point it to “smtp-pushover”. If it’s a stand-alone application running on the same host, you can point to localhost.

Notice that this is still insecure, and unencrypted, so you need to make your own decisions about your level of exposure. In my case, I block port 25 inside my network, and have one instance of that service running in each of my servers, so I can simply use “localhost” as the SMTP endpoint.

I hope this has been useful for you. And if you need any help with anything described here, feel free to ask your question in the comments below, or reach out to me on Mastodon.