Terry's journal

On Passing the CKA Certification

Wed, 14 Apr 2021 13:00:00 +1000

TL;DR

Passed the CKA (k8s v1.20); don’t know the score. The learning journey matters more than the outcome ;-)

main()

The CKA is my 1st IT certification in 7 years, last obtained was the CCA for XenServer 6.0 when working at Citrix (good old days).

As a hands-on type of guy holding the spartan style - show me the fking code / skill value, I am NO big fan of collecting IT certifications (badges). They don’t mean one can do the work well, let alone their productivity.

You can most likely validate if the candidate is smart but NOT if he/she can output productivity (optional - under pressure) -_-z

Will use this analogy to explain the reason behind getting the CKA certification:

Someone has been building for 20 years with success, he/she does NOT need a licence to be able to build, however, a proper builder licence is required so as to build LEGALLY (for self and clients).

Looking back, personally I’ve been dealing with k8s since Feb 2018 (v1.9.0 using the self-hosted kubeadm flavour. I’ve never done @kelseyhightower’s Kubernetes the hard way, it’s good to do but NOT necessary.

Preparing for the exam

I’ve completed the k8s fundamentals LFS258 course (best on market) and did most of the labs, focusing on areas that I am not that familiar with, considering the fact that I’ve been working on a specific flavour of k8s on a daily basis for 2+ years.

Doing the exam

I managed to finished 16.5 out of 17 tasks within 2 hours, didn’t know Network Policy good enough as the gravity flavour does not use it at all. Frankly speaking, I consider the exam intense and challenging considering my skill and background.

NOTE: The PSI exam user experience did have some negative impact on efficiency, network latency was OK. It’ll be good if LFS258 course can provide a simulation exam mock at the end.

Outcome

Passed, no score card or report (it’s a flaw)…

While the outcome is important, enjoy (and benefit from) the learning journey.

Some Thoughts

the CKA certification exam is unique in may ways:

2 hours, 17-20 hands-on tasks including k8s cluster troubleshooting and upgrade a cluster the kubeadm way
all hands-on tasks are performed in a web based Terminal Emulator (1 additional tab is allowed)
access to k8s docs and GitHub repos (never heard of before…)
freedom to install any package available from distro’s repo (e.g. vim, tmux, curl, jq, etc.) use your favourite tools to create the productivity workflow
Remotely proctored via chrome extension (via webcam + microphone), you can even request a break during the exam (I didn’t…)

Tips & Tricks

RTFQ - read the Questions carefully at least twice!
Check context, check context, check context!!!
Time management
develop a winning strategy to pass (make sure to get the ones you know first and correct!)
get Bash completion to work for kubectl and alias it to k or whatever (you don’t have to remember, all available in docs)
Know just enough Vim (or any editor that works you the best, e.g. nano, emacs and micro, etc.) to efficiently editing YAML. Create a minimal ~/.vimrc if that can help improve productivity.
I don’t find tmux useful in the exam, don’t get me wrong, I am a tmux fan, it’s been a part of my daily productivity workflow for 10+ years, it just doesn’t add much value to the exam. If you see it mentioned elsewhere, consider it bragging -_-z

Final Thoughts

I highly recommend the CKA for those who would like to pursue a career related to infrastructure, especially the popular and high demand container orchestration skill tree (and/or its ecosystem).

I’ll probably do the CK{AD,S} if I have the energy (incentive), and $ / time to spare.

Fun facts: As many who know me already know, I’ve been using Linux as my primary workstation OS, self-hosting countless Servers/Services since 2003 (a true distro-hopper), consider myself a poweruser (self-claimed Linux Ninja) but I’ve never held any Linux certifications (RHCE, LPI, LFCS, LFCE, or you name it, etc.) and no plan to do so ;-)

Not Yet - Resources for Continuous Improvement ;-)

Best free Training I’ve come across, from Gravitational: gravitational/workshop Deep dive into k8s networking, RBAC, security, logging, monitoring, etc.

learnk8s.io Full package ;-) (paid, well worth it)

Reference: Original Posted on LinkedIn.

EOL

vmtouch The Virtual Memory Toucher

Mon, 08 Feb 2021 18:09:00 +1100

TL;DR

Everyone loves touch.

vmtouch is the virtual memory toucher, portable file system cache diagnostics and control.

main()

OS kernel takes care of filesystem cache, generated from read / write. However, users don’t have much fine-grained control over what files / directories to cache / drop when more memory is needed to run new applications.

Take the Linux kernel exposes /proc/sys/vm/drop_caches for privileged users to signal the kernel to drop clean caches, as well as reclaimable slab objects like dentries and inodes to free up memory.

You only do

free pagecache echo 1 > /proc/sys/vm/drop_caches
free reclaimable slab objects (includes dentries and inodes) echo 2 > /proc/sys/vm/drop_caches
free slack objects and page cache echo 3 | sudo tee /proc/sys/vm/drop_caches

See Documentation sysctl/vm.txt for more info.

vmtouch offers more flexible and granular control over what files, directories (can be recursive) to load into memory. It is also possible to evict files from the cache.

Files loaded by vmtouch get locked into memory and stay there until system is restarted or vmtouch is terminated. It’s fun to use.

Keep in mind that the poweer of shooting yourself in the foot is in your own hands. Use it wisely and carefully.

NOTE: vmtouch is for page cache. If you are looking for solution to manage browser profile(s) in tmpfs / ramfs and periodically sync back to disk, take a U-turn and look at profile-sync-daemon AKA psd .

EOF

Time to Go All In for Podman

Wed, 03 Feb 2021 18:23:00 +1100

TL;DR

Docker Compose support lands in Podman 3.0.0 (dev as of 3rd Feb, GA soon). It can finally work seamlessly with Docker Compose as drop-in replacement to orchestrate containers.

NOTE: Initial support, rootless not yet supported.

main()

What is podman?

Daemonless container engine for managing OCI compliant containers on Linux.

For a long time, missing Docker Compose equivalent feature (to orchestrate multiple containers on single container runtime host) is a deal-breaker.

The podman-compose script is an option but it’s nowhere near drop-in replacement. Lost count on how many times I had to modify the docker-compose.yml file to make it work.

Docker CE had been working well for my use cases for a long time (Dockerfile, Compose, etc.).

Reason why I started using podman: Fedora 31 onwoards, Docker CE started to hit cgroup v2 support issue, the removal of docker-ce from official DNF repo ative package / repo issues (lag behind).

Enable podman.socket which implements Docker REST API and exposes a UNIX domain socket as backend, to provide feature parity (still daemonless?) ;-)

It can also generate YAML (and systemd units) to streamline the workflow to migrate from dev to production, from workstation to single host container runtime engine and/or production k8s clusters.

NOTE: Not all distributions provides podman-compose package (Arch / Manjaro and Fedora do AFAIK).

Podman alone cannot replace Docker CE. You’ll need the complete next generation container toolchain: Podman, Buildah (building OCI container images), Skopeo (working with remote contianer registries).

EOF

The Fall of CentOS

Wed, 09 Dec 2020 14:00:00 +1100

TL;DR

Use Oracle Linux (run RHCK) as free RHEL alternative.

main()

Goodbye CentOS, well done IBM (Red Hat).

There will be NO CentOS 9.
No updates for CentOS 8 after 31 December 2021.
No updates for CentOS 7 after 30 June 2024.

CentOS Steam will be the only release using CentOS badge moving forward, sitting in between Fedora (cutting edge) and RHEL (production stability), as a development rolling branch for RHEL.

Oracle Linux is

completely free to use, distribute and upgrade
application binary compatible with RHEL whether running the Unbreakable Enterprise Kernel or Red Hat compatible kernel.

see the FAQ for more information.

Funny enough, I have been maintaining Oracle Linux {5,6,7,8} VM images since I started using Vagrant as a part of the development & test workflow. Looks like it has paid off (facepalm), the switch is painless.

It’s a great opportunity for Oracle Linux to recovery lost territory, look forward to seeing the rise of Oracle Linux, ironically, thanks to IBM & Red Hat.

I don’t believe people would refuse to use free (& reliable, with good docs) Oracle Linux simply because they dislike (hate) Oracle (the company), same applies to VirtualBox.

The best tools are the ones that do what you want at the best cost.

As long as it does the job, who cares about the logo.

"Shut up and get the work done." - Linus Torvalds

2020-12-14 update

Founder of CentOS started a new RHEL rebuild called Rocky Linux, aiming to be a community driven, enterprise grade production ready Linux distribution.

In the meantime, Senior Directory of Oracle Linux & Virtualization wrote an interesting blog post with the intention to attract CentOS refugees as soon as possible by throwing the centos2ol 1-click switch script ;-)

Oracle Linux has been tracking RHEL releases consistently (since 2006), faster than CentOS, making it a better option if the Oracle Logo is not a concern.

errata typiclly within 24 hours
minor releases within 5 business days
major releases within 3 months

2 kernel options

RHCK (100% RHEL)
UEK (more modern kernel tracking mainline - UEK source code)

Supports workloads on both x86 and ARM systems, hip.

ISO images, Vagrant Boxes, container images and Raspberry Pi disk images can be directly downloaded from ---> YUM!

References

EOF

kitty tweaks for macOS

Thu, 23 Jul 2020 16:00:00 +1000

TL;DR

kitty has become my preferred secondary Terminal Emulator on macOS.

To make it as productive as it is under Linux, and in line with muscle memory tweaks are needed, add the following in ~/.config/kitty.conf

macos_option_as_alt yes
map alt+left send_text all \x1b\x62
map alt+right send_text all \x1b\x66

main()

Personally I started using kitty on Linux as an alternative to Konsole as it’s light-weight, featureful and GPU accelerated (offload rendering to GPU).

I still use other terminal emulaters like, Guake, Tilix, GNOME Terminal, alacritty, etc. under different scenarios but kitty has really grown on me.

See the full list ---> Productivity Tools

Adopting kitty on macOS has its own challenges, yes, it’s the ⌥ option key.

Primary terminal emulator iTerm2, I’ve been using Nature Text Editing key presets with left ⌥ option mapping to Esc+.

That way ⌥ + ←/→ behaves the same as alt + b/f, in line with Bash / readline key bindings which has already become a part of my muscal memory.

To do the same for kitty, in its config file ~/.config/kitty.conf

Use Option as alt

macos_option_as_alt yes

At the bottm append the following 2 key mappings and restart kitty, pat, pat ;-)

map alt+left send_text all \x1b\x62
map alt+right send_text all \x1b\x66

Now behaviours of ⌥ + ←/→ are in line with ⌥ (alt) + b/f.

Make kitty productive again ;-) Have fun petting kitty.

Reference: https://github.com/kovidgoyal/kitty/issues/838

Other Terminal Emulators I use from time to time, still quite like some of them (e,g, drop-down style like Guake, tilda).

EOF

iproute2 & systemd output in JSON format

Tue, 09 Jun 2020 18:00:00 +1000

TL;DR

Didn’t notice ip provided by iproute2 supports output in JSON format until I read about the motivation behind jc.

Example

ip -j a | jq
ip -j r | jq
ip -j -s link | jq

main()

Must have JSON CLI utils for more productivity:

jq - who doesn’t know? sed for JSON data
gron - makes JSON greppable. Super useful for people like me who cannot figure out the jq filters by simply looking at JSON code, get enlightened by gron and gron --ungron.
jc - ideal for COMMAND | jc PARSER [OPTIONS] use cases

In addition to iproute2, some systemd provided *ctl command supports JSON output format as well. Most commonly used ones

systemctl
journalctl
…

Examples

# systemctl status
systemctl status sshd.service -o json
systemctl status -l sshd.service -o json

# journalctl
journalctl --since yesterday --output json

Have fun processing JSON with turbocharged productivity ;-)

EOF

Make Windows Productive Again with WSL 2

Tue, 02 Jun 2020 14:00:00 +1000

TL;DR

Windows Terminal (Terminal Emulator) + Ubuntu (20.04 latest LTS) via WSL 2 + winget makes Windows 10 productive for Linux / macOS power users.

main()

I was given a Dell Latitude 5300 2-in-1 by a customer for a project. First impression: the build quality is not bad. What? it can be flipped to be a tablet, but no pencil (can’t find it attached).

It runs Windows 10, hmm…

BIOS is secured with password.
No local Administrator priviledge.
VPN needs to stay ON.
Seamless Active Directory integration and SSO.
Tight Domain security policies enforced.

NOTE: tight security control - give users no choice makes a lot of sense for a non-IT company, which can efficiently reduce helpdesk workload and operational cost.

I planned to install Ubuntu 20.04 in WSL 2 to run Ansible (which does not support Windows as control node) before getting the laptop.

It took a week to get approval to be given Power User.

Then the fun began

Added the domain user to local Administrators group (opted out for helpdesk as a result) - this needs to be repeated after each power cycle (domain security poilicy).
Installed Windows Terminal
Enabled WSL 2
Failed to install Ubuntu 20.04 from MicroSoft Store (not provisioned yet based on error message - PUR - not augumented)
Installed winget (instead of scoop)

Store doesn’t sell, one can still install from Ubuntu 20.04 focal wsl rootfs tarball, fine ;-)

# wsl --import <DistributionName> <InstallLocation> <FileName>

PS C:\Users\twang\Downloads> wsl --import Ubuntu-20.04 'C:\Users\twang\wsl\ubuntu' C:\Users\twang\Downloads\focal-server-cloudimg-amd64-wsl.rootfs.tar.gz

NOTE: the default login is root, bad even for WSL use case. Create a new user and launch WSL as non-root user:

# create user
useradd -m -U -s /bin/bash terry

# password-less sudo
echo 'terry ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/terry && sudo chmod 0440 /etc/sudoers.d/terry

# launch as user
# wsl -u terry
wsl -d Ubuntu-20.04 -u terry

Set Ubuntu Focal as default

wsl -s Ubuntu-20.04

winget works well (early stage) as basic package manager installing required software.

winget install terminal
winget install winscp

# winget search <keyword>
# winget show
# winget source list

NOTE: winget is known to be a AppGet fork, hopefully with Microsoft backed development, I won’t doubt that it’ll soon become homebrew equivalent existence on Windows. Good for users, sad for AppGet, scoop (may still have its place to install CLI utils) and Chocolatey.

Know Just Enough WSL to be Productive

Fun facts

Windows %userprofile% is mounted as /mnt/c/Users/<username> as FSTYPE —> drvfs.
\\wsl$ leads to the parent directory of the list of Linux distributions installed.
In Windows Terminal explorer.exe . launches Windows Explorer in current working directory.
Equivalent to
- open . on macOS
- xdg-open ., dolphin . or other_file_manager . on Linux.
Copying files from Windows to Linux (WSL 2), produces weird <filename>.Zone.Identifier files (NTFS related bug) which it should not, wtf…

Never mind, just get rid of it find ~ -type f -iname '*.Identifier' -exec rm -rfv {} +

Use fd if more comfortable with it ;-)
Don’t forget to symlink to Windows %userprofile% to make life in WSL easier
```
 ln -sf /mnt/c/Users/<username> ~/userprofile

 cd ~/userprofile
```

Overall Windows Terminal user experience is OK, it has huge room for improvements though, long way to go.

Last but not least, Linux in WSL 2 is NOT real Linux virtual machne. It is highly customised light-weight Hyper-V VM running customised kernel (Ubuntu 20.04 uses 4.4.0). It doesn’t use systemd as init to initialise userspace, as a result none of the systemd *ctl commands works. Some other GNU or open source utils may not work either, compatibility will improve for sure.

Ubuntu in WSL is definitely NOT perfect.

It definitely won’t replace Linux on workstations and laptops for me in any sense, however, I won’t look back at Cygwin or running Linux VM using VirtualBox if I have to use Windows machines to get things done and deliver.

NOTE: Cygwin popularity declines with the rise of WSL. It is still the preferred choice for people running Windows 7 or older who needs quick & dirty light-weight POSIX-compatible programming and runtime environment.

EOF

404 day, stay 200

Sat, 04 Apr 2020 15:43:00 +1100

TL;DR

It’s 404 Not Found day of the year 2020 (404 * 5), everyone stay 200 OK.

$ date -I
2020-04-04

main()

$ curl -LIs who.is/responsible/4/covid-19
...
Server: Vanish
...
HTTP/1.1 404 Not Found
...

$ curl -LIs 2020.life/family/stay?status=safe&condition=healthy
...
Server: FeelingLucky
...
HTTP/2 200 OK
...

EOF

sysctl drop_caches is write-only since Linux 5.5

Fri, 03 Apr 2020 15:52:00 +1100

TL;DR

terry@n54l:~$ cat /proc/sys/vm/drop_caches
cat: /proc/sys/vm/drop_caches: Permission denied

terry@n54l:~$ sudo -i
root@n54l:~# cat /proc/sys/vm/drop_caches
cat: /proc/sys/vm/drop_caches: Permission denied
root@n54l:~# whoami
root

# WTF?

Since Linux 5.5, drop_caches has become write-only (mode bits 0200) to avoid confusions when operating at scale.

From Kernel Newbies Linux 5.5: make drop_caches sysctl write-only commit

main()

From the git commit: kernel: sysctl: make drop_caches write-only

drop_caches mode bits changed from 0644 to 0200 which means write-only.

Justification

Currently, the drop_caches proc file and sysctl read back the last value written, suggesting this is somehow a stateful setting instead of a one-time command. Make it write-only, like e.g compact_memory.

It makes sense, drop_caches is one-off command, stateless. It really confuses if operating at scale, I’ve been in that boat before (many times).

Author explained a bit more with real world experience:

While mitigating a VM problem at scale in our fleet, there was confusion about whether writing to this file will permanently switch the kernel into a non-caching mode.

This influences the decision making in a tense situation, where tens of people are trying to fix tens of thousands of affected machines: Do we need a rollback strategy? What are the performance implications of operating in a non-caching state for several days?

It also caused confusion when the kernel team said we may need to write the file several times to make sure it's effective ("But it already reads back 3?").

Another sysctl syscall fun fact

Came across this in Linux 5.5(https://kernelnewbies.org/Linux_5.5) change log

Remove the sysctl system call (deprecated a long time ago) commit

This system call has been deprecated almost since it was introduced.

In a survey of the linux distributions I can no longer find any of them that enable CONFIG_SYSCTL_SYSCALL. The only indication that I can find that anyone might care is that a few of the defconfigs in the kernel enable CONFIG_SYSCTL_SYSCALL However this appears in only 31 of 414 defconfigs in the kernel, so I suspect this symbols presence is simply because it is harmless to include rather than because it is necessary.

As there appear to be no users of the sysctl system call, remove the code. As this removes one of the few uses of the internal kernel mount of proc I hope this allows for even more simplifications of the proc filesystem.

I decided to do a validation on the distributions I use daily. As you can see below, obviously Arch Linux, Fedora were fine, but Ubuntu, hmm… ;-)

# fedora 31
PRETTY_NAME="Fedora 31 (Thirty One)"
root@n54l:/boot# grep CONFIG_SYSCTL_SYSCALL config-$(uname -r)

# arch
terry@netbook:~$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Arch Linux"
terry@netbook:~$ zcat /proc/config.gz | grep CONFIG_SYSCTL_SYSCALL

# ubuntu 18.04
$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Ubuntu 18.04.4 LTS"
$ grep CONFIG_SYSCTL_SYSCALL /boot/config-$(uname -r)
CONFIG_SYSCTL_SYSCALL=y

EOF

systemctl isolate home.target && systemctl restart blog.service

Thu, 02 Apr 2020 20:49:18 +1100

`TL;DR`

I am resurrecting self-hosted personal blog, tweets are often instant spark of thoughts like spring, quick fun but hard to search and reference when needed. Contents and opinions of value should be saved and stored properly, in a better form, static pages are simple, indexable, searchable and can be easily hosted

So ssh, let me touch:

terry@terry.im:~$ sudo -i
root@terry.im:~# systemctl isolate home.target
root@terry.im:~# systemctl restart blog.service
root@terry.im:~# whoami
root

`main()`

Year 2020 turned out to be beyond most people’s wildest imagination. It’s 404 * 5, the whole world has been turned upside down by the COVID-19 outbreak. The world has NOT seen a pandemic at this scale since the 1918 H1N1 pandemic.

For Sydney (Australia), even worse, hit by the devastating bush fire across the country, worst in 30+ years, along with drought, followed by flood. Elements of magic: fire, water, earth, lightning, thunder… Human beings are so small in front of nature, let us show respect and be in awe of Mother Nature.

NOTE: 404 has special meaning for people live inside the GFW (or massive censorship), it can represent disappeared key words, posts (content of any type) or even human being.

Pulling all tweets, sync using cron job, or download and load archives periodically sounds viable. However, there is no good front-end UI (people like Linus and myself, look up, hmmm… can’t do UI). Indexing tweets using Elastic Search, visualise and analyse using Kibana sounds funky but apparently an overkill for someone who’s no one.

So, shorter blog posts, higher frequency & quality is the way to go. Furthermore, as the master of self-hosted blog with root of underlying infrastructure, I am the real owner of my thoughts (contents). echo 404 > /dev/null

Everyone (with extended family and loved ones) stay 200 (OK), stay healthy & safe.

EOF