I built Linux 6.10 from source, booted it in QEMU with a custom BusyBox initramfs, wrote a character-device driver, and attached GDB to a paused kernel to step through early boot at start_kernel. None of that description is the interesting part. The interesting part is that almost every step of it broke first, for reasons that had nothing to do with the kernel and everything to do with toolchain versions I hadn't thought to question.
Key technologies
Linux 6.10 · C · QEMU · GDB · BusyBox · GCC / Clang / LLVM
github.com/nhatminh06/linux-kernel-lab
Why I wanted to go this low
Most of my systems work sits above the kernel: containers, orchestration, service meshes. All of that assumes the kernel underneath just works. I wanted to spend time somewhere that assumption doesn't hold: building the thing everything else runs on top of, from source, and being responsible for every step between "here is some C code" and "here is a booting operating system."
The pieces
Two artifact chains come out of a kernel build: bzImage, the compressed image QEMU actually boots, and vmlinux, the uncompressed, symbol-bearing image GDB reads. You need both, for different reasons: QEMU doesn't want debug symbols bloating what it boots, and GDB can't resolve addresses without them. Alongside that, a BusyBox-built initramfs gives the booted kernel a minimal userspace to hand control to, and a character-device driver exposing /dev/mychardev gives me something concrete to test userspace-to-kernel communication against, using copy_to_user/copy_from_user instead of pretending pointers just work across that boundary.
The first real wall: GCC 15 and C23
The very first build failed in the boot decompressor, with errors that made no sense against kernel code that predates the compiler version I was using. The cause: GCC 15 defaults to a C23-flavored standard, and in C23, bool, true, and false became reserved keywords. Linux 6.10's boot decompressor defines its own bool/true/false, the way pre-C23 kernel code has always had to. A compiler released after the kernel was, silently changed its default standard, and broke code that had nothing wrong with it when it was written.
The fix was pinning -std=gnu11 explicitly in the decompressor's Makefile, the same fix that later landed upstream, which told me I wasn't imagining the problem. The lesson wasn't the specific flag. It was that "my code is broken" and "my toolchain's defaults changed out from under my code" produce identical-looking error output, and you cannot tell them apart without checking compiler versions first.
The same bug, wearing a different hat
resolve_btfids, a host-side build tool, failed next: a different symptom, the same root cause family: newer GCC treating a discarded const qualifier as a hard error under -Werror, in a libbpf-adjacent code path that predates that strictness. I disabled CONFIG_DEBUG_INFO_BTF rather than patch a host tool I didn't need for this lab's goals. Not every build failure deserves a deep fix; some deserve a scoped decision about what you actually need working.
GCC built it, Clang won't load it
The out-of-tree character-device module built cleanly with GCC and then failed to load. The running kernel was built with Clang and LTO; a GCC-built module carries LTO-specific compiler flags (-mllvm, -fsplit-lto-unit) that the kernel's module loader doesn't recognize and rejects outright. The kernel build system, the module build system, and the toolchain that produced the running kernel all have to agree, and nothing checks that for you upfront; it fails at load time, far from the actual mismatch. Rebuilding the module with LLVM=1 fixed it immediately once I understood what was actually being compared.
The panic that lied about its cause
The most instructive failure was the last one: the kernel booted, then immediately panicked with "Attempted to kill init" and an illegal-instruction fault, in PID 1, before anything useful had happened. That reads like a catastrophic kernel bug. It was QEMU's default qemu64 CPU model, which is deliberately conservative and doesn't expose every instruction-set extension my locally-built toolchain assumed it could use. The compiled binary contained instructions the emulated CPU didn't support, and that mismatch surfaced as "PID 1 died," not as anything that named the real cause. Adding -cpu max to the QEMU invocation resolved it in one line, after a debugging session that had nothing to do with the line that fixed it.
Attaching GDB at start_kernel
With the build finally stable, the debugging workflow was the payoff: booting QEMU paused with its GDB remote stub enabled (-s -S), attaching GDB against vmlinux for symbols, and setting a breakpoint at start_kernel: the first C function the kernel runs after early assembly setup. Hitting that breakpoint and stepping through, inspecting registers and the early-boot call stack, is a different kind of understanding than reading about boot sequences. You can see the exact state the machine is in before anything userspace-visible exists.
Why low-level debugging changed how I understand systems
Every failure in this project taught the same meta-lesson from a different angle: the error message tells you where something broke, not why. A boot decompressor error looked like a kernel bug and was a compiler default. A module load failure looked like a driver bug and was a toolchain mismatch. A boot panic looked like a kernel panic and was an emulator's CPU model. In each case, the fix was one line once I found the real cause, and finding the real cause meant refusing to trust the first plausible explanation.
That habit transfers directly to everything above the kernel, too: a failing pipeline stage, a pod stuck in an init state, a service that's "down" according to one signal and fine according to another. The instinct this project reinforced is to keep asking what layer actually owns the failure before trying to fix the layer that happened to report it.
Closing
I didn't set out to write a novel driver or discover a kernel bug. I set out to understand a system I'd been trusting blindly for years, and the way I actually got that understanding was by breaking my toolchain against it, repeatedly, and being forced to figure out exactly why each time.