20 Years in the making: Real-Time Linux finally makes it to the kernel

Vienna, Austria: No, I’m not kidding. After 20 years, Real-Time Linux (PREEMPT_RT) is finally–Finally–in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. Why is this a big deal? Let’s start by explaining what a real-time operating system (RTOS) is and what it’s good for. 

What is an RTOS?

RTOSs are specialized operating systems designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, “Real-time is the fastest worst-case scenario.” 

He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. Many people assume that RTOSs are for fast processes; they’re not. Speed is not the point in RTOSs — reliability is. This predictability is crucial in applications where timing is essential, such as industrial control systems, medical devices, and aerospace equipment.

Also: 5 Linux commands you should never run (and why)

Some popular examples of real-time operating systems in use today include  VxWorks, which is used in NASA’s Mars rovers to guide them, and the Boeing 787 Dreamliner aircraft to control avionics systems, ensuring real-time responsiveness for flight controls. Another example is QNX Neutrino, which is widely used in cars for infotainment, and advanced driver-assistance systems such as anti-lock brakes.

Real-Time Linux’s history

The Real-Time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. 

Real-time Linux began in the late 1990s when there was a growing need for Linux to support real-time applications. The initial efforts focused on creating separate real-time kernels that ran alongside the Linux kernel. This included academic projects such as KURT from the University of Kansas; RTAI, from the University of Milano; and New Mexico Institute of Mining and Technology’s RTLinux.

Ingo Molnar, a senior Linux kernel developer, started collecting and reshaping pieces of these technologies in 2004 to build the foundation for the real-time preemption patch set PREEMPT_RT.

This approach was different from earlier real-time Linux solutions as it modified the existing Linux kernel rather than creating a separate real-time kernel. By 2006, it had gained enough traction that Linus Torvalds observed, “Controlling a laser with Linux is crazy, but everyone in this room is crazy in his own way. So if you want to use Linux to control an industrial welding laser, I have no problem with you using PREEMPT_RT.” 

By 2009, a small team of kernel developers, including Thomas Gleixner, Peter Ziljstra, and Rostedt, had finished consolidating previous prototypic developments into a single out-of-tree patch set. It was then that many companies started using this patch set to build industrial systems requiring hard real-time properties with millisecond precision.

Also: These Linux distributions are best for developers – here’s why

As the project moved forward, many elements of it moved into the kernel. Rostedt told me in an interview that, in a way, it’s wrong to say that real-time is only now in Linux. Many of its features have been introduced into mainstream Linux over the years. Some of these, indeed, are essential to the Linux you use every day. 

For example, chances are you’ve never heard of “NO_HZ,”  which reduces power consumption in idle systems. This is what enabled Linux to run efficiently on machines with thousands of CPUs. “You don’t realize how much Linux improved because of the real-time patch,” Rostedt emphasized. “The only reason why Linux runs in data centers today is because of the work we did.”

Also: Linus Torvalds muses about maintainer gray hairs and the next ‘King of Linux’

So, without NO_HZ, Linux wouldn’t be running essentially all data centers. This, in turn, explains why Linux runs the cloud. I don’t know exactly what the world would look like without this real-time contribution, but it wouldn’t look anything like it does today. 

Real-Time Linux has also been really useful in ways no one ever dreamed of at the start. Rostedt remisenced, “Back in 2005, I got a real-time bug report, and I sent a patch and said, ‘Hey, here’s the fix. Can you apply it?’ And the guy’s like, ‘I don’t know what I’m doing.’ I replied, ‘Wait, aren’t you a kernel developer?’ He replied, ‘I’m a guitarist.'”

Also: 10 Linux apps I can’t do without – and why

It turned out he was using the early real-time patches because he was using JACK, the sound server for low-latency audio connections. He was using it because, like most musicians, he was too broke to buy high-end gear so “he got a cheap laptop, with Linux and JACK, because with the real time patch it would do good recording instead of skipping when the hard drive was writing.”  

It turns out that a lot of musicians were early Real-Time Linux users because it let them produce high-quality recordings on the cheap. Who knew? Other Real-Time Linux features that has slipped into the mainline kernel over the years include:  

1. The introduction of mutexes 

2. The ftrace debugging tool, which is arguably the most important Linux debugging tool

3. Priority inheritance for user-space applications

What took Real-Time Linux so long?

So, why is Real-Time Linux only now completely blessed in the kernel? “We actually would not push something up unless we thought it was ready,” Rostedt explained. “Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in.”

In addition, the path to the mainline wasn’t just about technical challenges. Politics and perception also played a role. “In the beginning, we couldn’t even mention real-time,” Rostedt recalled. “Everyone said, ‘Oh, we don’t care about real-time.”

Another problem was money. For many years funding for Real-Time Linux was, in a word, erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.

Also: Need to transfer files between Linux machines? Here are 5 dependable solutions

The final hurdle for full integration was reworking the kernel’s print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k. He wrote the original code and he still uses it for debugging to this day. However, print_k also puts a hard delay in a Linux program whenever it’s called. That kind of slowdown is unacceptable in real-time systems. 

Rostedt explained “print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that’s great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds.” That delay was the print_k message.

After a ton of work, several rejected proposals and heated discussions, a compromise was finally reached earlier this year. Torvalds is happy, the Real-Time Linux developers are happy, priint_K users are happy, and, at long last, Real-Time Linux is real. 

Also: 5 best Linux distributions for gamers in 2024

So, after two decades of development, the Linux real-time patch has finally been merged into the mainline kernel. This milestone marks the culmination of years of work by kernel developers to bring deterministic, low-latency performance to Linux.

With it, the Linux kernel is fully preemptible. This allows it to respond to events within microseconds. This capability is crucial for applications that require precise timing, such as industrial control systems, robotics, and audio production.

With the merging of the real-time patch, Linux is now poised to be a serious player in the RTOS world. This is a victory not just for real-time manufacturers but for all Linux users.