Enterprise Linux users face growing risks from software vulnerabilities, especially given their widespread reliance on open-source code in Linux applications and commercial software.
Live kernel patching minimizes the need for organizations to take down servers, reboot systems, or schedule disruptive maintenance windows. While these challenges are significant, live patching offers a practical solution to reduce downtime and improve operational efficiency.
Besides applying kernel patches, keeping up with security patches of known and newly discovered software vulnerabilities is a worsening problem for all computing platforms. The process is critical with the heavy integration of open-source code used in both Linux applications and commercial software.
According to Endor Labs’ Dependency Management Report released in September, security patches have a 75% chance of breaking an application. It reveals the dangers of emerging trends in open-source security and dependencies from third-party software components. Linux is used across industries to run organizations’ backend servers and applications.
These operations require better software development lifecycle security strategies, noted researchers. They found that managing open-source dependencies is impossible with public advisory databases alone. Fixing known code issues is either ignored or put off indefinitely. Studies show that a median delay of 25 days occurs between public patch availability and publication for 69% of vulnerabilities.
“The patching process is highly disruptive. When you need to update something, you have to take down the service, or you have to take down the entire server. If anything is visible to the end user, it is a problem. It requires scheduling, acceptance by all the stakeholders, and everybody signing off on it being at the right time,” Joao Correia, cybersecurity expert at TuxCare, told LinuxInsider.
TuxCare offers automated vendor-neutral enterprise Linux security patching with zero downtime and no required reboots. Its KernelCare solution supports multiple Linux distributions commonly used in organizations, providing flexibility across different environments.
Resistance to Change Keeps Software Insecure
According to Correia, live patching strategies were first developed in 2006. Various commercial-grade Linux server distributions such as Oracle, Red Hat, and Canonical’s Ubuntu have their own live patching services.
Organizations are good at adopting new technologies but not so good at changing how they handle software patching, he noted. The IT industry still does traditional patching as if nothing has changed in the past 30 years of computing.
Wrong approach, he warned. Nothing about our technology stack and the architecture of the things that we use is the same. We insist on shoehorning traditional patching into everything, he complained.
“The real problem is not live patching. The real problem is changing people’s mindset towards live patching. It is more than proven that it works,” Correia said.
Findings Support Broader Live Patching Adoption
The Endor Labs report supports his conviction that live patching needs to be adopted more widely. For example, across six ecosystems, 47% of advisories in public vulnerability databases do not contain any code-level vulnerability information. Without this, it is impossible for organizations to establish whether known vulnerable functions can be exploited in their applications.
For instance, updating the top 20 open-source components to non-vulnerable versions in the Python ecosystem would remove more than 75% of all vulnerability findings. For Java, it would remove 60% of the vulnerabilities.
The Node Package Manager or NPM, an open-source repository of tools engineers use to develop applications and websites, would remove 44% of vulnerabilities. Phantom dependencies, which are invisible to security tools, account for 56% of reported library vulnerabilities.
Endor Lab researchers also found that 95% of open-source dependency version upgrades have the potential to break applications.
Urgent Need for Live Patching Adoption
Correia asserted that live patching solves those issues and a whole lot more and is more than proven to secure your system much faster than traditional patching.
“It is non-disruptive, and there is still this unwillingness to change your methodology because it will impact your methodology. It will impact it in a good way. You do not need maintenance windows anymore,” he explained.
In some quarters, even IT experts are hesitant — or prevented — from adopting live patching. The new process is different from what they have done all these years and from what they were likely taught when they started the job.
“How do you overcome that? Well, we try to scream out from the highest hill we can find every single day. It has to be education,” he offered.
What exasperates Correia is the needless security breaches that occur because live patching is not universally applied. When an unpatched vulnerability impacts millions of servers worldwide, the problem could have been prevented or fixed within three or four minutes of its being made public.
“You could have had the patches applied, updating your systems within a couple of minutes after this was disclosed so none of your systems needed to be affected. You did not need to suffer a breach because of it,” Correia lamented.
This urgency touches on all computing levels with the caveat that both academia and government are more susceptible to this problem. U.S. government regulations now impose stricter security guidelines on government agencies.
Within academia, however, clever students and internal workers love to test their skills and try to breach systems on campus. Correia knows that firsthand from years of working as a systems administrator for universities. Other industries have their own compliance issues to meet.
TuxCare’s Live Patching Process
TuxCare achieves live patching through a four-step process that begins with monitoring announcements for new vulnerabilities affecting the Linux kernel:
- A patch is created to replace the insecure kernel code.
- The replacement code is compiled and prepared for deployment through TuxCare’s distribution servers.
- The KernelCare process running on the client-server checks for updates every four hours, unless manually initiated.
- The patch is applied. The detection and installation process can be configured automatically. The KCE kernel module pauses all processes, loads the updated binary into the secure kernel space, redirects all functions to the updated code, and resumes the kernel operation.
“Usually, the fixes are simple. Either it is off by one mistake, or a balance check that is missed, or it is a variable used when it has been freed. Basic issues that can be fixed very easily with one or a couple of lines of code,” Correia explained.
TuxCare’s code engineers look for the differences before and after the fix is deployed. They apply those differences to other Linux server distros with the same configurations.
Live Patching for Servers vs. Desktops
Correia noted that live kernel patching on Linux servers is much different from doing a live patch on the consumer end for a desktop Linux distribution. That big difference is the necessity of live patching with servers.
Linux desktops receive regular security patches and system updates, either on a rolling basis or according to a recurring schedule, depending on the distribution. Some systems, like Ubuntu and Linux Mint, even provide kernel updates and fixes without waiting for a new distribution release.
Unlike server environments, rebooting desktops for updates is not a significant issue. Desktop reboots typically complete within a few minutes, minimizing disruption.
“There is a difference between a patch and an update. An update is a new, more minor version of a package. It can contain bug fixes, performance improvements, new features, edits at the command line, and other enhancements,” Correia explained.
A patch is a partial snippet of code that fixes a vulnerability in the existing version. These patches fix vulnerabilities without latency so that the existing implementation can run more securely and system administrators can hold off on rebooting until the next regular maintenance window, he added.
Loading ...
