In 2016, CoreOS shipped an alpha release that let people SSH into accounts with any password. I remember standing in a hotel room with Matthew and a few other folks panicing about what to do next.
So when I read SSHhole, a newly disclosed authentication bypass involving OpenSSH and musl, the failure looked familiar. Different code, different mechanism, same pants shitting moment: the assumptions and choices that define the entire software contract between the packages in your Linux distribution.
This is work you delegate when you choose a distro. Someone decides how your kernel, libc, authentication stack, and userland fit together. You are rightfully saying "yo dogg, this is beyond my knowledge so I'm going to let you do the hard bits."
This is an anecdote about how even when you're crossing your "T"s, dotting your "I"s, and reacting quicky you can still screw the pooch.
We tested whether you could log in
The 2016 CoreOS incident started with a report of a compromised
machine. An attacker had logged in as operator and was sending spam. That
account had a disabled password.
We had added support for SSSD, the System Security Services Daemon, so people could authenticate against centralized services. That meant changing the PAM configuration. PAM lets a system combine authentication modules and specify what each module's result means for the overall login attempt.
Before the change, the relevant configuration looked like this:
auth required pam_env.so
auth required pam_unix.so nullok try_first_pass
auth optional pam_permit.so
Afterward:
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth sufficient pam_sss.so try_first_pass
auth optional pam_permit.so
The intent was straightforward: accept a valid local password or a valid password from the centralized service. Either authentication backend succeeding would be sufficient.
The configuration followed SSS documentation, but inherited a crucial piece
from Gentoo: the final pam_permit. The Red Hat configuration ended with a
required pam_deny instead.
With the old configuration, a failed required pam_unix check made the login
fail. With the new configuration, neither password check had to succeed. If
both failed and the environment module succeeded, the stack could still grant
access.
We had combined pieces from two distributions into a third authentication policy. The result accepted passwords that neither backend had validated. The Gentoo folks largely eschewed SSSD at the time, so we had to be the ones with the opinions. I was already engaging in enough friendly debate with ZeroChaos (Pentoo) and others about the :ahem: merits of systemd. In the same way if CoreOS wanted SSSD in Gentoo we were going to have to configure everything in both the build and the system composition correctly.
Testing showed that valid credentials worked. Unfortunately, that observation was also consistent with accepting any credentials.
!!!
Our review and tests missed the distinction.
The vulnerability affected CoreOS Alpha releases in the 104x.0.0 series. Machines were exploited. We stopped distribution and reconfigured the update service to send affected machines back to the last good release. More than 99% of affected systems were automatically patched within eight hours of the notification.
That recovery mattered. So did the fact that we needed it. In his postmortem, Matthew Garrett explicitly rejected the idea that being an alpha release made the security failure acceptable.
A password hash that isn't a password hash
SSHhole takes another route through this territory.
The disclosure describes a combination of musl, OpenSSH's non-PAM
password authentication path, enabled password authentication, and an account
whose stored password field is *. Another account's usable password hash
supplies the algorithm and salt for a timing defense.
The problem is musl's crypt() can return * on failure. The account's
stored password field is also *. In the affected path, an error result can
compare equal to the value intended to prevent password authentication. The
disclosure demonstrates this with passwords exceeding musl's 256-byte limit
for SHA-crypt.
A marker meaning "no password should work" meets an error result that was supposed to be unmatchable. They match.
The OpenSSH 10.6p2 release notes confirm an authentication bypass
affecting some Linux systems using musl with PAM disabled, and ship a workaround.
The OpenSSH developers place the root cause outside sshd, while recognizing
that it is exploitable through sshd and can be prevented there. If you operate
this combination, check your distribution's security update for the fix.
That boundary between projects is exactly where a distribution has work to do. The musl maintainer's advisory places the error in the caller's use of the API. An operator still needs the assembled system to reject the login, wherever the eventual fix belongs.
The individual decisions have understandable purposes: avoid crashing callers, limit expensive password hashing, conceal differences in authentication timing, support accounts that use SSH keys. Their interaction still needs to preserve the rule that a failed password check cannot grant access.
Now do ping
You can find these decisions in much less dramatic places. Take ping.
Should it run as root? Should the binary be setuid root? Should the package
grant it CAP_NET_RAW? Can ordinary users send echo requests through ICMP
datagram sockets instead?
Those choices depend on the implementation, the features you need, and the
kernel's configuration. Raw sockets require CAP_NET_RAW in the governing user
namespace. Linux also provides ICMP datagram sockets, with access controlled by
net.ipv4.ping_group_range. That path can support ordinary
echo requests without granting the process the broader raw-socket capability.
The ping manual documents the cases that still require it.
So "does ping work?" is an incomplete packaging test. You also care about who can run it, which privileges it receives, which features need those privileges, and whether the installed file permissions and sysctls implement that policy.
Somebody made those decisions on your behalf. Somebody also decided how users are resolved through the Name Service Switch, how authentication reaches PAM or bypasses it, and what a locked account means to each login service.
There are enough combinations here to easily fuck yourself while every individual package builds successfully.
Small is a maintenance commitment
I understand the attraction of a small distribution. I worked on CoreOS. Fewer components can mean fewer things to patch, fewer services to expose, and a system that is easier to reason about.
But removing a component also changes the paths through the components you keep. Disable PAM and OpenSSH uses different authentication code. Change libc and a failure can arrive in a different form. Change an account marker to make key authentication work and you need to check password authentication again.
An established distribution gives you access to accumulated maintenance work: packaging decisions, downstream patches, regression tests, and people who have already encountered some of the obscure ways these pieces disagree. You are standing on their collective experience, including their mistakes.
That deserves weight when choosing a base system. If you choose a smaller project, or assemble your own, you need confidence that someone owns the same work. Brilliant engineers can still have unknown unknowns about Unix account semantics. I have direct experience of what that costs.
The useful question is what evidence the project has for the system it ships. Does it test the installed image, with its actual libc, build flags, account database, and authentication configuration? Do tests prove that disabled passwords and invalid credentials are rejected? Can it get a correction onto machines when those tests miss something?
At CoreOS, unattended updates gave us a way to recover quickly. Negative authentication tests were part of what we added after failing to prevent the incident. Both belonged to the job of shipping the operating system.
When you choose a distribution, you choose the people and processes that decide how all these pieces fit together. When you assemble one yourself, you take on that job.