Far away to be identical

kurbu5: MIT Kerberos plugins in Rust

Sat, 04 Apr 2026 22:10:00 +0300

For a couple of years, Andreas Schneider and I have been working on a project we call the ‘local authentication hub’: an effort to use the Kerberos protocol to track authentication and authorization context for applications, regardless of whether the system they run on is enrolled into a larger organizational domain or is standalone. We aim to reuse the code and experience we got while developing Samba and FreeIPA over the past twenty years.

Local authentication hub

The local authentication hub relies on a Kerberos KDC available on demand on each system. We achieved this by allowing MIT Kerberos to communicate over UNIX domain sockets. On Linux systems, systemd allows processes to be started on demand when someone connects to a UNIX domain socket, and MIT Kerberos 1.22 has support for this mode.

A KDC accessible over a UNIX domain socket is not very useful in itself: it is only available within the context of a single machine (or a single container, or pod, if UNIX domain sockets are shared across multiple containers). Otherwise, it is a fully featured KDC with its own quirks. And we can start looking at what could be improved based on the enhanced context locality we have achieved. For example, a KDB driver can see host-specific network interfaces and thus be able to react to requests such as host/<ip.ad.dr.ess>@LOCALKDC-REALM dynamically—something that a centrally-managed KDC would only do through statically registered service principal names (SPNs), which are a pain to update as machines move across networks.

Adding support for dynamic features means new code needs to be written. MIT Kerberos is written in C, so our choices are either to continue writing in C or to integrate with whatever new language we choose. Initially, we kept the local KDC database driver written in C and decided to build the infrastructure we need in Rust. The end goal is to have most bits written in Rust.

The local KDC database isn’t supposed to handle millions of principal entries, but even for millions of them, MIT Kerberos has a pretty good default database driver built on LMDB: klmdb. We wanted to get out of the data store business and instead focus on higher-level logic. Thus, we made the same change I made in Samba around 2003 for virtual file system modules: we introduced support for stackable KDB drivers. This is also a part of the MIT Kerberos 1.22 release: a KDB driver implementation can ask the KDC to load a different KDB driver and choose to delegate some requests to it. The local KDC driver is using klmdb for that purpose.

With the database handled for us by klmdb, we focused on the local KDC-specific logic. We wanted to dynamically discover user principals from the operating system so that administrators do not need to maintain separate databases for them. systemd provides a userdb API to query such information over a varlink interface (also available over a UNIX domain socket) in a structured way, using JSON format. Thus, the Kirmes project was born. Kirmes is a Rust data library backed by the userdb API. It handles varlink communication through the wonderful Zlink library and exposes both asynchronous and synchronous access to user and group information.

The local KDC database driver prototype used the Kirmes C API. We demonstrated it at FOSDEM 2025: a user lookup is done over varlink, and if a user is present on the system, their Kerberos key is then looked up in klmdb using a specially-formatted userdb:<username> principal. You still need to handle those keys somehow, but there is a way to avoid that: use RADIUS.

Pre-authentication

A bit of historical reference. In 2012, Red Hat collaborated with MIT to introduce a KDC-side implementation of RFC 6560 (the OTP pre-authentication mechanism; at that point implemented in a proprietary solution by the RSA corporation). This mechanism allowed the KDC to get a hint out of a KDB driver and ask a RADIUS server to authenticate the credentials provided by the Kerberos client. Unlike traditional Kerberos symmetric keys, in this case, the client is sending a plain-text credential over the Kerberos protocol, and this credential can be forwarded to the RADIUS server. The plain-text nature of the RADIUS credential requires the use of a secure communication channel, and a good part of RFC 6560 relies on Flexible Authentication Secure Tunneling (FAST, RFC6113), where a pre-existing Kerberos ticket is used to encrypt the content of that tunnel.

Since ~2013, FreeIPA has used this mechanism to provide multi-factor authentication mechanisms: HOTP/TOTP tokens, RADIUS proxying to remote servers, the OAuth2 device authorization grant flow, and FIDO2 tokens. The list of mechanisms can be extended, as long as the model fits into the somewhat constrained Kerberos exchange flow. FreeIPA handles all communication from the KDC side via a local UNIX domain socket-activated daemon, ipa-otpd, which performs a user principal lookup and then decides on the details of how that user will be authenticated.

For the local KDC case, we used a similar approach but wrote a simplified version, localkdc-pam-auth, which uses PAM to authenticate user credentials. It works well and allows for a drop-in replacement: once the local KDC is set up, users defined on the system will automatically be able to receive Kerberos tickets, with no need to change any passwords or migrate their credentials into the Kerberos KDC. All we need now is the business logic to guide the KDC to use the OTP pre-authentication mechanism so that our RADIUS ‘proxy’ (localkdc-pam-auth) gets activated. This logic is implemented and will be available in the first localkdc release soon.

API bindings

But back to the KDC side. As mentioned above, our goal was to write the local KDC database driver in a modern, safe language. Interfacing Rust with the MIT Kerberos KDC means building an interface that allows aligning code on both sides. This is what this blog is actually about (sorry for the long prelude…): how to make an MIT Kerberos KDB driver in Rust.

Today I published Kurbu5, a project that aims to provide these API bindings to Rust. The name is a transliteration of “krb5” into Mesopotamian cuneiform phonology: Kurbu-ḫamšat-qaqqadī—”The Blessed Five-Headed One”.

Creating API bindings is tedious work: there are many interfaces, each representing multiple functions and structures. MIT Kerberos has 12 interfaces which altogether expose roughly 117 methods that plugin authors implement, backed by around 70 supporting types (data structures passed into and out of those methods). It all sounds like a Tolkien tale: nine interfaces for core Kerberos functionality (checking password quality, mapping hostnames to Kerberos realms, mapping Kerberos principals to local accounts, selecting which credential cache to use, handling pre-authentication on both the client and server side, enforcing KDC policy, authorizing PKINIT certificates, and auditing events on the KDC side), the database backend interface, and two administrative interfaces. This is something that could be automated with agentic workflows—which I did to allow a parallel porting effort. The resulting agent instructions are useful artifacts in themselves: they show how to work when porting MIT Kerberos C code to Rust.

The result is split over several Rust crates to allow targeted reuse. The bulk of the code lives in three crates. The core Kerberos plugin crate (kurbu5-rs) is the largest at around 12,600 lines. The database backend crate (kurbu5-kdb-rs) follows at 5,600 lines, and the administration crate (kurbu5-kadm5-rs) at 3,100 lines. The remaining crates—the proc-macro derives and the raw FFI sys crates—are much smaller, with the sys crates being almost trivially thin (the KDB and kadm5 ones are under 40 lines each, since they mostly just re-export bindings from the main sys crate).

All crates are available on crates.io and share the same MIT license as the original MIT Kerberos.

kurbu5-sys — Raw FFI bindings to the MIT Kerberos libkrb5 and KDB plugin API
kurbu5-derive — Proc-macro derives for kurbu5-rs non-KDB plugin interfaces
kurbu5-rs — Safe, idiomatic Rust API for writing MIT Kerberos non-KDB plugin modules
kurbu5-kdb-sys — KDB plugin API re-export — thin wrapper over kurbu5-sys adding libkdb5 linkage
kurbu5-kdb-derive — Proc-macro derive for kurbu5-kdb-rs KDB driver plugins
kurbu5-kdb-rs — Safe, idiomatic Rust API for writing MIT Kerberos KDB driver plugins
kurbu5-kadm5-sys — KADM5 plugin API bindings — links libkadm5srv_mit and re-exports kurbu5-sys types
kurbu5-kadm5-derive — Proc-macro derives for kurbu5-kadm5-rs KADM5_AUTH and KADM5_HOOK plugin interfaces
kurbu5-kadm5-rs — Safe, idiomatic Rust API for writing MIT Kerberos KADM5_AUTH and KADM5_HOOK plugin modules

In the localkdc project, we use kurbu5 to build a KDB driver and provide our audit plugin. We also have an experimental re-implementation of the OTP pre-authentication mechanism, both client and KDC sides, that was used to test interoperability with MIT Kerberos versions. The core of the KDB driver is ~520 lines of heavily documented Rust code, mostly handling business logic.

ASN.1 for legacy apps: Synta

Mon, 23 Mar 2026 10:33:00 +0200

Pretty much everything I deal with requires parsing ASN.1 encodings. ASN.1 definitions published as part of internet RFCs: certificates are encoded using DER, LDAP exchanges use BER, Kerberos packets are using DER as well. ASN.1 use is a never ending source of security issues in pretty much all applications. Having safer ASN.1 processing is important to any application developer.

In FreeIPA we are using three separate ASN.1 libraries: pyasn1 and x509 (part of PyCA) for Python code, and asn1c code generator for C code. In fact, we use more: LDAP server plugins also use OpenLDAP’s lber library, while Kerberos KDC plugins also use internal MIT Kerberos parsers.

The PyCA developers noted in their State of OpenSSL statement:

[…] when pyca/cryptography migrated X.509 certificate parsing from OpenSSL to our own Rust code, we got a 10x performance improvement relative to OpenSSL 3 (n.b., some of this improvement is attributable to advantages in our own code, but much is explainable by the OpenSSL 3 regressions). Later, moving public key parsing to our own Rust code made end-to-end X.509 path validation 60% faster — just improving key loading led to a 60% end-to-end improvement, that’s how extreme the overhead of key parsing in OpenSSL was.

That’s 16x performance improvement over the OpenSSL 3. OpenSSL did improve their performance since then but it still pays an overhead for a very flexible design to allow loading cryptographic implementations from dynamic modules (providers). Enablement for externally-provided modules is essential to allow adding new primitives and support for government-enforced standards (such as FIPS 140) where implementations have to be validated in advance and code changes cannot come without expensive and slow re-validation process.

Nevertheless, in FreeIPA we focus on integrating with Linux distributions. Fedora, CentOS Stream, and RHEL enforce crypto consolidation rules, where all packaged applications must be using the same crypto primitives provided by the operating system. We can process metadata ourselves but all cryptographic operations still have to go through OpenSSL and NSS. And paying large performance costs during metadata processing would be hurting to infrastructure components such as FreeIPA.

FreeIPA is a large beast. Aside from its management component, written in Python, it has more than a dozen plugins for 389-ds LDAP server, plugins for MIT Kerberos KDC, plugins for Samba, and tight integration with SSSD, all written in C. Its default certificate authority software, Dogtag PKI, is written in Java and relies on own stack of Java and C dependencies. We are using PyCA’s x509 module for certificate processing in Python code but we cannot use it and underlying ASN.1 libraries in C as those libraries aren’t exposed to C applications or intentionally limited in their functionality to PKI-related tasks.

For the 2026-2028, I’m focusing on enabling FreeIPA to handle post-quantum cryptography (PQC), as a part of the Quantum-Resistant Cryptography in Practice (QARC) project. The project is funded by the European Union under the Horizon Europe framework programme (Grant Agreement No. 101225691) and supported by the European Cybersecurity Competence Centre. One of well publicized aspects of moving to PQC certificates is their sizes. The following table 5 is from Post-Quantum Cryptography for Engineers IETF draft summarizes it well:

PQ Security Level	Algorithm	Public key size (bytes)	Private key size (bytes)	Signature size(bytes)
Traditional	RSA2048	256	256	256
Traditional	ECDSA-P256	64	32	64
1	FN-DSA-512	897	1281	666
2	ML-DSA-44	1312	2560	2420
3	ML-DSA-65	1952	4032	3309
5	FN-DSA-1024	1793	2305	1280
5	ML-DSA-87	2592	4896	4627

Public keys for ML-DSA-65 certificates 7.6x bigger than RSA-2048 ones. You need to handle public keys in multiple situations: when performing certificates’ verification against known certificate authorities (CAs), when matching their properties for validation and identity derivation during authorization, when storing them. FreeIPA uses LDAP as a backend, so storing 7.6 times more data directly affects your scalability when number of users or machines (or Kerberos services) grow up. And since certificates are all ASN.1 encoded, I naturally wanted to establish a performance baseline to ASN.1 parsing.

Synta, ASN.1 library

I started with a small task: created a Rust library, synta, to decode and encode ASN.1 with the help of AI tooling. It quickly grew up to have its own ASN.1 schema parser and code generation tool. With those in place, I started generating more code, this time to process X.509 certificates, handle Kerberos packet structures, and so on. Throwing different tasks at Claude Code led to iterative improvements. Over couple months we progressed to a project with more than 60K lines of Rust code.

Language	files	blank	comment	code
Rust	207	9993	17492	67284
Markdown	52	5619	153	18059
Python	41	2383	2742	7679
C	17	852	889	4333
Bourne Shell	8	319	482	1640
C/C++ Header	4	319	1957	1138
TOML	20	196	97	896
YAML	1	20	46	561
make	4	166	256	493
CMake	3	36	25	150
JSON	6	0	0	38
diff	1	6	13	29
SUM	364	19909	24152	102300

I published some of the synta crates yesterday on crates.io, the whole project is available at codeberg.org/abbra/synta. In total, there are 11 crates, though only seven are published (and synta-python is also available at PyPI):

Crate	Lines (src/ only)
synta	10572
synta-derive	2549
synta-codegen	17578
synta-certificate	4549
synta-python	8953
synta-ffi	7843
synta-krb5	2765
synta-mtc	7876
synta-tools	707
synta-bench	0
synta-fuzz	3551

Benchmarking, fuzzer, and tools aren’t published. They only needed for development purposes.

Performance

The numbers below were obtained on Lenovo ThinkPad P1 Gen 5, 12th Gen Intel(R) Core(TM) i7-12800H, 64 GB RAM, on Fedora 42. This is pretty much a 3-4 years old hardware.

Benchmarking is what brought this project to life, let’s look at the numbers. When dealing with certificates, ASN.1 encoding can be parsed in different ways: you can visit every structure or stop at outer shells and only visit the remaining nested structures when you really need them. The former is “parse+fields” and the latter is “parse-only” in the following table that summarizes comparison between synta and various Rust crates (and OpenSSL/NSS which were accessible through their Rust FFI bindings):

Library	Parse-only	Parse+fields	vs synta (parse-only)	vs synta (parse+fields)
synta	0.48 µs	1.32 µs	—	—
cryptography-x509	1.45 µs	1.43 µs	3.0× slower	1.1× slower
x509-parser	2.01 µs	1.99 µs	4.2× slower	1.5× slower
x509-cert	3.16 µs	3.15 µs	6.6× slower	2.4× slower
NSS	7.90 µs	7.99 µs	16× slower	6.1× slower
rust-openssl	15.4 µs	15.1 µs	32× slower	11× slower
ossl	16.1 µs	15.8 µs	33× slower	12× slower

“Parse+fields” tests access every named field: serial number, issuer/subject DNs, signature algorithm OID, signature bytes, validity period, public key algorithm OID, public key bytes, and version. The “parse+fields” speedup is the fair end-to-end comparison: synta’s parse-only advantage is large because most fields are stored as zero-copy slices deferred until access, while other libraries must materialise all fields eagerly at parse time.

The dominant cost in X.509 parsing is Distinguished Name traversal: a certificate’s issuer and subject each contain a SEQUENCE OF SET OF SEQUENCE with per-attribute OID lookup. synta defers this entirely by storing the Name as a RawDer<'a> — a pointer+length into the original input with no decoding. cryptography-x509 takes a similar deferred approach. The nom-based and RustCrypto libraries decode Names eagerly. NSS goes further and formats them into C strings, which is the dominant fraction of its 16× parse overhead.

For benchmarking I used certificates from PyCA test vectors. There are few certificates with different properties, so we parse them multiple times and then average numbers:

Certificate	synta	cryptography-x509	x509-parser	x509-cert	NSS
cert_00 (NoPolicies)	1333.7 ns	1386.7 ns	1815.9 ns	2990.6 ns	7940.3 ns
cert_01 (SamePolicies-1)	1348.8 ns	1441.0 ns	2033.4 ns	3174.3 ns	7963.8 ns
cert_02 (SamePolicies-2)	1338.6 ns	1440.1 ns	2120.1 ns	3205.6 ns	8206.8 ns
cert_03 (anyPolicy)	1362.4 ns	1468.3 ns	2006.2 ns	3194.5 ns	7902.4 ns
cert_04 (AnyPolicyEE)	1232.9 ns	1424.7 ns	1968.6 ns	3168.1 ns	7913.1 ns
Average	1323 ns	1432 ns	1989 ns	3147 ns	7985 ns

The gap between synta (1.32 µs) and cryptography-x509 (1.43 µs) is tighter here than in parse-only (3.0×) because synta’s field access includes two format_dn() calls (~800 ns combined) that cryptography-x509 does for effectively free (its offsets were computed at parse time). Synta leads by ~8% overall.

Now, when parsing PQC certificates, an interesting thing happens. First, it is faster to parse ML-DSA than traditional certificates.

Certificate	synta	cryptography-x509	x509-parser	x509-cert	NSS
ML-DSA-44	1030.9 ns	1256.4 ns	1732.2 ns	2666.0 ns	7286.9 ns
ML-DSA-65	1124.9 ns	1237.5 ns	1690.5 ns	2664.2 ns	7222.1 ns
ML-DSA-87	1102.6 ns	1226.5 ns	1727.2 ns	2696.6 ns	7284.6 ns
Average	1086 ns	1240 ns	1717 ns	2675 ns	7265 ns

synta’s ML-DSA parse+fields (1.09 µs) is faster than its traditional parse+fields (1.32 µs) because ML-DSA test certificates have shorter Distinguished Names (one attribute each in issuer and subject vs multiple attributes in traditional certificates in the test above). The signature BIT STRING — which is 2,420–4,627 bytes for ML-DSA — is accessed as a zero-copy slice with no size-dependent cost.

Processing CA databases

Imaging your app needs to test whether the certificate presented by a client is known to you (e.g. belongs to a trusted CAs set). A library like OpenSSL looks at the client’s certificate, extracts identifiers of the certificate issuer, looks up whether such issuer is known in the CA database. That would require looking up properties of the certificates in the database. The fast we can do that, the better.

All those numbers in the previous section are for a single certificate being parsed millions of times. In a real app we often need to validate the certificate against a system-wide database of certificate authorities. The database used by Fedora and other Linux distributions comes from Firefox. It contains 180 self-signed root CA certificates for all public CAs with diverse key types (RSA 2048/4096, ECDSA P-256/P-384) and DN structures. The median cert by DER size is “Entrust.net Premium 2048 Secure Server CA” (1,070 bytes); the benchmark uses this cert for single-certificate and field-access sub-benchmarks to get stable results that are not sensitive to certificate-size outliers.

Another data I tried to benchmark against is 9,898 certificates from the Common CA Database (CCADB), covering the full multi-level hierarchy used by Mozilla, Chrome, Apple, and Microsoft:

Depth	Count	Description
0	919	Root CAs (self-signed)
1	6,627	Intermediates issued directly by roots
2	2,212	Two levels deep
3	137	Three levels deep
4	3	Four levels deep

Intermediate CA certificates tend to have more complex DNs and more extensions than the root CAs in the Mozilla store. The CCADB median cert is “Bayerische SSL-CA-2014-01” (10,432 bytes). These certificates from CCADB cover past 30 years of certificate issuance on the internet.

To see how those benchmarks would behave if CA roots database would be built with post quantum cryptography, I rebuilt the CCADB corpus as ML-DSA certificates. Nine CCADB certificates were skipped: OpenSSL’s x509 -x509toreq -copy_extensions copy step failed to convert them to CSR form, typically because those certs use non-standard DER encodings or critical extensions that the x509toreq pipeline cannot copy into a PKCS#10 request. (The failures are in OpenSSL’s cert→CSR conversion; synta parses all 9,898 original CCADB certs without error.) This leaves 9,889 of the original 9,898 certs in the synthetic database.

The median cert by DER size is “TrustCor Basic Secure Site (CA1)” (6,705 bytes). ML-DSA certs range from 5,530 B to 16,866 B; the distribution is shifted left relative to the CCADB RSA/ECDSA median (10,432 B) because the smallest CCADB certs (compact root CAs with few extensions) become the new median position after ML-DSA key replacement enlarges all certs uniformly.

Benchmark	Library	Dataset	Time	Throughput
`synta_parse_all`	synta	Mozilla (180 certs)	87.8 µs	2.0 M/sec
`nss_parse_all`	NSS	Mozilla (180 certs)	1.577 ms	114 K/sec
`openssl_parse_all`	rust-openssl	Mozilla (180 certs)	3.552 ms	50.7 K/sec
`ossl_parse_all`	ossl	Mozilla (180 certs)	3.617 ms	49.8 K/sec
`synta_parse_and_access`	synta	Mozilla (180 certs)	261 µs	690 K/sec
`synta_build_trust_chain`	synta	Mozilla (180 certs)	11.6 µs	—
`synta_parse_all`	synta	CCADB (9,898 certs)	5.10 ms	1.94 M/sec
`nss_parse_all`	NSS	CCADB (9,898 certs)	106 ms	93 K/sec
`openssl_parse_all`	rust-openssl	CCADB (9,898 certs)	203 ms	48.8 K/sec
`ossl_parse_all`	ossl	CCADB (9,898 certs)	214 ms	46.3 K/sec
`synta_parse_and_access`	synta	CCADB (9,898 certs)	16.1 ms	615 K/sec
`synta_parse_roots`	synta	CCADB (919 roots)	457.7 µs	2.01 M/sec
`synta_parse_intermediates`	synta	CCADB (8,979 intermediates)	4.735 ms	1.90 M/sec
`synta_build_dependency_tree`	synta	CCADB (9,898 certs)	559 µs	—
`synta_parse_all`	synta	ML-DSA synth (9,889 certs)	5.78 ms	1.71 M/sec
`nss_parse_all`	NSS	ML-DSA synth (9,889 certs)	103 ms	96.4 K/sec
`openssl_parse_all`	rust-openssl	ML-DSA synth (9,889 certs)	239 ms	41.4 K/sec
`ossl_parse_all`	ossl	ML-DSA synth (9,889 certs)	256 ms	38.6 K/sec
`synta_parse_and_access`	synta	ML-DSA synth (9,889 certs)	17.5 ms	566 K/sec
`synta_parse_roots`	synta	ML-DSA synth (919 roots)	463 µs	1.98 M/sec
`synta_parse_intermediates`	synta	ML-DSA synth (8,970 ints.)	5.10 ms	1.76 M/sec
`synta_build_dependency_tree`	synta	ML-DSA synth (9,889 certs)	549 µs	—

NSS is 18–21× slower than synta across all three datasets; rust-openssl is 40–41× slower and ossl is 41–44× slower. All three C-backed libraries successfully parse ML-DSA certificates (NSS 3.120+ and OpenSSL 3.4+ support ML-DSA natively). NSS’s absolute parse time is nearly identical across CCADB traditional certs (106 ms) and ML-DSA synthetic certs (103 ms) — confirming that NSS’s dominant cost is eager DN formatting at parse time, which depends on DN attribute count rather than the signature algorithm. The slightly lower relative slowdown for NSS on ML-DSA (18× vs 21×) is entirely because synta is slower on ML-DSA (5.78 ms vs 5.10 ms), not because NSS is faster.

synta’s throughput is consistent at ~1.7–2.0 M certs/sec across all three datasets, confirming linear O(n) scaling. Parse rate is slightly lower for the ML-DSA synthetic hierarchy (1.71 M/sec) than for the CCADB traditional hierarchy (1.94 M/sec) because the larger ML-DSA SubjectPublicKeyInfo and signature BIT STRING fields add bytes to the tag+length-header scan that synta performs at parse time. The intermediates-only sub-benchmark is slightly lower than roots-only in each dataset (1.76 M/sec vs 1.98 M/sec for ML-DSA; 1.90 M/sec vs 2.01 M/sec for CCADB) because intermediate CAs tend to have more complex DNs and extension lists.

Finally, individual property access for a pre-parsed certificate, single field read, no allocation unless noted:

Field	Mozilla (1,070 B)	CCADB (10,432 B)	ML-DSA (6,705 B)	Notes
`issuer_raw` / `subject_raw`	4.1 / 4.1 ns	4.2 / 4.1 ns	4.5 / 4.4 ns	Zero-copy slice
`public_key_bytes` / `signature_bytes`	4.1 / 4.1 ns	4.2 / 4.2 ns	4.6 / 4.4 ns	Zero-copy slice
`signature_algorithm` / `public_key_algorithm`	5.9 / 5.4 ns	5.9 / 5.5 ns	6.3 / 6.4 ns	OID → `&'static str`
`serial_number`	10.9 ns	6.8 ns	7.5 ns	Integer → i64, length-dependent
`validity`	180 ns	206 ns	231 ns	Two time-string allocations
`issuer_dn`	401 ns	224 ns	246 ns	`format_dn()` → `String`
`subject_dn`	404 ns	292 ns	324 ns	`format_dn()` → `String`

Zero-copy fields (issuer_raw, subject_raw, public_key_bytes, signature_bytes) cost ~4–5 ns — the price of reading a pointer and length from a struct field. The slightly higher cost for CCADB and ML-DSA fields vs Mozilla is within measurement noise.

identify_signature_algorithm() and identify_public_key_algorithm() match the OID component array against a static table and return &'static str — no allocation, no string formatting. The ~5–6 ns cost is a few comparisons and a pointer return.

serial_number cost depends on the integer’s byte length: the Entrust Mozilla cert carries a 16-byte serial number (parsed via SmallVec<[u8; 16]>), while the CCADB and ML-DSA synthetic medians have shorter serials. At 10.9, 6.8, and 7.5 ns respectively, all are negligible.

validity (~180–231 ns) allocates two strings: UTCTime and GeneralizedTime are formatted from their raw DER bytes into owned Strings. The two calls account for essentially all of the cost; the YYMMDDHHMMSSZ to RFC 3339 formatting is the dominant work.

format_dn() is the most variable field: it walks the Name DER bytes, decodes each SEQUENCE OF SET OF SEQUENCE, looks up each attribute OID by name, and formats the result into an owned String. The Mozilla cert’s issuer DN is more complex (multiple attributes, longer values: 401 ns) than the CCADB median (224 ns) or the ML-DSA synthetic median (246 ns). The ML-DSA synthetic median’s subject DN (324 ns) is slightly more expensive than the CCADB median (292 ns) because a different cert occupies the median position after key replacement. format_dn() cost is proportional to the DN’s attribute count and string lengths.

Why C Libraries Are Slower

CERT_NewTempCertificate (NSS) and OpenSSL’s d2i_X509 perform significantly more work per certificate than synta:

Eager DN formatting — NSS formats the issuer and subject Distinguished Names into internal C strings during CERT_NewTempCertificate, even when the caller never reads them. Distinguished Name formatting is the single most expensive operation in certificate parsing; doing it unconditionally at parse time accounts for roughly 80% of NSS’s total parse cost. OpenSSL decodes DN structure eagerly as well.
Arena and heap allocation — each NSS certificate allocates a PLArena block and copies the full DER buffer into it (copyDER = 1). OpenSSL allocates from the C heap. These allocations are additional work beyond decoding.
Library state and locking — NSS acquires internal locks on every CERT_NewTempCertificate call to update the certificate cache, even when the resulting certificate is marked as temporary. This serialises concurrent parsing in multi-threaded applications.
FFI boundary costs — the rust-openssl and ossl measurements include the overhead of crossing from Rust into the C library via extern "C" calls and pointer marshalling.

synta defers all of (1): issuer and subject are stored as RawDer<'a> (borrowed byte spans) and decoded only when the caller calls format_dn(). There is no locking, no arena, and no FFI boundary.

In these tests I also found out that PyCA’s cryptography-x509 doesn’t have optimizations for multiple accesses to the same fields. It is typically not a problem if you are just loading a certificate and use it once. If you have to return back to it multiple times, that becomes visible and hurts your performance. So I submitted a pull request to apply some of the optimizations I found with synta. The pull request had to be split into smaller ones and few of them were already merged, so performance to access issuer, subject, and public key in certificates and to some attributes in CSRs was improved 100x. The rest waits for improvements in PyO3 to save some of memory use.

FreeIPA local tests and FOSDEM demos

Fri, 14 Feb 2025 13:15:00 +0200

FOSDEM 2025 is behind us. We ran Identity and Access Management devroom at FOSDEM. At the devroom, my team did few talks and demos about FreeIPA and Kerberos. While preparing to those talks, we tried to create demonstrations that could be repeated by others as well. First, this was an attempt to help ourselves, as we need to communicate our advances to others in the teams. Then we started to look at how to show our progress to folks outside of the development groups.

We iterated over our tools and finally ended up with something that is based on what we use in upstream CIs: we use podman containers to run what ends up being ephemeral VMs hosting the software. This doesn’t give ability to handle all possible scenarios. It is not a way to run actual production environments as well. Yet, it allows us a quick reuse and share:

descriptive definition of the deployment configuration
standard tooling to provision the configuration as containers with podman-compose
use of Ansible playbooks to run repeatable actions against the hosts, with inventory taken from the podman-compose integration

The tool, ipalab-config, quickly became flexible enough to be used in multiple scenarios. It powers ansible-freeipa’s own upstream CI, we aim to reuse it for new FreeIPA Web UI development and for the FreeIPA workshop.

For the demos at FOSDEM IAM devroom we put a separate repository that has all the scenarios and even recording files to reproduce the demos: freeipa-local-tests. You can try yourself how local authentication hub or IPA-IPA trust or IPA-IPA migration do work.

This project demonstrates how complex multi-system FreeIPA deployments can be tested locally or in your CI/CD. The test environment is built with the help of podman and orchestrated with ipalab-config and podman-compose tools. FreeIPA environment is deployed with the help of ansible-freeipa. Upstream, we run these tests in Github Actions as well.

Demo labs

Following configurations provided as ‘labs’ that can be reproduced using ipalab-config tool and the configurations from this project:

minimal deployment, consisting of a FreeIPA server and a FreeIPA client enrolled into it.
local KDC, consisting of two standalone machines, not enrolled into any domain. Each machine runs its own Kerberos KDC exposed to local applications over UNIX domain socket, with socket activation handled by systemd. See “localkdc - local authentication hub” talk at FOSDEM 2025. This is currently a work in progress.
FreeIPA deployment migration, demonstrating how IPA data can be migrated between separate test and production deployments. See “FreeIPA-to-FreeIPA Migration: Current Capabilities and Use Cases” talk at FOSDEM 2025.
FreeIPA trust, demonstrating how two separate IPA deployments can be set up to trust each other. See “Building Cross-Domain Trust Between FreeIPA Deployments” talk at FOSDEM 2025. This is currently a work in progress.

Demo recordings

Some of the demo labs have automated recording of the operations that could be performed on them. Video recording is built upon excellent VHS tool. A pre-built version for Fedora is provided in COPR abbra/vhs. This build also includes a fix from the upstream PR#551.

Minimal deployment demo

This demo recording includes a minimal use of FreeIPA command line:

an administrator logs into a client system over SSH using a password
Kerberos ticket is obtained automatically by the SSSD
IPA command line tool can authenticate to IPA server using Kerberos

Local KDC demo

The local KDC demo is more evolved:

a user logs into their own machine over SSH using a password
Kerberos ticket is obtained automatically by the SSSD from the local KDC which is activated on demand
User then uses a Kerberos ticket to authenticate to SUDO and obtain root privileges
The user also uses the Kerberos ticket to authenticate to Samba server running locally
Finally, the user authenticates with Kerberos IAKerb extension to a remotely running Samba server, removing completely a need for NTLM authentication protocol

IPA to IPA trust demo

This is a minimalistic demo of how users and groups from one IPA environment can be resolved in the other IPA environment. There is a trust agreement established between both IPA environments, similarly how IPA can establish a forest level trust with Active Directory.

Local authentication hub

Sun, 09 Feb 2025 13:00:00 +0200

FOSDEM 2025 is just behind us and it was a great event. I had a chance to talk about the local authentication hub project. The talk was well received and I got a lot of questions about the project. We ran Identity and Access Management devroom for the second time in row and it was a great success. I had two talks at the IAM devroom, both were process reports on the activity we have announced at FOSDEM 2024. Now that both recordings of the both talks published, I can share articles which go into more details.

Local authentication hub

Our FOSDEM talk is “localkdc - a general local authentication hub”. You can watch it and come back here for more details.

But before going into details, let me provide a bit of a background. It is 2025 now and we should go almost three decades back (ugh!).

History dive

Authentication on Linux systems is interwoven with the identity of the users. Once a user logged in, a process is running under a certain POSIX account identity. Many applications validate the presence of the account prior to the authentication itself. For example, the OpenSSH server does check the POSIX account and its properties and if the user was not found, will intentionally corrupt the password passed to the PAM authentication stack request. An authentication request will fail but the attempt will be recorded in the system journal.

This joint operation between authentication and identification sources in Linux makes it important to maintain a coherent information state. No wonder that in corporate environments it is often handled centrally: user and group identities stored at a central server and sourced from that one by a local software, such as SSSD. In order to consume these POSIX users and groups, SSSD needs to be registered with the centralized authority or, in other words, enrolled into the domain. Domain enrollment allows not only identity and authentication of users: both the central server and the enrolled client machine can mutually authenticate each other and be sure they talk to the right authority when authenticating the user.

FreeIPA provides a stable mechanism for building a centralized domain management system. Each user account has POSIX attributes associated with it and each user account is represented by the Kerberos principal. Kerberos authentication can be used to transfer the authentication state across multiple services and provides a chance for services to discover user identity information beyond POSIX. It also makes strong linking between the POSIX level identity and authentication structure possible: for example, a Kerberos service may introspect a Kerberos ticket presented by a user’s client application to see how this user was authenticated originally: with a password or some specific passwordless mechanism. Or, perhaps, that a client application performs operations on behalf of the user after claiming it was authenticated using a different (non-Kerberos) authentication.

Local user accounts’ use lacks this experience. Each individual service needs to reauthenticate a user again and again. Local system login: authenticate. Elevating privileges through SUDO? Authenticate again, if not explicitly configured otherwise. Details of the user session state, like how long this particular session is active, is not checked by the applications, making it also harder to limit access. There is no information on how this user was authenticated. Finally, overall user experience between local (standalone) authentication and domain-enrolled one differs, making it harder to adjust and educate users.

Local authentication is also typically password-based. This is not a bad thing in itself but depending on applications and protocols, worse choices could be made, security-wise. For example, contemporary SMB 3.11 protocol is quite secure if authenticated using Kerberos. For non-Kerberos usage, however, it is left to rely on NTLM authentication protocol which requires use of RC4 stream cipher. There are multiple attacks known to break RC4-based encryption, yet it is still used in majority of non-domain joined communications using SMB protocol simply because there was no (so far) alternative. To be correct, there was always an alternative, use of Kerberos protocol, but setting it up for individual isolated systems wasn’t practical.

The Kerberos protocol assumes the use of three different parties: a client, a service, and a key distribution center (KDC). In corporate environments a KDC is part of the domain controller system, a client and a service are both domain members, computers are enrolled in the domain. The client authenticates to KDC and obtains a Kerberos ticket granting ticket (TGT). It then requests a service ticket from the KDC by presenting its TGT and then presents this service ticket to the service. The service application, on its side, is able to decrypt the service ticket presented by the client and authenticate the request.

In the late 2000s Apple realised that for individual computers a number of user accounts is typically small and a KDC can be run as a service on the individual computer itself. When both the client and server are on the same computer, this works beautifully. The only problem is that when a user needs to authenticate to a different computer’s service, the client cannot reach the KDC hosted on the other computer because it is not exposed to the network directly. Luckily, MIT Kerberos folks already thought about this problem a decade prior to that: in 1997 a first idea was published for a Kerberos extension that allowed to tunnel Kerberos requests over a different application protocol. This specification became later known as “Initial and Pass Through Authentication Using Kerberos V5 and the GSS-API” (IAKerb). An initial implementation for MIT Kerberos was done in 2009/2010 while Apple introduced it in 2007 to enable remote access to your own Mac across the internet. It came in MacOS X 10.5 as a “Back to My Mac” feature and even got specified in RFC 6281, only to be retired from MacOS in 2019.

Modern days

In the 2020s Microsoft continued to work on NTLM removal. In 2023 they announced that all Windows systems will have a local KDC as their local authentication source, accessible externally via selected applications through the IAKerb mechanism. By the end of 2024, we have only seen demos published by Microsoft engineers at various events but this is a promising path forward. Presence of the local KDC in Windows raises an interoperability requirement: Linux systems will have to handle access to Windows machines in a standalone environment over SMB protocol. Authentication is currently done with NTLM, it will eventually be removed, thus we need to support the IAKerb protocol extension.

The NTLM removal for Linux systems requires several changes. First, the Samba server will need to learn how to accept authentication with the IAKerb protocol extension. Then, Samba client code needs to be able to establish a client connection and advertise IAKerb protocol extension. For kernel level access, the SMB filesystem driver needs to learn how to use IAKerb as well, this will also need to be implemented in the user space cifs-utils package. Finally, to be able to use the same feature in a pure Linux environment, we need to be able to deploy Kerberos KDC locally and do it in an easy manner on each machine.

This is where we had an idea. If we are going to have a local KDC running on each system, maybe we should use it to handle all authentication and not just for the NTLM removal? This way we can make both the local and domain-enrolled user experience the same and provide access locally to a whole set of authentication methods we support for FreeIPA: passwords, smartcards, one-time passwords and remote RADIUS server authentication, use of FIDO2 tokens, and authentication against an external OAuth2 Identity Provider using a device authorization grant flow.

How “local” a local KDC should be?

On standalone systems it is often not desirable to run daemons continuously. Also, it is not desirable to expose these services to the connected network if they really don’t need to be exposed. A common approach to solve this problem is by providing a local inter-process communication (IPC) mechanism to communicate with the server components. We chose to expose a local KDC via UNIX domain sockets. A UNIX domain socket is a well-known mechanism and has known security properties. With the help of a systemd feature called socket activation, we also can start local KDC on demand, when a Kerberos client connects over the UNIX domain socket. Since on local systems actual authentication requests don’t happen often, this helps to reduce memory and CPU usage in the long run.

If a local KDC is only accessible over a UNIX domain socket, remote applications could not get access to it directly. This means they would need to have help from a server application that can utilize the IAKerb mechanism to pass-through the communication between a client and the KDC. It would enable us to authenticate as a local user remotely from a different machine. Due to how the IAKerb mechanism is designed and integrated into GSS-API, this only allows password-based authentication. Anything that requires passwordless methods cannot obtain initial Kerberos authentication over IAKerb, at least at this point.

Here is a small demo on Fedora, using our localkdc tool to start a local KDC, obtain a Kerberos ticket upon login. The tickets can then be used effortlessly to authenticate to local services such as SUDO or Samba. For remote access we rely on Samba support for IAKerb and authenticate with GSSAPI but local smbclient uses a password first to obtain the initial ticket over IAKerb. This is purely a limitation of the current patches we have to Samba.

Make a pause here and think about the implications. We have an initial Kerberos ticket from the local system. The Kerberos ticket embeds details of how this authentication happened. We might have used a password to authenticate, or a smartcard. Or any other supported pre-authentication methods. We could reuse the same methods FreeIPA already provides in the centralized environment.

The Kerberos ticket also can contain details about the user session, including up to date group membership. It does not currently have that in the local KDC case but we aim to fix that. This ticket can be used to authenticate to any GSS-API or Kerberos-aware service on this machine. If a remote machine accepts Kerberos, it theoretically could accept a ticket presented by a client application running on the local machine as well. Only, to do that it needs to be able to communicate with our local KDC and it couldn’t access it.

Trust management

Luckily, a local KDC deployment is a full-featured Kerberos realm and thus can establish cross-realm agreements with other Kerberos realms. If two “local” KDC realms have trust agreements between each other, they can issue cross-realm Kerberos tickets which applications can present over IAKerb to the remote “local” KDC. Then a Kerberos ticket to a service running on the target system can be requested and issued by the system’s local KDC.

Thus, we can achieve passwordless authentication locally on Linux systems and have the ability to establish peer to peer agreements across multiple systems, to allow authentication requests to flow and operate on commonly agreed credentials. A problem now moves to the management area: how to manage these peer to peer agreements and permissions in an easy way?

Systemd User/Group API support

MIT Kerberos KDC implementation provides a flexible way to handle Kerberos principals’ information. A database backend (KDB) implementation can be dynamically loaded and replaced. This is already used by both FreeIPA and Samba AD to integrate MIT Kerberos KDC with their own database backends based on different LDAP server implementations. For a local KDC use case running a full-featured LDAP server is not required nor intended. However, it would be great if different applications could expose parts of the data needed by the KDB interfaces and cooperate together. Then a single KDB driver implementation could be used to streamline and provide uniform implementation of Kerberos-specific details in a local KDC.

One of the promising interfaces to achieve that is the User/Group record lookup API via varlink from systemd. Varlink allows applications to register themselves and listen on UNIX domain sockets for communication similar to D-Bus but with much less implementation overhead. The User/Group API technically also allows to merge data coming from different sources when an application inquires the information. “Technically”, because io.systemd.Multiplexer API endpoint currently does not support merging non-overlapping data representing the same account from multiple sources. Once it would become possible, we could combine the data dynamically and may interact with users on demand when corresponding requsts come in. Or we can implement our own blending service.

Blending data requests from multiple sources within MIT KDC needs a specialized KDB driver. We certainly don’t want this driver to duplicate the code from other drivers, so making these drivers stackable would be a good option. Support for one level of stacking has been merged to MIT Kerberos through a quickly processed pull request and will be available in the next MIT Kerberos release. This allows us to have a single KDB driver that loads other drivers specialized in storing Kerberos principals and processing additional information like MS-PAC structure or applying additional authorization details.

Establishing trusts

If Alice and Bob are in the same network and want to exchange some files, they could do this using SMB and Samba. But that Alice can authenticate on Bob’s machine, they would need to establish a Kerberos cross realm trust. With the current tooling this is a complex task. For users we need to make this more accessible. We want to allow users to request trust on demand and validate these requests interactively. We also want to allow trust to be present for a limited timeframe, automatically expiring or manually removed.

If we have a Kerberos principal lookup on demand through a curated varlink API endpoint, we also can have a user-facing service to initiate establishing the trust between two machines on demand. Imagine a user trying to access SMB share on one desktop system that triggers a pop-up to establish trust relationship with a corresponding local KDC on the remote desktop system. Both owners of the systems would be able to communicate out of band that provided information is correct and can be trusted. Once it is done, we can return back the details of the specific Kerberos principal that represents this trust relationship. We can limit lifetime of this agreement so that it would disappear automatically in one hour or a day, or a week.

Current state of local authentication hub

We started with two individual implementation paths early in 2024:

support IAKerb in MIT Kerberos and Samba
enable MIT Kerberos to be used locally without network exposure

MIT Kerberos did have support for IAKerb protocol extension for more than a decade but since Microsoft introduced some changes to the protocol, those changes needed to be integrated as well. This was completed during summer 2024, though no upstream release is available yet. MIT Kerberos typically releases new versions yearly in January so we hope to get some updates early 2025.

Samba integration with IAKerb is currently under implementation. Originally, Microsoft was planning to release Windows 11 and Windows Server 2025 with IAKerb support enabled during autumn 2024. However, the Windows engineering team faced some issues and IAKerb is still not enabled in the Windows Server 2025 and Windows 11 releases. We are looking forward to getting access to Windows builds that enable IAKerb support to ensure interoperability before merging Samba changes upstream. We also need to complete the Samba implementation to properly support locally-issued Kerberos tickets and not only do acquisition of the ticket based on the password.

Meanwhile, our cooperation with MIT Kerberos development team led to advancements in the local KDC support. The MIT Kerberos KDC can now be run over a UNIX domain socket. Also on systemd-enabled systems we allow socket activation, transforming local KDC into an on-demand service. We will continue our work on a dynamic database for a local KDC, to allow on-demand combination of resources from multiple authoritative local sources (Samba, FreeIPA, SSSD, local KDC, future dynamic trust application).

For experiments and ease of deployments, a new configuration tool was developed, localkdc. The tool is available at localkdc and COPR repository can be used to try the whole solution on Fedora.

If you want to get that test tried in a simple setup, you might be interested in a tool that we developed initially for FreeIPA: FreeIPA local tests. This tool allows to provision and run a complex test environment in podman containers. The video of the local KDC usage was actually generated automatically by the scripts from https://github.com/abbra/freeipa-local-tests/tree/main/ipalab-config/localkdc.

IPA-IPA trust progress report

Fri, 31 May 2024 12:30:00 +0300

FreeIPA and SSSD teams are working to enable IPA deployments to trust each other. This report outlines the progress we have so far.

Trust across enterprise domains

FreeIPA implements an enterprise domain management: systems enrolled into domain managed centrally and access resources available through the domain controllers. These resources include information about users and groups, machines, Kerberos services and different rules that can bind them. FreeIPA supports a trust to Active Directory forest by posing as a separate Active Directory forest with a single Active Directory domain (a forest root).

The approach helps to integrate with a majority of enterprise deployments. New deployments might not include Active Directory, though. Instead, they tend to be self-sufficient or integrate with externally managed data sources, often without retaining compatibility with traditional POSIX or Active Directory deployments. For example, cloud deployments might rely on OAuth2 identity providers or federate to social services.

FreeIPA can also integrate with OAuth2-based identity providers by bridging authentication over to them. Users would be native IPA users; IPA policies and rules to access IPA resources would apply to them but authentication and permission to issue a Kerberos ticket to the users would be delegated to an external IdP.

One useful approach is to have a single IPA environment that holds all users and groups while individual machines all placed in separate IPA deployments. These deployments would need to trust the environment where users defined. In this document we’d call such deployment an IPA-IPA trust.

IPA-IPA trust would look similar to the existing trust to Active Directory. A core technology is Kerberos: each deployment is a separate Kerberos realm; a trust on Kerberos level means cross-realm ticket granting tickets (TGT) issued by the realms’ KDCs. These cross-realm TGTs allow clients from one realm to request tickets for services located in the other realm.

Requesting Kerberos tickets is a base feature needed to authenticate across the trust. For proper relationship, identities of the authenticated objects (Kerberos principals and services) also need to be resolved. In POSIX environments these identities have to be associated with POSIX users to be able to store data on disk or to run processes with individual login sessions. The process requires not only authentication but also the ability to query information from a trusted domain’s domain controller.

In FreeIPA deployments, user and group information from the domain controllers is retireved by SSSD. When trust to Active Directory forest created, SSSD on IPA domain controllers is able to use a special entry in the trusted domain (trusted domain object, TDO) to authenticate against trusted domain’s domain controllers. SSSD knows how LDAP schema and directory information tree (DIT) on the Active Directory side are organized and can handle user and group object mapping for us.

If SSSD is able to resolve users and groups in the same way as with Active Directory trusts, this means we can reuse existing support for trusted Active Directory users and groups in IPA commands and keep the same user experience when adding HBAC or SUDO rules, managing IPA deployment, and so on. This would also mean we already would have a comprehensive test suite to validate IPA-IPA trust functionality.

Kerberos trust infrastructure

We first focused on making Kerberos infrastructure work when both realms represented by separate FreeIPA deployments. For past decade Microsoft, Samba Team, FreeIPA team, MIT Kerberos and Heimdal projects worked together to ensure there is a solid secure foundation for interoperability between non-POSIX (Active Directory) and POSIX deployments. In Active Directory world some crucial POSIX information is not available. When this information required on POSIX side, it is either synthesized or mapped onto existing objects by FreeIPA and Samba AD.

Kerberos protocol originally didn’t include any means to securely tie Kerberos principals to identities. For example, one was able to create a Windows machine named root in Active Directory and then use its machine account to login to a different machine as root POSIX user. This attack approach not possible anymore if all sides (Kerberos KDCs of the involved realms and a target machine) use authorization data in Kerberos tickets to exchange identity details of the original Kerberos principal’s operating system object. In this case it would mean that a Kerberos principal root would have its machine account information associated with the Kerberos ticket and the target machine would be able to inquire and see that the real Kerberos principal would be host/root.some.domain and that this principal’s operating system object type is a machine account.

The extensions also give ability to communicate group membership details and more information about the account. FreeIPA domain controllers already use these details to check and map properties of trusted Active Directory domains’ objects. For native IPA objects we also generate this information. IPA-IPA trust can rely on these details as well.

This observation allowed us to start with an experiment. Create a trusted domain object that represents a trusted IPA domain and see what does not work and needs a fix to make IPA-IPA trust a reality. We originally thought to replicate Active Directory infrastructure on IPA domain controller side so that existing ipa trust-add --type=ad .. command can be used to establish trust.

These changes are part of the tree I maintain at my github WIP tree.

Identity information retrieval

A part of Active Directory infrastructure that we implemented first was Global Catalog. Global Catalog is a separate, read-only, LDAP instance that contains a subset of the information about all Active Directory objects in the forest. Since FreeIPA LDAP tree uses different LDAP schema and structure than Active Directory LDAP tree, SSSD’s Active Directory identity provider cannot use normal FreeIPA LDAP tree as “yet another Active Directory domain’s LDAP tree”. We imagined that adding support for the Global Catalog (which SSSD is able to query) would make it possible to query IPA users and groups as “Active Directory” users and groups.

We soon found out two issues with this approach. It is not possible to force SSSD to use Global Catalog exclusively. SSSD uses the primary LDAP tree for retrieving users’ information and only switches to Global Catalog for retrieving groups information. Adding support for Global Catalog would not help: existing Active Directory domain identity provider in SSSD would not be able to work against a trusted IPA domain anyway. A proper support for trusted IPA domain in SSSD is needed.

Here we stumbled upon our next issue. SSSD implementation for IPA domain provider includes ability to handle all trusts to Active Directory domains that IPA deployment has. Trusted domains (subdomains, in SSSD speak), in theory, can come from any trust type. Since IPA has only a trust to Active Directory, SSSD implementation did not have an infrastructure to handle different types of subdomains. Instead, a single subdomain type is assumed: any subdomain discovered is an Active Directory domain. Even if we present IPA-IPA trust as an Active Directory trust, SSSD would still attempt to use Active Directory-specific LDAP schema and directory information tree arrangement when making LDAP queries. IPA users and groups from trusted domains aren’t found because LDAP server would not match that LDAP query to the IPA LDAP tree and schema.

SSSD needed a complete refactoring of the subdomains’ infrastructure to allow more than one subdomain type to be present. This work is still in progress. Justin Stephenson from the SSSD team tracks his progress in this WIP pull request.

Demo

With the refactoring work currently being done by Justin, it is possible to combine two parts: Global Catalog-based FreeIPA changes from my WIP source tree and SSSD changes from Justin’s WIP source tree.

We have these modifications to FreeIPA and SSSD prebuilt in Fedora COPR repository abbra/wip-ipa-trust. The packages there work in a specific environment where two FreeIPA deployments named ipa1.test and ipa2.test because we have so far no way to differentiate trusted IPA deployment from a trusted Active Directory deployment, thus SSSD hardcodes the names of the IPA domains to trust.

A trust between ipa1.test and ipa2.test can be established using standard ipa trust-add command. This approach allowed us to investigate what is working or not after the trust link is created. Both FreeIPA and SSSD teams can also work on validating that specific IPA functionality available to handle trusted Active Directory users and groups is also working with IPA-IPA trust.

I recorded a small demo on Fedora 40:

Two administrators establish trust between their environments and then allow use of resources across the trust. Since the exact details how trust is established will change, I omitted that step and show pre-created trust entries along with the ID range details. Only ‘Active Directory trust with POSIX attributes’ ID range type supported for now. It should not be a problem though because IPA deployments do provide POSIX information.

To allow access to IPA API, an administrator of the trusting domain needs to create ID overrides for user from the trusted domain. Once the overrides are in place, a user from the trusted domain can issue IPA API calls. In the demo we do ipa ping call and show that underlying Kerberos service tickets were requested and issued.

Future work

Now that we validated the approach, the road splits into two:

figure out how to get trust established
fix remaining bugs to make ‘day two’ operations possible.

Establishing trust is an interesting problem. Or two. For pure IPA-IPA trust we don’t need to provision the same infrastructure as with Active Directory trust: we don’t need to run Samba on each domain controller to be able to create trusted domain objects and respond to DCE RPC requests because we don’t use DCE RPC in SSSD.

On top of that, FreeIPA has extensive set of passwordless authentication methods which require use of a FAST channel when obtaining an initial Kerberos TGT. When trust to Active Directory created, we first authenticate to AD DC to get a Kerberos TGT of the Active Directory’s administrator. Windows does not have support for passwordless authentication methods through Kerberos except for PKINIT (smartcards), so in most cases we deal with password authentication. If an IPA administrative account is using passwordless method to authenticate, we first have to create a FAST channel or a trusted-to-be domain controller will not even expose supported pre-authentication methods. Any existing Kerberos ticket could be used to create the FAST channel but it must be from the trusted domain and we don’t have trust yet. In our own domain we can use a machine account (on a particular machine) or an Anonymous PKINIT to create the FAST channel. For a trusted-to-be domain we have to trust the CA who issued PKINIT certificates to their KDC. And KDCs might be accessible through a HTTPS proxy (MS-KKDCP protocol). This means some kind of pre-trust exchange needed to make sure we can authenticate against their KDCs successfully.

Also, we have no idea what exact pre-authentication methods will be available to the user. Practically, we only care about being able to authenticate to the trusted domain’s IPA API and then use that connection to set things up. For this it would be enough to authenticate to the trusted-to-be domain’s IPA API endpoint, save received cookies and use them. This method wouldn’t support passwordless authentication either, except an OTP method. FIDO2 and external IdP (OAuth2) authentication aren’t yet directly supported in IPA web UI and IPA API end-points.

With all those complications, we look at enabling OAuth2-based authentication and authorization endpoints in IPA first. We then can use them to request a token from a trusted-to-be domain’s OAuth2 endpoint and use it to establish the trust. A remote domain controller will handle the authentication of their users and will be able to supply all required details: trusted CA chains, ID ranges, scopes for authentication/ID mapping, etc.

And the OAuth2 endpoint will allow us to handle own passwordless authentication methods for IPA Web UI and applications on the enrolled systems like Cockpit which will be able to delegate to IPA for the user login.

CentOS Connect 2024 report

Tue, 13 Feb 2024 09:25:00 +0200

February 1st-4th I participated in two events in Brussels: CentOS Connect and FOSDEM. FOSDEM is getting closer to its quarter a century anniversary next year. With 67 mini-conferences and another 30 events around it, it is considered one of the largest conferences in Europe, any topic included. This report is about CentOS Connect.

CentOS Connect

CentOS Connect was a two-day event preceding FOSDEM. Organized by the CentOS project, it brought together contributors from multiple projects around CentOS Stream upstreams (such as Fedora Project) and downstreams (from Red Hat Enterprise Linux, AlmaLinux, Rocky Linux, and others), long-time users and community members. A lot of talks were given on both days and several special interest groups (SIGs) ran their workshops on the first day.

CentOS Integration SIG

I have attended a workshop organized by the CentOS Integration SIG. The focus of this group is to define and run integration tests around CentOS Stream delivery to the public. While CentOS Stream composes are built using the messages on how RHEL next minor version is composed after being thoroughly tested by Red Hat’s quality engineering group, there is a value in testing CentOS Stream composes by other community members independently as their use cases might differ. Right now the set of tests is defined as a t_functional suite maintained by the CentOS Core SIG but this suite is not enough for the complex scenarios.

The Integration SIG is looking at improving the situation. Two projects were present and interested in this work: RDO (RPM-based distribution of OpenStack) and RHEL Identity Management. Both teams have to deal with coordinated package deliveries across multiple components and both need to get multi-host deployments tested. RDO can be treated as a product built on top of CentOS Stream where its own RDO deliveries are installed on top of CentOS Stream images. RHEL Identity Management (IdM), on the other hand, is a part of RHEL. It is famously known as a ‘canary in the RHEL mine’ (thanks to Stephen Gallagher’s being poetic a decade ago): if anything is broken in RHEL, chances are it will be visible in RHEL IdM testing. With RHEL IdM integration scope expanding to provide passwordless desktop login support, this becomes even more important and also requires multi-host testing.

CentOS Integration SIG’s proposal to address these requirements is interesting. Since CentOS Stream compose event is independent of the tests, a proposal by Aleksandra Fedorova and Adam Samalik is to treat the compose event in a way similar to a normal development workflow. Producing CentOS Stream compose would generate a merge request of a metadata associated with it to a particular git repository. This merge request then can be reacted to by the various bots and individuals. Since CentOS Stream compose is public, it is already accessible to third-party developers to run their tests and report back the results. This way qualification of the compose promotion to CentOS Stream mirrors can be affected by all parties and will help to keep already published composes in a non-conflicting state.

Since most of the projects involved already have their own continuous integration systems, adding another trigger for those would be trivial. For RHEL IdM and its upstream components (FreeIPA, SSSD, 389-ds, Dogtag PKI, etc.) it would also be possible finally to react to changes to CentOS Stream well ahead of time too. In the past this was complicated by a relative turbulence in the CentOS Stream composes early in RHEL next minor release development time when everybody updates their code and stabilization needs a lot of coordination. I am looking forward to Aleksandra’s proposal to land in the Integration SIG’s materials soon.

Update: Aleksandra has published her proposal at the Fedora Discussion board

CentOS Stream and EPEL

Two talks, by Troy Dawson and Carl George, described the current state of EPEL repository and its future interactions with CentOS 10 Stream. Troy’s discussion was fun, as usual: a lot of statistics obtained from the DNF repository trackers shows that old habits are hard to get rid off and moving forward is a struggle to many users despite security and supportability challenges with older software. Both CentOS 7 and CentOS 8 Stream end of life dates are coming in 2024, adding enough pressure themselves to that.

There are interesting statistics from the EPEL evolution. In August 2023 at Flock to Fedora conference Troy and Carl pointed out that 727 packagers maintained 7298 packages in EPEL 7, 489 packagers handled 4968 packages in EPEL 8, and 396 packagers were handling 5985 packages in EPEL 9. Half a year later we have 7874 packages in EPEL 7, 5108 packages in EPEL 8, and 6868 packages in EPEL 9. Pace seems to pick up EPEL 9 with upcoming end of life dates for older versions. Since soon only EPEL 9 and EPEL 10 would be actively built, there are probably more packagers that would be active in them as well.

EPEL 10 is coming soon. It will bring a slight difference in package suffixes and overall reduction of complexity to packagers. It is great to see how close EPEL work tracks to ELN activities in Fedora. One thing worth noting is that every Fedora package maintainer is a potential EPEL maintainer as well because EPEL reuses Fedora infrastructure for package maintenance. Even if someone is not maintaining EPEL branches on their Fedora packages (I am not doing that myself – my packages are mostly in RHEL), it allows easy jump-in and collaboration. After all, if packages aren’t in RHEL but are in Fedora, their EPEL presence is just one git branch (and Fedora infrastructure ticket) away.

20 years of CentOS project

First day of the CentOS Connect event ended with a party to celebrate 20 years of the CentOS Project. First release (“CentOS version 2”) went out on May 14th, 2004 but since CentOS Connect is the closest big event organized by the project this year, getting a lot of contributors together to celebrate this anniversary seemed appropriate. A huge cake was presented, so big that it wasn’t possible to consume it completely during the party. It was delicious (like a lot of Belgian chocolate!) and next day coffee breaks allowed me to enjoy it a lot.

FreeIPA and CentOS project infrastructure

My own talk was originally planned to gather feedback from all projects which build on top of CentOS as they use a very similar infrastructure. CentOS Project infrastructure is shared with Fedora Project which is built around FreeIPA as a backend and Noggin as a user management frontend. I asked in advance for some feedback from Fedora, CentOS, AlmaLinux, and RockyLinux infrastructure teams and haven’t gotten that much which prompted my own investigation. It is not an easy job since most organizations aren’t really interested in telling the world details of their core infrastructure. Hope was that I’d be able to see real world usage and maybe take some lessons from it back to the development teams.

While working on my talk, we also experienced an outage in Fedora infrastructure related to the upgrade of the FreeIPA setup used there. My team has been helping Fedora infrastructure administrators so I finally got the feedback I was looking for. That led to several fixes upstream and they have recently been backported to RHEL and CentOS Stream as well. However, the overall lack of feedback is concerning – or at least I thought so.

During CentOS Connect I had the opportunity to discuss this with both AlmaLinux (Jonathan) and RockyLinux (Luis) projects’ sysadmins. “It just works” is a common response I get. Well, that’s nice to hear but what was more exciting to hear is that these projects went a bit further than we did in Fedora and CentOS Stream with their environments. Luis has published the whole RockyLinux infrastructure code responsible for FreeIPA deployment. It is based heavily on ansible-freeipa, reuses the same components we developed for Fedora Infrastructure. Rocky also runs FreeIPA tests in OpenQA instance, similarly to Fedora, and hopefully Luis would contribute more tests to cover passwordless login, already available in CentOS Stream. This would be a very welcome contribution in the light of the Integration SIG activities, helping us to test more complex scenarios community-wide. AlmaLinux infrastructure also uses ansible-freeipa but the code is not publicly available, for whatever reason. Jonathan promised to rectify this problem.

Hallway tracks

I had a number of discussions with people from all kinds of CentOS communities. FreeIPA sits at a unique position by providing secure infrastructure to run POSIX workloads and linking it with modern requirements to web authentication. Our effort to improve passwordless login to Linux environments with reuse of Kerberos to propagate authorization state across multiple machines helps to solve the ‘between the worlds’ problems. That is something that many academic institutions have noticed already and many talks I had in the hallways were related to it.

Multi-homed FreeIPA server investigation

Wed, 16 Aug 2023 10:04:00 +0300

Once in a while people come and ask for FreeIPA servers to work in multi-homed environments. A multi-homed environment in this context is a deployment where the same server is accessible through multiple network interfaces which connect together networks which are not routable to each other. This is typical for administrative and operational networks but there are other types of environments which employ disconnected networks for their operations. FreeIPA server right now has a single host name that resolves to the same IP address in all networks and if one cannot reach the server through that IP address, access to IPA server would not be possible. This typically assumes use of unicast networking as well.

A solution many people look for is to be able to access IPA servers by their interface-specific addresses. Since all secure communication over HTTPS and other protocols (LDAP, Kerberos, etc.) uses name-based resolution in the first place, use of different host names is implied. For example, if FreeIPA is deployed at DNS domain example.test, it would be using Kerberos realm EXAMPLE.TEST and then the original IPA server would be deployed at a host named ipa.example.test (the server host name is not that important here, rather the fact that is is an individual host name). Let’s look at possible communications with this server in a non-multihomed environment first.

Single-homed environment

An IPA client uses HTTPS to communicate with IPA management API, SSSD on the IPA client would use LDAP(S) and Kerberos protocols. In both HTTPS and LDAP(S) cases TLS negotiation would force checking server TLS certificate correctness. A hostname of the host we connect to (ipa.example.test) would have to be present as a dNS SAN record in the TLS certificate presented by the IPA server.

In Kerberos protocol case a different mechanism is used. Yet, Kerberos KDC must know the name of the service principal that a client is asking a service ticket for. If the client wants to acquire a service ticket to ldap/ipa.example.test@EXAMPLE.TEST, this service principal must exist in the Kerberos database that KDC is looking up at.

Multi-homed environment

There are multiple ways of exposing a single hostname in multi-homed environment but they generally involve use of DNS views specific to the individual networking. In such cases DNS serves visible to clients in one network would resolve ipa.example.test to an IP address in that specific network. FreeIPA DNS integration does not support DNS views; this means any of DNS manipulations would have to be done externally to FreeIPA. This is, of course possible, but it really is not then different from a single-homed environment from FreeIPA perspective.

Thus, we would have to have not a single ipa.example.test hostname but for each independent network’s address present on the IPA server a different host name must be present. Let’s assume these are ipa1.example.test and ipa2.example.test. Had we not done this split and simply added multiple addresses for the same ipa.example.test name, clients might resolve the name to an IP address which they could not reach through their own networking routing.

Requirements

Immediately we get a set of requirements here:

TLS certificates issued by IPA CA for HTTPS and LDAP(S) use on IPA server must include dNS SAN records for each hostname represented by the multi-homed server.
Kerberos principals for at least LDAP (ldap/), HTTP (HTTP/), and the system (host/) service principals must have aliases for all multi-homed hostnames.

The latter requirement means that if ipa1.example.test is the primary name, then ldap/ipa1.example.test should have an alias of ldap/ipa2.example.test, HTTP/ipa1.example.test should have an alias of HTTP/ipa2.example.test, and host/ipa1.example.test should have an alias of host/ipa2.example.test.

The same would apply to any other service hosted on IPA server: SMB (cifs/..) or DNS services would need those aliases as well. However, this is not enough.

Implementation considerations

Hostname aliases

FreeIPA does additional checks when issuing certificiates prior to passing the request to the Dogtag CA that is integrated into FreeIPA. For hosts and services on those hosts we also check whether a requestor is granted to issue these certificates. In FreeIPA terms, a host ipa1.example.test would be allowed to issue certificates with dNS SAN record of ipa2.example.test if a host object of ipa1.example.test manages the host object ipa2.example.test in FreeIPA.

Here lies our first problem. A host object in FreeIPA represents the host principal in Kerberos, host/ipa1.example.test. If we created two host objects, ipa1.example.test and ipa2.example.test, then they cannot be aliases to each other on Kerberos level because they’d be two completely different objects from FreeIPA perspective.

Perhaps, we can avoid creating two different host objects? DNS records for hosts are different from the host objects themselves, we only need to have different IP addresses for the hostnames represented by these host entries, not the host entries themselves. May be we could mark one hostname an alias of the other host object?

On Kerberos level FreeIPA does have Kebreros principal name aliasing already. However, it does not exist for hosts as this task has never appeared in past. We would need to add a way to add multiple names to the host object. One way to achieve that is to rely on the fact that fqdn LDAP attribute is multi-valued. Unfortunately, it is also enforced to be a primary key in IPA API – while the underlying LDAP attribute is a multi-valued one, IPA API will enforce its single value:

$ ipa host-mod ipa1.example.test --addattr fqdn=ipa2.example.test
ipa: ERROR: fqdn: Only one value allowed.

This happens because in IPA API any parameter which could be a multi-valued one should explicitly set multivalue=True in its definition. We probably would need to change the multi-valued state for fqdn parameter:

...
        Str('fqdn', hostname_validator,
            cli_name='hostname',
            label=_('Host name'),
            primary_key=True,
            normalizer=normalize_hostname,
>>>>>>      multivalue=True,
        ),
...

and review countless places where its single value is assumed through the code, like in the resolve_fqdn() helper below. LDAP does not guarantee a particular order of returned values for the multi-valued attributes. From LDAP protocol point of view they all equal, there is no particular order.

def resolve_fqdn(name):
    hostentry = api.Command['host_show'](name)['result']
    return hostentry['fqdn'][0]

An alternative would be to introduce a separate attribute purely for the hostname alias management. We don’t need to use it anywhere else at Kerberos level because there we use Kerberos-specific attributes to handle Kerberos principal names and aliases.

LDAP Access Controls

A big part of the FreeIPA access control mechanism relies on 389-ds LDAP server access control interface. Permissions and roles in FreeIPA effectively define a set of ACIs for 389-ds to check access rights. IPA servers verified to be present in certain resource groups (like cn=masters,cn=ipa,cn=etc,$BASEDN). For host aliases this means they should be present in the same groups to be able to operate as their own entities when checking permissions. This is important for internal logic in IPA LDAP plugins and in KDC driver. For the cases when authentication would be done via GSSAPI a resulting Kerberos principal will be normalized to he primary name of the system anyway.

Certificate issuance

Issuing a certificate is a whole separate topic which still awaits its write up. An abridged version of it can be found in my freeipa-users@ mailing list response from May 2022. I need to turn that into a proper document one day.

From the perspective of aliases, we would need to teach the certificate request processing code to look at the host and service aliases when validating SAN records.

Installer integration

There are two approaches to setting up this multi-homed environment. We can provide all information upfront or we can add a tool that adds individual aliases after deployment. This would mean to (re-)generate certificates, create host aliases and services, create configuration snippets and other details which are required to handle multiple host names for the same host from different networks.

The after-deployment case would cause re-issuance of certificates. For external CA providers it could be handled with existing tool that allows to replace existing TLS certificates with externally provided ones. We need to create a checker to verify that all required dNS SAN names were added and are available. In general, troubleshooting this environment would be non-trivial so a special module for ipa-healthcheck definitely would help.

Final thoughts

Multi-homed environments are hard to automate as many assumptions aren’t actually known to us. They are partly implicit to system and network administrator’s work and cannot be derived merely from the system state. It means administrators would need to aid IPA installers with an information. At this point, it is unclear how to structure this information and which of it going to be useful enough. In contemporary Linux environments you might have DNS resolution depend on a specific network interface thanks to systemd-resolved or VPN connection properties. We might not have that information for introspection in advance. While automatically issuing certificates with required names to cover multi-homed setup is not going to be easy, writing down requirements for external CAs and verifying those certificates before applying them at the second stage of external CA enrollment would further complicate things.

All these problems could be solved, of course. Prioritization of this work against other, more urging tasks, is what we need to figure first…

Flock to Fedora 2023 report

Fri, 11 Aug 2023 13:46:00 +0300

On August 2nd-4th, 2023, Fedora Project ran its annual contributors conference, Flock to Fedora, in Cork, Ireland. After a previous successful Flock in 2019 in Budapest, Fedora contributors did not meet in person due to rough pandemia years and had created Nest with Fedora online event instead. Nest ran for three years but online meetings aren’t a full replacement for face to face collaboration. Cork’s Flock was supposed to combine both online and offline events together.

I have been attending and presenting at various Flock and Nest events over past seven years. I was looking forward to see and collaborate with many other project participants and users and get to know new people as well.

My travel to Cork was unremarkable. I took a direct flight from Helsinki to Dublin and then an Aircoach bus to Cork. The ‘unremarkable’ part was really about the unexpected delays people did report over the Matrix channel. The only ‘trouble’ I had was to catch a taxi at 11pm after arrival to Cork to get to my B&B. The Aircoach bus from Dublin airport is very popular in summer and whatever taxi fleet is in Cork was DDoSed by the passengers.

Cork is hilly. I stayed in excellent B&B across the road from the conference hotel. The hotel and its conference facilities are in separate buildings; the events building is up hill from the hotel. Walking is helpful, climbing harder but given we are sitting most of the time, was a welcoming ‘struggle’. Perhaps, my stay outside of the conference hotel has also helped to avoid COVID-19 which few other participants, sadly, contracted. It is hit or miss every time.

Unfortunately, not everyone made to Cork. Marina Zhurakhinskaya passed away in June 2022. Ben Cotton, Fedora Program Manager, has been let go as a part of Red Hat’s layoffs earlier this year. Both had definitely changed Fedora project dramatically, in many ways, both leading to openness and friendliness Fedora is known for. Many presenters remembered both Marina and Ben during their sessions.

2023’s edition of Flock to Fedora was also the first Fedora Project’s event collocated with CentOS Connect. As a result, it brought together Red Hat Enterprise Linux distribution upstreams and downstreams together.

Talks

In total, there were up to four parallel tracks, dedicated to different areas of a distribution development and a project’s life and spanned over three days. That, unsurprisingly, made it challenging to visit all talks and activities. It is a common trait shared by many successful events. And for those who wanted to continue discussions after a talk has ended, there is always a ‘hallway track’.

State of Fedora 2023

The first talk was ‘State of Fedora 2023’ by the project leader, Matthew Miller. Recording is available here. I am linking to the re-take of the talk as the original streaming was off by 20 minutes and Matthew had to reprise it again.

A major announcement made during the talk was a hiring one. Fedora Operations Architect role has been introduced after a program manager role that Ben Cotton so masterfully executed was eliminated. Hopefully, this new role will be filled soon and will allow to capture the same benefits that Ben brought to Fedora. The role is a bit different, though, as it is focused on cross-project and cross-distro impact across Fedora and RHEL.

Fedora contributors’ survey results were also unveiled by Matthew in the talk. In general, contributors keep trust in the project and continue their participation at the pre-pandemia levels. Recent social networking turmoil around Red Hat actions hasn’t influenced the results too much. The screenshots below are from the video stream as the talk’s slides aren’t yet available.

The talk went into details on what Matthew and Fedora Council aim for Fedora Project’s future. Growing a project with thousands of contributors spread around the world and representing different cultures is hard. A lot of effort is put into making Fedora a welcoming place to everyone who is willing to work together towards a common goal.

`rpminspect`: Lessons from three distributions

David Cantrell created and maintains a tool that helps RHEL maintainers to keep their packages sane over years of maintenance. It is run as a part of CentOS Stream merge request process, as part of Fedora gating and pull request testing, and as a gating test for RHEL.

The talk itself is an excellent retrospective on what one should consider when creating a new Open Source project while working on it full time. David provided observations on how to sell the idea to your management, how to get people interested in becoming a community for your project, how to sustain development in a long run. This is one of rare gems of a ‘lone wolf’ maintainership stories that everybody needs to absorb when they start their new journey. Believe me, it is worth it.

Using AI/ML to process automated test results from OpenQA

Tim Flink from Fedora QE team decided to apply AI/ML to a problem of identifying hanging or failing jobs in OpenQA. OpenQA runs full-VM tests and records screencasts of everything what is shown on a VM screen. A crash of a graphical environment is abruptly visible there as graphics would be replaced by a terminal with a Wayland’s stacktrace. What followed is an experiment on processing these screens to reliably detect a particular type of a crash.

We spent some time with Tim discussing how these experiments can be applied to finding out possible issues in other system reports. Since Fedora is upstream of CentOS Stream or RHEL, it means certain issues – and their fixes – would often appear in Fedora first. If we could train a model on those issues in Fedora, can we detect automatically whether a particular fix is required in RHEL later? This is quite relevant to FreeIPA and SSSD as we do run their tests in OpenQA as a part of Fedora Server release criteria.

Another possible use case is to do a reverse training. Since we do know how a potential failure could look like, we can intentionally build an OpenQA test environment that would reproduce the failure and then train a model to recognize logs from such failures in real life scenarios. For example, establishing trust to Active Directory in FreeIPA is reliant on a working DNS setup, working firewall, etc. Failure to communicate through an incomplete firewall would be reflected with timeouts in the logs which we could train to recognize. There are endless possibilities here to aid through known errors…

Hallway track

On a similar note, I had discussion with Amazon’s David Duncan in the ‘hallway track’ which started from an observation that Cloud SIG would really benefit from our passwordless work: distributing VMs with pre-set passwords is not ideal, an ability to inject FIDO2 passkey information and have everything obey it at login in cloud would be great to have. Somewhere along this way, discussion switched to CoreOS-based environments and I realised my experiments with Fedora Silverblue to develop passwordless support for FreeIPA would probably be a subject to a talk that would be interesting to others as well.

I am running my own Silverblue images which source SSSD and FreeIPA upstream test builds to allow me easily to switch between different potential options in one go, without messing with an installation environment. It is quite important for the integration work we do and would be crucial for end-to-end testing of upcoming GNOME changes.

This also provided me an insight into what container-based environments need from FreeIPA and overall from enterprise domains to fit nicely. I should have submitted a talk about that to Flock! Well, I will do one next year, for sure. (And, TODO: file issues to track for that integration to FreeIPA upstream!)

Another interesting discussion we had with Jonathan Dieter. Jonathan is a long term Fedora contributor and FreeIPA user. For past several years Jonathan works with a local Irish company that provides services around the world to test local phone numbers. They maintain an infrastructure in more than 80 countries where there might be no global cloud providers at all. To keep that infrastructure reliable, they use FreeIPA (not alone, of course) and OStree-based images.

Asahi Linux and Fedora

It is one of the talks that I missed to attend in person as conflicts are inevitable: Mo Duffy’s Podman Desktop talk and Adam Williamson’s Fedora CI state talk were running at the same time.

Asahi Linux is a project which aims to upstream support for Apple’s ARM64 architecture, best known through Apple’s M1 and M2 systems. At the Flock Asahi Linux project members have announced that not only Fedora Asahi Remix will be the flagship distribution for the project, but also Fedora Discourse instance will be used to handle Asahi Linux community collaborations.

Asahi’s announcement also an example of how friendly has become Fedora Project as a community over years. I am definitely looking forward to see the remix to become one of official builds of Fedora.

Podman desktop: from Fedora to Kubernetes for beginners

Mo Duffy gave an outstanding talk about using Podman desktop to deliver workloads for non-technical people. It was a highlight of the conference, for sure. She also made few interesting points. For one, running cloud-based workloads locally to allow offline operations is nice. Mo demonstrated a Penpot instance, which is a design and prototyping application. Running it locally helps to maintain the same workflow while on an intercontinental flight. However, even more interesting is that this approach also allows to use a cloud software that otherwise is considered insecure. For example, running a Wordpress setup locally to benefit from its nice UI in a local browser and export static web site content to push to the actual web hosting.

By lowering a barrier to use containerised applications through Podman Desktop we may hope to get more people join our community and contribute. Starting with Podman Desktop’s friendliness would allow these newcomers to discover other Fedora flavors and features. It is certainly an interesting aspect we could expand further in a way similar how this ‘F’ in Fedora got expanded in Mo’s presentation.

Panel: Upstream collaboration & cooperation in the Enterprise Linux ecosystem

Another conference highlight was the panel that brought representatives of Fedora, RHEL, Rocky Linux, Alma Linux, and CentOS Stream together on stage. Distributions upstream and downstream of RHEL presented their views on various development and community topics. It is worth to watch the stream.

State of EPEL

Trow Dawson and Carl George did present another state of EPEL. EPEL has a solid contributors’ base who keep thousands of packages available to RHEL and downstream distributions’ users. EPEL is using Fedora infrastructure and for many packages it shares maintainers with Fedora (EPEL branches are branches in Fedora dist-git for the same package, if this package is not in RHEL). So all EPEL contributors are Fedora contributors. ;)

One interesting aspect in every “State of EPEL” talk is a long tail of the EPEL demographics. Much like “State of Fedora” shows demographics of Fedora releases, EPEL statistics includes details on who is running the lowest number of downstream systems:

Passwordless Fedora

My talk was on the morning of the second day. People were still recovering from the night of International Candy Swap and table games so at start I had may be a couple of attendies. Eventually, we’ve got more people in the room and there were also online attendees so it wasn’t so feeling so lonely.

My talk was similar to previous ones at FOSDEM and SambaXP. What was new is a demo from Ray Strode on how potentially a user experience could look like in GNOME for a passwordless login. Ray implemented a prototype of external identity provider login flow that Allan Day has shared recently. This flow could be used for login through Microsoft’s Entra ID (a.k.a. Azure AD) or any OAuth2 provider supported by FreeIPA. We aren’t fully there yet but the goal is to do this work once for GNOME and reuse for various passwordless authentication approaches supported through SSSD.

I also showed an old demo from my FOSDEM and Flock 2016. It shows how we integrated 2FA tokens (Yubikeys in this example) with FreeIPA to authenticate and obtain Kerberos tickets through a KDC proxy over HTTPS. These tickets then were used to login onto a VPN. This is something that is possible in Fedora and RHEL for almost a decade now.

OpenQA hacking

Before Flock, Adam Williamson started to work on integrating Samba AD tests into OpenQA for Fedora. It almost worked well but there were few issues Adam wasn’t able to resolve so we set down at the Flock and figured out at least few of those. The only remaining one was an apparent race condition within a test that enrolls a system to Samba AD using kickstart. SSSD, it seems, starts before networking is up and stable, and decides that it is offline. When the test tries to resolve an Active Directory user, SSSD fails to do so as it thinks to be offline.

Interestingly, the same test against FreeIPA works fine. The same test done past kickstart works fine as well, for both FreeIPA and Samba AD. There is probably a need to add a waiting period to settle a network state. We saw this in past too but never found a good way to trigger a proper event for SSSD to recover.

I am trying to reduce candy consumption so I skipped the social events on the first day but attended the conference dinner on the second day. All social events during the Flock well organized and this one wasn’t exception either. We had interesting discussions with Fedora and Rocky folks, getting to know there is a lot of similarity in how people do their lives across the world.

On Friday’s night another social event was a Ghost Tour. However, we skipped it and together with few other people went to do a bit of memorabilia road through another Mexican place and (of course!) a local bar. Life in IT and development in 90’s and early 2000’s weren’t that much different in US and Europe, really. Thanks to Spot and Amazon for covering the dinner, thanks to other folks for beer and a company.

I left on Saturday at noon using the same Aircoach bus towards Dublin airport. The bus was full – make sure you have booked your seat online in advance. My flight back to Finland was uneventful as well. Overall, it was a great conference, as usual. I’d like to say thank you to all volunteers and organizers who keep Flock so wonderful and Fedora project so welcoming. Thank you!

FreeIPA authentication improvements and Fedora Infrastructure part 2

Fri, 28 Oct 2022 09:50:00 +0300

This article continues the discussion about FreeIPA authentication improvements and how they could benefit Fedora Infrastructure.

FreeIPA 4.9.10 has added support for relaying authentication to OAuth2 identity providers (IdPs). Users would get their access to FreeIPA resources mediated by an external OAuth2 identity provider which supports OAuth2 device authorization grant flow (RFC 8628). This is not too dissimilar from how smart TVs connect to Youtube and other media services on your behalf. A user would be able to grant access to a scoped information to a FreeIPA OAuth2 client registered with such IdP. In order to authorize the access, the user might need to login to the IdP first and this is performed with the help of a browser running elsewhere. Most common browsers do have support for Webauthn/FIDO2 tokens, thus it is possible to build a system where a login to FreeIPA-enrolled system is authenticated by the FIDO2 token exclusively.

Based on the result of the external IdP authorization grant, FreeIPA would then issue a Kerberos ticket to the user. The ticket can then be used for single sign-on across all FreeIPA services.

The functionality to authenticate against an external IdP is already available in Fedora 36 versions of FreeIPA and SSSD. It is also available in RHEL 8.7 and 9.1 beta versions published in late September 2022.

Relying on an external identity provider to authenticate your FreeIPA users would also mean rethinking the relationship between the IdP and the FreeIPA deployment. Typically, when people integrated FreeIPA with OAuth2 or SAML identity providers, an assumption was made that user authentication is handled by the FreeIPA servers and identity provider issues a token for a web application. Turning this situation around means FreeIPA only becomes a user data store – maybe even not the data store for the IdP itself. IdP would only need to know about the user identity from its own point of view while FreeIPA would have a POSIX identity of the same user, consumed by other applications. Whether it is visible through the IdP grants or via POSIX interfaces is more of an application implementation detail. Decoupling these details leads to a bit of duplication, of course, but allows for better flexibility. A user might come from a federated source and have their authentication handled by a different identity provider – their public service provider like Google, Github, Gitlab, or Microsoft Azure. Or Facebook and any other social media.

All this, however, would mean a change to the way Fedora Accounts system and Fedora IdP, Ipsilon, interact with each other. Right now Ipsilon sources accounts information from the FreeIPA instance directly and delegates authentication to it as well (though PAM service provided by SSSD). Fedora Accounts system, in its turn, manages passwords and tokens in the FreeIPA instance. Since FreeIPA does not have support for Webauthn/FIDO2 web-based authentication yet, Ipsilon would need to manage that itself. So some user accounts would need to source their authentication details from FreeIPA and some would need to use Ipsilon’s internal source. That is a bit complicated as Ipsilon has no way to combine data sources.

Ipsilon does not have any support for both OAuth2 device authorization grant flow and Webauthn/FIDO2 tokens. Even if the FreeIPA instance used by both Fedora project and CentOS project would be updated to provide external IdP authentication, the Ipsilon instance would not be able to utilize this new feature.

Another alternative would be to use Keycloak (free software project upon which Red Hat Single Sign-On product is built). Keycloak 21 is adding multiple user stores support and the FreeIPA team works on a new provider to utilize it. This new Keycloak user store plugin supports SCIM v2 API and also needs a new FreeIPA application that abstracts out access to FreeIPA resources as a SCIM v2 API. In the end, it is the same SSSD as a backend as in case of the Ipsilon IdP but with the ability to split the data store across multiple sources. Keycloak already has support for both OAuth2 device authorization grant flow and the Webauthn/FIDO2 tokens. How this all works together is shown in our Nest with Fedora 2022 talk. Talk slides can be found here but the video includes a live demo session.

On mobile devices front, both Google and Apple are gearing towards introduction of software tokens for Webauthn/FIDO2, often called ‘passkeys’ in the user-oriented marketing materials. These passkeys effectively shared across devices where a user is logged in and uses a shared password wallet. The Google Security team goes at length explaining why this is secure, although some researchers aren’t convinced (this is more about Apple implementation, though).

Consuming software tokens stored in a mobile device from a computer for authentication would need additional support from the identity providers. Neither Ipsilon nor Keycloak implement that. For native Android applications Google just announced they’ll provide an API in what remains of 2022.

Finally, going outside of Fedora Project’s infrastructure, supporting Webauthn/FIDO2 to login to Linux servers and desktops in a centrally managed way is still not there.

Both FreeIPA and SSSD teams are working on a more direct integration with FIDO2 tokens. Instead of relying on an OAuth2 identity provider to handle Webauthn/FIDO2 operations in the browser, SSSD is integrating with libfido2 library to make FIDO2 token’s use on the machine locally. This would replace the pam_u2f module and would allow FreeIPA to provide centralized management of the FIDO2-based passkeys. On top of that, FreeIPA will provide Kerberos integration: login with the FIDO2 token would be turned into a Kerberos ticket and single sign-on operations would be ensured across multiple services, like it is done today for traditional Kerberos pre-authentication methods. A benefit is that passwords can be completely removed for the users with direct FIDO2 token enrollment.

FreeIPA-enrolled clients (Fedora 36+, RHEL 8.7/9.1 betas) already can login via external IdP as outlined above but only for the SSH or a direct console login. Login over graphical desktop is not working well: a message telling to visit an IdP page and authorize the device access does not fit into a gdm login entry which is used to show the message coming from the PAM stack. GDM is also not able to go to that URL directly as there is still no support for opening a browser instance pre-login in GNOME or any other desktop manager. We are looking at implementing this with GNOME developers similar to how a smartcard selection was added in 2017. Hopefully, this will come to one of the next Fedora versions …

Notwithstanding open issues, we are pretty much close to enabling FreeIPA-enrolled Linux systems to go passwordless. It is not yet a year of passwordless Linux like not a year of Linux on the desktop but making it reality is not that far away. Perhaps I am too optimistic, though – like we were all before with Linux on the desktop.

FreeIPA authentication improvements and Fedora Infrastructure part 1

Fri, 28 Oct 2022 08:50:00 +0300

The Fedora project exists because of its contributors. Their contributions shape the landscape of Linux distributions in a direct way but they also have made a significant influence on the Open Source projects themselves. Fedora contributors are not only people who participate in package maintenance, there are upstream developers, documentation writers, quality assurance engineers across multiple industries, students, volunteers and many many others. As with many other areas, this participation is bi-directional and practices established in the Fedora project may apply elsewhere too.

One area dear to me is authentication. The FreeIPA project serves as an umbrella to provide a consistent centralized identity management and authentication solution for Linux systems (in the first place, though standards-compliant, UNIX-like operating systems benefit from its use as well). FreeIPA’s core is built around Kerberos authentication protocol and SSSD daemon. It makes use of Kerberos’ features for single sign-on ease on the client side.

Many Free Software and Open Source projects use FreeIPA to deploy centralized identity management, authentication and authorization for their own contributors. Examples can be found small and large: GNOME project was one of the earliest, migrating its infrastructure to FreeIPA in 2014. The Fedora project is not an exception, it has been using FreeIPA for quite some time, though FreeIPA deployment was only handling Kerberos – user accounts were stored in a different place and synchronized with FreeIPA. The old Fedora account system was gradually rewritten to be built on top of FreeIPA and in 2021 all accounts were migrated to the new system.

Fedora accounts system allows users to login with a password and optionally to use two-factor authentication with the help of TOTP tokens. Users can also associate GnuPG or SSH public keys with themselves. These details get consumed by various Fedora applications but there are several important use cases:

Issue Kerberos ticket which can be used to authenticate against a build system used by Fedora
Access servers over SSH protocol, with Kerberos tickets or SSH keys
Use Kerberos ticket or authenticate with password to Fedora Project’s identity provider and authorize Fedora applications to access user data.

FreeIPA supports more authentication methods in its Kerberos implementation but they aren’t used by the Fedora project. On the other hand, the CentOS Project shares its FreeIPA instance with Fedora and uses certificates issued by the FreeIPA CA for its authentication. These certificates can be used to obtain Kerberos tickets as well. For both projects this FreeIPA instance is maintained by the Community Platform Engineering team who looks after both communities’ needs.

Since the Fedora Project is an important gateway to Red Hat Enterprise Linux, many Fedora contributors were recently raising security concerns around secure software delivery and asking how we can improve authentication and authorization for Fedora infrastructure. For gory details, the thread about removing access of inactive packagers after Fedora 37 release has a few examples. A common agreement is that use of Webauthn and FIDO2 tokens would definitely improve the security, especially coupled with the prevention of an authentication methods’ downgrades.

Moving to Webauthn/FIDO2 would probably be relatively easy if every single application to access in the Fedora infrastructure would be a web-based. For example, OAuth2 clients could be forced to operate on specific scopes of the user data which would only be granted if strong authentication methods were used to authenticate. For SSH-based access it would also be possible to get SSH keys based on a native FIDO2 support in OpenSSH and prevent use of other key types. However, a single sign-on functionality would be lost then as currently it is not possible to convert SSH public key access to Kerberos tickets nor is it possible to use FIDO2 keys natively in Kerberos flows.

Use of FIDO2 tokens introduces another problem, though, this time a social one. Fedora Project is famous for its wide contributor base, spanning many countries and continents. Fedora Project contributors come from all kinds of backgrounds and so far the only requirement to enable their contributions from a technical point of view was internet access. Even enforcement of the two-factor authentication with the help of TOTP tokens could have been mitigated by use of a software token on a mobile device or simulated on a computer. In many countries access to mobile devices is easier than to any hardware token. It is still a cost and somebody has to pay for it. The cheapest FIDO2 token is around 10 EUR, although a single token can be used against many resources (websites). There is a cost to enforce stronger authentication methods and somebody would need to pay it if the Fedora project chooses to raise the requirements. Perhaps, companies benefiting from the secure software development processes around Fedora distribution would be willing to step up?

In the next article I will look at how FreeIPA could help with the technical side of supporting Webauthn/FIDO2 use by Fedora contributors.

Far away to be identical

kurbu5: MIT Kerberos plugins in Rust

Local authentication hub

Pre-authentication

API bindings

ASN.1 for legacy apps: Synta

Synta, ASN.1 library

Performance

Processing CA databases

Why C Libraries Are Slower

FreeIPA local tests and FOSDEM demos

Demo labs

Demo recordings

Minimal deployment demo

Local KDC demo

IPA to IPA trust demo

Local authentication hub

Local authentication hub

History dive

Modern days

How “local” a local KDC should be?

Trust management

Systemd User/Group API support

Establishing trusts

Current state of local authentication hub

IPA-IPA trust progress report

Trust across enterprise domains

Kerberos trust infrastructure

Identity information retrieval

Demo

Future work

CentOS Connect 2024 report

CentOS Connect

CentOS Integration SIG

CentOS Stream and EPEL

20 years of CentOS project

FreeIPA and CentOS project infrastructure

Hallway tracks

Multi-homed FreeIPA server investigation

Single-homed environment

Multi-homed environment

Requirements

Implementation considerations

Hostname aliases

LDAP Access Controls

Certificate issuance

Installer integration

Final thoughts

Flock to Fedora 2023 report

Talks

State of Fedora 2023

rpminspect: Lessons from three distributions

Using AI/ML to process automated test results from OpenQA

Hallway track

Asahi Linux and Fedora

Podman desktop: from Fedora to Kubernetes for beginners

Panel: Upstream collaboration & cooperation in the Enterprise Linux ecosystem

State of EPEL

Passwordless Fedora

OpenQA hacking

Social events

FreeIPA authentication improvements and Fedora Infrastructure part 2

FreeIPA authentication improvements and Fedora Infrastructure part 1

`rpminspect`: Lessons from three distributions