I’ve attended annual SambaXP conference on May 19th-21st. I’ve presented about FreeIPA ID Views and this year we also had quite a few Red Hat’s talks in the program so that organizers even made a 'Red Hat track' on the last day, with all speakers in that track coming with a Shadowman.

SambaXP is a conference run by SerNet GmbH in Goettingen, Lower Saxony, Germany. SerNet is one of core contributors to Samba project, organized by, among others, Volker Lendecke who is founder of Samba project along with Andrew Tridgell and Jeremy Allison.

The conference is well attended; Microsoft is both sponsoring and sending participants for last six or more years — including key managers and developers of Windows Server and SMB protocols stack.

The conference itself is spanning two days. However, the day before the official start there are workshops and tutorials. A tutorial by Stefan Kania was dedicated to the Samba 4 migration experience. Tutorials and workshops are paid-for events, as well as the conference itself, but very valuable to all attendees.

This year we had protocols interop event on the 'minus one' day where Microsoft team ran a number of detailed presentations about changes in SMB protocol coming with SMB 3.1.1 specification and Windows 10, Windows Server 2016. Enhancements in SMB 3.1.1 clearly point to the direction of running HyperV workloads directly over SMB protocol in the cloud. There are multiple extensions to allow to map HyperV semantics for serving large volumes of VMs off scale out file systems. Another area of improvements in SMB 3.1.1 is related to getting security tightened to the point of forcing pre-authentication integrity. This is a continuation of a more general effort publicized by Microsoft to 'get rid of passwords' for Windows 10. More on SMB 3.1.1 extensions can be read in the Jose Barreto’s blog.

Another part of the protocols interop event was familiarizing with Microsoft’s Protocol Test Suite. The suite is available already from MSDN and is important part of validating correctness of implementations because its tests are generated out of the actual specifications. A hint was given on some positive news to be expected in June in time for Interop Plugfest (June 20-25th) in Redmond with regards to the suite itself but already now there is a change as running the Protocol Test Suite does not require having access to Visual Studio anymore.

SMB 3.1.1 pre-authentication integrity makes harder to analyze traffic. Microsoft decided to use ETW tracing facilities available since Windows 7 which produce a log output of the internal SMB code just before encrypting packets and handing them over to NDIS layer. ETW tracing format is then possible to load into Message Analyzer and watch both encrypted and unencrypted content side to side. We’ve been told ETW tracing also contains debugging output from SMB routines otherwise not seen at all. This feature does not require installing additional software and one can run analysis on a different machine than the trace was obtained.

It would be interesting to add support of reading ETW traces to Wireshark. With PCAPNG format its is possible to associate comments and 'notes' with the networking packets so having both ETW and NDIS level traces in the same capture is certainly possible.

Continuing Microsoft tune, Tom Talpey, Microsoft’s architect in File Server team, also demonstrated the performance improvements with SMB 3.1.1, storage quality of service additions, and laid out the problem space with newer memory types like persistent and phase memory. For next step of this work at SDC conference in September, Tom promised 'paradigm changes in the protocol'.

More on SMB 3 and HyperV integration can be found in this post of links by Jose Barreto.

A final note on Microsoft’s take is that they want to outlaw SMB1 protocol. When this finally will happen, is unclear, but there is huge incentive to move away from NTLMSSP variants and anonymous data access. This will in particular affect RHEL 6 after 2016-2018, especially in the area of domain controllers because Windows Server 2016 at some point will move to drop SMB1 communication between the DCs. A more detailed overview of plans is available in this blog post. These changes should not be taken with a light heart — majority of consumer NAS devices use SMB1 protocol and force anonymous access by default.

Jeremy Allison also made a loud rant about how we deliver media from these NAS devices to our screens. All Smart TVs, he said, now run Linux. It means they are SMB clients or can be made so with relative ease. SMB3 as a protocol is perfect on low latency high throughput often needed for streaming media. He wants to work with TV manufacturers and turn DLNA from HTTP to SMB media serving.

The keynote by Marc Muehlfeld (Samba Team) gave us results of a survey of Samba community. The presentation is available at Samba site. More than 50% of respondents run Samba 4.x, with ~38% on Samba 3.6 — the version which is not supported by upstream anymore. About 60% of respondents are planning to migrate to Samba AD in next two years, with 25% are targeting the upcoming half a year, so migration seem to happen naturally. Samba AD is not an easy feat, most of responses mention that majority of issues they experienced are in Authentication (44%), File Serving (30%), and Active Directory backend (28%). While issues is what typically makes us in infrastructure world visible to upper management, it was interesting to see that 15% of respondents never experienced any issues.

One of defining topics for this year SambaXP is cloud. We had two separate talks on SMB in the cloud from Google and SuSE Linux, clear message on cloud readiness of the SMB protocol from Microsoft (they now have a separate implementation of SMB protocol for Azure cloud, tuned for cloud-specific workloads). Majority of talks during the first day were related to clustering and high availability (IBM, Red Hat, Nutanix) as well.

Red Hat’s talks also centered on using libraries provided by Samba project in other solutions. SSSD is building on talloc, tevent, ldb, and tdb as key components on our identity management client-side solution. We also implement various pluggable interfaces to augment what Samba daemons see themselves when interoperating with FreeIPA and SSSD.

Steve French (Primary Data) presented his view on the client side of SMB — he is working on cifs.ko kernel driver but also looking beyond Linux. Apple went with first SMB 3 extensions related to OS X-specific features and Steve is currently designing an equivalent of what we had as UNIX extensions in SMB1 for SMB3. A first rough cut into the client-side code was done during the conference and we identified few handy extensions which would allow to reduce greatly memory consumption and string manipulations in case of both server and client running in UTF-8 environment.

SMB 2 and particularly SMB 3 protocol families are interested in that it is much cleaner spec to start with — there is a number of startups that have their own SMB protocol implementations, all SMB2+ only, without legacy support at all. They may not have all the required features yet but it looks like SMB3 is gaining a good base across cloud/storage startups as a good starting point for a modern performance-oriented networking file and block system, extensible enough to cover specific workloads. This, on the other hand, creates a possibility to fragment the spec and it will be interesting to see if common protocol testing platform will help in keeping protocol forks close to each other.

Aside from talks there were intense development and hacking sessions, often well into after midnight. Many bugs were fixed and some of important long term development branches were reviewed in face to face sessions. Below are few examples relevant to identity management work:

  • Our effort to move Samba AD to MIT Kerberos is nearing completion. We are hoping to get it polished in next ~6-8 months but the current patchset is passing all but few tests in the Samba testsuite which is run on every commit to the upstream git tree (~1750 different testing scenarios). There is now a common agreement upstream to move to MIT Kerberos as the primary Kerberos implementation. Both Heimdal and MIT Kerberos will be supported but it seems that Heimdal upstream situation is not that healthy since Apple moved full way with the project.

  • We are very close to land initial implementation of cross-forest trusts in Samba AD, sponsored by Red Hat and implemented by SerNet. Some of key interoperability bugs were fixed during the conference sprints and I have now an environment where FreeIPA 4.1 can establish trust to Samba AD. Our long term goal of allowing users to maintain Windows workstations in Samba AD and Linux machines in FreeIPA is on track.

  • There is active work on scalability. Volker Lendecke presented his progress of improving the messaging system used by Samba that allows to scale both in number of processes on the single host and with a number of hosts beyond currently supported by CTDB. At the same time, Jakub Hrozek (Red Hat, SSSD) demonstrated some of his findings when trying to move ldb database (key part of SSSD caching system and crucial component of Samba AD DC) to a faster backend based on LMDB from OpenLDAP.

  • A new version of patchset to re-target Samba AD to use OpenLDAP was published by Nadezhda Ivanova (Symas). The work has stalled a bit due to other customer-related work by Nadezhda but general work to externalize Samba AD components is moving forward. As usual, projects like this aren’t easy and pretty hard to achieve without being able to dedicate months of sustained attention to details.

  • There is a common effort to make predictable and manageable release cycles. While with autobuild we have always buildable and releasable git master tree, it is currently tested only in a single environment. Michael Adam (Red Hat) presented his set of Vagrant-based scripts to quickly test Samba under multiple distributions, increasing developer productivity and test coverage.

I’ve been lucky at finding a reason for a long standing bug in IPv6 support that caused some of gripes in testing FreeIPA as far as two years ago. Samba client libraries support Active Directory Domain Controller’s resolution via CLDAP pings but the code that actually tried to reach multiple servers in order to find the one with right capabilities was not taking into account a case when it is impossible to establish actual connection due to IPv6 routes missing in the environment. DNS might return you IPv6 hosts but opening sockets to them may subtly fail with network being unreachable or blocked at firewall level. Samba client libraries weren’t paying attention to the greater context and the whole CLDAP request processing got aborted even though more possible targets were available to test. The fix is in git master and both stable branches now.

Finally, an SMB 3 panel on the last day demonstrated that Samba is in fairly good shape with regards to SMB 3 implementation and there is a healthy communication with Microsoft on the protocol development. Now that Windows Server release cycle is decoupled from Windows (client) release cycle, there seems to be less pressure from the schedule point of view in avoiding discussion of protocol improvements well in advance — a definite change in behavior over last five years.

Slides and audio recordings of the talks will start appearing on sambaxp.org in upcoming weeks. It was impossible to visit all the talks of three parallel tracks so I’m looking forward to listen to them.