[Ukfreebsd] opentech - 11th Sept - London

[Robert Watson]

On Sat, 31 Jul 2010, Sam Smith wrote:

> Is everyone still running FreeBSD 4.0 since there's nothing significantly 
> different since then? :)

:-).

I think you could reasonably track a number of themes in FreeBSD development 
over the last decade or two, some of which align with industry trends as well:

- Adoption of threading as the concurrency model of choice throughout the OS,
   userspace and kernel.  This means high-performance pthreads in userspace,
   and a pthreads-like model in kernel, priority propagation on locks to
   address priority inversion, and a data-centric locking model.  In the
   mid-1990's, it was unclear what concurrent programming model will win, and
   for its limitations, threading has proven incredibly powerful.  With our
   pthreads project in the mid-1990's, followed by KSE, an M:N threading model,
   and most recently, libthr, maturing thread models have been a theme in
   FreeBSD development over the last decade.  The initial goal was to support
   high levels of concurrency effectivley (KSE), but as application models
   evolved, it became clear that a simpler and less concurrent 1:1
   implementation offered greater benefit by simplifying scheduler data
   structures, allowing them to use more complex heuristics as hardware became
   more parallel.

   Most recently, the FreeBSD Project has engaged with Apple to adapt Grand
   Central Dispatch (GCD), a new concurrent programming framework, to run on
   FreeBSD.  This builds on threads and the FreeBSD kqueue model, now found in
   many other operating systems including Mac OS X, but offers a more
   programmer-accessible, and ultimately, scalable, programming model.

- Tracking processor architecture changes, including a transition from
   instruction-centric performance to cache-centric performance and support for
   newer and widespread RISC architectures (ARM, MIPS, ...).  These
   architectures have transformed the "embedded" hardware space from a world of
   8-bit microprocessors with 64K of memory to one with hundreds of megabytes
   of memory and multi-GB disks, and perhaps most importantly, introduced
   virtual memory support required to run UNIX-like operating systems.  This
   has opened the door to mature, FreeBSD-derived operating systems all over
   the embedded market, from Apple's FreeBSD-derived iOS on the iPhone and iPad
   to FreeBSD-derived JunOS running on low-power switches.  These systems
   resemble our server UNIX systems from the late 1990's in terms of CPU
   capacity, memory, etc, and so FreeBSD has "grown down" quite effectively.
   Power use has been an increasing focus in OS design as a result.

- Another critical hardware trend has been the move to increasingly parallel
   server and appliance hardware.  Ten years ago, we ran on two-core systems,
   now we run on 32-core systems, with 64- and 128-core configurations in our
   immediate future.  This has meant not just a transition from a Giant locked
   kernel to granular, data-oriented locking (the highly successful 10-year
   SMPng project), but more recently work on algorithms and approaches that
   scale to high core counts through complex work distribution,
   hardware-assisted work distribution on high-performance network cards, etc.
   FreeBSD's performance in multiprocessing is cutting edge, as benchmarks over
   the last few years have shown, and we continue to aggressively exploit new
   parallelism.  Most recently, we've been collaborating with network vendors
   such as BlueCoat and Juniper, as well as hardware vendors such as Chelsio
   and Intel, to develop a highly scalable network stack able to run on dozens
   (perhaps someday hundreds) of kernel threads.  Likewise, NUMA is becoming a
   reality, and our kernel memory allocator has recently been updated to
   introduce NUMA-awareness.

- Virtualization has been another industry trend that FreeBSD has both led,
   and responded to.  FreeBSD's Jail model, introduced around ten years ago,
   led a revolution in OS-centric virtualization, with imitators such as
   Solaris Zones and Linux Virtual Servers.  With Jails, virtual hosts could
   delegate "root" access to hundreds or even thousands of virtual machines per
   physical hosts, at a time when Internet growth was occuring dramatically,
   providing a stable and powerful hosting platform that remains in use
   throughout the ISP industry (from providers such as Yahoo!, Verio, New York
   Internet, and countless others).  More recently, FreeBSD has slightly less
   agressively jumped on the Xen bandwagon, runs on VirtualBox, VMWare, and
   many other virtual systems.  While true "virtualization" provides strong
   isolation, another influential virtualization trend has been
   para-virtualization, in which hardware virtualization boundaries are blurred
   in order to improve performance through specially adapted device drivers
   that bypass slow hardware emulation and talk directly to the hypevisor and
   host OS.

- Security has been a critical area in which the FreeBSD Project has been
   effective.  We recognized early that the UNIX model was powerful, but
   limiting for many evolving applications such as firewall appliances, smart
   phones, and more.  The TrustedBSD Project, announced roughly ten years ago,
   developed access control list support (ACLs), first to the POSIX spec, and
   more recently, the NT-compatible NFSv4 ACL model, as NFSv4 becomes more
   influential.  We jointly developed security audit support with Apple,
   allowing FreeBSD and Mac OS X to meet the requirements of the CAPP common
   criteria standard (Mac OS X has been formally evaluated to that standard,
   although FreeBSD, as a general-purpose OS, has not been).  Our MAC Framework
   allows flexible and extensible access control, supporting advanced security
   features in the base OS, but also allowing product vendors to extent the OS
   model without modifying the OS.  Companies like Apple, Juniper, and McAfee
   use this feature extensively to provide alternative access control models,
   such as sandboxing, for third-party code, and to build system assurance.
   Most recently, we've announced the Capsicum project, developed in
   collaboration with Google, to support capability-oriented OS security.
   This work will be presented at USENIX Security 2010 in August.

- Storage has been another area of significant work in the FreeBSD Project,
   both in continuing to advance the UFS file system (64-bit support,
   snapshots, background file system checking, extended attributes, advanced
   security features such as ACLS and MAC, and most recently, journaling).  Our
   GEOM framework, developed under sponsorship from DARPA in the early 2000's,
   allows easily pluggable and extensible storage transforms, bringing storage
   multipathing, new RAID integration, and features like full file-system
   journaling and replication for fault tolerance.  We've also ported Sun's ZFS
   file system to FreeBSD, where its self-healing and management features power
   countless servers and storage appliances in a tightly-integrated way.

- Improvements in development tools, and more generally, introspection, have
   played an important role as well.  From moving to adopt the new clang/LLVM
   compiler suite, adopting Coverity's Prevent static analysis tool (unlike
   many open source projects, we have our own Prevent server system and
   actively re-analyze our source code nightly).  We've also ported Sun's
   DTrace, offering introspection tools for performance and behavioural
   analysis -- the FreeBSD Foundation recently announced a contract to Rui
   Paulo to complete userspace DTrace support, which will allow similar
   analysis of userspace applications.  Other tools, such as our hardware PMC
   (performance monitoring counters) support, system tracing tools, kernel
   debugging tools such as integrated debugging, dynamic lock order analysis,
   and much more make FreeBSD one of the best OS platforms for kernel feature
   development.  We've talked to many developers who actually write their Linux
   kernel code on FreeBSD so that they can use our debugging tools, before
   porting to Linux!

- Another really interesting growth area for FreeBSD has been the develoment
   of open source spin-off systems that take FreeBSD as a baseline and then
   extend our system in various ways.  Commercial vendors have always done this
   (NetApp, Juniper, Apple, Isilon, Panasas, ...), but the open source side has
   greatly expanded our user reach as an open source system.  These include
   PC-BSD, a desktop-oriented system, pfSense, a firewall system, and FreeNAS,
   a network attached storage system.  These, and many other, systems take
   FreeBSD's powerful OS core, and then customize it for specific purposes.
   We're proud of their work, and eager to integrate back core OS features that
   the've developed.

- There are some other interesting points to be made about open source project
   structure.  Dealing with project growth has been a critical challenge, both
   socially and technically.  Roughly ten years ago, we moved for the first
   time to an elected core team model, a social experiment that has proven
   extremely successful, allowing us to renew project leadership over time.
   We've undertaken some huge OS development projects, one of the most
   interesting our SMPng project, which involved dozens of developers (and
   companies) collaborating to improve multiprocessor scalability.  We've
   increasingly had a delegated project leadership structure, in which
   chartered teams take responsibility for portions of the project's work:
   release engineering, application porting, security advisories, system
   adminstration, bug-busting, documentation, etc, moving away from the idea
   that a small "core team" of developers does it all.  We've also had to
   evolve our tools, migrating from CVS with many extensions, to Subversion
   with fewer extensions.  Along the way we've adopted Perforce as a way to
   improve our branched development methodology for side-projects with long
   life cycles, as well as to support dozens of summer students each year
   sponsored by Google.

Surely there are many more areas, as well, that I've missed in the above list. 
I think it's safe to say that the world is exactly the same as ten years ago, 
only also entirely different!

Robert

>
>
> Cheers
> Sam
>
> On 29 Jul 2010, at 14:06, Mark Blackman wrote:
>
>> On 29 Jul 2010, at 12:08, Sam Smith wrote:
>>
>>>
>>> hey all,
>>>
>>> Details of this year's OpenTech are below - many of you will
>>> be interested in coming along.
>>>
>>> There's a session on "Modern X" for finding out about things
>>> that you heard about a decade ago, but which have radically
>>> changed since you last looked at them... If someone wants to
>>> offer about 12 minutes on FreeBSD, it would fit in well.
>>
>> Is FreeBSD radically changed from 10 years ago? :)
>>
>> - Mark
>>
>> _______________________________________________
>> Ukfreebsd mailing list
>> Ukfreebsd at uk.freebsd.org
>> http://mailman.uk.freebsd.org/mailman/listinfo/ukfreebsd
>> UK BSD Groups http://www.bsdgroups.org.uk
>> Hosted by EXOnetric http://www.exonetric.net
>
> _______________________________________________
> Ukfreebsd mailing list
> Ukfreebsd at uk.freebsd.org
> http://mailman.uk.freebsd.org/mailman/listinfo/ukfreebsd
> UK BSD Groups http://www.bsdgroups.org.uk
> Hosted by EXOnetric http://www.exonetric.net
>