2021-07-04
In this article we’ll learn how to build shared libraries and install them properly on several platforms. For guidance, we’ll examine the goals and history of dynamic linking on UNIX-based operating systems.
Content for the article comes from researching how to create a shared library, wading through sloppy conventions that people recommend online, and testing on multiple Unix-like systems. Hopefully it can set the record straight and help improve the quality of open source libraries.
The design typically used nowadays for dynamic linking (in BSD, MacOS, and Linux) came from SunOS in 1988. The paper Shared Libraries in SunOS neatly explains the goals, design, and implementation.
The authors’ main motivations were saving disk and memory space, and upgrading libraries (or the OS) without needing to relink programs. The resource usage motivation is probably less important on today’s powerful personal computers than it was in 1988. However, the flexibility to upgrade libraries is as useful as ever, as well as the ability to easily inspect which library versions each application uses.
Dynamic linking is not without its critics, and isn’t appropriate in all situations. It runs a little slower because of position-independent code (PIC) and late loading. (The SunOS paper called it a “classic space/time trade-off.”) The complexity of the loader on some systems offers increased attack surface. Finally, upgraded libraries may affect some programs differently than others, for instance breaking those that rely on undocumented behavior.
At compile time the link editor resolves symbols in specified libraries, and makes a note in the resulting binary to load those libraries. At runtime, applications call code to map the shared library symbols in memory at the correct memory addresses.
SunOS and subsequent UNIX-like systems added compile-time flags to
the linker (ld) to generate – or link against – dynamically linked
libraries. The designers also added a special system library (ld.so)
with code to find and load other libraries for an application. The
pre-main() initialization routine of a program loads ld.so
and runs it from within the program to find and load the rest of the
required libraries.
As mentioned, applications can take advantage of updated libraries without needing recompilation. Library updates can be classified in three categories:
An application linked against a library at a given major release will continue to work properly when loading any newer minor or patch release. Applications may not work properly when loading a different major release, or an earlier minor release than that used at link time.
Multiple applications can exist on a machine at once, and each may require different releases of a single library. The system should provide a way to store multiple library releases and load the right one for each app. Different systems have different ways to do it, as we’ll see later.
Each library release can be marked with a version identifier (or “version”) which seeks to capture information about the library’s release history. There are multiple ways to map release history to a version identifier.
The two most common mapping systems are semantic versioning and libtool versioning. Semantic versioning counts the number of releases of various kinds that have happened, and writes them in lexicographic order. Libtool versioning counts distinct library interfaces.
Semantic versioning is written as major.minor.patch and
libtool as current:revision:age. The intuition is that
current counts interface changes. Any time the interface
changes, whether in a minor or major way, current
increases. Here’s how each system would record the same history of
release events:
| Event | Semver | Libtool |
|---|---|---|
| Initial | 1.0.0 | 1:0:0 |
| Minor | 1.1.0 | 2:0:1 |
| Minor | 1.2.0 | 3:0:2 |
| Patch | 1.2.1 | 3:1:2 |
| Major | 2.0.0 | 4:0:0 |
| Patch | 2.0.1 | 4:1:0 |
| Patch | 2.0.2 | 4:2:0 |
| Minor | 2.1.0 | 5:0:1 |
Here’s how applications answer the question, “Can I load a given library?”
current interface number of the library I linked
with between current - age and current of the
library to be loaded?
We’ll be using semantic versioning in this guide, because libtool versioning is only relevant to libtool, a tool to abstract library creation across platforms. I believe we can make portable libraries without libtool. I mention both systems only to show that there’s more than one way to build version identifiers.
One final note: version identifiers say that things have changed, but omit what changed. More complicated systems exist to track library compatibility. Solaris, for instance, developed a system called symbol versioning. Symbol versioning chases space savings at the expense of operational complexity, and we’ll consider it later.
One subtlety of versioning is that changes can happen in either a library’s programming interface (API) or binary interface (ABI). A C library’s programming interface is defined through its header files. A backward-incompatible API change means a program written for the previous version would not compile when including headers from the new version.
By contrast, a binary interface is a runtime concept. It concerns the calling conventions for functions, or the memory layout (and meaning) of data shared between program and library. The ABI ensures compatibility at load and run-time, while the API ensures compatibility at compile and link time.
The two interfaces usually change hand-in-hand, and people sometimes confuse them. It’s possible for one to change without the other, though.
Examples of breaking ABI, but API stability:
In these library changes, application code doesn’t need to change, but does need to be recompiled with the new library headers in order to work at runtime.
Examples of ABI stability, but breaking API:
In these library changes, application code would need to be modified to compile successfully against the new library, even though code compiled before the change could load and call the library without issue.
const foo * to
foo *. A pointer to a const object cannot be implicitly
converted to a pointer to a non-const object. The ABI doesn’t care
though, and moves the same bytes. (If the library does in fact modify
the dereferenced value, it may be an unpleasant surprise to the
application of course.)It’s usually easy to tell when you’ve added functionality vs broken backward compatibility, but there are tools to check for sure. For instance, the ABI Compliance Checker can detect breakages in C and C++ libraries.
In light of the versioning discussion earlier, which changes should the version identifier describe? At the very least, the ABI. When the loader is searching for a library, the ABI determines whether a library would be compatible at runtime. However, I think a more conservative versioning scheme is wise, where you bump a version when either the API or ABI change. You’ll end up with potentially more library versions installed, but each shared API/ABI version will provide guarantees at both compilation and runtime.
After compiling object files, the compiler front-end (gcc, clang, cc, c99) will invoke the linker (ld, lld) to find unresolved symbols and match them across object files or in shared libraries. The linker searches only the shared libraries requested by the front-end, in the order specified on the command line. If an unresolved symbol is found in a listed library, the linker marks a dependency on that library in the generated executable.
The -l option adds a library to the list of candidates
for symbol search. To add libfoo.so (or
libfoo.dylib on Mac), specify -lfoo. The
linker looks for the library files in its search path. To add
directories to the default search path(s), use -L, for
instance -L/usr/local/lib.
What happens if multiple versions of a library exist in the same
directory? For instance two major versions, libfoo.so.1 and
libfoo.so.2? OpenBSD knows about version numbers, and would
pick the highest version automatically for -lfoo. Linux and
Mac would match neither, because they’re looking for an exact match of
libfoo.so (or libfoo.dylib). Similarly, what
if both a static and dynamic library exist in the same directory,
libfoo.a and libfoo.so? All systems will
choose the dynamic one.
Greater control is necessary. GCC has a colon option to solve the
problem, for instance -l:libfoo.so.1. However clang doesn’t
have it, so a truly portable build shouldn’t rely on it. Some systems
solve the problem by creating a symlink from libfoo.so to
the specific library desired. However when done in a system location
like /usr/local/lib, it nominates a single inflexible
link-time version for the whole system. I’ll suggest a different
solution later that involves storing link-time files in a separate place
from load-time libraries.
At launch time, programs with dynamic library dependencies load and run ld.so (or dyld on Mac) to find and load the rest of their dependencies. The load library inspects DT_NEEDED ELF tags (or LOAD_DYLIB names in Mach-O on Mac) to determine which library filename to find on the system. Interestingly, these values are not specified by the program developer, but by the library developer. They are extracted from the libraries themselves at link-time.
Dynamic libraries contain an internal “runtime name” called SONAME in
ELF, or install_name in Mach-O. An application may link against a file
named libfoo.so, but the library SONAME can say, “search
for me under the filename libfoo.so.1.2 at load time.” The loader cares
only about filenames, it never consults SONAMES. Conversely, the
linker’s output cares only about SONAMES, not input library
filenames.
Loaders in different operating systems go about finding dependent libraries slightly differently. OpenBSD’s ld.so is very true to the SunOS model, and understands semantic versions. For instance, if asked to load libfoo.so.1.2, it will attempt to find libfoo.so.1.x with the largest x ≥ 2. FreeBSD also claims to have this behavior, but I didn’t observe it in my tests.
In 1995, Solaris 2.5 created a way to track semantic versioning at the symbol level, rather than for the entire library. With symbol versioning there would be a single e.g. libfoo.so file that simply grows over time. Every function inside is marked with a version number. The same function name can even exist under multiple versions with different implementations.
The advantage of symbol versioning is that it can save space. In the alternative, where versioning is per-library rather than per-symbol, a large percentage of object code is often copied unchanged from one library version to the next. The disadvantages of symbol versioning are:
Symbol versioning quickly found its way into Linux, and became a staple of Glibc. Because of Linux’s symbol versioning preference, its ld.so doesn’t make any effort to rendezvous with the latest minor library version (à la SunOS or OpenBSD). Ld.so searches for an exact match between SONAME and filename.
However, even on Linux, most libraries don’t use symbol versioning.
Also, their SONAMEs typically record only a major version (like
libfoo.so.2). Within that major version, you just have to hope the
hidden minor version is new enough for all applications compiled or
installed on the system. If an app relies on functions added in a later
minor library version, it’ll crash when it attempts to call them.
(Setting the environment variable LD_BIND_NOW=1 will
attempt to resolve all symbols at program start instead, to detect the
failure up front.)
MacOS uses an entirely different object format (Mach-O rather than
ELF), and a differently named loader library (dyld rather than ld.so).
Mac’s dynamically linked libraries are named .dylib, and
their version numbers precede the extension.
Native Mac applications are usually installed into their own
dedicated directories, with libraries bundled inside. Thus the loader
has special provisions for finding libraries, like the keywords
@executable_path, @loader_path and
@rpath in the install_name. MacOS supports
system libraries too, with dyld consulting the
DYLD_FALLBACK_LIBRARY_PATH, by default
$(HOME)/lib:/usr/local/lib:/lib:/usr/lib.
Like Linux, Mac does an exact name match – no minor version rendezvous. Unlike Linux, libraries can record their full semantic version internally, and a “compatibility” version. The compatibility version gets copied into an application at link time, and says the application requires at least that version at runtime.
For example, libfoo.1.dylib with full version 1.2.3
should have a compatibility version of 1.2.0 according to the rules of
semantic versioning. An application linked against it would refuse to
load libfoo with lesser minor version, like 1.1.5. At load time, the
user would see a clear error:
dyld: Library not loaded: libfoo.1.dylib
Referenced from: myapp
Reason: Incompatible library version: myapp requires version 1.2.0 or later,
but libfoo.1.dylib provides version 1.1.5Standard practice is to create symlinks libfoo.so -> libfoo.so.x -> libfoo.so.x.y.z in a shared system directory. The first link (without the version number) is for linking at build time. Problem is, it’s pinned to one version. There’s no portable way to select which version to link against when there are multiple versions installed.
Also, standard practice gives even less care to versioning header files. Sometimes whichever version was most recently installed overwrites them in /usr/local/include. Sometimes the headers are maintained only at the major version level, in /usr/local/include/libfoo-n.
To solve these problems, I suggest bundling all development (linking) library files together into a different directory structure per version. Since I advocated earlier that the “total” library version should be bumped whenever the API or ABI changes, the same version safely applies to headers and binaries.
First choose an installation PREFIX. If the system has an /opt directory, pick that, otherwise /usr/local. In this directory, add dynamic and/or static libraries, headers, man pages, and pkg-config files as desired:
$PREFIX/libfoo-dev.x.y.z
├── libfoo.pc
├── libfoo-static.pc
├── include
│ └── foo
│ ├── ...
│ └── ...
├── lib
│ ├── libfoo.so (or dylib or dll)
│ └── static
│ └── libfoo.a
└── man
├── ...
└── ...Linking against libfoo.x.y.z is easy. In a Makefile, set your flags like this:
CFLAGS += -I/opt/libfoo-dev.x.y.z/include
LDFLAGS += -L/opt/libfoo-dev.x.y.z/lib
LDLIBS += -lfoo
# an example suffix rule using the flags
.c:
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $< $(LDLIBS)Pkg-config
can allow an application to express a range of acceptable library
versions, rather than hardcoding a specific one. In a configure script,
we’ll test for the library’s presence and version, and output the flags
to config.mk:
# supposing we require libfoo 1.x for x >= 1
pkg-config --print-errors 'libfoo >= 1.1, libfoo < 2.0'
# save flags to config.mk
cat > config.mk <<-EOF
CFLAGS += $(pkg-config --cflags libfoo)
LDFLAGS += $(pkg-config --libs-only-L libfoo)
LDLIBS += $(pkg-config --libs-only-l libfoo)
EOFThen our Makefile becomes:
include config.mk
.c:
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $< $(LDLIBS)To choose a specific version of libfoo, we can add it to the pkg-config search path and run the configure script:
# make desired libfoo version visible to pkg-config
export PKG_CONFIG_PATH="/opt/libfoo-dev.x.y.z:$PKG_CONFIG_PATH"
./configure
makeTo create pkg-config .pc files for a library, see Dan
Nicholson’s guide. In order to offer both a static and dynamic
library, the best way I could imagine was to release separate files,
libfoo.pc and libfoo-static.pc that differ in
their -L flag. One uses lib and another
lib/static. (Pkg-config’s --static flag is a
bit of a misnomer, and just passes items in Libs.private
in addition to Libs in the build process.)
This section talks about installing dynamic libraries for system-wide loading. Libraries installed for this purpose are not meant to link with at compile time, but to load at runtime.
ELF objects don’t have much version metadata. SONAME is about it. That, combined with the lackluster behavior of loaders on some systems, means the traditional installation technique doesn’t work too well.
Let’s review the traditional way to install ELF libraries, and then a safer method I designed.
Traditional installation method
This way allows a sysadmin to see exactly which versions are installed, and to have multiple major versions installed at once. It doesn’t allow multiple minor versions per major (although usually only the latest minor is needed), and more importantly doesn’t offer protection against loading too old a minor version.
Safer installation method
For version x.y.z, compile libfoo.so with SONAME libfoo.so.x.y
# use compilation flags
-shared -Wl,-soname,libfoo.so.${MAJOR}.${MINOR}Copy libfoo.so to /usr/local/lib/libfoo.so.x.y.z
Backfill minor version symlinks in DEST:
i=0
while [ $i -le "$MINOR" ]; do
ln -fs "libfoo.so.$VER" "$DEST/libfoo.so.$MAJOR.$i"
i=$((i+1))
doneAt the cost of potentially a lot of minor version symlinks, this technique emulates the SunOS and OpenBSD behavior of minor version rendezvous. Also, because the SONAME has major.minor granularity, it will protect against loading too old a minor version.
(As an alternative to the symlinks, FreeBSD has libmap.conf)
Mach-O has more version metadata inside than ELF, so a traditional install works fine here.
For version x.y.z, compile libfoo.dylib with
# use compilation flags
-dynamiclib -install_name "libfoo.${MAJOR}.dylib" \
-current_version ${VER} \
-compatibility_version ${MAJOR}.${MINOR}.0Copy libfoo.dylib to /usr/local/lib/libfoo.x.dylib
It’s important to set the compatibility version correctly so that Mac’s dyld will prevent loading too old a minor version. To upgrade the library, overwrite libfoo.x.dylib with one of a later internal minor release.
For an example of how to build a library portably, and install it conveniently for the linker and loader, see begriffs/libderp. It’s my first shared library, where I tested the ideas for this article.
Written by Joe "begriffs" Nelson