Using Private Libraries to Compile and Run Applications

Overview

Software libraries distributed with standard operating system releases do not always provide the necessary functionality for building or running applications.  This may be due to newer libraries being required by the software for additional capabilities or, at times, an older library is required due to a lack of backwards compatibility. 

Often, a new library cannot be installed in the system area (e.g. /usr/lib) because it conflicts with an existing version of that library or it interacts with other system libraries or applications in a detrimental manner.  When this occurs, a private library may be required.

A private library is a library that is built using a non-administrative account.  It is private because it is installed in the user's own file space instead of a system library directory.  This allows anyone to create a custom library and use it without impacting the operating system or other users, or requiring intervention by a system administrator.  This document is written to assist in the use of private libraries to compile and run programs requiring them.

Types of private libraries

Libraries are normally used in two ways.  The first is statically.  This means that the library routines are included in an application's executable code when it is created.  It allows the application to run on any machine with which it is compatible without additional files.  A down side to using statically linked libraries is that any changes to library routines require relinking (and possibly recompiling) applications that reference them.    Static libraries can be recognized by a .a extension.

The second way to use a library is shared.  In this case, the library routines are symbolically linked to an application, but reside in separate files, called shared libraries.  This allows the libraries to be updated independently of the application.  If the routines do not change in their external behavior, the changes are transparent to any code linked to them.  The application does not need to be relinked to use the newer library routines since this is done when it is loaded into memory for execution.  The library files, however, are required to be installed on any machine on which the application is run, must be in a directory that is readable by the application's user and must be within the user account's library search path.  Shared libraries are designated with a .so extension.

Where to install private libraries

Since the system library areas (/usr/lib, /usr/local/lib, etc.) are not writeable by non-administrative accounts,  private libraries must be installed in a directory available to the user.  A good choice is to create a 'lib' directory within the home directory of an account.  This provides a single point to reference the private libraries and can easily be accessed using '$HOME/lib' from any host.  If the library is to be used by other accounts, the directory must be readable by any accounts needing to access it.  The additional accounts can access the library using '~account_owning_library/lib'.  Research groups or classes using private libraries may dedicate a shared space for private libraries if needed.

Compiling packages using private libraries

When compiling (actually linking) packages, the system library directories are normally searched for any required libraries.  To add a private library, the path to it must be included in this list of locations.  This is done in C and C++ by including the library directory (-L) flag along with the path in which the library exists.  This must be repeated for each non-system library directory, which is why putting private libraries in one location is a good idea because then, only one directory path needs to be included.  Individual libraries within the directory(s) are designated in the linking phase with the -l flag.  This is necessary for using both static and shared libraries.  For instructions on the appropriate use of the flags for a specific compiler, the compiler documentation should be consulted.

Running applications with private libraries

If the private libraries are statically linked to an application, the application will run with no further preparation since the library code is already incorporated into the executable file.  However, if a private library is linked dynamically, the library's directory must be included in the library search path so that the application can locate it.

The search path for run time libraries contains, by default, the following system directories:

/lib
/usr/lib
/usr/local/lib

To use a private shared library, the directory containing the library must be added to this list.  An easy way to do this is to set the environment variable LD_LIBRARY_PATH.  The following will set the path for an individual invocation of an application:

 LD_LIBRARY_PATH=path_to_library:$LD_LIRARY_PATH application

This will prepend the directory path_to_library to any existing search path and then load and execute the application.

To modify LD_LIBRARY_PATH for all applications, it can be set within an account's startup scripts.  This will cause all applications, including those already on the system, to use the new search path.  The instructions on setting this variable are specific to the shell used by the account.  This method should only be used if there is no conflict with operating system library names since it may affect the performance of other programs using those libraries.