Building RPM Packages with the ECN Template

Overview

This document describes the process of building an RPM package from source using the ECN spec template file.  The template reduces the amount of editing required to produce a file that will unpack, configure, compile, and build a package from a file containing a package's source.  Examples will be used from the spec file for GCC version 2.8.0.

RPMBUILD Environment

The top directory for all files used and produced by rpmbuild is either ${HOME}/rpmbuild, or the directory defined as %_topdir in ${HOME}/.rpmmacros.  Unless specifically noted, directories mentioned later in this document are located below this point.

Source files

Copy all source files into the SOURCES directory.  The files should be in either tarball (compressed .tar) or zip format, and unedited.  The build process will try to determine the correct unpacking method based on the file.  Any changes should be applied by patching.  All patch files created must also be copied into the SOURCES directory.

The SPEC file

The spec file is the workhorse of the RPM building process.  It provides information on the software being packaged, the instructions to build the package, and where and how the software is installed.  Since spec files can become quite complicated, and to standardize the package building process as much as possible, a template has been developed to make packaging software easier.  The file is template.spec, and is located in the ECN source file directory.

Editing the Template File

The ECN template file contains macros to ease the building of packages.  Most software can be built and packaged with little change, other than filling out the configuration block and the %changelog section.  The template is designed to build packages using either a configure script or cmake, based on files contained in the tarball.  It is also set up to automatically add the version to the package name, add a Provides: tag with the package name and version, and create a list of files to be included in the RPM.
The template.spec file should be copied into the SPECS directory and renamed to reflect the package being built.  The version of the package should not be included in this name, and the name should have the extension .spec.  This file will be edited to perform unpacking, configuring, building, installing, and packaging the software.  It will also be used to create a source RPM (SRPM) that can be used to update the software in the future.  In the example, the file is gcc.spec.

The configuration block

The configuration block is where you will set up the building and packaging process by defining the following macros:

• pkg_name - the 'base' name of the package (no version or sub-package included)
• pkg_version - the version of the software package
• pkg_release - the release number, or version of the spec file
  • The architecture and 'ecn' will be appended to this in the final release field
• pkg_summary - short description of the software
• pkg_description - longer description of the software's function
  • If multi-line, delete this definition and place it in the %description section of the spec file
• pkg_builder - the name of the packager
• pkg_builder_email - email address of the packager
• pkg_license - licensing information for the software
• pkg_URL - the web site that hosts the software
• pkg_source - ideally, the full URL of the source file
  • at minimum, the file name of the source file, although the complete path is preferred
  • the file name must match the name of the file in SOURCES

Here is the configuration block for GCC version 8.2.0:

%define   pkg_name           gcc
%define   pkg_version        8.2.0
%define   pkg_release        1
%define   pkg_summary        Gnu Compiler Collection
%define   pkg_description    The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages (libstdc++,...). GCC was originally written as the compiler for the GNU operating system. The GNU system was developed to be 100% free software, free in the sense that it respects the user's freedom.
%define   pkg_builder        J. Nelson Howell
%define   pkg_builder_email  nelson@purdue.edu
%define   pkg_license        GPLv3+
%define   pkg_URL            https://gcc.gnu.org/
%define   pkg_source         https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.xz

If the software is simply built, using either configure or cmake, this should be sufficient to build and package it. 

The %changelog section

At the end of the spec file is the %changelog section.  This records any changes to the spec file.  It has a specific format that allows the RPM package manipulation commands to recognize it.  The date format is the output of date +'%a %b %d %Y'.   Entries are added at the top of the list (newest first) in the following format:

* date builder_name <builder_email> version-release
- Description of build, or update notes

For the GCC example, which has updates, the %changelog is:

* Tue Jul 26 2018  J. Nelson Howell <nelson@purdue.edu> 8.2.0-1.el7.ecn
- Updated from new sources
* Tue Jun 12 2018  J. Nelson Howell <nelson@purdue.edu> 7.2.0-1.el7.ecn
- Updated from new sources
* Mon Jun 11 2018  J. Nelson Howell <nelson@purdue.edu> 7.1.0-1.el7.ecn
- Initial build  from sources

Building the Package

The rpmbuild command is used to build the package once the spec file is complete.  To build the GCC package, the command is:

rpmbuild -bb gcc.spec

This will unpack the source, apply any necessary patches, configure and compile the software, install it into BUILDROOT, and create a binary RPM package file.  The resulting RPM file for the gcc example given would be:

gcc8.2.0-8.2.0-1.el7.ecn.x86_64.rpm

Indicating that the package name is gcc8.2.0, the version is 8.2.0, and the release is 1.el7.ecn.x86_64.  This file will be written to the RPMS/x86_64 directory.

Options to the rpmbuild command

A few options are available to modify the rpmbuild procedure.  To simply unpack the software into BUILD and then exit, use the flag -bp.  This can be used to edit files if needed, and create a patch file.  It also provides a test of the unpacking method and can be used to inspect the package's source files.  To test the compilation phase of the build without creating an RPM file, -bc is used.  If an SRPM is needed (one should be built after the binary package build is successful) -bs will create it, or -ba will build both the RPM and SRPM files in one command.

ECN macros in the template file also allow a lua file for lmod (modules) to be created, a separate -local sub-package that contains symbolic links to /usr/local for inclusion in the standard execution and library paths, and a -pkg sub-package for installation on a package server.  The flags and their arguments are:

--with local: create a -local sub-package containing symbolic links to /usr/local

--with modules: add a .lua file to be used with lmod and the module command

--define 'packagedir package_directory_on_server' :
   - create a -pkg sub-package containing the software that will be installed on a package server in package_directory_on_server
   - create an RPM that will contain symbolic links to the software on the package disk

GCC is an example using all of these.  The base package (gcc2.8.0) contains only links to the package disk and the .lua file, and is installed on  all hosts.  The -pkg sub-package is installed only on the package servers, in /export/package/special/GCC/8.2.0.  The -local sub-package contains links to /usr/local, and is optionally installed.  Additionally, an SRPM is created containing all of the files required for building the package..  All of this is done with the following command:

rpmbuild -ba define 'packagedir /export/package/special/GCC' --with local --with modules gcc.spec

The resulting files are

RPMS/x86_64/gcc8.2.0-8.2.0-1.el7.ecn.x86_64.rpm
RPMS/x86_64/gcc8.2.0-pkg-8.2.0-1.el7.ecn.x86_64.rpm
RPMS/x86_64/gcc8.2.0-local-8.2.0-1.el7.ecn.x86_64.rpm
SRPMS/gcc8.2.0-8.2.0-1.el7.ecn.src.rpm

The SRPM is saved in the ECN source tree for any future changes to the package.

Special Cases

Sometimes a software package will not build in a standard manner, and requires special handling.  In this case, more rigorous editing of the template spec file may be needed.  Some examples are:

• Different, non-standard unpacking directory
  • If the software does not unpack into name-version, edit the name in the %setup macro with -n option.  This does not affect the RPM package name or version.
  • If the software unpacks into the current directory (tarbomb), use -c instead of -n so that a new directory is created for it.
• Packages contain their own installation programs or scripts
  • You may be required to completely rewrite the %build and %install sections to run the appropriate commands.
  • If the unpacking is also non-standard, you will need to edit the %prep section as well.
• Custom editing of package files required
  • Sometimes a file, or files,  in the package may require editing for localization or for fixing errors.  Create a patch file or files with all applicable changes and include it in SOURCES.
• The package requires other files that are not included in the source distribution
  • Copy the files to SOURCES, add a SourceN: tag for the file, and add the following command in %prep to copy them into BUILDROOT if it will not be used in the building process:
    cp %{SOURCEN}  %{buildroot}
    
    or  add the following command if the file will be used in the building process:
    cp %{SOURCEN}  .
     
• The package requires other packages to be installed for use or compiling
  • The Requires: tag should be used to ensure that all requisite packages are installed.  This also prevents accidentally removing a package that may break the functionality of the software.
  • If a package is only required to build the RPM, but not for runtime use, the BuildRequires: tag should be used.  An example is a -devel RPM required to provide headers for compilation, but not needed to run the software.
• Pre or post installation or uninstallation scripts
  • %pre, %post, %preun, or %postun sections contain scripts for these situations.  These are used for additional configuration or operations to be performed at installation or removal time, that cannot performed by the building process.

Creating patch files

Since the original source package should remain unedited (pristine), any changes performed on files should use patch to make the changes.  The easiest way to do this is to run rpmbuild -bp to generate the build tree, then make a copy of this tree, renaming it to something like package-version.orig.  Then make any changes within the first copy, and from the BUILD directory run diff -ru package-version.orig package-version > ../SOURCES/patch_description.patch.

In the spec file add the tag

Patch0: patch_description.patch

and in the %prep section add

%patch0 -p 1

In the GCC example, the patch file is named dependency.patch, so the tag is:

Patch0: dependency.patch

This process keeps the original source files available, and provides an audit trail of any changes made to it for fixing errors, or for localization.  Many times, the patches are applicable to later versions as well, and the patch files can be reused with little or no changes.  If multiple patches are to be applied, increment the patch number for each patch's tag and macro (Patch1/%patch1:Patch2/%patch2:. . .)

Updating Packages

When updating a package, rebuilding the spec file from the template is usually not necessary.  Running rpm -i SRPM-file will unpack a copy of the software into the build environment.  This includes the spec file and any files that are in  SOURCES.  To update the package, copy the new tarball into SOURCES, edit the Source: tag(s) in the spec file to reflect the new file name(s), change the version number (and reset the release number if applicable), add the update to the top of the %changelog section, and rebuild the package with rpmbuild.  If patches were applied, they should be reviewed.

Appendix A: The ECN SPEC Template File (template.spec)

# Disable pre-compiling of python files
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')

# Disable creation of debug package
%global         debug_package %{nil}

# Create links to /usr/local
%define         make_local      0%{?_with_local:1}

# Create .lua file for modules
%define         make_modules    0%{?_with_modules:1}

# Create a package disk version
%define         make_package    0%{?packagedir:1}

################################################
#
#   CONFIGURATION BLOCK
#  
#   Define all variables in this block
#   Add ***LONG DESCRIPTION*** below
#   Add new changelog entry
#
#   Optional: or as needed
#   Add Requires: and/or BuildRequires: tags
#   Modify %%prep, %%build, or %%install scripts
#   Modify %%setup script if there is a name match problem
#   Create and add Patch: files and instructions
#   Add 'module load' commands if needed
#
###############################################

%define   pkg_name          
%define   pkg_version       
%define   pkg_release        1
%define   pkg_summary       
%define   pkg_description   
%define   pkg_builder       
%define   pkg_builder_email 
%define   pkg_license       
%define   pkg_URL           
%define   pkg_source        

###############################################
#
#       END CONFIGURATION BLOCK
#
###############################################

%define         moduledir /etc/modulefiles/Linux/%{pkg_name}

# Preamble
Version:        %{pkg_version}
Name:           %{pkg_name}%{version}
Release:        %{pkg_release}%{?dist}.ecn
Summary:        %{pkg_summary}
Packager:       %{pkg_builder} (%{pkg_builder_email})
License:        %{pkg_license}
URL:            %{pkg_URL}
Source0:        %{pkg_source}
Provides:       %{pkg_name} = %{version}

%if %{make_package}
%define         _prefix %{packagedir}/%{version}
%define         pkg_prefix /package/%{pkg_name}/%{version}
%else
%define         _prefix /opt/%{pkg_name}/%{version}
%define         pkg_prefix %{_prefix}
%endif

%define         modulefile %{moduledir}/%{version}.lua
%define         filelist /tmp/rpmbuild-%{pkg_name}-%{version}.files
%define         local_filelist /tmp/rpmbuild-%{pkg_name}-%{version}.local
%define         locallinks /tmp/rpmbuild-%{pkg_name}-%{version}.links

%description
%{pkg_description}

%if %{make_local}
%package local
Group:          ECN Local Links
Summary:        Links to /usr/local
Provides:       %{name}-local = %{version}
Requires:       %{name} = %{version}
Provides:       %{pkg_name}-local = %{version}
%description local
Links to /usr/local for placing in a standard PATH location
%endif

%if %{make_package}
%package pkg
Group:          ECN Package Disk Installation
Summary:        Installation to /export/package on server
Provides:       %{name}-package = %{version}
Provides:       %{pkg_name}-package = %{version}
%description pkg
Files installed to package disk
%endif

%prep
# This will work with most packages.  Modify as necessary to match unpacked directory name.
%setup -q -n %{pkg_name}-%{version}

%build
# this configuration assumes a simple cmake or ./configure
# modify as needed for the particular package building process
  if [ -f CMakeLists.txt ]
  then
    module load gcc
    module load cmake
    mkdir build
    cd build
    cmake -DCMAKE_INSTALL_PREFIX=%{_prefix} ..
  else
    if [ -x configure ]
    then
      %configure --prefix=%{_prefix}
    else
      echo "Check configuration mechanism"
      exit 1
    fi
  fi

make %{?_smp_mflags}

%install
# this install assumes a simple 'make install'
# modify as needed for the particular package installation process
rm -rf ${RPM_BUILD_ROOT}
if [ -f CMakeLists.txt ]
then
   module load gcc
   module load cmake
   cd build
fi

%make_install

# Get a list of files and links created
/usr/bin/find ${RPM_BUILD_ROOT} -type f -printf '/%%P\n' > %{filelist}
/usr/bin/find ${RPM_BUILD_ROOT} -type l -printf '/%%P\n' >> %{filelist}

# Some packages insist on spaces in file names
# Wildcard them to '?'s
sed -i -e 's/ /?/g' %{filelist}

%if %{make_modules}
# Create .lua file for module()
# Add or change paths and/or environment variables as needed
mkdir -p ${RPM_BUILD_ROOT}%{moduledir}

echo "whatis([==[Configures environment variables for using %{pkg_name} version %{version} ]==])" > ${RPM_BUILD_ROOT}%{modulefile}

if [ -d ${RPM_BUILD_ROOT}%{_prefix}/bin ]
then
   echo 'prepend_path{"PATH","%{pkg_prefix}/bin",delim=":",priority="0"}' >> ${RPM_BUILD_ROOT}%{modulefile}
fi

for MODULELIB in lib lib64
do
   if [ -d ${RPM_BUILD_ROOT}%{_prefix}/${MODULELIB} ]
   then
      echo 'prepend_path{"LD_LIBRARY_PATH","%{pkg_prefix}/'${MODULELIB}'",delim=":",priority="0"}' >> ${RPM_BUILD_ROOT}%{modulefile}
   fi
done
%if !%{make_package}
echo %{modulefile} >> %{filelist}
%endif
%endif

%if %{make_local}
# Build list of files to link for installation in /usr/local
rm -f %{local_filelist} %{locallinks}
for LOCALDIR in bin lib lib64 share include
do
   grep ^%{_prefix}/${LOCALDIR}/ %{filelist} | sed -e 's!%{_prefix}/!!' >> %{local_filelist}
done

for filename in $(cat %{local_filelist})
do
   echo /usr/local/${filename} >> %{locallinks}
   mkdir -p $(dirname ${RPM_BUILD_ROOT}/usr/local/${filename})
   ln -s %{pkg_prefix}/${filename} ${RPM_BUILD_ROOT}/usr/local/${filename}

done
%endif

%if %{make_package}
# Create link from package disk to /opt
mkdir -p ${RPM_BUILD_ROOT}/opt/%{pkg_name}
ln -s %{pkg_prefix} ${RPM_BUILD_ROOT}/opt/%{pkg_name}/%{version}

%files pkg -f %{filelist}
%defattr(-,root,root)
%files
/opt/%{pkg_name}/%{version}
%{modulefile}
%defattr(-,root,root)
%else
%files -f %{filelist}
%defattr(-,root,root)
%endif

%if %{make_local}
%files local -f %{locallinks}
%defattr(-,root,root)
%endif

%doc

%clean
rm -rf %{filelist} %{locallinks} %{local_filelist} ${RPM_BUILD_ROOT}

%changelog
################################################
#
#       Newest first
#       Increment Release: tag and match to this entry
#       Macros will not expand to fill these fields
#
#       date +'%a %b %d %Y' will provide the proper syntax
#       for the date stamp
#
################################################       
* date builder builder-email version-release
- Initial build from sources

Appendix B: SPEC File Sample (gcc.spec)

Edits in bold

# Disable pre-compiling of python files
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')

# Disable creation of debug package
%global         debug_package %{nil}

# Create links to /usr/local
%define         make_local      0%{?_with_local:1}

# Create .lua file for modules
%define         make_modules    0%{?_with_modules:1}

# Create a package disk version
%define         make_package    0%{?packagedir:1}

################################################
#
#   CONFIGURATION BLOCK
#  
#   Define all variables in this block
#   Add ***LONG DESCRIPTION*** below
#   Add new changelog entry
#
#   Optional: or as needed
#   Add Requires: and/or BuildRequires: tags
#   Modify %%prep, %%build, or %%install scripts
#   Modify %%setup script if there is a name match problem
#   Create and add Patch: files and instructions
#   Add 'module load' commands if needed
#
###############################################

%define   pkg_name           gcc
%define   pkg_version        8.2.0
%define   pkg_release        1
%define   pkg_summary        Gnu Compiler Collection
%define   pkg_description    The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages (libstdc++,...). GCC was originally written as the compiler for the GNU operating system. The GNU system was developed to be 100% free software, free in the sense that it respects the user's freedom.
%define   pkg_builder        J. Nelson Howell
%define   pkg_builder_email  nelson@purdue.edu
%define   pkg_license        GPLv3+
%define   pkg_URL            https://gcc.gnu.org/
%define   pkg_source         https://ftp.gnu.org/gnu/gcc/gcc-8.2.0/gcc-8.2.0.tar.xz

###############################################
#
#       END CONFIGURATION BLOCK
#
###############################################

%define         moduledir /etc/modulefiles/Linux/%{pkg_name}

# Preamble
Version:        %{pkg_version}
Name:           %{pkg_name}%{version}
Release:        %{pkg_release}%{?dist}.ecn
Summary:        %{pkg_summary}
Packager:       %{pkg_builder} (%{pkg_builder_email})
License:        %{pkg_license}
URL:            %{pkg_URL}
Source0:        %{pkg_source}
Patch0:         dependency.patch
Provides:       %{pkg_name} = %{version}
Requires:       gmp
Requires:       mpfr
Requires:       libmpc
BuildRequires:  gmp-devel
BuildRequires:  mpfr-devel
BuildRequires:  libmpc-devel

%if %{make_package}
%define         _prefix %{packagedir}/%{version}
%define         pkg_prefix /package/%{pkg_name}/%{version}
%else
%define         _prefix /opt/%{pkg_name}/%{version}
%define         pkg_prefix %{_prefix}
%endif

%define         modulefile %{moduledir}/%{version}.lua
%define         filelist /tmp/rpmbuild-%{pkg_name}-%{version}.files
%define         local_filelist /tmp/rpmbuild-%{pkg_name}-%{version}.local
%define         locallinks /tmp/rpmbuild-%{pkg_name}-%{version}.links

%description
%{pkg_description}

%if %{make_local}
%package local
Group:          ECN Local Links
Summary:        Links to /usr/local
Provides:       %{name}-local = %{version}
Requires:       %{name} = %{version}
Provides:       %{pkg_name}-local = %{version}
%description local
Links to /usr/local for placing in a standard PATH location
%endif

%if %{make_package}
%package pkg
Group:          ECN Package Disk Installation
Summary:        Installation to /export/package on server
Provides:       %{name}-package = %{version}
Provides:       %{pkg_name}-package = %{version}
%description pkg
Files installed to package disk
%endif

%prep
# This will work with most packages.  Modify as necessary to match unpacked directory name.
%setup -q -n %{pkg_name}-%{version}
%patch0 -p 1
%build
# this configuration assumes a simple cmake or ./configure
# modify as needed for the particular package building process
if [ -f CMakeLists.txt ]
then
  module load gcc
  module load cmake
  mkdir build
  cd build
  cmake -DCMAKE_INSTALL_PREFIX=%{_prefix} ..
else
 if [ -x configure ]
 then
    %configure --prefix=%{_prefix} --disable-multilib
 else
    echo "Check configuration mechanism"
    exit 1
 fi
fi

make %{?_smp_mflags}

%install
# this install assumes a simple 'make install'
# modify as needed for the particular package installation process
rm -rf ${RPM_BUILD_ROOT}
if [ -f CMakeLists.txt ]
then
   module load gcc
   module load cmake
   cd build
fi

%make_install

# Get a list of files and links created
/usr/bin/find ${RPM_BUILD_ROOT} -type f -printf '/%%P\n' > %{filelist}
/usr/bin/find ${RPM_BUILD_ROOT} -type l -printf '/%%P\n' >> %{filelist}

# Some packages insist on spaces in file names
# Wildcard them to '?'s
sed -i -e 's/ /?/g' %{filelist}

%if %{make_modules}
# Create .lua file for module()
# Add or change paths and/or environment variables as needed
mkdir -p ${RPM_BUILD_ROOT}%{moduledir}

echo "whatis([==[Configures environment variables for using %{pkg_name} version %{version} ]==])" > ${RPM_BUILD_ROOT}%{modulefile}

if [ -d ${RPM_BUILD_ROOT}%{_prefix}/bin ]
then
   echo 'prepend_path{"PATH","%{pkg_prefix}/bin",delim=":",priority="0"}' >> ${RPM_BUILD_ROOT}%{modulefile}
fi

for MODULELIB in lib lib64
do
   if [ -d ${RPM_BUILD_ROOT}%{_prefix}/${MODULELIB} ]
   then
      echo 'prepend_path{"LD_LIBRARY_PATH","%{pkg_prefix}/'${MODULELIB}'",delim=":",priority="0"}' >> ${RPM_BUILD_ROOT}%{modulefile}
   fi
done
%if !%{make_package}
echo %{modulefile} >> %{filelist}
%endif
%endif

%if %{make_local}
# Build list of files to link for installation in /usr/local
rm -f %{local_filelist} %{locallinks}
for LOCALDIR in bin lib lib64 share include
do
   grep ^%{_prefix}/${LOCALDIR}/ %{filelist} | sed -e 's!%{_prefix}/!!' >> %{local_filelist}
done

for filename in $(cat %{local_filelist})
do
   echo /usr/local/${filename} >> %{locallinks}
   mkdir -p $(dirname ${RPM_BUILD_ROOT}/usr/local/${filename})
   ln -s %{pkg_prefix}/${filename} ${RPM_BUILD_ROOT}/usr/local/${filename}

done
%endif

%if %{make_package}
# Create link from package disk to /opt
mkdir -p ${RPM_BUILD_ROOT}/opt/%{pkg_name}
ln -s %{pkg_prefix} ${RPM_BUILD_ROOT}/opt/%{pkg_name}/%{version}

%files pkg -f %{filelist}
%defattr(-,root,root)
%files
/opt/%{pkg_name}/%{version}
%{modulefile}
%defattr(-,root,root)
%else
%files -f %{filelist}
%defattr(-,root,root)
%endif

%if %{make_local}
%files local -f %{locallinks}
%defattr(-,root,root)
%endif

%doc

%clean
rm -rf %{filelist} %{locallinks} %{local_filelist} ${RPM_BUILD_ROOT}

%changelog
################################################
#
#       Newest first
#       Increment Release: tag and match to this entry
#       Macros will not expand to fill these fields
#
#       date +'%a %b %d %Y' will provide the proper syntax
#       for the date stamp
#
################################################       
* Thu Jul 26 2018  J. Nelson Howell <nelson@purdue.edu> 8.2.0-1.el7.ecn
- Updated from new sources
* Tue Jun 12 2018  J. Nelson Howell <nelson@purdue.edu> 8.1.0-1.el7.ecn
- Updated from new sources
* Mon Jun 11 2018  J. Nelson Howell <nelson@purdue.edu> 7.2.0-1.el7.ecn
- Initial build from sources