[tex-live] bruno@clisp.org: relocatable packages
Karl Berry
karl at freefriends.org
Sun Mar 30 13:56:35 CEST 2003
Bruno has just developed a general system for making packages
relocatable, see his detailed messages below.
Since we've expended a lot of effort on making the TeX system
relocatable (for TeX Live), I thought it would be worth forwarding. It
would be nice if we could all use the same system at some point.
My only comment is that all this infrastructure is very nice, but it
would still not necessarily be obvious to me how to relocate a package,
because I don't know all the files that comprise it.
That is, say I installed a package in /usr/local on my machine (thus it
is intermixed with zillions of others), and then want to copy it to my
home directory on a shared machine. If the package is relocatable, then
I can do this -- if I can find all the files. But how do I know what
the files are?
For that matter, I've often wanted the file list for other reasons, too
-- for instance, to remove an old package.
Perhaps make install could write a manifest to $pkgdatadir/something?
K
Date: Sat, 29 Mar 2003 13:49:35 +0100 (CET)
From: Bruno Haible <bruno at clisp.org>
To: bug-gnulib at gnu.org
cc: Alexandre Duret-Lutz <duret_g at lrde.epita.fr>
cc: Akim Demaille <akim at epita.fr>
Subject: [Bug-gnulib] relocatable packages
Hi all,
It has been a pain for many users of GNU packages for a long time that
packages are not relocatable. It means a user cannot copy a program,
installed by another user on the same machine, to his home directory,
and have it work correctly (including i18n). So many users need to go
through "configure; make; make install" with all its dependencies,
options and hurdles.
rpms and similar package systems solve the "ease of installation"
problem but have a drawback: they hardwire path names, usually to /usr
or /usr/local, with the consequences that
- Users need root privileges to install a binary package,
- It is impossible to have two different versions of the same
package installed (which makes Unix not better than Woe32 in this
aspect).
So I've now made GNU gettext relocatable. Thanks to automake and
gnulib, it is foreseeable that other GNU packages, like GNU tar, GNU
coreutils, GNU libiconv etc., could support --enable-relocatable
without much maintainer effort. With some support in automake the
effort will be even less than it is now.
Here is a writeup:
- how I came to the current implementation in
http://www.haible.de/bruno/gnu/gettext-0.12-pre2.tar.gz
- how the user/installer sees relocatability
- what the maintainer has to do in order to provide --enable-relocatable.
All kinds of comments are welcome.
How to make GNU packages relocatable?
=====================================
Issues:
I) explicit references to data file (in $pkgdatadir), to executables
(in $pkglibdir) etc.
II) shared libraries linked to executables or referenced by other shared
libraries
Solutions:
I) Pathname hacking based on one of
1) A package dependent environment variable.
2) Symlink from /var/installprefix/<package> to where the package is.
(Not Woe32.)
3) An explicit way to get the executable's pathname
- Linux: readlink /proc/self/exe
- Woe32: GetModuleFileName(NULL,...)
- other Unixes (except BeOS): analyze argv[0], using PATH and
realpath.
II) Influence shared library runtime path:
1) A package dependent environment variable (only Linux, Solaris, OSF/1).
2) Symlink from /var/installprefix/<package> to where the package is.
(Not Woe32.)
4) Use a wrapper program around each program that sets the
variable LD_LIBRARY_PATH (most Unixes, or LIBPATH on AIX, or
SHLIB_PATH on HP-UX, or LIBRARY_PATH on BeOS, or
DYLD_LIBRARY_PATH on MacOS X, or PATH on Woe32).
5) Force sysadmin to modify /etc/ld.so.conf (not all Unixes).
6) Put all shared libraries into $bindir. (Only Woe32.)
Evaluation/Drawbacks:
1) usability problem
2) Installation requires root privilege.
3) ok
4) expensive at runtime but easy to use
5) Installation requires root privilege.
6) ok
Optimal solution:
Linux: I.3 + II.4
Woe32: I.3 + II.6
Relocatability of libraries:
Scenario:
1) Install library L,
2) Move it to a new prefix,
3) Install package P that uses L,
4) Move L and P to new locations.
Does P still work? There are three cases:
4a) Move P but keep L where it is.
4b) Move P and L independently (assumes they were installed in different
prefixes in step 3).
4c) Move P and L together to a new location (assumes they were installed
in the same prefix in step 3).
Evaluation if L is a shared library:
4b) will never work because P will not know where to find L - issue II -
except if the user sets LD_LIBRARY_PATH or similar. Not important.
4c) should work. Issue II is already done because is is the same as if
step 2 had been omitted. But for issue I, we see that the data
references of L and of P have to be relocated differently: For L,
it must take into account the installation directory of step 1,
while for P, it must take into account the installation directory of
step 3. Still, the new location of P can be used to compute the new
relocation rules for L.
4a) should work. Issue II can be solved by hardwiring library pathnames,
as is done usually with --enable-rpath. Issue I requires that data
references of L and P are relocated differently, as for case 4c),
but also that L must find its new location without referring to
P's new location. Only works on Linux (via /proc/self/maps) and
Woe32 (via GetModuleFileName in DllMain).
Evaluation if L is a normal ar library:
4a) will never work.
4b) will never work.
4c) should work.
So we implement a solution for 4c) on all platforms and for 4a) on
Linux, Woe32.
configure arguments:
--enable-relocatable
User's, installer's view:
=========================
--enable-relocatable makes the entire installed packages relocatable.
This means, it can be moved or copied to a different location on the
filesystem.
It is possible to make symlinks to the installed and moved programs,
and invoke them through the symlink. It is possible to do the same thing
with a hard link _only_ if the hard linked file is in the same directory
as the real program.
For reliability it is best to give together with --enable-relocatable a
--prefix option pointing to an otherwise unused (and never used again)
directory, for example, --prefix=/tmp/inst$$. This is recommended
because on some OSes the executables remember the location of shared
libraries (and prefer them over LD_LIBRARY_PATH !), therefore such an
executable will look for its shared libraries first in the original
installation directory and only then in the current installation directory.
Installation with --enable-relocatable will not work for setuid / setgid
executables. (This is because such an executable kills its LD_LIBRARY_PATH
variable when it is launched.)
The runtime penalty and size penalty are nearly zero on Linux (just one
system call more when an executable is launched), and small on other
systems (the wrapper program just sets an environment variable and execs
the real program).
Maintainer's view:
==================
How to add relocatable support to a package?
I assume you use automake. (Without automake the Makefile.in changes
get so big that I'd recommend to switch to automake first.)
I assume you have a lib/ directory in which you have auxiliary modules,
from gnulib and others.
* config.libpath
* reloc-ldflags
* install-reloc
Add these scripts to your aux directory (AC_AUX_DIR argument), or to
the top level directory if you have none.
- config.libpath is used during autoconfiguration to determine the handling
of shared library search paths by your system. It is a separate script so
support for new systems can be added to it without the need to regenerate
all configure files.
- reloc-ldflags is invoked by the Makefiles to determine some additional
flags to be used when linking, for systems supporting this method of
relocatability.
- install-reloc is invoked by the Makefiles to install a program in a
relocatable way.
* lib/ sources
You need to add the following source files:
+ For use by your programs:
- relocatable.h, relocatable.c
Provides the function relocate() that transforms an installation-time
pathname to a run-time pathname.
- progname.h, progname.c
Provides the function set_program_name() that you must call. It
performs the preparations for relocate() to work later.
- xreadlink.h, xreadlink.c
Part of set_program_name()'s job is to determine the full pathname
of the program at run time. On Linux, it does this by reading the
symlink /proc/self/exe provided by the kernel. The xreadlink()
function is a safe variant of readlink(), for this purpose.
- canonicalize.h, canonicalize.c
Part of set_program_name()'s job is to determine the full pathname
of the program at run time. On other platforms than Linux, this is
done by looking at argv[0]. But argv[0] can be a symlink, which must
be resolved first. If we're unlucky enough that this symlink contains
"../" components, simple string operations are not enough for
resolving it; we need the full power of realpath() or
canonicalize_file_name().
- xmalloc.h, xmalloc.c, xstrdup.c
Failsafe memory allocation. The modules above need it.
- error.h, error.c
Pretty error signalling routine. xmalloc needs it.
- pathmax.h
Usual pathname size, needed by canonicalize.c.
- alloca_.h, alloca.c
Memory allocation on the stack, needed by canonicalize.c.
- memmove.c
Needed by canonicalize.c.
- stdbool.h.in
Needed by progname.c.
+ For the wrapper programs:
- relocwrapper.c
This is the wrapper itself. Its purpose is to find out where
the real program is, and invoke it.
- setenv.h, setenv.c, unsetenv.c
The wrapper program needs to set the LD_LIBRARY_PATH variable (or
another, similar variable depending on the OS) to ensure that the
real program will find its shared libraries after it has moved
(together with its shared libraries).
- strerror.c
The wrapper program needs to print an error message when it cannot
execute the real program. It needs strerror().
* lib/Makefile.am
Assuming that your Makefile gathers all modules in a libfoo.a, you'll
have to add:
- to libfoo_a_SOURCES
error.h error.c
pathmax.h
progname.h progname.c
xmalloc.h xmalloc.c xstrdup.c
xreadlink.h xreadlink.c
- to libfoo_a_LIBADD
@ALLOCA@ @LIBOBJS@
(or @LTALLOCA@ @LTLIBOBJS@ if you link libfoo with libtool)
- to EXTRA_DIST
alloca_.h alloca.c
canonicalize.h canonicalize.c
memmove.c
relocatable.h relocatable.c
relocwrapper.c
setenv.h setenv.c unsetenv.c
strerror.c
- the rules for creating alloca.h if necessary
- the rules for creating stdbool.h if necessary
* m4/ macros
Add these files:
m4/alloca.m4
m4/canonicalize.m4
m4/error.m4
m4/lib-ld.m4
m4/onceonly.m4
m4/pathmax.m4
m4/relocatable.m4
m4/setenv.m4
m4/stdbool.m4
m4/strerror.m4
m4/xalloc.m4
m4/xreadlink.m4
* m4/Makefile.am
Add the added files to EXTRA_DIST.
* configure.ac
Ensure it has the following invocations:
AC_RELOCATABLE
gl_FUNC_ALLOCA
gl_CANONICALIZE
gt_FUNC_ERROR_AT_LINE
AC_REPLACE_FUNCS([memmove])
gl_PATHMAX
gt_FUNC_SETENV
gt_STDBOOL_H
gl_FUNC_STRERROR
gl_XALLOC
gl_XREADLINK
* Your sources
1) In every program, add to main() as the first statement (even before
setting the locale or doing anthing related to libintl):
set_program_name (argv[0]);
2) Everywhere where you use a constant pathname from installation-time,
wrap it in relocate() so it gets translated to the run-time situation.
Example:
bindtextdomain (PACKAGE, LOCALEDIR);
becomes:
bindtextdomain (PACKAGE, relocate (LOCALEDIR));
3) In every shell script, add near the beginning the "Support for
relocatability" section from gettextize.in. You can take the two
functions without modifications; only the following need to be
adapted:
if test "@RELOCATABLE@" = yes; then
exec_prefix="@exec_prefix@"
bindir="@bindir@"
orig_installdir="$bindir" # see Makefile.am's *_SCRIPTS variables
func_find_curr_installdir # determine curr_installdir
func_find_prefixes
# Relocate the directory variables that we use.
gettext_dir=`echo "$gettext_dir/" | sed -e "s%^${orig_installprefix}/%${curr_installprefix}/%" | sed -e 's,/$,,'`
fi
Here you adapt the definition of orig_installdir, depending on where
the script gets installed. And at the end, instead of gettext_dir you
transform those variables that you need.
* Your Makefile.am
For every program foo, that gets installed in, say, $(bindir), you add
foo_CFLAGS = -DINSTALLDIR=\"$(bindir)\"
if RELOCATABLE_VIA_LD
foo_LDFLAGS = `$(RELOCATABLE_LDFLAGS) $(bindir)`
endif
And the following also also necessary, once per Makefile.am that installs
a program:
# Support for relocatability.
RELOCATABLE_LIBRARY_PATH = $(libdir)
RELOCATABLE_SRC_DIR = $(top_srcdir)/lib
RELOCATABLE_BUILD_DIR = ../lib
RELOCATABLE_CONFIG_H_DIR = ..
@SET_RELOCATABLE@
Here RELOCATABLE_LIBRARY_PATH denotes the directories in which your
package (or any package you rely on, for example, gettext-runtime)
has installed libraries. It is a colon separated list.
RELOCATABLE_SRC_DIR denotes where to find relocwrapper.c and the related
sources, so you need them only once in your package, even if you have
multiple directories which install programs.
RELOCATABLE_CONFIG_H_DIR denotes where to find the config.h which defines
the various HAVE_* macros that relocwrapper.c and the related sources
need.
@SET_RELOCATABLE@ is some black magic that is filled in by the
AC_RELOCATABLE macro.
_______________________________________________
Bug-gnulib mailing list
Bug-gnulib at gnu.org
http://mail.gnu.org/mailman/listinfo/bug-gnulib
More information about the tex-live
mailing list