Build environment isolation and file system diffing

Question

Alright so after trying to chase down the dependencies for various pieces of software for the n-th time and replicating work that various people do for all the different linux distributions I would like to know if there is a better way of bundling various pieces of software into one .rpm or .deb file for easier distribution.

My current set up for doing this is a frankenstein monster of various tools but mainly Vagrant and libguestfs (built from source running in Fedora because none of the distributions actually ship it with virt-diff). Here are the steps I currently follow:

1. Spin up a base OS using either a Vagrant box or by create one from live CDs.
2. Export the .vmdk and call it base-image.
3. Spin up an exact replica of the previous image and go to town: use the package manager, or some other means, to download, compile, and install all the pieces that I need. Once again, export the .vmdk and call it non-base-image.
4. Make both base images available to the Fedora guest OS that has libguestfs.
5. Use virt-diff to diff the two images and dump that data to file called diff.
6. Run several ruby scripts to massage diff into another format that contains the information I need and none of the stuff I don't like things in /var.
7. Run another script to generate a command script for guestfish with a bunch of copy-out commands.
8. Run the guestfish script.
9. Run another script to regenerate the symlinks from diff because guestfish can't do it.
10. Turn the resulting folder structure into a .deb or .rpm file and ship it.

I would like to know if there is a better way to do this. You'd think there would be but I haven't figured it out.

Hellish

So you want a single file installer with all dependencies like Google Chrome, Opera does. Am I right ?. Are you interested in portable app (or portable directory) ? – Radian

@totti: Ideally a chroot like environment that can be shipped and run on any compatible linux environment. – Hellish

Answer 1

I would definitely consider something along the lines of:

A)

• yum list (select your packages/dependencies whatever)
• use yumdownloader on the previous list (or use th pkgs you have already downloaded)
• createrepo
• ship on media with install script that adds the cd repo to repolist, etc.

or B)

first two steps as above, then pack the rpms into an archive build a package that contains all of the above and kicks off the actual install of the rpms (along the lines of rpm -Uvh /tmp/repo/*) as a late script (in the cleanup phase, maybe). Dunno if this can be done avoiding locks on the rpm database.

Consecration

Answer 2

I think you reached the point of complexity - indeed a frankenstein monster - where you should stop fearing of making proper packages with dependencies. We did this in my previous work - we had a set of fabricated rpm packages - and it was very easy and straightforward, including:

• pre/post install scripts
• uninstall scripts
• dependencies

We never had to do anything you just described. And for the customer, installing even a set of packages was very easy!

You can follow a reference manual of how to build RPM package for more info.

EDIT: If you need a single installation package, then create this master packge, that would contain all the other packages (with dependencies set properly) and installed them in the post-install script (and uninstalled them in the uninstall script).

Amimia

I can build RPM packages just fine that is not the issue. The issue is some customers do not have internet access and need a self-contained package. – Hellish

I didn't say easier for the person building those packages. It's definitely easier from the customers perspective if I just give them a single package that they can just install with rpm -i. – Hellish

@davidk01 then you could prepare all those packages as I recommended and then create master one, that would contain all of them and installed them in the post-install script (and uninstalled them in the uninstall script). – Amimia

Answer 3

Is the problem primarily that of ensuring that your customers have installed all the standard upstream distro packages necessary for your package to run?

If that's the case, then I believe the most straightforward solution would be to leverage the yum and apt infrastructure to have those tools track down and install the necessary prerequisite packages.

If you supply a native yum/apt repository with complete pre-req specs (the hard work you've apparently already completed). Then the standard system install tool takes care of the rest. See link below for more on creating a personal repository for yum/apt.

For off-line customers, you can supply media with your software, and a mirror - or mirror subset - of the upstream distro, and instructions for adding them to yum config/apt config.

Yum Creating a Yum Repository in the Fedora Deployment Guide

Apt How To Setup A Debian Repository on the Debian Wiki

Pressurecook

There is no network access. Shipping distributions on a cd or some other medium is possible but then again might as well have one package instead of three different packages. – Hellish

Answer 4

There are mainly 3 steps to make a package with all dependencies (let it be A, B & C).

A. Gather required files.
There are many way to gather files of the main software and its dependencies. In order to get all the dependices and for error free run you need to use a base OS (i.e live system)
1. Using AppDirAssistant
This app is used by www.portablelinuxapps.org to create portable app directory. They scan and watch for the files accessed by the app to find required.
2. Using chroot & overlayfs
In this method you don't need to boot into live cd instead chroot into it.
a. mount the .iso @ /cdrom and
b. mount the filesystem(filesystem.squashfs) @ another place, say @ /tmp/union/root
c. Bind mount /proc @ /tmp/union/root/proc
d. Overlay on it
mount -t overlayfs overlayfs /tmp/union/root -o lowerdir=/tmp/union/root,upperdir=/tmp/union/rw
e. Chroot
chroot /tmp/union/root

Now you can install packages using apt-get or another method (only from the chrooted terminal). All the changed files are stored @ /tmp/union/rw. Take files from there.
3. Using manually collected packages
Use package manager to collect dependencies. For example apt-get install package --print-uris will print download uris for dep packages. Using this uris download packages and extract all (dpkg -x 1.deb ./extracted).

B. Clean garbage files
After gathering files remove unwanted files

C. Pack files
1. Using appImageAssistance
If you manually gathered files then you need to copy appname.desktop file from ./usr/share/applications to root of directory tree. Also copy file named AppRun from another app or extract it from AppDirAssistance.
2. Make a .deb or .rpm using gathered files.

Radian

aufs requires kernel upgrades and other kinds of trickery and unionfs-fuse is buggy as hell and the only thing I managed to do with it was freeze up the box every time I tried to mount some directories. – Hellish

Which OS, Year ?. All above mentioned works well in my Ubuntu 12.10. – Radian

Answer 5

So your customers aren't ever going to install any other software that might specify a different version of those dependencies that you are walking all over, right?

Why not just create your own distro if you're going to go that far?

Or you can just give them a bunch of packages and a single script that does rpm -i dep1 dep2 yourpackage

Talebearer

Saying why not create your own distribution is not very helpful and there is no reason for a full blown distribution. Giving them a bunch of packages and a script could work but if I'm going to do that then might as well package the whole thing in one instead of X, Y, Z packages. – Hellish