rpmbuild change compression format

Question

I try to pack some map files for our geoserver in an internal rpm package. For the build part, this is just to copy the files. I think this works as expected. But it takes terribly long to pack those 20GB of images.

I've read that rpm internally compresses the data and that this can be done with several different compression algorithms. But, I don't have a clue which compression my rpm chooses and how I can influence this. I could not find any options for the rpmbuild command, nor for the specfile nor for the general rpm options I can list with rpmbuild --showrc

I´m not very experienced with rpmbuild and specfiles, but after reading lots of man pages and tutorials on rpm.org I have no further ideas.

The specfile I use looks like:

%define debug_package %{nil}

%global mapsversion 0.9
# If this is a snapshot, put the date here and uncomment
#global snapshot_version 20100519

# This is the version in a form acceptable
# an an RPM version string (i.e. no '-')
# Hier werden die Makros definiert.
%global rpmversion %(echo %{mapsversion} | tr '-' '_')
%global pkgversion %{mapsversion}%{?snapshot_version:-SNAPSHOT}
%global pkgname %{name}

Name:           geoserver-maps-part2
Version:        %{rpmversion}
Release:        1%{?dist}
Summary:        Swiss Maps for GeoServer
Group:          Application/ourApp
License:        Copyright (c) 2011
URL:            http://doc.polyalert.local
#Source0:        %{name}-%{version}.tgz
BuildArch:  noarch
BuildRoot:  %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
Requires:   geoserver

%define mapshome /opt/geoserver/swisstopo
%define mapssource /home/user/polyalert_env/geoserver/swisstopo

%description
Swiss Maps for GeoServer

%prep

%build
/bin/true

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT%{mapshome}
cp -a %{mapssource}/pk100 $RPM_BUILD_ROOT%{mapshome}

%clean
rm -rf $RPM_BUILD_ROOT

%pre

%post

%preun


%files
%{mapshome}/pk100

%changelog
* Tue Feb 14 2012 user - 1.0
- First version of specfile

I call rpmbuild like this:

rpmbuild -bb --define "_topdir $TOP_DIR" --define "_gpg_name ourkey" --define "_signature gpg" --sign $TOP_DIR/SPECS/$SPEC_FILE_NAME $RPM_BUILD_PARAMETER

Any suggestions?

Paginal

The only compression I am aware of is /usr/lib/rpm/brp-compress will compress man pages. – Sumerlin

since the RPMs are much much smaller than the provided input data, I'm pretty sure that all the data in there is compressed. I even figured out that zip archives (in my case jar files) contained in an RPM will be compressed again, so that the jar installed from the RPM is different compared to the input (even if its content is identical).. – Paginal

OK, I should have prefaced that with "the only user-controllable compression" then... ;) – Sumerlin

Answer 1

Eleven years later...

In the comments it was noted that you can now put the option:

%define _source_payload w0.ufdio
%define _binary_payload w0.ufdio

To totally bypass the compression. I verified on RPM 4.11.3 / CentOS 7.9 (the fdio option didn't work).

With rpm -qp --qf "%{PAYLOADCOMPRESSOR}\n" <filename>, I was able to confirm it said (none) compared to the previous build's xz.

An internet search finds nothing useful concerning these flags and the most popular result seems to come right back here...

---

The original answer:

I was working with some RPM stuff today and accidentally stumbled upon the answer for ya!

Put these in your spec file:

%define _source_payload w0.gzdio
%define _binary_payload w0.gzdio

That will still use gzip but pass it -0 for a level, which should just store. On my RPM, it made it grow from 21MB to 76MB, so I'm pretty sure this is your answer!

BTW, I found that in one of the macro files - you can also do bzdio and any number from 0 to 9 to use bzip2. This was on RHEL4; later versions of RPM seem to support more compression options; but again, for what you want, the above should be what you need.

Sumerlin

would like to see someone validate this on Fedora 20 or RHEL7. – Moisten

Those should support xz - see here. – Sumerlin

Hm, is there a value not to use a compression algorithm at all?.. – Belva

@imz--IvanZakharyaschev that's what this answer is - gzip at 0-level compression is no compression. I don't believe there's a way to "not send it to a compression engine." – Sumerlin

w.fdio or w.ufdio must be the I/O types that don't send to a compression engine at all. And my experiments seem to confirm that they are a bit faster on a non-compressible data. Logically, fdio is simpler than ufdio, but perhaps not in the read/write implementations, but just in how the path is opened (ufdio can interpret URLs), as I could see from the sources. (BTW w0.lzdio, w0.xzdio, w0T8.xzdio are way slower than w0.gzdio, w.ufdio, or w.fdio.) – Belva

@imz--IvanZakharyaschev never heard of it and can't easily find anything in a search nor do I have a box to play with it right now. – Sumerlin

Answer 2

Please check the file /usr/lib/rpm/macros in your build machine, (the file maybe diff in path), it has a total support list of compression methods there: e.g.:

329 #       Compression type and level for source/binary package payloads.
330 #               "w9.gzdio"      gzip level 9 (default).
331 #               "w9.bzdio"      bzip2 level 9.
332 #               "w7.xzdio"      xz level 7, xz's default.
333 #               "w7.lzdio"      lzma-alone level 7, lzma's default
334 #
335 #%_source_payload       w9.gzdio
336 #%_binary_payload       w9.gzdio

so here just as Aaron said, you can set it here for universal, or set specifically for your proj. spec.

Jephum

New versions of rpm can do multiple threading for xz with "w7T16.xzdio" xz level 7 using 16 thread (xz only). Looking at my /usr/lib/rpm/macros! – Nematode

Answer 3

I ran into the same issue with Ant building a runnable Jar RPM with Spring Boot Loader complaining of this:

Caused by: java.lang.IllegalStateException: Unable to open nested entry 'BOOT-INF/lib/accessors-smart-1.2.jar'. It has been compressed and nested jar files must be stored without compression. Please check the mechanism used to create your executable jar file

My ant build task was like this:

<exec executable="rpmbuild"  failonerror="true"> 
  <env key="version" value="${fullversion}" /> 
  <arg value="-ba" /> 
  <arg value="--clean" />
  <arg value="${specfile}" />
</exec>

My solution to build an RPM with a runnable JAR was to disable the repacking, setting the macro definitions on the spec file did not do it for me.

Adding this to the spec file was what worked for me:

#Disable jar unpacking
%define __jar_repack 0

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=219731

Thanks to the previous posters for helping to focus in on the issue too.

Erion

Trilateral

Answer 4

I have used "%define _binary_payload w9.xzdio" on RHEL 6.6. As I understand, the default compression tool used in RHEL 6 is xz, but the default compression level appears to be 2, even though 7 is supposed to be xz's default. I kicked it up to 9 and some giant RPMs went from 653MB to 439MB. I was able to save a total of 1 gigabyte over the default compression.

Shipp

This was noted in my comment above from Sep 2014. – Sumerlin