Varnish DAEMON_OPTS Options Errors

Question

When using inline C with Varnish I've not been able to get /etc/varnish/default
to be happy at startup.

I've tested inline C with varnish for two things: GeoIP detection and Anti-Site-Scraping functions.

The DAEMON_OPTS always complains even though I'm following what other seem
to indicate works fine.

My problem is that this command line start up works:

varnishd -f /etc/varnish/varnish-default.conf -s file,/var/lib/varnish/varnish_storage.bin,512M -T 127.0.0.1:2000 -a 0.0.0.0:8080 -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'

But it errors out with trying to start up from default start scripts:

/etc/default/varnish has this in it:

DAEMON_OPTS="-a :8080 \
             -T localhost:2000 \
             -f /etc/varnish/varnish-default.conf \
             -s file,/var/lib/varnish/varnish_storage.bin,512M \
             -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"

The error is:

# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error:
Unknown parameter "'cc_command".

If I try change the last line to:

-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"

It's error is now:

# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error: Unknown storage method "hared"

It's trying to interpret the '-shared' as -s hared and 'hared' is not a storage type.

For both GeoIP and the Anti-Site-Scrape I've used the exact recommended daemon options
plus have tried all sorts of variations like adding ' and '' but no joy.

Here is a link to the instruction I've followed that work fine except the DAEMON_OPTS part.
http://drcarter.info/2010/04/how-fighting-against-scraping-using-varnish-vcl-inline-c-memcached/

I'm using Debian and the exact DAEMON_OPTS as stated in the instructions.

Can anyone help with a pointer on what's going wrong here?

Tamarind

Answer 1

Even if Jacob will probably never read this, visitors from the future might appreciate what I'm going to write.

I believe I know what's wrong, and it looks like a Debian-specific problem, at least verified on Ubuntu 11.04 and Debian Squeeze.

I traced the execution from my /etc/default/varnish that contains the $DAEMON_OPTS to the init script. In the init script /etc/init.d/varnish, the start_varnishd() function is:

start_varnishd() {
    log_daemon_msg "Starting $DESC" "$NAME"
    output=$(/bin/tempfile -s.varnish)
    if start-stop-daemon \
        --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
        -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
        log_end_msg 0
    else
        log_end_msg 1
        cat $output
        exit 1
    fi
    rm $output
}

So I modified it to print the full start-stop-daemon command line, like:

 start_varnishd() {
    log_daemon_msg "Starting $DESC" "$NAME"
    output=$(/bin/tempfile -s.varnish)
+   echo "start-stop-daemon --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"
    if start-stop-daemon \
        --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
        -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
        log_end_msg 0

So I got a command line echoed on STDOUT, and copied-pasted it into my shell. And, surprise! It worked. WTF?

Repeated again to be sure. Yes, it works. Mmh. Could it be another of those bash/dash corner cases? Let's try feeding the start-stop-daemon command line to bash, and see how it reacts:

start_varnishd() {
    log_daemon_msg "Starting $DESC" "$NAME"
    output=$(/bin/tempfile -s.varnish)
    if bash -c "start-stop-daemon \
        --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
        -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then
        log_end_msg 0
    else
        log_end_msg 1
        cat $output
        exit 1
    fi
    rm $output
}

Yes, it works just fine, at least for my case. Here's the relevant part of my /etc/default/varnish:

...
## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request.  Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :6081 \
             -T localhost:6082 \
             -f /etc/varnish/geoip-example.vcl \
             -S /etc/varnish/secret \
             -s malloc,100M \
             -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o %o %s'"
...

I've seen posts where someone tried to work around this problem by moving the compile command into a separated shell script. Unfortunately that doesn't change the fact that start-stop-daemon is going to pass the $DAEMON_OPTS var through dash, and that will result in mangled options.

Would be something along the lines of:

-p 'cc_command=exec /etc/varnish/compile.sh %o %s'"

And then the compile.sh script as:

#!/bin/sh
cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o $@

but it doesn't work, so just patch your init scripts, and you're good to go! Hope you can find this information useful.

Walburga

There might be a special syntax or workaround to make this work on dash too. Unfortunately, I don't know much about dash. – Walburga

Take a look at this perhaps? #1661693 – Unit

I have to say that I didn't understand what's the proposed solution there at the end of the story... :-| – Walburga

This answer works! thank you so much! the proposed solution is to patch /etc/init.d/varnish with bash – Illinium

Answer 2

You can try using :- DAEMON_OPTS="-a :8080 \ -T localhost:2000 \ -f /etc/varnish/varnish-default.conf \ -s file,/var/lib/varnish/varnish_storage.bin,512M \ -p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"

Ceremonious

Answer 3

Obviously, your startup script interpreting the DAEMON_OPTS is not prepared for whitespace (even within single quotes). At my Fedora (15) installation, the suggested solution works fine; the arguments get interpreted correctly because the "$*" bash parameter is passed in /etc/init.d/varnish and in /etc/init.d/functions in daemon().

Did you get your startup scripts from a package or did you make custom scripts?

Arkhangelsk

Answer 4

This isn't directly related to the question, but you may find yourself here if you are working through the Varnish Tutorial - Put Varnish on port 80.

For recent installs of Varnish on Debian systems the configuration for varnishd startup options can be found in /etc/systemd/system/multi-user.target.wants/varnish.service. The documented way of changing the port via /etc/default/varnish still exists, but is no longer functional unless you change your system to use init scripts rather than systemd.

After you've changed your options in /etc/systemd/system/multi-user.target.wants/varnish.service, don't forget to run systemctl daemon-reload, which will catalog the changes for executing the program.

Ecclesiastes