What happens in BASH when you do Ctrl-C (hint, it's not simply sending a SIGINT)

Question

A little background first - When I do apt-get install downloads from my company internet it provides a high burst of speed (400-500KB/s) for the first 10 seconds or so before dropping down to a tenth of that (40-50KB/s), and then after a few minutes to a truly miserable (4-5KB/s). This makes me think that the sysadmin has implemented some sort of a network throttling scheme.

Now I know that the network is not simply erratic, because if I start an apt-get install foo, Ctrl-C it after 10 seconds and immediately run apt-get install foo again (by doing an up arrow and enter to use bash history), and then keep repeating this process for a few minutes till all packages are downloaded, I can download even large packages very fast. In particular, even after aborting a download with Ctrl-C, apt-get seems to be able to resume the download in the next invocation.

Of course, staring at the screen doing Ctrl-C Up Enter every 10 seconds gets really boring real fast, so I wrote a shell script -

#!/bin/sh
for i in `seq 1 100` ; do
    sudo apt-get install foo -y &
    sleep 10
    sudo kill -2 $!
done

This seems to work. It spawns apt-get, runs it for 10 seconds and then kills (by sending a SIGINT) it and starts it up again. However, it doesn't really work because now apt-get does not resume downloads on subsequent invocations!

An an experiment I ran sudo apt-get install foo from one terminal and then ran kill -2 <PID of apt-get> from another terminal. And even in that case, when I restart apt-get, it does not resume the download.

So clearly a Ctrl-C is not equivalent to SIGINT. And something else is happening when I do Ctrl-C manually which gives apt-get a chance to save the state of the download. The question is - what is it?

Edit

These are the suggestions I have received so far, but no cigars. The mystery deepens! -

1. On sudo kill -2 $! the signal might be going to sudo instead of apt-get. This is not the reason because as mentioned above I also tried sending SIGINT specifically to apt-get's PID and even that prevented apt-get from saving its state.

2. Sudo catches the signal and sends some other signal to apt-get. I tried sending apt-get all the signals I can think of! It still does not resume the download for any of them. It only resumes downloads when I do Ctrl-C to kill it.

3. Apt-get handles SIGINT differently if it is from a script instead of an interactive shell. Again, the "experiment" above proves that this is not true.

Castle

One might speculate that apt-get installs a SIGINT handler that just processes its signal differently depending on whether or not it's an interactive shell it's running on or not... – Appetitive

Well it can't be just that because then running apt-get in one shell and killing it from another should still allow it to capture SIGINT and do whatever it needs to save download state. – Castle

Answer 1

Okay mystery solved! Thanks to the helpful folks over at the Indian Linux Users Group.

The answer here is two-fold -

Firstly, apt-get invokes another program called http for downloading data.

[~] ➔ file /usr/lib/apt/methods/http

/usr/lib/apt/methods/http: ELF 32-bit LSB executable, Intel 80386, version 1
(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15,
stripped

Note that that's an executable, not even a script, probably to support downloading files during system installation when none of perl/python/ruby etc. are yet available.

Secondly, when you press Ctrl-C after running apt-get, the SIGINT gets sent to http, and not to apt-get. When http receives the SIGINT, it saves the download state before shutting down.

Here's the updated script that works perfectly -

#!/bin/sh
for i in `seq 1 100` ; do
    sudo apt-get install foo -y &
    sleep 10
    sudo kill -2 `ps -ae | grep " http" | awk '{print $1}'`
done

Castle

Answer 2

Hint, it's not simply sending a SIGINT).

Yes it is just sending a SIGINT :-) Throttling is what is happening, you've got that right. Here's what I suspect is happening:

• Something is limiting the bandwidth of connections. To track connections it's also including the source port (which is a bad idea IMO) among other parameters

• When you kill apt-get and restart it, it will naturally get a new TCP source port and the evil entity throttling you will think, "Oh, it's a new connection", which it actually is

So how do you speed things up ? Well the real solution would be to use multiple parallel downloads. I never used it myself, but I have heard of a tool called "apt-fast" (actually a bash script itself) which does something like this.

EDIT

After reading the question again I suspect the signal is not sent to apt-get.

sudo apt-get install foo -y &
sudo kill -2 $! # sends signal to sudo, which sends whatever it wants to `apt-get`

So I believe sudo catches the signal and sends something else (sigterm? sighup?) to apt-get.

Jewel

Then why does the download resume when doing an up arrow and enter to use bash history? – Iciness

@Thaddee Tyl Because it's starting a new process, picking a new source port, new TCP connection, new "throttle" start time. Read my answer again :-) – Jewel

I mean, does apt-get store a cache of what it downloads somewhere? Does it normally resume downloads? I wouldn't know, I don't use it on a regular basis. I'm wondering why it wouldn't resume the download when launched from a different terminal, too. – Iciness

@Thaddee Tyl I don't know how apt-get resumes downloads (nor do I care to be honest) but a new TCP connection is what I suspect to be the cause of all this. – Jewel

Well the question is not about how the throttling works (I really don't care), but what is the difference between Ctrl-C and kill -2 pid. – Castle

@Anupam Jain Then you didn't ask the right question. The right question would have been: "what does apt-get do when receiving these signals?" – Jewel

@cnicutar, I asked exactly what I wanted to ask. Please read the question again, especially the last line of the question. – Castle

@Anupam Jain I'm sorry, my eyes felt "apt-get specific stuff" and I stopped reading. Only the title is misleading, I give you that. Still, I don't understand the downvote :-/ Perhaps you should read the title of your question again ? – Jewel

I'm not sure what you mean. The title of the question couldn't have been clearer - what is the difference between Ctrl-C and SIGINT (i.e. kill -2 pid). And your answer does not even try to address the question, hence the downvote. – Castle

@Jewel let us continue this discussion in chat – Castle

Okay this was a good suggestion. However it does not explain the behaviour. Please see the edit to my question. – Castle

Answer 3

Alright, as cnicutar said, it is just sending a SIGINT. Now, the key to what was said is here:

I suspect the signal is not sent to apt-get

which is true. Let me explain.

Running sudo foo starts a sudo process which then(that is after you insert your passwd) invokes foo; it's argument. Once sudo accepts the passwd and invokes foo you're into foo's "space". Whatever you do while waiting for foo to finish is done upon foo and not sudo.

So, sending a CtrlC while waiting for apt to do its job, that signal is sent to apt.

Now, if you start sudo, its pid is stored in $! var. Sending a kill signal to that pid/var is sending a kill signal to sudo, not apt which was later started by sudo. If you want to send a kill signal to apt you'd probably want to use pidof utility.

Test for yourself:

sudo do-something
echo $!
pidof do-something

the output should be two different pid s.
I hope that helps a bit.

Coseismal

What you are saying might have been a factor. That's why I did the "experiment" mentioned in my question. Even when I get the PID of apt-get (not of sudo) and kill it from another terminal, it still prevents apt-get from saving the state. Clearly something else is going on. – Castle