Core dump file is not generated

Question

Every time, my application crash a core dump file is not generated. I remember that few days ago, on another server it was generated. I'm running the app using screen in bash like this:

#!/bin/bash
ulimit -c unlimited
while true; do ./server; done

As you can see I'm using ulimit -c unlimited which is important if I want to generate a core dump, but it still doesn't generate it, when I got an segmentation fault. How can I make it work?

Nepenthe

it doesn't look the case, but be careful if you use sudo (and probably other kind of subshells): in ulimit -c unlimited ; sudo ./server-crashing , the new limit won't have effect when server-crashing crashes. – Amorphous

Check journalctl -f I found e.g. systemd-coredump[345476]: Resource limits disable core dumping for process 345459 – Pryer

Answer 1

Make sure your current directory (at the time of crash -- server may change directories) is writable. If the server calls setuid, the directory has to be writable by that user.

Also check /proc/sys/kernel/core_pattern. That may redirect core dumps to another directory, and that directory must be writable. More info here.

Baseborn

Yes, core_pattern is tricky. When Arch Linux switched to systemd, I ran into that problem. Now I am using echo "core" > /proc/sys/kernel/core_pattern to get my core dumps as expected (by default it was written to the systemd-journal). You can spend lots of time to figure that out... – Healey

@PhilippClaßen: I need. That's what I did too. Figuring out how to do it the other way is too hard I guess. I tried, but I couldn't. – Forelady

As a side note, that info is in man 5 core manual page. The pattern supports %p and other such flags. – Sharynshashlik

In addition to making sure those directories are writable, make sure you're looking in that location to find the core file – Jaimie

Answer 2

This link contains a good checklist why core dumps are not generated:

• The core would have been larger than the current limit.
• You don't have the necessary permissions to dump core (directory and file). Notice that core dumps are placed in the dumping process' current directory which could be different from the parent process.
• Verify that the file system is writeable and have sufficient free space.
• If a sub directory named core exist in the working directory no core will be dumped.
• If a file named core already exist but has multiple hard links the kernel will not dump core.
• Verify the permissions on the executable, if the executable has the suid or sgid bit enabled core dumps will by default be disabled. The same will be the case if you have execute permissions but no read permissions on the file.
• Verify that the process has not changed working directory, core size limit, or dumpable flag.
• Some kernel versions cannot dump processes with shared address space (AKA threads). Newer kernel versions can dump such processes but will append the pid to the file name.
• The executable could be in a non-standard format not supporting core dumps. Each executable format must implement a core dump routine.
• The segmentation fault could actually be a kernel Oops, check the system logs for any Oops messages.
• The application called exit() instead of using the core dump handler.

Supercilious

Also: if the application sets signal handler for SIGSEGV, then without further tricks (see stackoverflow.com/questions/16697361) core dumps will not be created. – Cassady

One thing to add: when a program calls setuid() e.g. to drop root privileges, it is no longer core-dumpable (the executable DOES NOT have to be suid). Tested on Linux 3.12 with default Arch Linux configuration. I have no idea why this happens, it's not docummented anywhere. Calling prctl(PR_SET_DUMPABLE, 1, ...) after setuid fixes this, so it's not a filesystem permission issue. – Apodosis

Actually, this is documented on the prctl man page, under the PR_SET_DUMPABLE section: man7.org/linux/man-pages/man2/prctl.2.html – Fiance

The core(5) manpage is the (hopefully) authoritative list on why core dumps aren't created. – Wizen

I would add that limits.conf is only honored by PAM. So if you have daemons getting started by systemd (or some other init) limits.conf will not be honored. Systemd configuration options can be found in:/etc/systemd/system.conf – Nalda

Also in case you application generates SIGPIPE then it will go down abruptly without generating core dump .. – Countryandwestern

Answer 3

For systemd systems¹, install the package systemd-coredump. Coredumps can be found via:

ls /var/lib/systemd/coredump

---

Furthermore, these coredumps are compressed in the lz4 format. To decompress, you can use the package liblz4-tool like this: lz4 -d FILE. To be able to debug the decompressed coredump using gdb, I also had to rename the utterly long filename into something shorter...

^{1 Debian 9 Stretch}

Invertase

You can use the command coredumpctl list to see whether core dumps have been generated. – Guajardo

This is why I'm here... Why the standard mechanism isn't working on Debian 9? I spent a few hours trying to figure out why I have no dumps crated and finally also went to the systemd-coredump solution. – Autoionization

Same here. Solved the same problem I had, but in Ubuntu 18.04. – Hypsography

Damn, I cannot create the core file with either apport or ulimit. ONLY systemd-coredump works. Thank you a lot – Coatbridge

If anyone reading this still couldn't get it to work, you perhaps forgot one last bit - run the executable with sudo privilege to generate core dump inside /var/lib/systemd/coredump – Condescend

Answer 4

Check:

$ sysctl kernel.core_pattern

to see how your dumps are created (%e will be the process name, and %t will be the system time).

For Ubuntu, dumps are created by apport in /var/crash, but in different format (see inside file).

You can test it by:

sleep 10 &
killall -SIGSEGV sleep

If core dumping is successful, you will see “(core dumped)” after the segmentation fault indication.

Read more:

• How to generate core dump file in Ubuntu
• https://wiki.ubuntu.com/Apport

Predetermine

Answer 5

Remember if you are starting the server from a service, it will start a different bash session so the ulimit won't be effective there. Try to put this in your script itself:

ulimit -c unlimited

Unitary

Answer 6

If one is on a Linux distro (e.g. CentOS, Debian) then perhaps the most accessible way to find out about core files and related conditions is in the man page. Just run the following command from a terminal:

man 5 core

Papotto

Answer 7

Also, check to make sure you have enough disk space on /var/core or wherever your core dumps get written. If the partition is almos full or at 100% disk usage then that would be the problem. My core dumps average a few gigs so you should be sure to have at least 5-10 gig available on the partition.

Pareto

Answer 8

Note: If you have written any crash handler yourself, then the core might not get generated. So search for code with something on the line:

signal(SIGSEGV, <handler> );

so the SIGSEGV will be handled by handler and you will not get the core dump.

Unitary

Answer 9

The answers given here cover pretty well most scenarios for which core dump is not created. However, in my instance, none of these applied. I'm posting this answer as an addition to the other answers.

If your core file is not being created for whatever reason, I recommend looking at the /var/log/messages. There might be a hint in there to why the core file is not created. In my case there was a line stating the root cause:

Executable '/path/to/executable' doesn't belong to any package

To work around this issue edit /etc/abrt/abrt-action-save-package-data.conf and change ProcessUnpackaged from 'no' to 'yes'.

ProcessUnpackaged = yes

This setting specifies whether to create core for binaries not installed with package manager.

Baxley

Answer 10

If you call daemon() and then daemonize a process, by default the current working directory will change to /. So if your program is a daemon then you should be looking for a core in / directory and not in the directory of the binary.

Kay

Answer 11

Although this isn't going to be a problem for the person who asked the question, because they ran the program that was to produce the core file in a script with the ulimit command, I'd like to document that the ulimit command is specific to the shell in which you run it (like environment variables). I spent way too much time running ulimit and sysctl and stuff in one shell, and the command that I wanted to dump core in the other shell, and wondering why the core file was not produced.

I will be adding it to my bashrc. The sysctl works for all processes once it is issued, but the ulimit only works for the shell in which it is issued (maybe also the descendents too) - but not for other shells that happen to be running.

Rounce

Answer 12

Just in case someone else stumbles on this. I was running someone else's code - make sure they are not handling the signal, so they can gracefully exit. I commented out the handling, and got the core dump.

Caribou

Answer 13

In centos,if you are not root account to generate core file: you must be set the account has a root privilege or login root account:

vim /etc/security/limits.conf

account soft core unlimited
account hard core unlimited

then if you in login shell with securecrt or other:

logout and then relogin

Charterhouse

Answer 14

Allow Dump from Daemons To allow all daemons witch are started by systemd to core dump.

Edit: /etc/systemd/system.conf add following

DefaultLimitCORE=infinity Edit: /etc/sysctl.d/core.conf add following

kernel.core_pattern = /var/lib/coredumps/core-%e-sig%s-user%u-group%g-pid%p-time%t kernel.core_uses_pid = 1 fs.suid_dumpable = 2

more detail: https://pve.proxmox.com/wiki/Enable_Core_Dump_systemd

Larcener

Answer 15

Our application stopped producing core dumps when a capability was set to it.

setcap 'cap_sys_nice=eip' /usr/bin/${our_app}

Removing it allowed the re-generation of coredumps.

setcap '-r' /usr/bin/${our_app}

See also: How do I get a coredump from a setcap executable?

Wards

You win some (capability), you lose some (capability) – Capitalism