systemd/udev dependency failure when auto mounting separate partition during startup

Question

EDIT: This occurs when I am trying to use a separate partition with ANY mount point, not just /var.

I am using Buildroot to build an embedded linux system. I am trying to use a separate partition for /var, but this occurs for ANY mount point. My init system is systemd, udev for device management. I've edited my fstab to:

# <file system> <mount pt>     <type>   <options>                <dump> <pass>
/dev/root       /              ext2     rw,noauto                0      1
/dev/mmcblk0p4  /var           ext2     defaults                 0      2
proc            /proc          proc     defaults                 0      0
devpts          /dev/pts       devpts   defaults,gid=5,mode=620  0      0
tmpfs           /dev/shm       tmpfs    mode=0777                0      0
tmpfs           /tmp           tmpfs    defaults                 0      0
sysfs           /sys           sysfs    defaults                 0      0
tmpfs           /run           tmpfs    defaults                 0      0

Once the system boots, I receive a message "Expecting device dev-mmcblk0p4.device...", eventually followed by:

"Timed out waiting for device dev-mmcblk0p4.device."

"Dependency failed for /var."

"Dependency failed for Local File Systems."

Same thing happens if I use /var2, or anything else, as the mount point instead of /var. Once the boot fails, it enters emergency mode and the mount will then successfully occur. I can then exit system maintenance and everything works as expected. What is causing this delayed mount?? Have I missed a step?

Screenshot of output:

Piegari

There's Fedora bug982007 But should mention that with Systemd bundled in ALTLinux (209, 210) I haven't been aware of this bug on a pretty complex configuration (/usr, /var, /home on separate pertitions and all are on raid volumes of different types). So I guess the problem may be caused by "non-elaborate" configuration and/or initrd. I suppose that you're booting from a SD card, right? I guess you to introduce an artificial dependency on the corresponding udev add event? – Driver

@Driver yes, booting from SD card. Unfortunately I am forced to work with an older, non-mainline kernel version, so I'm worried there is a bug that may be contributing to this as you've suggested. It seems like such a simple change, though... I'll keep trying to find more information. – Piegari

Can you check the version of systemd and simply update it if it's less than 208? – Driver

Current version is 202. I'll try updating and post results this afternoon. – Piegari

Updated systemd to 208, same issue. – Piegari

Turns out this is occurring when mounting any separate partition from fstab...not related to /var. I've updated the question to reflect this. – Piegari

Upsteam report – Lilialiliaceous

Answer 1

I had a similar issue here using systemd-208 with kernel 3.13.1 and 3.14.4 under Exherbo (similar to Gentoo): my separate partitions (/var, /home, /boot and even /swap) were mounted under the desired root location. Everything worked fine. Then, upgrading to systemd 213 the boot process stopped at the point, where the partitions should have been mounted with the message: 'A start job is running for dev-sdxx.device' until timeout of 1 min 30 sec. Afterwards I was in Emergency Mode. Excerpt of

$ journalctl -xb

Jun 09 13:50:29 exathlon systemd[1]: Job dev-disk-by\x2dlabel-BAK_A4.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-disk-by\x2dlabel-BAK_A4.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /mnt/BAK_A4.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Local File Systems.
Jun 09 13:50:29 exathlon systemd[1]: Triggering OnFailure= dependencies of local-fs.target.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /export/LinuxMint13_KDE_64.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /export/SystemRescueCD-x86.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/disk/by-label/BAK_A4.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb11.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb11.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /var.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Update UTMP about System Boot/Shutdown.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for Load/Save Random Seed.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sdb11.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb8.device/start timed out.
Jun 09 13:50:29 exathlon systemd-journal[1044]: Forwarding to syslog missed 15 messages.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb8.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /home.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sdb8.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sda6.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sda6.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /boot.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for File System Check on /dev/sda6.
Jun 09 13:50:29 exathlon systemd[1]: Job dev-sdb5.device/start timed out.
Jun 09 13:50:29 exathlon systemd[1]: Timed out waiting for device dev-sdb5.device.
Jun 09 13:50:29 exathlon systemd[1]: Dependency failed for /dev/sdb5.

shows the failed attempts of mounting by systemd.

After looking up the internet I was able to find a remedy in configuring the kernel with

CONFIG_FHANDLE=y (before: n)

see 'systemd System and Service Manager' under

REQUIREMENTS:
 CONFIG_FHANDLE (libudev, mount and bind mount handling)

source: http://cgit.freedesktop.org/systemd/systemd/tree/README

Afterwards the problem was solved, all partitions mounted again.

I cannot tell, why this was not happening / or was not required with systemd 208.

Fairyfairyland

Same problem here updating from Opensuse 13.1 to 13.2 ! Needed a long time to find this solution! THX !!!!! – Metcalf

Doesn't work for me on Archlinux: zgrep CONFIG_FHANDLE /proc/config.gz shows the option is enabled (y), but the problem still happens. – Lilialiliaceous

@Lilialiliaceous do you happen to use LUKS? – Bohaty

@webKnjaZ nope. – Lilialiliaceous

Answer 2

Was able to work around this, although it is a sort of a hack. Would still love to know why this is occurring in the first place, but it appears as though udev isn't mounting mmcblk partitions until after systemd init is complete, which causes dependency errors if /etc/fstab calls for an mmcblk partition. udev checks the fstab, waits for an mmcblk device to be mounted and times out, THEN attempts to mount the device.

My Solution:

1. Create systemd service to handle mounting the partition:

#/etc/systemd/system/mount-data-partition.service
[Unit]
Description=Mount Data Partition
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/mount /dev/mmcblk0p4

2) Add a `Wants` dependency to this service within `systemd-udev-trigger.service`:

#/usr/lib/systemd/system/systemd-udev-trigger.service
[Unit]
Description=udev Coldplug all Devices
Documentation=man:udev(7) man:systemd-udevd.service(8)
DefaultDependencies=no
Wants=systemd-udevd.service mount-data-partition.service
After=systemd-udevd-kernel.socket systemd-udevd-control.socket
Before=sysinit.target
ConditionCapability=CAP_MKNOD

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/udevadm trigger --type=subsystems --action=add ; /usr/bin/udevadm trigger

This causes `mount-data-partition.service` to be called by and executed before `systemd-udev-trigger.service`. The `mount` command will then look for `/dev/mmcblk0p4` in `/etc/fstab`, and mount as specified (in my situation, `/var`). Since `/dev/mmcblk0p4` is now mounted, udev recognizes that the device exists and no longer times out while waiting for it. System continues to boot as normal.

I hate to answer my own question, but hopefully this information helps someone out. If someone can enlighten me on why this is occurring in the first place, please do.

Piegari

The right answer have been given by @marsh, but your workarount helped me a lot since I cannot easily update my kernel in my context. I added wants one sysinit.service as symlinks since I dont use udev, but it works fine. Thank you very much ! – Foreclosure