Parallel make: set -j8 as the default option

Question

I can set number of threads for the build process using -j argument. For example, I have 4 cores +4 virtual. When I write: make -j8 the speed increases 4 times.

Is it possible to set that value as default? (For example, in Linux Gentoo, in config file, it's possible to set this default value).

p.s. I have Arch Linux

Berry

So.. the default C++ make option is 'run 4 times slower than is possible'? What a strange decision! – Barty

@MartinJames Parallel make has many flaws and so it makes a bad default. – Oh

Pubby: Most makefile authors have numerous flaws and don't get the dependencies right, ultimately leading to bad experiences when trying parallel builds ;) – Yellowstone

But often, just rerunning make once more makes everything right while still having built most of the stuff much faster. – Fassold

@Fassold I have a question that illustrates what you have said: Error building wxWidgets using -j (jobs) option with MinGW – Kelpie

Does this answer your question? Make "make" default to "make -j 8" – Nannettenanni

Answer 1

Your question is not about threads, but processes (jobs) executed by make.

The simple, way to set this, when make is used from the console is adding:

alias make="/usr/bin/make -j 8"

to your .profile file.

You can also use setenv MAKEFLAGS '-j 8', but MAKEFLAGS can ignore this parameter in some scenarios, because keeping desired number of processes requires communicating with recursive make calls. Happily this method works with current versions of GNU Make.

Scissel

Since this mentioning of "recursive make" doesn't stop, I'd like to once again point to aegis.sourceforge.net/auug97.pdf ... besides, the make file needs to be well-written to even allow for parallel make. Often, especially automated methods, disregard the actual dependencies and cause targets to be built multiple times and the likes ... – Yellowstone

In case the link is not working in future, the paper is titled "Recursive Make Considered Harmful" and should be easy enough to find. It is well worth a read. – Interject

What's setenv here? The csh builtin? – Educationist

Most people should NOT pass the -j option value down to recursive calls. Jobserver in the top-level make does this much better: all the child processes combined get exactly as many jobs as the cores assigned. Now imagine if you set -j16 for 16 cores, and EACH of 10 child makes gets -j16 (which overrides jobserver). Now you have 160 jobs doing context switches on a CPU-bound tasks. – Angelineangelique

Answer 2

setenv MAKEFLAGS '-j8'

Hope this helps!

Sverige

Note that setenv belongs to (t)csh and is not available in bash (see unix.stackexchange.com/a/85100/200009). – Saar

Answer 3

Here's how I've done it:

CORES ?= $(shell sysctl -n hw.ncpu || echo 1)

all:; @$(MAKE) _all -j$(CORES)
_all: install lint test
.PHONY: all _all
…

I've basically "aliased" my default target all to a "private" _all. The command to figure out the number of cores is OSX specific, AFAIK, so you could just improve it to be more cross platform if you will. And because of the ?= assignment, we can just override it with and env variable if/when needed.

EDIT:

You can also append to your MAKEFLAGS from within the makefile itself, like so:

CPUS ?= $(shell sysctl -n hw.ncpu || echo 1)
MAKEFLAGS += --jobs=$(CPUS)
…

EDIT 2:

You may also use the following, ff you want it to be more cross-platform:

CPUS ?= $(shell (nproc --all || sysctl -n hw.ncpu) 2>/dev/null || echo 1)

Hobnob

This is IMHO better than the other solutions, since you can tailor this per Makefile. – Whatnot

You can't alter MAKEFLAGS this way. Try it: gist.github.com/sanmai/dcc31ae20afa6e8ba4721f174fe05fd9 – Profitsharing

@Profitsharing The Linux kernel Makefile (github.com/torvalds/linux/blob/master/Makefile) modifies MAKEFLAGS in a similar way, and I have a similar jobs modification that works for me in my project. – Pectize

@Pectize true you can do that in the current version of the make program; before there was a long time where you could not – Profitsharing

@Profitsharing I was doing it back in the late 90's with Slackware 3 and the 2.1 kernel, because I didn't know I could just pass it as an argument, and it still works on RHEL7 (Make 3.82). Must have been a bug introduced where it didn't work for a while. – Pectize

@Pectize it kinda worked like if you supplied -j without any arguments even if you did, so you were lucky not having met an accidental forkbomb. Here's a changelog entry with the fix. – Profitsharing

I think I like this answer. It lets me have better control when I do it. For instance, I often do "make clean all", and I would definitely not want them running in parallel. I also have rules that set up temp directories and such which should be done left-to-right, not parallel. Nice. Note, however, the sysctl won't work in a docker container, but this answer far predates that being a common thing. – Cauley

@JosephLarson, I've been using CPUS ?= $(shell (nproc --all || sysctl -n hw.ncpu) 2>/dev/null || echo 1) since then to address what you mentioned. [maybe I'll just update the answer :-)] – Hobnob