How do I automatically (re)compile ELPA packages?

Question

I'm now installing as much as I can through MELPA and Marmalade, and I manage my ~/.emacs.d using git. However, I have git ignore *.elc files.

This means that when I install a package on one system and then start using another system, git pull only gives me the *.el files. Using these files is often slower than using *.elc.

I tried adding the following to ~/.emacs.d/init.el:

;; load the packages we've installed. Note that package-install
;; byte-compiles the packages, but .elc is ignored by git so we force recompilation here
(byte-recompile-directory (expand-file-name "~/.emacs.d/elpa") 0)
(package-initialize)

Unfortunately, this isn't equivalent to the compilation done by package.el. For example, if I install emacs-eclim, package.el doesn't compile emacs-eclim/company-emacs-eclim.el, and I get the following error:

Leaving directory `/home/wilfred/.emacs.d/elpa'

Compiling file /home/wilfred/.emacs.d/elpa/emacs-eclim-20130310.1237/company-emacs-eclim.el at Mon Mar 11 15:40:01 2013
Entering directory `/home/wilfred/.emacs.d/elpa/emacs-eclim-20130310.1237/'
company-emacs-eclim.el:35:1:Error: Cannot open load file: eclim
Warning: reference to free variable `multiple-cursors-mode'
Warning: reference to free variable `mc--read-char'
Warning: assignment to free variable `mc--read-char'
Warning: reference to free variable `multiple-cursors-mode'
Warning: reference to free variable `mc--read-quoted-char'
Warning: assignment to free variable `mc--read-quoted-char'
Warning: reference to free variable `rectangular-region-mode'
Warning: reference to free variable `rectangular-region-mode'

How do I make Emacs byte-compile only the same files as package.el would?

Bucket

Answer 1

package.el actually uses exactly the same method, it's just recompiling on startup makes the errors more noticable.

The function used is package--make-autoloads-and-compile, which calls:

(byte-recompile-directory pkg-dir 0 t)

So the original code in the question is correct. However, to recompile a directory that isn't yet compiled, you can do the following:

(require 'dash)
(require 'f)

(defun was-compiled-p (path)
  "Does the directory at PATH contain any .elc files?"
  (--any-p (f-ext? it "elc") (f-files path)))

(defun ensure-packages-compiled ()
  "If any packages installed with package.el aren't compiled yet, compile them."
  (--each (f-directories package-user-dir)
    (unless (was-compiled-p it)
      (byte-recompile-directory it 0))))

(ensure-packages-compiled)

Bucket

Answer 2

I recommend not keeping the packages in version control (you wouldn't put .o files under revision control, would you?). Here is the code I use to keep my packages in sync:

(setq jpk-packages
      '(
        ac-dabbrev
        ...
        yasnippet
        ))

(package-initialize)
(add-to-list 'package-archives
             '("melpa" . "http://melpa.milkbox.net/packages/"))
(add-to-list 'package-archives
             '("org" . "http://orgmode.org/elpa/"))

(when (not package-archive-contents)
  (package-refresh-contents))

(dolist (pkg jpk-packages)
  (when (and (not (package-installed-p pkg))
           (assoc pkg package-archive-contents))
    (package-install pkg)))

(defun package-list-unaccounted-packages ()
  "Like `package-list-packages', but shows only the packages that
  are installed and are not in `jpk-packages'.  Useful for
  cleaning out unwanted packages."
  (interactive)
  (package-show-package-list
   (remove-if-not (lambda (x) (and (not (memq x jpk-packages))
                            (not (package-built-in-p x))
                            (package-installed-p x)))
                  (mapcar 'car package-archive-contents))))

I put the above in init.el (which of course is under version control), and it installs any packages that don't exist yet when emacs starts up. Updating is done from the buffer package-list-packages creates. package-list-unaccounted-packages shows all the packages that are installed but not in my jpk-packages list, and makes it easy to remove the ones that I took out of the list.

To answer your specific question, I'd just delete the elpa directory and reinstall everything (using the above code).

Precancel

It seems like your jpk-packages is equivalent to a Carton manifest (as @francesco has described with Pallet above), but you lose the ability to specify exact package versions. – Bucket

I don't use Carton or Pallet, but from what I understand, what I use is a very simple version of them. I like my way because I'd need almost all of that code just to bootstrap carton/pallet, and so far I haven't had issues with not specifying versions of packages. – Precancel

"you wouldn't put .o files under revision control, would you?" I don't feel this is a fair comparison. Building .o files should always be reproducible. However, there doesn't seem to be a way to specify a specific version of a package. I check in package files in order to be able to deploy a consistent snapshot of my emacs config that I know to work together. – Progestational

Answer 3

The solution I like to the more general question of "How do I automatically re-compile any outdated .elc file" is to use https://github.com/tarsius/auto-compile

I used to have a handful of custom solutions which covered most situations I ran into, but this library covers everything I was doing and more besides. Just initialise it before loading anything else, and you're sorted.

As to the original question of not compiling files which were not originally compiled, that's what the 0 argument to byte-recompile-directory does, so you're explicitly asking for that behaviour. The default behaviour is to only recompile existing outdated .elc files, so you should simply remove that 0.

Unreserved

Answer 4

I do not know the answer to your specific issue about byte-compiling packages exactly the same way package.el would.

However, for your more general problem about maintaining your emacs configuration directory in git while installing extensions through Melpa or marmalade, I suggest you give a look to pallet, which has been designed to solve this very problem.

Plutus

Thanks, I didn't know about Pallet. However, I'd really like my .emacs.d to be self-contained -- I've had grief in the past with rarely-updated elisp files no longer being available where I originally downloaded them from. – Bucket

Answer 5

(byte-compile-file) worked for me when I need to manually recompile packages I have previously installed using elpa.

Particularly for this situation: https://github.com/yjwen/org-reveal/issues/76t

Spunk

Answer 6

The problem you have is simply that you byte-compile files before initializing your packages. If you switch the two lines, your problem should disappear. The error you get (Cannot open load file: eclim) is because ~/.emacs.d/elpa/emacs-eclim-20130310.1237/ is not yet in your load-path, and package-initialize will add it to your load-path (along with a few other things).

Tiv