Recursively adding .org files in a top-level directory for org-agenda-files takes a long time

Question

I'm trying to find a way to quickly recurse through every subdirectory searching for org files. I've found several solutions (Elisp Cookbook, and several solutions on github), but they don't handle my real world usage (hundreds of directories (and subdirectories) and hundreds of org files). They seem to run forever on my system (Windows 7, with max-lisp-eval-depth = 10000). My work around is to add each directory manually to my org-agenda-list, but it's annoying and I know I've probably forgotten some. Any suggestions would be appreciated.

Applique

Answer 1

How about storing the list of org-agenda directories in a file that you (automatically) update every once in a while, when you know the direcory structure has changed and you have some time.

You could for example use something like this:

;; From http://www.emacswiki.org/emacs/ElispCookbook#toc58
(defun directory-dirs (dir)
  "Find all directories in DIR."
  (unless (file-directory-p dir)
    (error "Not a directory `%s'" dir))
  (let ((dir (directory-file-name dir))
        (dirs '())
        (files (directory-files dir nil nil t)))
    (dolist (file files)
      (unless (member file '("." ".."))
        (let ((file (concat dir "/" file)))
          (when (file-directory-p file)
            (setq dirs (append (cons file
                                     (directory-dirs file))
                               dirs))))))
    dirs))


(setq my-org-agenda-root "~/org")
(setq my-org-agenda-files-list "~/.emacs.d/org-agenda-list.el")

(defun my-update-org-agenda-files ()
  "Create or update the `my-org-agenda-files-list' file.

This file contains elisp code to set `org-agenda-files' to a
recursive list of all children under `my-org-agenda-root'. "
  (interactive)
  (with-temp-buffer
    (insert
     ";; Warning: this file has been automatically generated\n"
     ";; by `my-update-org-agenda-files'\n")
    (let ((dir-list (directory-dirs my-org-agenda-root))
          (print-level nil)
          (print-length nil))
      (cl-prettyprint `(setq org-agenda-files (quote ,dir-list))))
    (write-file my-org-agenda-files-list)))

(load my-org-agenda-files-list)

Every once in a while, run M-xmy-update-org-agenda-files to update the list.

Hawger

Answer 2

Haven't seen other people post this, so I will do. Have you tried load "find-list" library and use its "find-lisp-find-files" function? I added these lines in my org config and it works, but it may not fit your performance requirement:

(load-library "find-lisp") (setq org-agenda-files (find-lisp-find-files "FOLDERNAME" "\.org$"))

source: http://emacs-orgmode.gnu.narkive.com/n5bQRs5t/o-multiple-recursive-directories-with-org-agenda-files

Bewley

Answer 3

The following code works well in emacs 24.3+:

;; Collect all .org from my Org directory and subdirs
(setq org-agenda-file-regexp "\\`[^.].*\\.org\\'") ; default value
(defun load-org-agenda-files-recursively (dir) "Find all directories in DIR."
    (unless (file-directory-p dir) (error "Not a directory `%s'" dir))
    (unless (equal (directory-files dir nil org-agenda-file-regexp t) nil)
      (add-to-list 'org-agenda-files dir)
    )
    (dolist (file (directory-files dir nil nil t))
        (unless (member file '("." ".."))
            (let ((file (concat dir file "/")))
                (when (file-directory-p file)
                    (load-org-agenda-files-recursively file)
                )
            )
        )
    )
)
(load-org-agenda-files-recursively "/path/to/your/org/dir/" ) ; trailing slash required

It does not require intermediate files creation and you can put it on a shortcut as well.

To be able to refile to any file found add this:

(setq org-refile-targets
      '((nil :maxlevel . 3)
        (org-agenda-files :maxlevel . 1)))

Garrulous

It is also possible to configure emacs to get the agenda files list for refiling. See edit above. – Garrulous

Just added a fix to add to the list only directories, which already contain .org files. – Garrulous

Answer 4

As of Emacs 25, you can use directory-files-recursively which returns all files matching a regex from a root directory.

(defun org-get-agenda-files-recursively (dir)
  "Get org agenda files from root DIR."
  (directory-files-recursively dir "\.org$"))

(defun org-set-agenda-files-recursively (dir)
  "Set org-agenda files from root DIR."
  (setq org-agenda-files 
    (org-get-agenda-files-recursively dir)))

(defun org-add-agenda-files-recursively (dir)
  "Add org-agenda files from root DIR."
  (nconc org-agenda-files 
    (org-get-agenda-files-recursively dir)))

(setq org-agenda-files nil) ; zero out for testing

(org-set-agenda-files-recursively "~/Github") ; test set

(org-add-agenda-files-recursively "~/Dropbox") ; test add 

You can view the contents of org-agenda-files by typing C-hv org-agenda-files.

Drewdrewett

Answer 5

Little late to the party, and probably not the answer you were hoping for, but the only way I found to speed up my agenda load times was by not including some directories which had thousands of org files.

(setq org-directory "~/org/"
      my-agenda-dirs '("personal" "projects" "todos" "work")
      org-agenda-files (mapcan (lambda (x) (directory-files-recursively
                                             (expand-file-name x org-directory)
                                             "\.org$"))
                               my-agenda-dirs))

Essentially rather than recursing over my entire org dir I only recurse through a select few of it's subdirs.

Chausses

Answer 6

I also tried "recursive directory listing" at Emacs startup. It simply is way to loooong to be usable. So, try to stick with a limited number of "root" directories where you put your agenda files.

Even better is sticking with "~/org", and possibly a few subdirs like "work" and "personal", and that's it. You can put there your agenda files, and have links Inside them to your real project root dirs.

I know, this is not optimal, but I don't have anything better right now.

Generalize