deft mode terribly slow

Question

I just started using emacs24 (the one that comes with Ubuntu) in order to use deft mode to mimic the behavior of notational velocity on the Mac. My ~/.emacs-d/init.el looks like this:

(add-to-list 'load-path "~/.emacs.d")
(require 'deft)
(setq deft-directory "~/Notes")
(setq deft-use-filename-as-title t)

in order to load my folder with txt files. I have a few thousand text files in the folder with file names that include CJK characters. Now, running deft mode is now /extremely/ slow, it's impossible to even type to search among the files. After typing a word I have to wait for several minutes before it shows up on the screen and searches through the notes. Even turning emacs off takes a long time. My computer is a Lenovo W540 64-bit i7-4800MQ CPU @ 2.70GHz × 8 and 23.2 GiB memory (it tells me). I'm running Ubuntu 14.04. What might be wrong with my emacs?

I'm new to emacs and frankly I don't even understand what I just wrote above, I'm just trying to piece the setup I want together from tutorials I find online, so forgive any stupidities that might have slipped into my question. Thanks in advance!

Pepi

Answer 1

I'd be happy to be proven wrong, but I don't think deft is intended to work with "a few thousand text files"! You would surely want some kind of specialised indexing and searching tool to efficiently work with that quantity of data.

I'm a bit out of date, but I believe deft is just opening the files in Emacs and using regexp searching to find matches (which is just fine for smaller numbers of files, but it doesn't scale to your use-case).

The only suggestion I can make is to ensure that deft.el is byte-compiled (i.e. you have a deft.elc file). Use M-x byte-compile-file otherwise. But even if you're currently using an uncompiled version, I don't believe that is going to provide the sort of performance improvements you'd need to make this a viable solution.

Mochun

Thank you very much. On the top of your head, do you know of any program that would allow for full text searches in such a corpus of text files and handles unicode well? – Morphosis

Nothing straightforward, I'm afraid (but it's not something I've ever looked into, so I wouldn't know what's available). FWIW the nearest thing I've used myself is Apache Solr which is certainly effective at what it does, but it's non-trivial to set up, and I don't know offhand how you'd configure it for this sort of usage. Hopefully someone else can offer better suggestions. – Mochun

An 8-core i7 with 24 Gb of RAM is a pretty beefy machine. I can't imagine Emacs having any trouble opening a thousand text files on that hardware unless each individual file is huge. (I've opened text files of over 1 Gb on a much less powerful computer, one at a time--they were slow to initially open, but could be edited like any other file once they were in the window). My suspicion is that there's something wrong with deft-mode that makes it inefficient for this particular case. For an alternative, how about Projectile to organize your files in directories and ag to search? – Predator

@Pepi and phils, isn't the "problem" (depending on the use case) that deft does not really index the files prior to searching? It would be great if it worked more like nvALT and indexed in parallell with keeping the files as individual text files. – Sklar

Answer 2

I found this solution that limits the number of notes shown when using incremental search: https://github.com/jrblevin/deft/issues/43

  ;; Overwrite `deft-current-files` for the `deft-buffer-setup` and limit it to 30 entries
  (defun anks-deft-limiting-fn (orig-fun &rest args)
    (let
        ((deft-current-files (-take 30 deft-current-files)))
      (apply orig-fun args)))

  (advice-add 'deft-buffer-setup :around #'anks-deft-limiting-fn)

It helps me somewhat, although deft is still not really snappy when using a couple of thousand notes, as I also do.

Sklar

Answer 3

As of version 0.9, a variable deft-file-limit was added, setting the "maximum number of files to list in the Deft browser."

So to replicate trmdttr's configuration answer for 30 files, you'd add this to your configuration:

(setq deft-file-limit 30)

Cerell

Answer 4

notdeft is a package that was created with this sort of use case in mind; it uses an external backend for indexing and searching (and can handle a huge number of huge files)

EDIT: Here’s a link to the GitHub: https://github.com/hasu/notdeft

It’s a little bit less intuitive to setup (that’s the biggest highlight of Deft I think). You have to get the backend compiled, but the basic interface is the same; you can just type words, and it updates the results in real-time. There’s an advanced way to use the query syntax, but it’s not necessary. Just searching for “noteContent1 noteContent2 :tagName:” will list files that have both the contents and the tag somewhere, and it scales to gigabytes of notes.

Othilie

Could you add links, please. – Mochun

Answer 5

I found deft searching to be unusably slow with Emacs 28 despite setting deft-file-limit. I used notdeft, which worked reasonably well, but didn't work well with my cross-platform .emacs.d directory. I'm sure I could have hacked notdeft to look in an OS + arch specific directory, but it wasn't critical enough for me strong textto invest the time.

Emacs 29 restores deft to something close to its previous speed. Deft with Emacs 29 might not be sufficient for the "thousands of files" case, but it works reasonably well with around a thousand files. It's not super fast, but entirely livable.

Emacs 29 isn't officially released yet, but I've found its various speed improvements (like not be crippled by long lines) to be worth jumping on the Emacs 29 bandwagon.

Mote