How does a progn form in font-lock-keywords work?

Question

Following code will visually replace "hello world" with "HW" by passing a progn form to font lock keywords.

(font-lock-add-keywords
 nil '(("\\(hello world\\)"
        (0 (progn (put-text-property (match-beginning 1) (match-end 1)
                                     'display "HW")
                  nil)))))

I've look into C-h v font-lock-keywords to see if this is a documented feature of font lock. The hello world element seemed to be of this form:

(MATCHER HIGHLIGHT ...)

which would mean that (0 ...) is HIGHLIGHT and the doc says

HIGHLIGHT should be either MATCH-HIGHLIGHT or MATCH-ANCHORED.

and

MATCH-HIGHLIGHT should be of the form:

 (SUBEXP FACENAME [OVERRIDE [LAXMATCH]])

So I guessed 0 was SUBEXP and (progn ...) was FACENAME. But if (progn ..) were a valid FACENAME, the following code would work but it doesn't work.

;;  (MATCHER . FACENAME)
(font-lock-add-keywords
 nil '(("goodbye lenin"
        . (progn (put-text-property (match-beginning 1) (match-end 1)
                                    'display "GL")
                 nil))))

That brings me to the question of how the first code works and whether it is relying on an undocumented feature.

---

Update:

Side note: simpler way of visual replacement without font lock errors

(font-lock-add-keywords
 nil '(("my llama"
        (0 (progn (put-text-property (match-beginning 0) (match-end 0)
                                     'display "ML")
                  nil)))))

Brenneman

Answer 1

It does work - but your MATCHER is not correct - the result of the match is not stored. This, for example does not work:

(font-lock-add-keywords
 nil '(("goodbye lenin"
        (0 (progn (put-text-property (match-beginning 1) (match-end 1)
                                    'display "GL")
                  nil)))))

while this does:

(font-lock-add-keywords
 nil '(("\\(goodbye lenin\\)"
        . (progn (put-text-property (match-beginning 1) (match-end 1)
                                    'display "GL")
                 nil))))

The documentation says: "FACENAME is an expression whose value is the face name to use. Instead of a face, FACENAME can evaluate to a property list of the form (face FACE PROP1 VAL1 PROP2 VAL2 ...) in which case all the listed text-properties will be set rather than just FACE."

Here, the FACENAME expression (progn) evaluates to nil, so no properties or faces are being set - the only effect that caused by put-text-property.

Trantrance

The ("\\(goodbye lenin\\)" . (progn..)) example works but somehow it generates Error during redisplay: (jit-lock-function 1) signaled (invalid-function progn) in Messages, and when I type (defun goodbye lenin), the word defun stops getting highlighted with font-lock-keyword-face as soon as I finish the closing paren. – Brenneman

Indeed. It seems to me that in the case of the latter syntax, the form is evaluated at font-lock.el line 1542 (emacs 24.2). There is a funcall, and progn is not a function, hence the error. Changing the funcall to eval removes the error, but I am not an expert on emacs font lock, so I cannot promise this does not break something. Also, I cannot find the reason why the whole line loses highlights when you type the next character after "lenin". – Trantrance

Anyway, I think the gist here is that yes, the definitions are equal so they should behave identically - but the two cases are implemented by two distinct pieces of code and they do not produce 100% identical results. – Trantrance

The reason why the whole line loses highlight is probably because of the font lock error. When an element in font-lock-keywords causes error, font lock sometimes give up on processing other elements after that element, and the element for highlighting the word defun is placed somewhere after the most recently added element for goodbye lenin. – Brenneman

Even after fixing the error by changing the funcall to eval, the highlights are lost, so that is not the (only) reason. – Trantrance

Answer 2

One problem with ("goodbye lenin" . (progn (put-text-property (match-beginning 1) (match-end 1) 'display "GL") nil)) is that it is just another way to write: ("goodbye lenin" progn (put-text-property (match-beginning 1) (match-end 1) 'display "GL") nil), and these equivalences can lead to ambiguities, which is why in this case you get errors.

So the form (MATCHER . HIGHLIGHT), (match . FACENAME) and such should only be used when HIGHLIGHT and FACENAME are not themselves lists.

Limnetic

Does this mean that ("bob" . 'warning) is also ambiguous because it is the same as ("bob" quote warning) and that it should rather be written as ("bob" (0 'warning)) or ("bob" 0 'warning)? – Brenneman

Yes. It might work as is, but the longer form is not that much longer, saving you from having to worry if it's ambiguous or not. – Limnetic