Lisp: How to MAPCAR "#x" over a list of HEX?

Question

Using #x... like below one obtains the decimal of hex value

> #xB1 
177
> #xA5
165
> #xFF
255

Say we have a list of hex, what is the correct syntax using mapcar #x... over the list? Below doesn't work:

> (mapcar #'(lambda (hex) `(#x,hex)) '(B1 A5 FF))

Reader error: Malformed number in a #b/#o/#x/#r macro. [Condition of type SIMPLE-ERROR]

Thanks.

Hebetate

I found that this also works: > (mapcar #'(lambda (hex) (eval (read-from-string (concatenate 'string "#x" (string hex))))) '(B1 A5 FF)) – Hebetate

The shown attempt shows a very basic misunderstanding of how code is represented and read in Lisp. Please try to understand the differences between read time, macro expansion time, compile time and run time. – Anasarca

#x... doesn't get the decimal value of a hex string; it's a notation for writing numbers. When the reader reads #xB1, it returns the number 177. Then the REPL is evaluating the form 177, which evaluates to itself, and then prints that. The way it's printed depends on a number of things. E.g., see what happens if you (setf *print-base* 16) and then evaluate #xB1. You'll see B1 printed. – Denude

Answer 1

The #x is what's called a "reader macro". It is very similar to using quotations (ie "") to represent strings. They are executed when the code is read/compiled. What you actually want is a procedure which can convert from hexadecimal strings at run time. The procedure you are looking for is parse-integer, which takes a string and returns the value it represents. The mapcar with it should look something like this:

(mapcar (lambda (hex) 
           (parse-integer hex :radix 16))
        '("B1" "A5" "FF"))

Note that this is using strings, if you want to use symbols as in your suggestion you would have to do something like this:

(mapcar (lambda (hex) 
           (parse-integer (symbol-name hex) :radix 16))
        '(B1 A5 FF))

If you don't know the difference between a symbol and a string, I would suggest reading this: What exactly is a symbol in lisp/scheme?

Ardrey

Thanks. #x is reader macro, i knew how to mapcar to function, but mapcar to a macro such a concept I am not sure; so generally I can use: lambda (macro-thing) (eval (read-from-string "...computed macro thing...")) so can achieve mapcar over reader macro, is it correct – Hebetate

It does work, but it is extremely inefficient and is also a huge security vulnerability. – Ardrey

Early Lisp had first class macros (fexpr), but there is a problem when you send a macro to mapcar. It's complicating compilation so much they ended up with defmacro and that they are not first class objects. – Fireproof

@Ardrey I entirely agree that using (eval (read-from-string …)) here would be very dangerous. It is possible, though, to obtain the reader macro function and call it directly with the known inputs. That's not as good for this specific case, but it might be helpful for dispatch macro characters in general, so I added an answer with that approach. As noted, though, it's more brittle because implementations don't guarantee what assumptions they're making about when those functions are called. – Denude

Answer 2

It occurs to me that while the best solution for this problem is probably one using parse-integer as mentioned in malisper's answer, there is a sense in which this could be solved with a mapping based approach.

When we write something like #xB1, we're not explicitly invoking a function. Instead, we're using the fact that # is a dispatching read macro character, and that there's a function installed for the subcharacter x that reads numbers written in hexadecimal. That means that by the time the evaluator or compiler gets a form, the number is already there. However, we do have access to the function that is doing the processing of the hexadecimal string, using get-dispatch-macro-character. Viz.:

CL-USER> (get-dispatch-macro-character #\# #\x)
#<FUNCTION SB-IMPL::SHARP-X>                    ; in SBCL

CL-USER> (get-dispatch-macro-character #\# #\x)
#<SYSTEM-FUNCTION SYSTEM::HEXADECIMAL-READER>   ; in CLISP

What can we do with that function? How would we use it?

2.1.4.4 Macro Characters

… If a character is a dispatching macro character C1, its reader macro function is a function supplied by the implementation. This function reads decimal digit characters until a non-digit C2 is read. If any digits were read, they are converted into a corresponding integer infix parameter P; otherwise, the infix parameter P is nil. The terminating non-digit C2 is a character (sometimes called a ``sub-character'' to emphasize its subordinate role in the dispatching) that is looked up in the dispatch table associated with the dispatching macro character C1. The reader macro function associated with the sub-character C2 is invoked with three arguments: the stream, the sub-character C2, and the infix parameter P. For more information about dispatch characters, see the function set-dispatch-macro-character.

That means that when we write something like #xB1, the function above is getting called with a stream from which it can read B1, the character x, and nil. We can try calling that function with arguments like that, although we can't be quite sure what will happen, because implementations might make different assumptions about where the function will be called from.

For instance, this works without a problem in CLISP, but SBCL assumes that the function should be called recursively from read (which we're not doing):

CL-USER> (funcall (get-dispatch-macro-character #\# #\x)
                  (make-string-input-stream "B1")
                  #\x 
                  nil)
177                     ; in CLISP

CL-USER> (funcall (get-dispatch-macro-character #\# #\x)
                  (make-string-input-stream "B1")
                  #\x
                  nil)
; Evaluation aborted on #<SB-INT:SIMPLE-READER-ERROR "~A was invoked
; with RECURSIVE-P being true outside of a recursive read operation."
; {1005F245B3}>.  ; in SBCL

That said, for implementations where this will work, we can easily write a mapcar-like function to extract a dispatch macro character function and map it over some strings. Thus, in an implementation where this works:

(defun map-dispatch-macro-character (disp-char
                                     sub-char
                                     list
                                     &optional (readtable *readtable*))
  "Retrieve the dispatch macro character for DISP-CHAR and SUB-CHAR and
map it over the elements in LIST. Each element in LIST is either a
string designator or a two-element list of a string-designator and a
prefix argument."
  (flet ((to-list (x)
           (if (listp x) x
               (list x))))
    (let ((fn (get-dispatch-macro-character disp-char sub-char readtable)))
      (mapcar (lambda (x)
                (destructuring-bind (str &optional prefix) (to-list x)
                  (with-input-from-string (in (string str))
                    (funcall fn in sub-char prefix))))
              list))))

CL-USER> (map-dispatch-macro-character #\# #\x '(B1 "A5" (FF nil)))
(177 165 255)

And of course, if you really want to be able to write #x, you could of course define a version that just extracts the characters from a string of length two, so that you could do:

CL-USER> (map-dispatch-macro-character* "#x" '(B1 A5 FF))
(177 165 255)

Denude