How does str(list) work?

Question

Why does str(list) returns how we see list on the console? How does str(list) work? (any reference to the CPython code for str(list))?

>>> x = ['abc', 'def', 'ghi']
>>> str(x)
"['abc', 'def', 'ghi']"

To get the original list back from the str(list) I have to:

>>> from ast import literal_eval
>>> x = ['abc', 'def', 'ghi']
>>> str(x)
"['abc', 'def', 'ghi']"
>>> list(str(x))
['[', "'", 'a', 'b', 'c', "'", ',', ' ', "'", 'd', 'e', 'f', "'", ',', ' ', "'", 'g', 'h', 'i', "'", ']']
>>> literal_eval(str(x))
['abc', 'def', 'ghi']

Why doesn't list(str(list)) turns the str(list) back to the original list?

Or I could use:

>>> eval(str(x))
['abc', 'def', 'ghi']

Is literal_eval the same as eval? Is eval safe to use?

How many times can I do the following? Does the code break if it keep on doing str(list(str(list))))? E.g.

>>> x = 'abc'
>>> list(x)
['a', 'b', 'c']
>>> str(list(x))
"['a', 'b', 'c']"
>>> list(str(list(x)))
['[', "'", 'a', "'", ',', ' ', "'", 'b', "'", ',', ' ', "'", 'c', "'", ']']
>>> str(list(str(list(x))))
'[\'[\', "\'", \'a\', "\'", \',\', \' \', "\'", \'b\', "\'", \',\', \' \', "\'", \'c\', "\'", \']\']'
>>> list(str(list(str(list(x)))))
['[', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'a', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'b', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'c', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ']']
>>> str(list(str(list(str(list(x))))))
'[\'[\', "\'", \'[\', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \'a\', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \',\', "\'", \',\', \' \', "\'", \' \', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \'b\', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \',\', "\'", \',\', \' \', "\'", \' \', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \'c\', "\'", \',\', \' \', \'"\', "\'", \'"\', \',\', \' \', "\'", \']\', "\'", \']\']'
>>> list(str(list(str(list(str(list(x)))))))
['[', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '[', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'a', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'b', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", 'c', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', "'", '"', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", '"', "'", ',', ' ', "'", ',', "'", ',', ' ', "'", ' ', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ',', ' ', '"', "'", '"', ',', ' ', "'", ']', "'", ']']

Barrios

You appear to have the expectation that creating a string from a list is round-trippable. It is not meant to be; lists are not end-user presentable objects and you get the same output as repr(listobject); debug information for developer consumption only. – Demarcusdemaria

Lists have not __str__ method at all, its tp_str slot is actually empty. So, instead its tp_repr slot is used to get the repr representation of the list and its items(including cylic objects): hg.python.org/cpython/file/e8783c581928/Objects/… – Camey

Answer 1

Well you have a total of 4 questions, let us go one by one.

1. Why does str(list) returns how we see list on the console? How does str(list) work?

What is str() and __str__()?

The str() callable is to return a printable form of the object only! From the docs

str(object) does not always attempt to return a string that is acceptable to eval(); its goal is to return a printable string.

The __str__() function in a class is called whenever you call str() on an object. Again from the documentation

object.__str__(self)

Called by the str() built-in function and by the print statement to compute the “informal” string representation of an object.

What is the list callable?

The list() callable is to create a list from an iterable passed as an argument. Again from the docs

Return a list whose items are the same and in the same order as iterable‘s items

Thus, str(list) gives you a printable form and list(str(list)) will iterate over the string. That is list(str(list)) will give you a list of the individual characters of the printable form of the argument passed.

A small walk-through between the nested calls,

Given list, l = ['a','b'] _{(Apologies for taking a smaller example than that in your question)}.

When you call str(l), it returns a printable form of the list l, that is "['a','b']".

Now you can see clearly that "['a','b']" is a string and is indeed an iterable. Now when you call list on this i.e. list("['a','b']") you get a weird list like ['[', "'", 'a', "'", ',', "'", 'b', "'", ']']. Why does this happen? This happens because the string iterates over its characters, you can test this by using a dummy string,

>>> 'dummy'
'dummy'
>>> list('dummy')
['d', 'u', 'm', 'm', 'y']

Thus when you call the list on a string you get a list of character. Note that again here, when you call str() on list('dummy'), you will not get back your original string 'dummy', so again you will have to use join! Thus recalling the same function will NOT get you back your original object!

So, Calling str() over a list calls the builtin __str__() method of the list?

The answer is NO!

What happens internally when you call str() on a list?

Whenever you call str() on an list object, the steps followed are

1. Call the repr() of each of the list element.
2. Add a fancy [ at the front and another ] at the end of the list.
3. Join all of them with a comma.

As you can see from the source code of the list object in cpython on github. Going through the source code of cpython in hg.python, which is more clear, you can see the following three comments. _{(Thanks to Ashwini for the link on that particular code)}

/* Do repr() on each element.  Note that this may mutate the list,
   so must refetch the list size on each iteration. */ line (382)

/* Add "[]" decorations to the first and last items. */ line (398)

/* Paste them all together with ", " between. */ line (418)

These correspond to the points I mentioned above.

Now what is repr()?

repr() prints the string representation of all the objects. Again from the documentation

Return a string containing a printable representation of an object.

and also note this sentence!

For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object.

And now your second question here,

2. Why doesn't list(str(list)) turns the str(list) back to the original list?

Internally, str(list) actually creates the repr() representation of the list object. So to get back the list after calling str on the list, you actually need to do eval on it and not a list call.

Workarounds

But we all know that eval is evil, so what is/are the workaround(s)?

1. Using literal_eval

The first work-around would be to use ast.literal_eval. That brings us to your 3rd question,

3. Is literal_eval() the same as eval()? Is eval() safe to use?

ast.literal_eval() is safe unlike the eval() function. The docs themselves mention that it is safe --

Safely evaluate an expression node or a string containing a Python literal or container display

2. Using string functions and builtins

Another workaround can be done using str.split()

>>> x = ['abc', 'def', 'ghi']
>>> a = str(x)
>>> a[2:-2].split("', '")
['abc', 'def', 'ghi']

This is just a simple way to do that for a list of strings. For a list of integers you will need map.

>>> x = [1,2,3]
>>> a =str(x)
>>> list(map(int,a[1:-1].split(', '))) # No need for list call in Py2
[1, 2, 3]

Thus unlike literal_eval these are simple hacks given that you know the elements of the list. If they are heterogeneous in nature like [1, "a", True] then you will have to loop through the split list and discover the element type and then convert it and append the converted element to a final list.

Another place where this fails is when the string itself contains quote characters. As mentioned by nneonneo in a comment

The str.split solution is very fragile and will break if the input contains e.g. strings that contain ", ", or tuples, or other lists, ... It is much better to use ast.literal_eval because that will deal with all the subtleties of the syntax.

And for your final question,

4. Does the code break if you do str(list(str(list)))) again and again?

Not really. The output will grow longer and longer as each time you are creating a list of a str and then again getting the printable version of it. The limitation is your physical machine's limitation only. (which will be soon reached as each step the string length is multiplied by 5.)

Soubise

About the What happens internally when you call str on a list? part, why isn't do 3 before 2 like ', '.join(i.__repr__() for i in l).join('[]')? – Hippie

@KevinGuan 1. Efficiency 2. Simplicity. It is easier to add the brackets to the first and last elements at once. Once they are added the individual elements can be directly passed to _PyString_Join and return the result. As you can see in your example you are joining twice. On an average this method works out better. – Soubise

The str.split solution is very fragile and will break if the input contains e.g. strings that contain ", ", or tuples, or other lists, ... It is much better to use ast.literal_eval because that will deal with all the subtleties of the syntax. – Nidifugous

Answer 2

You appear to have the expectation that creating a string from a list is round-trippable. It is not meant to be; lists are not end-user presentable objects and you get the same output as repr(listobject); debug information for developer consumption only.

The list() callable creates a new list object from any arbitrary iterable object; Python strings are iterable producing individual characters when you do so, list(stringobject) always produces a list with individual characters.

As such, list() will never attempt to interpret a string argument as Python syntax; and doing so would not even work if the original list contained objects without a Python literal notation. Take for example:

>>> def foo(): return 'bar'
... 
>>> alist = [foo]
>>> alist
[<function foo at 0x106c748c0>]

You cannot take that debug string output and turn that back into the original list, especially if you run this in a Python interpreter where there is not even such a function defined.

Demarcusdemaria

Answer 3

The str() function in python is used to turn a value into a string. The simple answer to what the str() does to a list is that it creates a string representation of the list (square brackets and all).

As for the list(str(list)), all you are doing is telling python to convert the original list to a string then you are splitting that string and putting it into a list so that each index has one character. So you could nest list and str calls as many times as you want (assuming your computer is has enough memory).

Tremayne