Equivalent for pop on strings

Question

Given a very large string. I would like to process parts of the string in a loop like this:

large_string = "foobar..."
while large_string:
    process(large_string.pop(200))

What is a nice and efficient way of doing this?

Ideate

What exactly do you want? The first 200 characters? The 200th character? Something else? – Revivify

@KarlKnechtel Isn't the while loop clearly showing that I want to process all characters of the string? – Ideate

It looks like you want to remove and process the 200th character of large_string until pop throws "IndexError: pop index out of range" – Angilaangina

Answer 1

You can wrap the string in a StringIO or BytesIO and pretend it's a file. That should be pretty fast.

from cStringIO import StringIO
# or, in Py3/Py2.6+:
#from io import BytesIO, StringIO

s = StringIO(large_string)
while True:
    chunk = s.read(200)
    if len(chunk) > 0:
        process(chunk)
    if len(chunk) < 200:
        break

Chartulary

If he want to consume the string from the end this does not work. – Dyna

@ms4py fortunately the order of the chunks does not matter at all for my task – Ideate

@ms4py: true. In that case, I'd slice the string up into a list and iterate over it in reverse: [large_string[i:i+200] for i in xrange(0, len(large_string), 200)] – Chartulary

@larsmans: Or, you could use the buffer's seek method to read the last n bytes: s.seek(-200, 2); chunk = s.read()... – Bendicta

You don't need Py3 for io.StringIO - it exists from 2.6. – Bobbobb

One of the benefits of pop is that it discards the elements it returns, releasing the memory they occupied. StringIO does not discard the parts of the string that were already read. Is there a way to get this aspect of the pop functionality for strings? – Milkmaid

@Joe: from what I know of CPython internals, I don't think any solution will partially deallocate the string and retain linear time complexity for taking slices off it. – Chartulary

Answer 2

you can convert the string to a list. list(string) and pop it, or you could iterate in chunks slicing the list [] or you can slice the string as is and iterate in chunks

Aghast

Answer 3

You can do this with slicing:

large_string = "foobar..."
while large_string:
    process(large_string[-200:])
    large_string = large_string[:-200]

Dyna

This is pretty wasteful. Not only because it does the slicing twice, but because it uses an O(n²) time algorithm. – Chartulary

Answer 4

To follow up on dm03514's answer, you can do something like this:

output = ""
ex = "hello"
exList = list(ex)
exList.pop(2)
for letter in exList:
    output += letter

print output # Prints 'helo'

Kristof