Python and threading: Why does "large" list entity get lost when accessing from other method?

Question

A large entity (list) is created in one method (foo) and bound to self.result. The attempt to access this entity in a second method (transmit) fails starting at a certain size (something between 150,000 and 155,000 characters in the list). Printing (print self.result) from inside transmit leaves me with None. I guess this is important: self.foo is directly called in a separate thread.
Please help. How do I get such "large" entity from a separate thread back into the main thread without such limitation?

...

    def apply(self):
        self.get_data()
        self.start_foo_thread()

    def start_foo_thread(self):
        self.foo_thread = threading.Thread(target=self.foo)
        self.foo_thread.daemon = True
        self.progressbar.start()
        self.foo_thread.start()
        self.master.after(20, self.check_foo_thread)

    def check_foo_thread(self):
        if self.foo_thread.is_alive():
            self.master.after(20, self.check_foo_thread)
        else:
            self.progressbar.stop()

    def foo(self): 
        s = self.stringinput
        n = self.numberinput
        list = multiply_str_into_list(s, n)
        self.result = list_to_text(list)
        print self.result # output is not None 

    def transmit(self):
        print self.result # output is None for more than about 155,000 characters in the list
        return self.result

def multiply_str_into_list(string, n): #takes a string and multiplies it by n and writes into list
    n_string = []
    for i in range(0,n):
        n_string.append(string)
    return n_string

def list_to_text(list): #takes a list as input and joins it into str with each list item on a new line
    a = '\n'.join(list)
    return a

Sopher

Although as far as I can tell it's not the cause of your problem here, generally you shouldn't name things list and string because those are already used for built-in types or modules and doing so can cause unintended side-effects, as well as make it hard for others to read. It can also make debugging harder. – Unpopular

@martineau: +1 for that hint – Sopher

Answer 1

You don't really provide enough information to reproduce the problem, let alone debug it, but my guess is that at some point, you are doing something like this:

self.result = self.result.append(x)

Since .append() modifies the list in place, and returns None, this will clobber the reference to the list. It needn't be .append() either -- all of the methods that mutate lists return None.

As to why it is happening only at a certain size, perhaps you have some code like the above that is triggered only at a certain size, or else it is a red herring.

Tamp

@kindall: yes i use .append() at some point (i have edited my question). I still don't get the connection to some certain size when it happens... – Sopher

Python names are pointers to objects -- the name is being changed to point at None instead of the list. Python does not magically change pointers to lists to point to None when the lists get too big. Something in your code is doing that. I suggested a way it might be happening, but there could be other ways. Find it and fix it. – Tamp

@kindall: i have given more code now, maybe you can spot what i messed up? – Sopher

@solarisman: How does the rest of your code determine that the self.foo_thread has finished, so it's OK to now call transmit()? self.result shouldn't be accessed by any other threads until that time. – Unpopular

@solarisman: self.request could get changed to None if the object it is part of (some sort of TKinter Dialog I suspect) has gone out of scope since it was created and apply() got called. Not sure why that would only happen when the number in the list exceeds a certain size, though. – Unpopular

@martineau: i see that point. so now in my case, either way: progressbar works but this issue above or progressbar doesn't show up...i think i forget about multi threading for now – Sopher