GCD serial queue does not seem to execute serially

Question

I have a method that at times can be invoked throughout my code. Below is a very basic example, as the code processes images and files off of the iphone photo gallery and marks them already processed when done with the method.

@property (nonatomic, assign) dispatch_queue_t serialQueue;

....

-(void)processImages
{
    dispatch_async(self.serialQueue, ^{
        //block to process images
        NSLog(@"In processImages");

        ....

        NSLog(@"Done with processImages");
    });
}

I would think that each time this method is called I would get the below output... "In processImages" "Done with processImages" "In processImages" "Done with processImages" etc...

but I always get

"In processImages" "In processImages" "Done with processImages" "Done with processImages" etc...

I thought a serial queue would wait till the first block is done, then start. To me it seems it is starting the method, then it gets called again and starts up before the first call even finishes, creating duplicates of images that normally would not be processed due to the fact that if it really executed serially the method would know they were already processed. Maybe my understanding of serial queues is not concrete. Any input? Thank you.

EDIT:MORE Context below, this is what is going on in the block...Could this cause the issue???

@property (nonatomic, assign) dispatch_queue_t serialQueue;

....

-(void)processImages
{
    dispatch_async(self.serialQueue, ^{
        //library is a reference to ALAssetsLibrary object 

        [library enumerateGroupsWithTypes:ALAssetsGroupSavedPhotos usingBlock:^(ALAssetsGroup *group, BOOL *stop)
        {
            [group enumerateAssetsUsingBlock:^(ALAsset *asset, NSUInteger index, BOOL *stop)
            {
             ....
             //Process the photos here
            }];
        failureBlock:^(NSError *error) { NSLog(@"Error loading images from library");
        }];

    });
}

-(id)init
{
    self = [super init];
    if(self)
    {
        _serialQueue = dispatch_queue_create("com.image.queue",NULL);
    }
    return self;
}

this object is only created once, and as far as I can tell can never be created again based off my code...I will run tests to make sure though.

UPDATE 2: WHAT I THINK IS HAPPENING, please comment on this if you agree/disagree....

Obviously my main issue is that it seems this block of code is being executed concurrently, creating duplicate entries (importing the same photo twice) when it wouldn't normally do this if it was run serially. When a photo is processed a "dirty" bit is applied to it ensuring the next time the method is invoked it skips this image, but this is not happening and some images are processed twice. Could this be due to the fact I am enumerating the objects in a second queue using enumerategroupswithtypes: within that serialQueue?

1. call processImages
2. enumerateObjects
3. immediately return from enumerateObjects since it is async itself
4. end call to processImages

processImages is not really done though due to the fact that enumerategroups is probably still running but the queue might thing it is done since it reaches the end of the block before enumerategroups is finished working. This seems like a possibility to me?

Phlegethon

I wonder if you're accidentally calling dispatch_queue_create multiple times. Perhaps put a NSLog statement there. Your expectation is correct (that it should be strictly serial), but there's something else simple going on. – Corbin

without more context it's hard to answer but your assumption is true. I wonder if you're using multiple objects that have their own queues. Make sure your queue is shares across all of them. Try printing the current thread and the value of serialQueue in your NSLogs – Twister

Can you set a breakpoint at the dispatch_async call and verify serialQueue is the same object each time processImages is called. Sounds like your queue is being recreated. – Lacunar

I added more context to what is happening in the block...I wouldn't think what I am doing can cause an issue, but you never know – Phlegethon

You're either creating multiple queues or not creating a serial queue. Could you share the actual code where you create the queue? – Onestep

Please see additional context above... – Phlegethon

Answer 1

Serial Queues ABSOLUTELY will perform serially. They are not guaranteed to perform on the same thread however.

Assuming you are using the same serial queue, the problems is that NSLog is NOT guaranteed to output results in the proper order when called near simultaneously from different threads.

here is an example:

1. SQ runs on thread X, sends "In processImages"
2. log prints "In proc"
3. SQ on thread X, sends "Done with processImages"
4. SQ runs on thread Y, sends "In processImages"
5. log prints "essImages\n"

After 5., NSLog doesn't necessarily know which to print, 3. or 4.

If you absolutely need time ordered logging, You need a dedicated queue for logging. In practice, I've had no problems with just using the main queue:

dispatch_async(dispatch_get_main_queue(), ^{
    NSLog(@"whatever");
});

If all NSlog calls are the on the same queue, you shouldn't have this problem.

Mussulman

Answer 2

enumerateGroupsWithTypes:usingBlock:failureBlock: does its work asynchronously on another thread and calls the blocks passed in when it's done (on the main thread I think). Looking at it from another perspective, if it completed all the synchronously by the time the method call was complete, it could just return an enumerator object of the groups instead, for instance, for a simpler API.

From the documentation:

This method is asynchronous. When groups are enumerated, the user may be asked to confirm the application's access to the data; the method, though, returns immediately. You should perform whatever work you want with the assets in enumerationBlock.

I'm not sure why you're trying to accomplish by using the serial queue, but if you just want to prevent simultaneous access, then you could just add a variable somewhere that keeps track of whether we're currently enumerating or not and check that at first, if you don't have to worry about synchronization issues. (If you do, perhaps you should look into using a GCD group, but it's probably overkill for this situation.)

Underwriter

The reason why I am using a serial queue/GCD is because without it, it does lock up the UI when enumerating through pictures. Lots of image processing is going on from resizing to facial recognition within that block. When the phone first loads the app it calls this method. In the background the method can be called. Sometimes these events could possibly conflict, that is why I wanted to create a serial queue, so that if the event kicks off, and then while its processing the event is called again, I can place it in a serial queue to execute when done. – Phlegethon

Currently I do use a BOOL variable to determine if we are currently enumerating. I just figured using a serial queue would work better. But it looks like it won't. See my reason under "UPDATE 2: WHAT I THINK IS HAPPENING" in my original post. – Phlegethon

Answer 3

If the question is "Can serial queue perform tasks asynchronously?" then the answer is no. If you think that it can, you should make sure that all tasks are really performing on the same queue. You can add the following line in the block and compare the output:

dispatch_async(self.serialQueue, ^{
    NSLog(@"current queue:%p current thread:%@",dispatch_get_current_queue(),[NSThread currentThread]);

Make sure that you write NSLog in the block that performs on your queue and not in the enumerateGroupsWithTypes:usingBlock:failureBlock: Also you can try to create your queue like this

dispatch_queue_create("label", DISPATCH_QUEUE_SERIAL);

but I don't think that will change anything

EDIT: By the way, method

enumerateGroupsWithTypes:usingBlock:failureBlock:

is asynchronous, why do you call it on another queue?

UPDATE 2: I can suggest something like this:

dispatch_async(queue, ^{
    NSLog(@"queue");

    pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER, *pmutex = &mutex;
    pthread_mutex_lock(pmutex);

    ALAssetsLibraryGroupsEnumerationResultsBlock listGroupBlock = ^(ALAssetsGroup *group, BOOL *stop) {
        NSLog(@"block");
        if (group) {
            [groups addObject:group];
        } else {

            [self.tableView performSelectorOnMainThread:@selector(reloadData) withObject:nil waitUntilDone:NO];
            dispatch_async(dispatch_get_current_queue(), ^{
                pthread_mutex_unlock(pmutex);
            });
        }
        NSLog(@"block end");
    };

    [assetsLibrary enumerateGroupsWithTypes:groupTypes usingBlock:listGroupBlock failureBlock:failureBlock];
    pthread_mutex_lock(pmutex);
    pthread_mutex_unlock(pmutex);
    pthread_mutex_destroy(pmutex);
    NSLog(@"queue end");
});

Zacarias

so you are saying an asynchronous queue cannot be run serially like I have done above? – Phlegethon

Serial queue will always perform serially. Your problem is somewhere else. – Zacarias

thank you, please see my update 2 based off of your EDIT statement. – Phlegethon

Is it possible, that you call -(void) processImages method twice? If it is, I guess enumerateGroupsWithTypes can be executing several times concurrently, it may cause duplicates. I just found this line in the documentation: "When the enumeration is done, enumerationBlock is invoked with group set to nil." Maybe you should use some kind of lock to wait until enumeration is finished. – Zacarias

I definitely call processImages twice, three times, or more. That is the main reason I wanted to use a serial queue, to ensure its processed in a FIFO order. So I guess it is possible that it is serially running, but enumerateGroupsWithTypes has yet to finish. – Phlegethon

Answer 4

I hit an issue like this, and the answer for me was to realize that asynchronous calls from a method on the serialized queue goes to another queue for processing -- one that is not serialized.

So you have to wrap all the calls inside the main method with explicit dispatch_async(serializedQueue, ^{}) to ensure that everything is done in the correct order...

Looming

Answer 5

Using Swift and semaphores to illustrate an approach to serialization:

Given: a class with an asynchronous ‘run’ method that will be run on multiple objects at once, and the objective is that each not run until the one before it completes.

The issue is that the run method allocates a lot of memory and uses a lot of system resources that can cause memory pressure among other issues if too many are run at once.

So the idea is: if a serial queue is used then only one will run at a time, one after the other.

Create a serial queue in the global space by the class:

let serialGeneratorQueue: DispatchQueue = DispatchQueue(label: "com.limit-point.serialGeneratorQueue", autoreleaseFrequency: DispatchQueue.AutoreleaseFrequency.workItem)

class Generator {

    func run() {
         asynchronous_method()
    }

    func start() {

        serialGeneratorQueue.async {
            self.run()
        }
    }

    func completed() {
       // to be called by the asynchronous_method() when done
    }
}

The ‘run’ method of this class for which very many objects will be created and run will be processed on the serial queue:

serialGeneratorQueue.async {
    self.run()
}

In this case an autoreleaseFrequency is .workItem to clean up memory after each run.

The run method is of some general form:

func run() {
   asynchronous_method()
}

The problem with this: the run method exits before the asynchronous_method completes, and the next run method in the queue will run, etc. So the objective is not being achieved because each asynchronous_method is running in parallel, not serially after all.

Use a semaphore to fix. In the class declare

let running = DispatchSemaphore(value: 0)

Now the asynchronous_method completes it calls the ‘completed’ method:

func completed() {
   // some cleanup work etc.
}

The semaphore can be used to serialized the chain of asynchronous_method’s by add ‘running.wait()’ to the ‘run’ method:

func run() {
    asynchronous_method()

    running.wait() 
}

And then in the completed() method add ‘running.signal()’

func completed() {
   // some cleanup work etc.

    running.signal()
}

The running.wait() in ‘run’ will prevent it from exiting until signaled by the completed method using running.signal(), which in turn prevents the serial queue from starting the next run method in the queue. This way the chain of asynchronous methods will indeed be run serially.

So now the class is of the form:

class Generator {

    let running = DispatchSemaphore(value: 0)

    func run() {
         asynchronous_method()

         running.wait() 
    }

    func start() {

        serialGeneratorQueue.async {
            self.run()
        }
    }

    func completed() {
       // to be called by the asynchronous_method() when done

       running.signal()
    }
}

Contagious

Answer 6

I thought a serial queue would wait [until] the first block is done ...

It does. But your first block simply calls enumerateGroupsWithTypes and the documentation warns us that the method runs asynchronously:

This method is asynchronous. When groups are enumerated, the user may be asked to confirm the application's access to the data; the method, though, returns immediately.

(FWIW, whenever you see a method that has a block/closure parameter, that’s a red flag that the method is likely performing something asynchronously. You can always refer to the relevant method’s documentation and confirm, like we have here.)

So, bottom line, your queue is serial, but it is only sequentially launching a series of asynchronous tasks, but obviously not waiting for those asynchronous tasks to finish, defeating the intent of the serial queue.

So, if you really need to have each tasks wait for the prior asynchronous task, there are a number of traditional solutions to this problem:

1. Use recursive pattern. I.e., write a rendition of processImage that takes an array of images to process and:

   • check to see if there are any images to process;
   • process first image; and
   • when done (i.e. in the completion handler block), remove the first image from the array and then call processImage again.

2. Rather than dispatch queues, consider using operation queues. Then you can implement your task as an “asynchronous” NSOperation subclass. This is a very elegant way of wrapping an asynchronous task This is illustrated in https://mcmap.net/q/260059/-nsurlsession-with-nsblockoperation-and-queues.

3. You can use semaphores to make this asynchronous task behave synchronously. This is also illustrated in https://mcmap.net/q/260059/-nsurlsession-with-nsblockoperation-and-queues.

Option 1 is the simplest, option 2 is the most elegant, and option 3 is a fragile solution that should be avoided if you can.

Corbin

Answer 7

You might have more than one object, each with its own serial queue. Tasks dispatched to any single serial queue are performed serially, but tasks dispatched to different serial queues will absolutely be interleaved.

Another simple bug would be to create not a serial queue, but a concurrent queue...

Ptyalin