Close multiple goroutine if an error occurs in one in go
Asked Answered
D

4

28

consider this function :

func doAllWork() error {

    var wg sync.WaitGroup

    for i := 0; i < 2; i++ {

        wg.add(1)
        go func() {

            defer wg.Done()
            for j := 0; j < 10; j++ {
                result, err := work(j)
                if err != nil {
                    // can't use `return err` here
                    // what sould I put instead ? 
                    os.Exit(0)
                }
            }
        }()
    }
    wg.Wait()

    return nil
}

In each goroutine, the function work() is called 10 times. If one call to work() returns an error in any of the running goroutines, I want all the goroutines to stop immediately, and the program to exit. Is it ok to use os.Exit() here ? How should I handle this ?


Edit: this question is different from how to stop a goroutine as here I need to close all goroutines if an error occurs in one

Dragone answered 4/8, 2017 at 7:37 Comment(4)
It's okay if you want to just shutdown the whole program. If you would like to isolate the effect to just closing to the spawned goroutines, you can use a channel to indicate all the goroutines to stop execution. Only change is that the goroutines will finish the execution of the current work() being executed before exiting.Tarnopol
Possible duplicate of how to stop a goroutineCryptogram
@Flimzy This is not a duplicate as in the marked duplicate only one goroutine (the controlling goroutine) may initiate cancel, but in this question multiple goroutines (any of the workers) may do so. Techniques applied there cannot be applied here directly (without additional work).Renick
BTW, I think it should be wg.Add(2) since the for loop would be executed for at most twice.Negotiant
R
58

You may use the context package which was created for things like this ("carries deadlines, cancelation signals...").

You create a context capable of publishing cancelation signals with context.WithCancel() (parent context may be the one returned by context.Background()). This will return you a cancel() function which may be used to cancel (or more precisely signal the cancel intent) to the worker goroutines.
And in the worker goroutines you have to check if such intent has been initiated, by checking if the channel returned by Context.Done() is closed, easiest done by attempting to receive from it (which proceeds immediately if it is closed). And to do a non-blocking check (so you can continue if it is not closed), use the select statement with a default branch.

I will use the following work() implementation, which simulates a 10% failure chance, and simulates 1 second of work:

func work(i int) (int, error) {
    if rand.Intn(100) < 10 { // 10% of failure
        return 0, errors.New("random error")
    }
    time.Sleep(time.Second)
    return 100 + i, nil
}

And the doAllWork() may look like this:

func doAllWork() error {
    var wg sync.WaitGroup

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel() // Make sure it's called to release resources even if no errors

    for i := 0; i < 2; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()

            for j := 0; j < 10; j++ {
                // Check if any error occurred in any other gorouties:
                select {
                case <-ctx.Done():
                    return // Error somewhere, terminate
                default: // Default is must to avoid blocking
                }
                result, err := work(j)
                if err != nil {
                    fmt.Printf("Worker #%d during %d, error: %v\n", i, j, err)
                    cancel()
                    return
                }
                fmt.Printf("Worker #%d finished %d, result: %d.\n", i, j, result)
            }
        }(i)
    }
    wg.Wait()

    return ctx.Err()
}

This is how it can be tested:

func main() {
    rand.Seed(time.Now().UnixNano() + 1) // +1 'cause Playground's time is fixed
    fmt.Printf("doAllWork: %v\n", doAllWork())
}

Output (try it on the Go Playground):

Worker #0 finished 0, result: 100.
Worker #1 finished 0, result: 100.
Worker #1 finished 1, result: 101.
Worker #0 finished 1, result: 101.
Worker #0 finished 2, result: 102.
Worker #1 finished 2, result: 102.
Worker #1 finished 3, result: 103.
Worker #1 during 4, error: random error
Worker #0 finished 3, result: 103.
doAllWork: context canceled

If there would be no errors, e.g. when using the following work() function:

func work(i int) (int, error) {
    time.Sleep(time.Second)
    return 100 + i, nil
}

The output would be like (try it on the Go Playground):

Worker #0 finished 0, result: 100.
Worker #1 finished 0, result: 100.
Worker #1 finished 1, result: 101.
Worker #0 finished 1, result: 101.
Worker #0 finished 2, result: 102.
Worker #1 finished 2, result: 102.
Worker #1 finished 3, result: 103.
Worker #0 finished 3, result: 103.
Worker #0 finished 4, result: 104.
Worker #1 finished 4, result: 104.
Worker #1 finished 5, result: 105.
Worker #0 finished 5, result: 105.
Worker #0 finished 6, result: 106.
Worker #1 finished 6, result: 106.
Worker #1 finished 7, result: 107.
Worker #0 finished 7, result: 107.
Worker #0 finished 8, result: 108.
Worker #1 finished 8, result: 108.
Worker #1 finished 9, result: 109.
Worker #0 finished 9, result: 109.
doAllWork: <nil>

Notes:

Basically we just used the Done() channel of the context, so it seems we could just as easily (if not even easier) use a done channel instead of the Context, closing the channel to do what cancel() does in the above solution.

This is not true. This can only be used if only one goroutine may close the channel, but in our case any of the workers may do so. And attempting to close an already closed channel panics (see details here: How does a non initialized channel behave?). So you would have to ensure some kind of synchronization / exclusion around the close(done), which will make it less readable and even more complex. Actually this is exactly what the cancel() function does under the hood, hidden / abstracted away from your eyes, so cancel() may be called multiple times to make your code / use of it simpler.

How to get and return the error(s) from the workers?

For this you may use an error channel:

errs := make(chan error, 2) // Buffer for 2 errors

And inside the workers when an error is encountered, send it on the channel instead of printing it:

result, err := work(j)
if err != nil {
    errs <- fmt.Errorf("Worker #%d during %d, error: %v\n", i, j, err)
    cancel()
    return
}

And after the loop, if there was an error, return that (and nil otherwise):

// Return (first) error, if any:
if ctx.Err() != nil {
    return <-errs
}
return nil

Output this time (try this on the Go Playground):

Worker #0 finished 0, result: 100.
Worker #1 finished 0, result: 100.
Worker #1 finished 1, result: 101.
Worker #0 finished 1, result: 101.
Worker #0 finished 2, result: 102.
Worker #1 finished 2, result: 102.
Worker #1 finished 3, result: 103.
Worker #0 finished 3, result: 103.
doAllWork: Worker #1 during 4, error: random error

Note that I used a buffered channel with a buffer size equal to the number of workers, which ensures sending on it is always non-blocking. This also gives you the possibility to receive and process all errors, not just one (e.g. the first). Another option could be to use a buffered channel to hold only 1, and do a non-blocking send on it, which could look like this:

errs := make(chan error, 1) // Buffered only for the first error

// ...and inside the worker:

result, err := work(j)
if err != nil {
    // Non-blocking send:
    select {
    case errs <- fmt.Errorf("Worker #%d during %d, error: %v\n", i, j, err):
    default:
    }
    cancel()
    return
}
Renick answered 4/8, 2017 at 9:4 Comment(3)
Hey I loved your answer, had a query. In your doAllWork func your cancel function was created OUTSIDE of the goroutines. Is that good practice? I tried creating the CancelFunc inside the goroutine scope but my IDE throws the the cancel function is not used on all paths (possible context leak)lostcancel warning. I feel it is safe in this situation(All goroutines should stop if one fails, but continue if sucesses). What could be the drawbacks of this method?Enucleate
This doesn't feel right to me. ctx.Done() is checked before initiating work(), but what if one goroutine experiences an error while another is in the middle of its work() function? I was expecting it to be possible to send the equivalent of a SIGTERM to the other goroutines.Adulteration
@CameronHudson You can't cancel a goroutine, see cancel a blocking operation in Go. The goroutine has to finish voluntarily (return).Renick
S
14

A more clear way to go here is to use errgroup (documentation).

Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.

You can check it out in this example (playground):

    var g errgroup.Group
    var urls = []string{
        "http://www.golang.org/",
        "http://www.google.com/",
        "http://www.somestupidname.com/",
    }

    for _, url := range urls {
        // Launch a goroutine to fetch the URL.
        url := url // https://golang.org/doc/faq#closures_and_goroutines
        
       g.Go(func() error {
            // Fetch the URL.
            resp, err := http.Get(url)
            if err == nil {
                resp.Body.Close()
            }
            return err
        })
    }
   
    // Wait for all HTTP fetches to complete.
    if err := g.Wait(); err == nil {
        fmt.Println("Successfully fetched all URLs.")
    
    } else {

        // After all have run, at least one of them has returned an error!
       // But all have to finish their work!
       // If you want to stop others goroutines when one fail, go ahead reading!
        fmt.Println("Unsuccessfully fetched URLs.")
    }

But attention: The first call to return a non-nil error cancels the group phrase in the Go documentation is a little bit misleading.

In fact, errgroup.Group if created with a context (WithContext function), will call the cancel function of the context returned by WithContext when a goroutine in the group will return an error, otherwise nothing will be done (read the source code here!).

So, if you want to close your different goroutines, you must use the context returned my WithContext and manage it by yourself inside them, errgroup will just close that context! Here you can find an example.

To summarize, errgroup can be used in different ways, as shown by the examples.

  1. "just errors", as the above example: Wait wait that all goroutines end, and then returns the first non-nil error if any from them, or return nil.

  2. In parallel: You have to create the group with the WithContext function and use the context to manage the context closing. I created a playground example here with some sleeps! You have to manually close each goroutines, but using the context you can end them when one close the context.

  3. Pipelines (see more in the examples).

Schmaltzy answered 5/3, 2020 at 14:50 Comment(2)
Hi, I have a question for you, I modified your example a little play.golang.org/p/ky2DkCZBkyR, and if I understand correctly all goroutines (2,3) should be canceled as soon as goroutine (4) returns the error. But it's not happened, goroutines (2,3) still works until their end. What am I missing?Silvey
Hi @Silvey ! You made me think about this :) thank you! I edited my answer, I hope it's correct now and more clear! Let me know if you understand now!Schmaltzy
M
2

Since go1.20, context package now has WithCancelCause which can be used for error propagation. See https://pkg.go.dev/context#WithCancelCause

ctx, cancel := context.WithCancelCause(parent)
cancel(myError)
ctx.Err() // returns context.Canceled
context.Cause(ctx) // returns myError
Micamicaela answered 1/11, 2023 at 4:9 Comment(0)
G
0

Another way to go here is to use errgroup.WithContext. You can check it out in this example.

In short, g.Wait() waits for the first error to happen or for all to finish without errors. When error happens in any of the goroutines (timeout in the provided example), it cancels the execution in other goroutines through ctx.Done() channel.

Glossary answered 15/11, 2019 at 13:10 Comment(2)
however, the go routine spin using this context do not accept argument :(Dilate
It's the same scope so you don't have to pass in any arguments.Gough

© 2022 - 2024 — McMap. All rights reserved.