runtime/cgo: pthread_create failed: Resource temporarily unavailable

Question

I currently have this function, that pipes multiple youtubedl commands thru ffmpeg, and then pipes the output of ffmpeg to an HTTP client.

func pipeThruFfmpegToMp4(vi *VideoInfo, rw web.ResponseWriter) error {
    var ffmpeg *exec.Cmd
    ffmpeg = exec.Command(
        "ffmpeg",
        "-i", "-",
        "-i", "pipe:3",
        "-c:v", "copy", "-c:a", "copy",
        "-preset", "veryfast",
        "-metadata", fmt.Sprintf(`title=%s`, vi.GetTitle()),
        "-movflags", "frag_keyframe+empty_moov",
        "-f", "mp4",
        "-")


    youtubevideo := exec.Command(YoutubeDLPath, "-c", "-f", fmt.Sprintf("%s/bestvideo[ext=mp4]/bestvideo/best", vi.GetFormat()), "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))
    fmt.Println(youtubevideo)

    youtube := exec.Command(YoutubeDLPath, "-c", "-f", "bestaudio[ext=m4a]/bestaudio/best", "--no-cache-dir", "--restrict-filenames", "--hls-prefer-native", "-o", "-", fmt.Sprintf("%s", vi.GetVideoUrl()))
    fmt.Println(youtube)

    var ytvbuf, ytbuf, ffbuf bytes.Buffer
    youtubevideo.Stderr = &ytvbuf
    youtube.Stderr = &ytbuf
    ffmpeg.Stderr = &ffbuf

    video, err := youtubevideo.StdoutPipe()
    if err != nil {
        log.Printf("pipeThruFfmpegToMp4: %v\n", err)
        return err
    }

    pipe3, err := youtube.StdoutPipe()
    if err != nil {
        log.Printf("pipeThruFfmpegToMp4: %v\n", err)
        return err
    }

    ffmpeg.Stdin = video
    ffmpeg.ExtraFiles = []*os.File{pipe3.(*os.File)}
    ffmpeg.Stdout = rw

    // Headers sent, no turning back now
    rw.Header().Set("Content-Type", "video/mp4")
    rw.Header().Set("Content-Disposition", fmt.Sprintf("attachment;filename=\"%s.mp4\"", vi.GetSlug()))
    rw.Flush()

    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    ffmpeg.Wait()
    youtubevideo.Wait()
    youtube.Wait()

    // check ytvbuf, ytbuf, ffbuf for stderr errors

    if ffbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ffbuf.String()})
        log.Printf("pipeThruFfmpegToMp4: %v\n", ffbuf.String())
    }
    if ytvbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ytvbuf.String()})
        log.Printf("pipeThruYouTubevDLToMp4: %v\n", ytvbuf.String())
    }
    if ytbuf.Len() != 0 {
        rollbar.Error(rollbar.ERR, err, &rollbar.Field{"stderr", ytbuf.String()})
        log.Printf("pipeThruYouTubeDLToMp4: %v\n", ytbuf.String())
    }

    return nil
}

The problem is that everything runs fine, but after a while the ram starts filling up on the server, till it gets to the point that it errors out with runtime/cgo: pthread_create failed: Resource temporarily unavailable

I'm unsure if it's a memory leak, or if either instance of youtube-dl is not closing right or if ffmpeg isn't closing right and just consuming more and more ram as the program runs more, until the program crashes with this error

runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f083501fe97 m=128 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f083501fe97
stack: frame={sp:0x7f05ff7fd7d0, fp:0x0} stack=[0x7f05feffe288,0x7f05ff7fde88)
00007f05ff7fd6d0:  00007f05ff7fd700  0000ffff00001fa0
00007f05ff7fd6e0:  00007f05ff7fdbe0  00007f05f8000da0
00007f05ff7fd6f0:  0000000000000000  000000000093032c
00007f05ff7fd700:  0000000000000000  00007f0835600ec3
00007f05ff7fd710:  0000000000000005  0000000000000000
00007f05ff7fd720:  000000c0000ce120  00007f0834ff1ce0
00007f05ff7fd730:  00007f05ff7fdaf0  00007f083560870a
00007f05ff7fd740:  0000000000000000  0000000000000000
00007f05ff7fd750:  0000000000000000  00007f05ff7fdbe0
00007f05ff7fd760:  2525252525252525  2525252525252525
00007f05ff7fd770:  000000ffffffffff  0000000000000000
00007f05ff7fd780:  000000ffffffffff  0000000000000000
00007f05ff7fd790:  000000c00010d1a0  000000c000953740
00007f05ff7fd7a0:  000000c0000ce120  000000c000cf2300
00007f05ff7fd7b0:  000000c00010d260  000000c001f4e180
00007f05ff7fd7c0:  000000c001f4e000  000000c00169f680
00007f05ff7fd7d0: <0000000000000000  000000c001f34180
00007f05ff7fd7e0:  6e75720000000000  6f67632f656d6974
00007f05ff7fd7f0:  0000000000000000  0000000000000000
00007f05ff7fd800:  000000c000cf2300  000000c00010d260
00007f05ff7fd810:  000000c001f4e180  000000c001f4e000
00007f05ff7fd820:  000000c00169f680  000000c00169f500
00007f05ff7fd830:  000000c001f34180  000000c001f34000
00007f05ff7fd840:  000000c000c92780  000000c001ec2600
00007f05ff7fd850:  fffffffe7fffffff  ffffffffffffffff
00007f05ff7fd860:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd870:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd880:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd890:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8a0:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8b0:  ffffffffffffffff  ffffffffffffffff
00007f05ff7fd8c0:  ffffffffffffffff  ffffffffffffffff
runtime: unknown pc 0x7f083501fe97
stack: frame={sp:0x7f05ff7fd7d0, fp:0x0} stack=[0x7f05feffe288,0x7f05ff7fde88)
00007f05ff7fd6d0:  00007f05ff7fd700  0000ffff00001fa0
00007f05ff7fd6e0:  00007f05ff7fdbe0  00007f05f8000da0
00007f05ff7fd6f0:  0000000000000000  000000000093032c
00007f05ff7fd700:  0000000000000000  00007f0835600ec3
00007f05ff7fd710:  0000000000000005  0000000000000000

I've also tried building the binary with CGO_ENABLED=0 even though I don't even have a import "c" but that also ends up erroring out as well with

runtime: failed to create new OS thread (have 21 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

my limit's are already very generouse, which maybe could also be a problem? Or do I maybe have to increase the pipe size?

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 192907
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 65536
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63883
virtual memory          (kbytes, -v) unlimited

Either way, I apprecate any and all the help I can get on this. Here is a link to the entire program and here is a thread in the golang-nuts forum regarding this questions

Adhibit

I implemented the rest of your program with mocks, and indeed it doesn't seem to release resource the way it should. There must be some data race in your program. Can you run it with the -race option (go build -race or go run -race). – Peirce

@Peirce When I run with -race it compiles fine – Adhibit

It compiles fine, does it run fine as well? – Peirce

@Peirce It does run with the -race build, but the error still shows the same. It spits out the block then go routines, showing the os/exec line – Adhibit

Answer 1

So after some debugging and with the help of Kevin's answer. It seems youtube-dl was not closing after it was successful.

Given that if ffmpeg finished, then it would not make any sense to have youtube-dl instance running, and since the Wait() function is waiting for youtube-dl to finish, but it never seems to send it's finish signal, I went ahead and had to add a kill to also kill the child sub processes.

So I had to add change this

    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    ffmpeg.Wait()
    youtubevideo.Wait()
    youtube.Wait()

To this

    youtube.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    youtubevideo.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    ffmpeg.Wait()
    syscall.Kill(-youtube.Process.Pid, syscall.SIGKILL)
    syscall.Kill(-youtubevideo.Process.Pid, syscall.SIGKILL)
    youtubevideo.Wait()
    youtube.Wait()

Please note, since it seems it makes some sort of child subprocess, youtube.Process.Kill() didn't actually kill the process, that's why I had to use syscall, and also Setpgid: true

Adhibit

Answer 2

I sometimes had the issues, where your code would not kill the youtube-dl processes. The video was converted, but the process was kept. As a first indicator watch the process count, like this:

ps | grep youtube-dl | wc -l

It is possible to let the subprocesses timeout, for one subprocess this would look like this:

package main

import (
    "bytes"
    "fmt"
    "os/exec"
    "time"
)

func main() {
    // Sleep is used, so we can control how long it runs.
    cmd := exec.Command("sleep", "2")

    // Use a bytes.Buffer to get the output
    var buf bytes.Buffer
    cmd.Stdout = &buf

    cmd.Start()

    // Use a channel to signal completion so we can use a select statement
    done := make(chan error)
    go func() { done <- cmd.Wait() }()

    // Start a timer, should be higher for your video conversion, 
    // I suggest you use a value that fits both your videos and server capabilities
    timeout := time.After(2 * time.Second)

    // The select statement allows us to execute based on which channel
    // we get a message from first.
    select {
    case <-timeout:
        // Timeout happened first, kill the process and print a message.
        cmd.Process.Kill()
        fmt.Println("Command timed out")
    case err := <-done:
        // Command completed before timeout. Print output and error if it exists.
        fmt.Println("Output:", buf.String())
        if err != nil {
            fmt.Println("Non-zero exit code:", err)
        }
    }
}

In your case you could replace

    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    ffmpeg.Wait()
    youtubevideo.Wait()
    youtube.Wait()

with

    ffmpeg.Start()
    youtubevideo.Start()
    youtube.Start()
    commands := 3

    done := make(chan error)
    go func() { done <- ffmpeg.Wait() }()
    go func() { done <- youtubevideo.Wait() }()
    go func() { done <- youtube.Wait() }()

    timeout := time.After(1 * time.Hour)

Loop:
    for {
        select {
        case <-timeout:
            ffmpeg.Process.Kill()
            youtubevideo.Process.Kill()
            youtube.Process.Kill()
            log.Println("Conversion timed out")
            break Loop
        case err := <-done:
            if err != nil {
                log.Println("Non-zero exit code:", err)
            }
            commands = commands - 1
            if commands == 0 {
                break Loop
            }
        }
    }

Caeoma

I will try this, one question though, since I don't know how long/large the video will be with youtube-dl, will this break youtube-dl earlier than it's suppose to? – Adhibit

One more question, now that you meantion that youtube-dl is hanging, would it not be the same instead of the time outs to do something like this instead ffmpeg.Start() youtubevideo.Start() youtube.Start() ffmpeg.Wait() youtubevideo.Wait() youtube.Wait() pipe3.Close() video.Close() – Adhibit

.Wait() will loop over the sub processes channel and wait until they will exit. So the pipe3.Close() and video.Close() will never be reached until the processes already stopped. Using a timeout youtube-dl might be stopped earlier especially when downloading longer videos, like 10h of nyan cat. – Caeoma

time.After(1 * time.Hour) breaks after an hour, if you think is too soon, make the timeout longer. – Caeoma

I tried this, and it never kills the youtube-dl processes. They still end up hanging and the program still end's up crashing the same way. I am currently trying it built with the -race condition. Perhaps instead of a timeout, kill youtube-dl as soon as ffmpeg exists? Since it does not appear that ffmpeg hangs. – Adhibit

Actually your solution and the command line timeout did not fix the problem, it still kept on crashing. I posted what finally fixed the problem. – Adhibit