Download content video from video stream with a path of .TS or .m3u8 file through actual code so i can make chrome extension [closed]

Question

Videos on most sites make use of progressive downloading, which means that the video is downloaded to my computer, and easy to trace. There are lots of extensions out there to do this, and even in the dev-tools this is easily done.

On certain websites videos are streamed. which means that we do no just download 1 file, we download lots of small packages. In the dev-tools these packages can be traced. The website I'm interested in is: http://www.rtlxl.nl/#!/goede-tijden-slechte-tijden-10821/c8e2bff7-5a5c-45cb-be2b-4b3b3e866ffb.

-The packages have a .TS extension.

-Packages can be saved by copying the url of the request

-I can not play these files.

I must have done something wrong, or I'm missing something. I want to know what I am doing wrong. I want to create a chrome extension for personal use which captures the urls of all the packages. when I have all the urls I want to pass them on to a php scripts which downloads them and uses ffmpeg to paste them into a mp4 file.

Please guide me how to download the content of these files.

Isogonic

if you are on linux see this post gistpages.com/posts/download-streaming-video-ts-files – Preventer

downloadhelper.net/install chrome/firefox addon, downloads streaming videos in one click – Preventer

There are 2 things keep in mind: [1] You probably need decrypt .ts file with AES key, this one still can be done by simple script. [2] Need remuxing or else will not seekable with few seconds and corrupt at the end because of some parts. This is the hard part and need ffmpeg. – Guillotine

you can use hls-downloader chrome extension – Preventer

Answer 1

Easy youtube-dl example on macOS (in the command line Terminal; Windows supported too):

# List variants (resolutions/bitrates)
$ youtube-dl -F https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Requesting header
[generic] f08e80da-bf1d-4e3d-8899-f0f6155f6efa: Downloading m3u8 information
[info] Available formats for f08e80da-bf1d-4e3d-8899-f0f6155f6efa:
format code           extension  resolution note
audio-English_stereo  mp4        audio only [en] 
628                   mp4        320x180     628k , avc1.42c00d, video only
928                   mp4        480x270     928k , avc1.42c00d, video only
1728                  mp4        640x360    1728k , avc1.42c00d, video only
2528                  mp4        960x540    2528k , avc1.42c00d, video only
4928                  mp4        1280x720   4928k , avc1.42c00d, video only
9728                  mp4        1920x1080  9728k , avc1.42c00d, video only (best)

# Choose a variant to download, and use its format code below
$ youtube-dl --format 628 https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
...
frame= 5257 fps=193 q=-1.0 Lsize=    6746kB time=00:03:30.16 bitrate= 263.0kbits/s speed=7.73x    
video:6679kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.998669%
[ffmpeg] Downloaded 6907810 bytes
[download] 100% of 6.59MiB in 00:29

$ open f08e80da-bf1d-4e3d-8899-f0f6155f6efa-f08e80da-bf1d-4e3d-8899-f0f6155f6efa.mp4

Use the browser's Developer Tools > Network to get the m3u8 (HLS manifest) URL when starting a streaming video.

Unduly

Worked for me. I had to install also ffmpeg. As linux debian user had to exec "apt-get install ffmpeg" Just a claritication. On the 2nd command, the value of paramenter--format, (628) means resolution and should be chosen from the previous list – Darcee

This is awesome! I wonder if this can be automated to find the "best"? – Enclose

@frank-fu Just omit the "format" parameter, and youtube-dl will automatically select the "best" i.e. youtube-dl https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8 – Relict

Have to brew install ffmpeg because of this error on macos ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one. – Vish

Worked perfectly. Required installation of youtube-dl and ffmpeg brew install youtube-dl brew install ffmpeg and then youtube-dl "<url-to-m3u8-file>" and just like magic it combined both audio and video of the file – Armstead

Answer 2

Addition to @aalhanane and @Micheal Espinola Jr

As m3u8x is only available for windows. Once you have identified the m3u8 url you can also use Jdownloader2 or VLC Media Player to download and concatenate the stream.

Jdownloader2: Just copy the m3u8 url when it the Jdownloader is open. It will recognize the stream in Linkgrabber tab.

VLC 3:

Open Network -> Paste m3u8 url -> Checkmark Streamoutput -> Select Settings. Choose output file, container , video and audio encoding. (e.g output.mp4, container: mpeg4, video: h264, audio: mp4a) Start Stream. It will not play the video, but encode it, showing the encoding progress by moving the video play back progress bar.

WARNING: Previously suggesteed chrome extension Stream Video Downloader contains malware. See reddit post

Corral

Thanks, Downloaded full video with program jDownloader2 , even host used Wowza Streaming Engine – Craftwork

I am trying to download video from elearning website. I coped m3u8 file and tried all methods like VLC, youtube-dl, Jdownloader2 etc. Still unable to download but I can stream. For this site I have to login to stream videos. Even Internet Download Manager IDM fails to download saying "cannot download this protected stream". So, is there anyway I can download protected/encrypted streams without video recording. Please help. – Diffidence

pls post the website. if you have to login to watch it, the site can take additional measures to protect the content only to a logged in user. The methods above wont work then. Also there are DRM measures like widevine which make sure, only your browser gets proper decryption key, but not any other program. In this case, the only way is screen and oudio output capture – Corral

Answer 3

---> Open Firefox

---> open page the video

---> Play Video

Click ---> Open menu

Click ---> open web developer tools

Click ---> Developer Toolbar

Click ---> Network

---> Go to Filter URLs ---> Write "M3u8" --> for Find "m3u8"

---> Copy URL ".m3u8"

========================

Now Download software "m3u8x" ----> https://tajaribsoft-en.blogspot.com/2016/06/m3u8x.html#downloadx12

---> open software "m3u8x"

---> paste URL "m3u8"

---> chose option "One...One"

---> Click Download

---> Start Download

========================

image "Open menu" ===>

image "Developer Toolbar" ===>

image "m3u8x" ===>

Gauntlett

There is no .m3u8 file in the network tab, there are only .ts files. – Acquittal

That's awesome but you did not mention that after "paste URL "m3u8"" you have to type name for the file, for example "video" then click on icon of hand next to "quality" and only after that you can select "one on one" and "download". As for the rest, works good! Thanks – Southdown

If you only see .ts files, you need to refresh the page to get the .m3u8 link which should be the first url to load – Prophet

Answer 4

using this post

• Open Firefox / chrome

• open page the video

• Play Video

• click F12 on keyboard -> network

• in Filter URLs ts

• copy link of ts

• remove index and ts extension from link

  for example:

  http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
  

  will be copied as

   http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000
  

insert in below script under LINK

#!/bin/bash

# insert here urls
LINK=(

'http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000' # replace this with your url 

)

mkdir my-videos
cd mkdir my-videos

CNT=0

for URL in ${LINK[@]}
do
  # create folder for streaming media
  CNT=$((CNT + 1))
  mkdir $CNT
  cd $CNT

  (

   DIR="${URL##*/}"

   # download all videos
   wget $URL'_'{0..1200}.ts

   # link videos
   echo $DIR'_'{0..1200}.ts | tr " " "\n" > tslist
   while read line; do cat $line >> $CNT.mp4; done < tslist

   rm -rf media* tslist
   ) &
   cd ..

done

wait

EDIT

adding script in python - runs on windows and linux

import urllib.request
import os
import shutil

my_lessons = [
   #  http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000_454.ts
    "http://vid.com/vod/mp4:vod/PRV/Yg0WGN_6.mp4/media_b180000" # replace this with your url 


]

lesson_dir = "my_vids"
try:
    shutil.rmtree(lesson_dir)
except:
    print "ok"

os.makedirs(lesson_dir)
os.chdir(lesson_dir)

for lesson, dwn_link in enumerate(my_lessons):
    print ("downloading lesson  %d.. " % (lesson), dwn_link)
    file_name = '%04d.mp4' % lesson
    f = open(file_name, 'ab')
    for x in range(0, 1200):
        try:
            rsp = urllib.request.urlopen(dwn_link + "_%04d.ts" % (x) )
        except:
            break
        file_name = '%d.mp4' % lesson
        print "downloading  %d.ts" % (x)
        f.write(rsp.read())
    f.close()



print "done good luck!! ==================  "

if the script fails, or downloads empty file, try removing the try wrap to see what fails

Preventer

Very nice. Result was not very good for my videos if I concatenated like that though. There were pretty annoying visual artefacts at junctions when I watched with VLC. The best result I got with ffmpeg "concat protocol". It was not perfect either, as VLC had some trouble moving backward and forward in big steps. – Irresponsive

... that forward/backward thing was not a problem for all videos though. – Irresponsive

This should've been the accepted answer, because it actually shows what is going on behind the scenes (more or less, at least, if the concatenation is indeed not that reliable), instead of just pointing at an already finished tool, without explaining how the tool makes this work, in the first place. – Padraig

Answer 5

While this shouldn't have ever been asked on SO and got through the vetting processing in the first place, I have no idea... but I'm giving my answer anyway.

After exploring basically all of the options presented here, it turns out the simplest is often the best.

First download ffmpeg from: https://evermeet.cx/ffmpeg/

Next, after you have got your .m3u8 playlist file (most probably from the webpage source or network traffic), run this command:

ffmpeg -i "http://host/folder/file.m3u8" -bsf:a aac_adtstoasc -vcodec copy -c copy -crf 50 file.mp4

I tried running it from a locally saved m4u8 file, and it didn't work, because the ffmpeg download procedure downloads the chunks which are relative to the URL, so make sure you use the website url.

Darceydarci

Ho yeah! Like a charm! – Or

Here's more of what is happening from my GIST - gist.github.com/vrdriver/bf9746af1b1ca8a544826a2be87d33ae – Darceydarci

It would've helped a lot, if you would've explained, what the command line options are doing and why they are needed or how they could possible be adjusted. – Padraig

You can read about the options in the ffmpeg documentation. I found an example that worked. ffmpeg.org/ffmpeg.html – Darceydarci

This is the best answer. – Frightened

Elegance for the win. – Falchion

Superb! Exactly what I was looking for ⭐⭐⭐⭐⭐ – Postman

Answer 6

As simple as these two commands:

wget https://example.com/videos/tschunks_{0..10}.ts

cat tschunks_{0..10}.ts > video.mp4

*- where "https://example.com/videos/tschunks_{0..10}"-part can be found in m3u8 file

Vani

Answer 7

You would need to download all of the transport stream (.ts) files, and concatenate them into a single mpeg for playback. Transport streams such as this have associated playlist files (.m3u8) that list all of the .ts files that you need to download and concatenate. If available, there may be a secondary .m3u8 playlist that will separately list subtitle steam files (.vtt).

Sonometer

Answer 8

I needed to download HLS video and audio streams from a e-learning portal with session-protected content with application/mp2t MIME content type.

Manually copying all authentication headers into the downloading scripts would be too cumbersome.

But the task got much easier with help of Video DownloadHelper Firefox extension and it's Companion App. It allowed to download both m3u8 playlists with TS chunks lists and actual video and audio streams into mp4 files via a click of button while correctly preserving authentication headers.

The resulting separate video and audio files can be merged with ffmpeg:

ffmpeg -i video.mp4 -i audio.mp4 -acodec copy -vcodec copy video-and-audio.mp4 

or with mp4box:

mp4box -add audio.mp4#audio video.mp4 -out video-and-audio.mp4

Tried Video DownloadHelper Chrome extension too, but it didn't work for me.

Coracorabel

forum.videohelp.com/threads/… also mentions livestreamer alternative which was recently forked as streamlink, but it doesn't handle authentication. – Coracorabel

2020 vid helper 7.3.9. and ff 80 and this works one click :) – Duet

Answer 9

• Download VLC Player
• Media
• Convert/Save
• Network (Tab)
• Enter URL of [playlist].m3u8
• Follow remaining wizard steps to set the stream destination (File)
• Set appropriate transcoding profile (MP4 at the time of this answer)
• Watch video

Dangle

Answer 10

Many websites package their videos as multiple Transport Stream files (.ts). If you wanted to output the full length video you would need to download all .ts file pieces and assemble them. The assembling can easily be done using the FFmpeg command line tool.

In case anyone is looking for a Java way of downloading TS files from video stream here's a good utility example and article that also goes into detail on how TS and M3U8 files work >> https://jet-cabral.medium.com/java-ts-video-downloader-a0fcf23ab84a

Once all .ts files are downloaded the ffmpeg command to assemble/concatenate the videos looks like so:

ffmpeg -i "concat:C:\users\your-name\file_0.ts|C:\users\your-name\file_1.ts|C:\users\your-name\file_2.ts|C:\users\your-name\file_3.ts

If you already know the url of the .m3u8 file that contains the .ts files you want to download. Then the following command downloads all the .ts files and outputs a single .mp4 file

ffmpeg -i "http://www.some-website.com/video/someM3U8_file.m3u8" -c copy -bsf:a aac_adtstoasc "full_video.mp4"

Freddafreddi

Answer 11

I came up with an efficient parralelized one-line that concatenate a sequence of .ts files into one .mp4 file using GNU Parallel:

 parallel -k curl https://example.com/video/seg-{}-f4-v1-a1.ts ::: {1..279} >> result.mp4

The key is to replace the variant part of your url with {} and to set {a..b} with the lower and upper bound. That's it!

Physicality

Answer 12

Copy and paste one of the .ts video files into a new tab in Chrome. Remove the identifying number of the .ts file (0,1,2,3 etc. or whatever number it is) and change the extension from ".ts" to ".mp4". That should bring up the video file in your browser as usual.

Comedian

Answer 13

I made some changes to dina's answer to avoid attempting to download/combine 1200 parts if there aren't that many.

I also found it helpful to sort by waterfall in the network tab of chrome. This will sort by the time the files are downloaded, so when you are streaming a video the most recently downloaded parts will be at the top, making it easy to find the .ts links.

#!/bin/bash

# Name of the containing folder
GROUP="My Videos"

# Example link: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE_0.ts
# Insert below as: https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE

# INSERT LINKS TO VIDEOS HERE
LINK=(
'Title for the video link'
'https://vids.net/ABCAED/AADDCDE/m3u8/AADDCDE/AADDCDE'
'Title for the next video'
'https://vids.net/EECEADFE/EECEADFE/m3u8/EECEADFE/EECEADFE'
)

# ------------------------------------------------------------------------------
mkdir "$GROUP"
cd "$GROUP"

I=0
while [ $I -lt ${#LINK[@]} ]
do
  # create folder for streaming media
  TITLE=${LINK[$I]}
  mkdir "$TITLE"
  cd "$TITLE"
  mkdir 'parts'
  cd 'parts'

  J=$((I + 1))
  URL=${LINK[$J]}

  I=$((I + 2))

  DIR="${URL##*/}"

  # download all streaming media parts
  VID=-1
  while [ $? -eq 0 ];
  do
    VID=$((VID + 1))
    wget $URL'_'$VID.ts
  done

  # combine parts
  COUNTER=0
  while [  $COUNTER -lt $VID ]; do
    echo $DIR'_'$COUNTER.ts | tr " " "\n" >> tslist
    let COUNTER=COUNTER+1
  done
  while read line; do cat $line >> $TITLE.ts; done < tslist

  rm -rf tslist
  mv "$TITLE.ts" "../$TITLE.ts"

  cd ..
  rm -rf 'parts'
  cd ..

done

Tetartohedral

Answer 14

1) Please read instructions by @aalhanane (after "paste URL m3u8" step you have to type name for the file, eg "video" then click on "hand" icon next to "quality" and only after that you should select "one on one" and "download").

2) The stream splits video and audio, so you need to download them separately and then use the same m3u8x to join them https://youtu.be/he-tDNiVl2M (optionally convert to mp4).

3) m3u8x can download video without any issues but in my case it cannot extract audio links. So I simply downloaded the *.m3u8 file and searched for line which contains GROUP-ID="audio-0" and then scroll right and copied the link (!including token!) and paste it straight into "Quality URL" field of m3u8x app. Then "one on one" and download it similar to video stream.

Once I had both video and audio, I joined and success =)

p.s. in case automatic extraction will stop working in the future, you can use the same method to extract video links manually.

Southdown

Answer 15

• Get one Link from Network tab of developer tools
• Remove index and ts extension from link

With following script you can save movie to Videos folder

Example usage:

download-video.sh https://url.com/video.mp4 video-name

download-video.sh

#!/bin/bash
LINK=$1
NAME=$2

START=0
END=2000

help()
{
    echo "download-video.sh <url> <output-name>"
    echo "<url>: x.mp4 (without .ts)"
    echo "<output-name>: x (without .mp4)"
} 

create_folders()
{
    # create folder for streaming media
    cd ~/Videos
    mkdir download-videos
    cd download-videos
}

print_variables()
{
    echo "Execute Download with following parameters"
    echo "Link $LINK"
    echo "Name $NAME"
}

check_video()
{
    i=$START
    while [[ $i -le $END ]]
    do
        URL=$LINK'-'$i.ts
        STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
        if [ "$STATUS_CODE" == "200" ]; then
            break
        fi
        ((i = i + 1))
    done

    if [ "$STATUS_CODE" == "200" ]; then
        START=$i
        echo "START is $START"
    else 
        echo "File not found"
    fi
} 


download_video()
{
    i=$START
    e=$END
    while [[ $i -le $END ]]
    do
        URL=$LINK'-'$i.ts
        STATUS_CODE=$(curl -o /dev/null --silent --head --write-out '%{http_code}\n' $URL)
        if [ "$STATUS_CODE" != "200" ]; then
            break
        fi
        wget $URL
        e=$i
        ((i = i + 1))
    done

    END=$e
}

concat_videos()
{
    DIR="${LINK##*/}"

    i=$START
    echo "i is $i"
    while [[ $i -le $END ]]
    do
        FILE=$DIR'-'$i.ts
        echo $FILE | tr " " "\n" >> tslist
        ((i = i + 1))
    done
    while read line; 
    do 
        echo "gugu"$line
        cat $line >> $NAME.mp4; 
    done < tslist

    rm *.ts tslist
}

if [ "$1" == "" ]; then
    echo "No video url provided"
    help
else
    LINK=$1
    if [ "$2" == "" ]; then
        echo "No video output-name provided"
        help
    else
        NAME=$2
        create_folders
        print_variables
        check_video
        download_video
        concat_videos
    fi
fi

Hurricane

script works good, but not always, some site put some restrictions. If I run wget $URL, i get 403. If try to download from chrome it works. Looks like a site does some validation who and how the file has been accessed... – Stifling

Answer 16

You can use Xtreme Download Manager(XDM) software for this. This software can download from any site in this format. Even this software can change the ts file format. You only need to change the format when downloading.

like:https://www.videohelp.com/software/Xtreme-Download-Manager-

Plethora