How to download streaming video from xhr

Question

I'd like to download a video from a site that uses flowplayer. When I inspect the element, I get something like: So we can say this video is streamed via xhr. Is there a way to get full video from these files? I tried Flash Player Downloader and it downloads the file in .mp4 but it's broken and can't repair it via some tools.

Halcomb

some news about this?? I have similar issue :( – Deviant

Answer 1

I don't have a real answer but I, too, tried to download such a fragmented stream. It's a pain. What helped me the answer to this: https://superuser.com/questions/1204497/download-all-m4s-files-of-a-mpeg-dash-stream

The idea is to get the stream's URL from the initial file IS.mp4. All subsequent parts numbered 000000.m4s, 000001.m4s, 000002.m4s, ... share the same location as IS.mp4

Generating the download links for all the parts, downloading and merging them, can be easily scripted, as shown in the linked post:

echo "IS.mp4" >"links.txt"
seq -f "%06g.m4s" 0 394 >>"links.txt"
wget -i "links.txt" -O "audio.mp4" -B "http://80.188.78.212/aa/ffc8a55fe6f203b0bffecb73efacb69b/1494953435522/eda9c6f7b1e3de68db8e3e5dc0d14fc7/dna-61924494877285694-pc/1002-1502/"

You have to download all parts, including IS.mp4 and then merge them to one file. Sometimes you'll also need to download the audio track in such way, too. After you have the video and audio track, combine them in one container.

Gehman

How did you solve this in the end? You said the answer from SuperUser helped, but what was the final solution? How did you access the .xhr files from the TAB / session? – Genro

In the browser I used the "Inspect Element" from the context menu and then I went to the Network-Monitor tab and selected XHR as filter type. – Gehman

Answer 2

well, i have a semi-manual solution that surely works with smaller video files:

1. open the video page in firefox (mine is 75.0). do not start the playout.
2. open the network panel (F12).
3. refresh the video page. scroll down to the player and start the playout.
4. wait for the video to end. do not touch your browser's controls in the meantime.
5. click on the "HAR" button in the top-right corner of the network panel.
6. save the whole network dump as a HAR file.

1 minute video is about 10 MB of JSON-encoded text. now, you should write a script that decodes and concatenates the response bytes ranges (HTTP 206) from the saved network dump. sample PHP code:

// for demonstration purposes only // no checks // at all //

$har = $argv[1]; // HAR input file //
$nam = $argv[2]; // video name from URL //
$out = $argv[3]; // MP4 output file //

$text = file_get_contents($har);
$json = json_decode($text);

$map = array(); // range map //
foreach ($json->log->entries as $e) {
  if (!is_object($e) OR !property_exists($e, 'request') OR !property_exists($e, 'response')) {
    continue;
  }
  if ($e->request->method != 'GET') {
    continue;
  }
  if (strpos($e->request->url, $nam) === false) {
    continue;
  }
  if ($e->response->status != 206) {
    continue;
  }
  if ($e->response->content->encoding != 'base64') {
    continue;
  }
  foreach ($e->response->headers as $h) {
    if ($h->name == 'content-range' AND 0 < preg_match('/^bytes ([0-9]+)-([0-9]+)\\/([0-9]+)$/', $h->value, $match)) {
      $map[$match[1]] = array(
        'in' => $match[1],
        'out' => $match[2],
        'text' => $e->response->content->text,
      );
      break;
    }
  } // foreach //
} // foreach //

file_put_contents($out, '');
ksort($map);
$err = 0;
$j = -1;
foreach ($map as $k => $v) {
  $i = $j + 1;
  $j = $v['out'];
  $part = base64_decode($v['text']);
  file_put_contents($out, $part, FILE_APPEND);
} // foreach //

i have just successfully saved some short videos. their durations vary between 4 and 8 minutes. the output file is an mp4 stream, so i transcoded it to mkv:

ffmpeg -i har2bin.mp4 -c copy -y har2bin.mkv

now, you can play the mkv file with vlc player.

Resinate

i tried in chrome as well recently and it works. furthermore, firefox truncates the XHR responses at 1M while chrome does not. – Resinate

I think you can change or disable the limit in Firefox in the about:config. Setting devtools.netmonitor.responseBodyLimit to 0 disables the limit. – Bramlett

thank you for the information. – Resinate