Getting Bitmap from Video decoded with Nestream AppendBytes (AS3)?

Question

I am wondering if someone who has handled NetStream.appendBytes in Flash knows how to get the bitmapData from a decoded video frame? I have already looked at this question but that is from 3 years ago and the more recent comment/answer seems to be gone. In 2014 has anyone managed to turn those bytes into a bitmap? I am working with Flash Player 11.8 and this is not a desktop/AIR app.

In the image below I can do steps 1) and 2) but there's a brick wall at step 3)

The problem is that simply using bitmapdata.draw(video_container); does not work but instead it throws a Policy File error even though I am using a byteArray (from local video file in the same directory as the SWF). No internet is even involved but Flash tells me that "No Policy File granted permission from the server" or such nonsense. I think the error is just a bail-out insteading of straight up saying "You are not allowed to do this.."

I have tried: trying to appease this Crossdomain.xml issue anyway and looking into all known security/domain settings. I came to the conclusion that the error is not the problem but a side effect of the issue.. The issue here being that: Flash Player is aware of the SWF's location and of any files local to it. That's okay when you pass a String as URL etc but when the Netstream data is not local to the SWF domain then it becomes a Policy File issue. Problem is my data is in the Memory not in a folder like the SWF and therefore cannot alllow bitmapData.draw since it cannot "police" an array of bytes, any known fixes for this?... (I can't even say the words I really wanted to use).

What I am trying to achieve: Is to essentially use Netstream as an H.263 or H.264 image decoder in the same way Loader is a JPEG-to-Bitmap decoder or LoadCompressed.. is an MP3-to-PCM decoder. You know, access the raw material (here RGB pixels), apply some effects functions and then send to screen or save to disk.

Bromidic

Do you want to extract an image from a video stream as snapshot ? or doing that using NetStream.appendBytes which do the inverse ( inject data into a NetStream object ) ? – Underbrush

Hi, I want to extract from not put into, making a video frame from existing bitmapdata is the easy part (ie: no hassle). It's when I bring in an FLV/MP4 file and append a frame then suddenly I'm not allowed to snapshot those decoded bytes to some bitmapData. It's really strange... – Bromidic

Hi, +1 for the good effort but it's not the correct answer. To quote your answer "To take a snapshot... we don't need NetStream.appendBytes which inject data into a NetStream object" well that's the problem and whole point of the question.. I'm injecting bytes to be decoded so I can take a snapshot of the output image. There is no URL to put in a String but instead I have a Byte Array (which is why the security error makes no sense, how can anyone "Policy File" a bytearray??). Anyway I think I found a possible solution just need time to double check it.. – Bromidic

Hi, sorry man if I have not understood your question. I think what your are looking for is not dowable now because simply it's an Adobe bug. Take a look here, the bug is here and a bug workaround here and here. – Underbrush

I think I found a solution for your problem, take a look to my new answer. – Underbrush

Answer 1

I know it is a little late, but I think I found a solution for your problem.

So, to avoid the Security sandbox violation #2123 Error, you have just to do like this :

// ...

net_stream.play(null);

net_stream.play('');

// ...

Hope that can help.

Underbrush

Thanks!! This works great and it also avoids security error for computespectrum too (when using appendBytes). Genius. VC – Bromidic

Answer 2

To take a snapshot from a video stream we don't need NetStream.appendBytes which inject data into a NetStream object.

For that we can use BitmapData.draw which has some security constraints. That's why in many times we get a flash security error. About that, Adobe said :

"... This method is supported over RTMP in Flash Player 9.0.115.0 and later and in Adobe AIR. You can control access to streams on Flash Media Server in a server-side script. For more information, see the Client.audioSampleAccess and Client.videoSampleAccess properties in Server-Side ActionScript Language Reference for Adobe Flash Media Server. If the source object and (in the case of a Sprite or MovieClip object) all of its child objects do not come from the same domain as the caller, or are not in a content that is accessible to the caller by having called the Security.allowDomain() method, a call to the draw() throws a SecurityError exception. This restriction does not apply to AIR content in the application security sandbox. ...".

For crossdomain file creation and some other security config for AMS server, you can take a look on this post : Crossdomain Video Snapshot - Fixing BitmapData.draw() Security Sandbox Violation.

After allowing our script to get data from our video stream, we can pass to the code.

I wrote a code that play a video stream ( rtmp or http ) and take a snapshot to show it in the stage or save it as a file after applying a pixel effect :

const server:String = null; //'rtmp://localhost/vod'
const stream:String = 'stream'; // 'mp4:big_buck_bunny_480p_h264.mp4';
var nc:NetConnection;
var ns:NetStream;
var video:Video;
const jpg_quality:int = 80;
const px_size:int = 10;

nc = new NetConnection();
nc.addEventListener(AsyncErrorEvent.ASYNC_ERROR, function(e:AsyncErrorEvent):void{});
nc.addEventListener(NetStatusEvent.NET_STATUS, function(e:NetStatusEvent):void{
    if(e.info.code == 'NetConnection.Connect.Success'){
        ns = new NetStream(nc);
        ns.addEventListener(NetStatusEvent.NET_STATUS, function(e:NetStatusEvent):void{});
        ns.addEventListener(AsyncErrorEvent.ASYNC_ERROR, function(e:AsyncErrorEvent):void{});
        video = new Video(320, 180);
        video.x = video.y = 10;
        video.attachNetStream(ns);
        addChild(video);
        ns.play(stream);
    }
})
nc.connect(server);

btn_show.addEventListener(
    MouseEvent.CLICK,
    function(e:MouseEvent): void{
        var bmp:Bitmap = pixelate(video, px_size);
            bmp.x = 10;
            bmp.y = 220;
        addChild(bmp);
    }
)
btn_save.addEventListener(
    MouseEvent.CLICK,
    function(e:MouseEvent): void{

        var bmp:Bitmap = pixelate(video, px_size);

        var jpg_encoder:JPGEncoder = new JPGEncoder(80);
        var jpg_stream:ByteArray = jpg_encoder.encode(bmp.bitmapData);

        var file:FileReference = new FileReference();
            file.save(jpg_stream, 'snapshot_'+int(ns.time)+'.jpg');

    }
)

function pixelate(target:DisplayObject, px_size:uint):Bitmap {

    var i:uint, j:uint = 0;
    var s:uint = px_size;
    var d:DisplayObject = target;

    var w:uint = d.width;
    var h:uint = d.height;

    var bmd_src:BitmapData = new BitmapData(w, h);
        bmd_src.draw(d);

    var bmd_final:BitmapData = new BitmapData(w, h);

    var rec:Rectangle = new Rectangle();
        rec.width = rec.height = s;

    for (i = 0; i < w; i += s){
        for (j = 0; j < h; j += s){
            rec.x = i;
            rec.y = j;                  
            bmd_final.fillRect(rec, bmd_src.getPixel32(i, j));
        }
    }

    bmd_src.dispose();
    bmd_src = null;

    return new Bitmap(bmd_final);
}

Of course, this is just a simple example to show the manner to get a snapshot from a video stream, you should adapt and improve it to your needs ...

I hope all that can help you.

Underbrush

Answer 3

I know this question is a couple months old, but I wanted to post the correct answer (because I just had this problem as well and other will too).

Correct answer:

It's a bug that has been open at adobe for almost 2 years

Link to the bug on Adobe

Work Around until the bug gets fixed (I am using this and it works great):

Workaround using Sprite and graphics

Bezel

Sorry for the late reply. That workaround link was indeed helpful. I had already found it (as mentioned in the last sentence here ). My problem then was it came with extra issues for what I was particularly doing (h264 bytes, aligning source vid with drawn vid, problems with drawing black if metadata-less files whilst showing a "drawable" picture etc. Thinking to provide an answer with working example + code, I'll still hook you up with "correct answer" rep though – Bromidic

I think I found a solution for your problem, take a look to my answer. – Underbrush