Now, there are plenty of example of me answering questions with essays on why following validation rules are important. I've also said that sometimes you just have to be a rebel and break the rules, and document the reasons.
You can see in this example that framesets do work in HTML5 still. I had to download the code and add an HTML5 doctype at the top, however. But the frameset element was still recognized, and the desired result was achieved.
Therefore, knowing that using framesets is completely absurd, and knowing that you have to use this as dictated by your professor/teacher, you could just deal with the single validation error in the W3C validator and use both the HTML5 video element as well as the deprecated frameset element.
<!DOCTYPE html>
<html>
<head>
</head>
<!-- frameset is deprecated in html5, but it still works. -->
<frameset framespacing="0" rows="150,*" frameborder="0" noresize>
<frame name="top" src="http://www.npscripts.com/framer/demo-top.html" target="top">
<frame name="main" src="http://www.google.com" target="main">
</frameset>
</html>
Keep in mind that if it's a project for school, it's most likely not going to be something that will be around in a year or two once the browser vendors remove frameset support for HTML5 completely. Just know that you are right and just do what your teacher/professor asks just to get the grade :)
UPDATE:
The toplevel parent doc uses XHTML and the frame uses HTML5. The validator did not complain about the frameset being illegal, and it didn't complain about the video element.
index.php:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Frameset//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-frameset.dtd">
<html>
<head>
</head>
<frameset framespacing="0" rows="150,*" frameborder="0" noresize>
<frame name="top" src="http://www.npscripts.com/framer/demo-top.html" target="top">
<frame name="main" src="video.html" target="main">
</frameset>
</html>
video.html:
<!doctype html>
<html>
<head>
</head>
<body>
<div id="player-container">
<div class="arrow"></div>
<div class="player">
<video id="vid1" width="480" height="267"
poster="http://cdn.kaltura.org/apis/html5lib/kplayer-examples/media/bbb480.jpg"
durationHint="33" controls>
<source src="http://cdn.kaltura.org/apis/html5lib/kplayer-examples/media/bbb_trailer_iphone.m4v" />
<source src="http://cdn.kaltura.org/apis/html5lib/kplayer-examples/media/bbb400p.ogv" />
</video>
</div>
</body>
</html>
<iframe seamless="seamless">
– Tellurize