W3C took care of that in 1997 explaining how frames should be implemented in "Implementing HTML Frames":
Any frame that attempts to assign as its SRC a URL used by any of its ancestors is treated as if it has no SRC URL at all (basically a blank frame).
Iframe recursion bug/attack history
As kingdago found out and mentioned in the comment above, one browser that missed to implement a safeguard for this was Mozilla in 1999. Quote from one of the developers:
This is a parity bug (and a source of possible embarrasment) since MSIE5
doesn't have a problem with these kinds of pages.
I decided to dig some more into this and it turns out that in 2004 this happened again. However, this time JavaScript was involved:
This is the code, what causes it: <iframe name="productcatalog"
id="productcatalog" src="page2.htm"></iframe> directly followed by
a script with this in it:
frames.productcatalog.location.replace(frames.productcatalog.location
+ location.hash);
...
Actual Results: The parent window gets recursively loaded into the
iframe, resulting sometimes in a crash.
Expected Results: Just show it like in Internet Explorer.
Then again in 2008 with Firefox 2 (this also involved JavaScript).
And again in 2009. The interesting part here is that this bug is still open and this attachment: https://bugzilla.mozilla.org/attachment.cgi?id=414035
(will you restrain your curiosity?) will still crash/freeze your Firefox (I just tested it and I almost crashed the whole Ubuntu). In Chrome it just loads indefinitely (probably because each tab lives in a separate process).
As for the other browsers:
- In 2005 Konqueror had a bug in it's safeguard that allowed to render iframes one inside another (but it seems that somehow it wasn't freezing/crashing the whole app).
- IE6, Opera 7.54 and Firefox 0.9.3 are also reported to be susceptible to attacks basing on iframe recursion.