It is a convenience control, useful for point-and-click UI design. But sure, it is very wasteful. Although it doesn't hold a candle to the Label control. You are burning up an entire Windows window, just to draw an image. Native window objects are very expensive system resources. The alternative is one line of code in your OnPaint() method, e.Graphics.DrawImage().
And it is not a very smart control either as Rick points out. It rescales the image to fit the control every single time it needs to paint itself. And it doesn't optimize the pixel format of the image either, forcing GDI+ to make the pixel conversion every single time as well. The net effect can certainly be a slow UI, getting an image to draw 100x slower than necessary is certainly not unusual. Otherwise the kind of trade-off that is difficult to make in a general-purpose control; such optimizations don't come for free, potentially doubling the memory requirements.
For comparison, a Microsoft Office program like Outlook uses about 50 windows, most of them are toolbars. That's what you slam together in Winforms in less than 10 minutes. Convenience has a price. So does an Office program, it takes a lot of programmers.
PictureBox
can do, you would be stuck at some feature because it's not easy as simply draw the image yourself. – Expellant