Disclaimer: I'm making an (hopefully educated) guess about the inner workings of the Swift compiler here, so apply grains of salt.
Aside from value semantics, ask yourself: Why do we have structs? What is the advantage?
One advantage is that we can (read: want to) store them on the stack (resp. in an object frame), i.e. just like primitive values in other languages. In particular, we don't want to allocate dedicated space on the heap to point to. That makes accessing struct values more efficient: we (read: the compiler) always knows where exactly in memory it finds the value, relative to the current frame or object pointer.
In order for that to work out for the compiler, it needs to know how much space to reserve for a given struct value when determining the structure of the stack or object frame. As long as struct values are trees of fixed size (disregarding outgoing references to objects; they point to the heap are not of interest for us), that is fine: the compiler can just add up all the sizes it finds.
If you had a recursive struct, this fails: you can implement lists or binary trees in this way. The compiler can not figure out statically how to store such values in memory, so we have to forbid them.
Nota bene: The same reasoning explains why structs are pass-by-value: we need them to physically be at their new context.