The specific reasoning is given in JEP 395
Static members of inner classes
It is currently specified to be a compile-time error if an inner class declares a member that is explicitly or implicitly static, unless the member is a constant variable. This means that, for example, an inner class cannot declare a record class member, since nested record classes are implicitly static.
We relax this restriction in order to allow an inner class to declare members that are either explicitly or implicitly static. In particular, this allows an inner class to declare a static member that is a record class.
In other words, it was necessary to remove the restriction on static members of inner classes for a particular case; i.e. to allow record
classes to be declared in inner classes. But they decided to take the opportunity to remove the restriction in all cases.
This implies that the designers have concluded that the original restriction as a whole was neither necessary for technical reasons or desirable.
why did this restriction exist in the first place?
That is a more difficult question. The decision to make that restriction would have been made in 1996 or early 1997 when Java 1.1 was being designed. It is unlikely that anyone can still accurately remember the reasons behind original decision. So unless someone can find a contemporaneous written source, we will never know for sure.
(Brian Goetz commented above: "... at the time nested was added (Java 1.1), there were multiple possible interpretations of static within another class, so the question was deferred.". That certainly makes sense, but this could be (just) one person's recollection of something that happened ~25 years ago. If it was me, I wouldn't be confident in my memory from that far back. Unless I had contemporaneous minutes, notes, etc to refer to.)
There is some speculation about the rationale for the original restriction here:
static
they could keep the illusion up a bit longer... – Ulpian