Some folks have already dug up the relevant OpenJDK artifacts that relate to this question, namely enhancement request JDK-6278287 and this email thread. Here's a full discussion of the topic.
Fundamentals
First, there's no fundamental reason why EnumSet couldn't implement SequencedSet or even NavigableSet. (And similarly for EnumMap implementing SequencedMap or NavigableMap. Henceforth I'll just talk about the Set types, but the discussion also mostly applies to the corresponding Map types.)
Enum types have a defined ordering, and thus the enums present in an EnumSet will always have a defined encounter order. EnumSet's iterator is already defined to iterate them in order. It would be perfectly sensible to add the SequencedSet operations that operate on the first and last elements and to provide a reversed view.
An interesting wrinkle is what the type of the reversed view should be. The obvious type would be SequencedSet<E extends Enum<E>>. This will work, but this would drop the "EnumSet"-ness from the return type. EnumSet doesn't add any new instance methods over the Set interface, but EnumSet is used in some places in the API, e.g. EnumSet::complementOf. In addition, some of the EnumSet method implementations check the argument type, e.g., they use instanceof RegularEnumSet and choose a fast path implementation for that case. That would be lost if the reversed-view wrapper didn't implement EnumSet. (Well, those implementations could be retrofitted with additional instanceof checks to accommodate the reversed views.)
An alternative would be to modify the specification of EnumSet somehow so that it would allow iteration to be in reverse order. This would allow reversed() to have a return type of EnumSet. This would mitigate the above problems, but it might cause new problems for existing code that relies on iteration of any EnumSet always being in forward order. Such code might be broken if a reversed view were passed to it.
NavigableSet provides a superset of the operations of SequencedSet. In addition to access to first/last elements and reversed view, it also provides various ways to get subset views (headSet, subSet, tailSet). These are also perfectly sensible given that enum values are totally ordered.
Usefulness
The primary use case for EnumSet seems to be as an array of flags, as a replacement for storing bits in an int or long and using bitwise boolean operations. The most common usage mode is to set and query these flags individually. For example, "Is this resource writable?" can be answered by using bitwise operations on an int, or by using EnumSet.contains(). The main advantage of EnumSet over bits is that it's type-safe. Adding SequencedSet doesn't help anything with this use case (but it doesn't hurt, either).
Another use case for EnumSet is as a set of commands or functions. One of the respondents on the email thread talked about using enum values to represent commands and processing them in-order by iterating an EnumSet. It's a different use case, but it's mostly already supported by the currently available forward iteration. He pretty much admitted that retrofitting SequencedSet wouldn't help all that much.
I suppose it's a possibility that somebody might want to operate on the first or last elements of an EnumSet, or iterate it in reverse order, but I haven't heard of such use cases. (That doesn't mean they don't exist of course. If they do exist, I'd appreciate hearing about them.) Thus far, though, this doesn't seem compelling. It's even harder to imagine use cases for the various subset views of NavigableSet.
Cost
Adding a reversed view probably isn't terribly difficult, but it's more than a one-liner. Most of the effort involves adding the reversed view, which often involves a fairly thin wrapper class that mostly delegates to the backing implementation. There might be two wrappers, one for RegularEnumSet and one for JumboEnumSet. Otherwise there's not much work other than some bit twiddling to get the last bit set and to iterate them in reverse order.
In addition, there is the testing and specification work that's required when adding APIs to the JDK. (Most people tend to underestimate the effort involved with this.)
NavigableSet's subset views are rather more work. Implementing the subset views is rather fiddly, and there's no "AbstractNavigableSet" that would make it easy to use a shared implementation.
There is also the need to consider interaction with other potential enhancements, such as a potential unmodifiable EnumSet, which would probably also need companion classes to RegularEnumSet and JumboEnumSet. Adding SequencedSet implementations and reversed views might make it harder to add unmodifiable implementations in the future. This probably isn't insurmountable, but it does add complexity.
Completeness
Since enums are totally ordered, shouldn't a collection of them implement the best type already available, for the sake of completeness? After all, the net API footprint is essentially zero -- no new APIs are being added to the system -- and no new concepts are being added.
I'm a bit sympathetic to this. When you're using a system, you expect that the pieces will fit together orthogonally and interchangeably. You can store enums in regular Collection, SequencedSet, or NavigableSet; but if you store enums in an EnumSet, which is specialized for storing enums, you lose the possibility of those more powerful interfaces. This can impose uncomfortable tradeoffs on users.
On the other hand, there are an arbitrary number of things that could be done in the name of completeness or consistency, and time and effort are in fact scarce. It follows that we can do only a subset of what's ideal, even if that means parts of the system are inconsistent or incomplete. How is that subset chosen? That leads us to....
Priorities
Here's where we have to apply judgment. First, there's the direct cost-benefit tradeoff. The cost of EnumSet implementing SequencedSet is moderate, and the cost of NavigableSet is rather higher. The benefit of implementing SequencedSet seems rather low, and NavigableSet even lower.
There's also the concept of what I'll call net benefit. EnumSet doesn't implement SequencedSet, but what if you needed something that SequencedSet provides, like a reversed view? You could probably write something like
List.copyOf(enumSet).reversed()
It's a bit more code, it involves some memory allocation, some operations are probably less efficient, and it's a copy instead of a view. The point is that getting a reversed view of EnumSet isn't impossible; it's merely a bit inconvenient and a bit less efficient. So, the net benefit of having the EnumSet implement SequencedSet is rather lower.
There's also opportunity cost: if we did this, what other thing would we not do? I'm not going to enumerate all the things that my team and I are working on, but I think they're all pretty important, and likely more important than having EnumSet implement SequencedSet.
One thing I mentioned earlier was the possibility of an unmodifiable EnumSet: see JDK-5039214. (An unmodifiable EnumSet would in fact be immutable, since enum values are immutable.) Given the platform's emphasis on thread-safety and unmodifiable data, this seems fairly important. There are also some reasonable optimizations that this would offer compared to the current "dumb" unmodifiable wrappers. I'd argue that doing this is likely more important than implementing SequencedSet, although I have to admit that nobody is working on this one either. However, if we were to do something in this area, it seems likely we'd want to implement an unmodifiable EnumSet before we enhanced it to implement SequencedSet.
Summary
Sometimes, the answer to "Why wasn't this done?" is something like, well this is a bad idea because.... That isn't the case here.
This case, which is rather more typical, falls into the large category of things that might be good ideas, but whose net benefits seem lower than other things. Thus it isn't being worked on now, but it might be worked on in the future. Or not, depending on what other things happen in the future.
reversed()doesn't appear to mandate the returned object must be the same type, just a reversed view. For example,TreeSet::reversedis only defined to return aNavigableSet, not necessarily aTreeSet. – Titanism