There is a semi-official explanation in an Intel talk https://software.intel.com/en-us/articles/hyper-threading-technology-and-multi-core-processor-detection
Basically after Intel introduced SMT/hyperthreading in the final versions of the Pentium 4, they created the Pentium D, which being multi-core was considered an evolution of the hyperthreading model because a core has more independence from another core than SMT units have relative to each other, and thus multi-core performs better than SMT at the same thread count. So they used the same bit to indicate SMT/hyperthreading or multi-core for that reason, i.e. multi-core was seen as an improved form of hyperthreading. The Intel Xeon E5420 that you ask about is a multi-core processor, so that's why the bit is set.
Of course, once you can have both hyperthreading and multi-core in the same package, it's sometimes necessary to tell them apart, for example because you'd rather schedule a new thread on an unloaded core rather than on the other hyper-threaded half of an already loaded core. So new cpuid bits (or rather leaves) were eventually introduced for the purpose of describing a combined SMT and multi-core topology. The most recent of these newer cpuid features for querying processor topology is the EAX=0xB leaf.
As an aside, you should not use the topology enumeration algorithm given in that old Intel talk for processors made in 2010 and thereafter because it will give incorrect/inflated core counts. Use the updated method given at https://software.intel.com/en-us/articles/intel-64-architecture-processor-topology-enumeration/ instead. Alas this newer page is much more dry and doesn't for example answer the question that you asked, whereas the old talk does...