The reason this happens is due to inheritance, not specificity.
Look at it this way, if the span didn't have that class, it would inherit the color red from the parent <p> element and "world" would be red. But note that that's due to inheritance.
When you set color for the span, via the class, that overrides the inherited value.
Specificity is for determining which rule to use in multiple competing rules. In your example, there are no competing rules for <span>, so specificity does not come into play. However, if you added this to your styles:
#my_id span {color: orange}
you would see that "world" is orange because of the specificity of the id being more than the class.