You are correct.
An Internet Gateway is a logical connection between a VPC and the Internet. If there is no Internet Gateway, then there is no connection between the VPC and the Internet.
Every Amazon EC2 instance in a VPC has a private IP address. They can optionally also have a public IP address. This can either be a random one assigned when an instance is started, or a static Elastic IP address can be assigned to the instance.
Actually, these public IP addresses are assigned to an Elastic Network Interface (ENI). An instance can have multiple ENIs.
The instance itself doesn't actually 'know' that it has a public IP address / Elastic IP address. Instead, when traffic comes from the Internet destined for the public IP address, the Internet Gateway performs a reverse NAT and sends the traffic to the private IP address of the instance. Similarly, any traffic going from the instance to the Internet Gateway comes from the private IP address of the instance, which the Internet Gateway then forwards as coming from the instance's public IP address.
Thus, the Internet Gateway 'owns' the public IP address, but forwards it to the instance. It's all quite magical, so sometimes it's just easier to imagine as the instance having the public IP address.
One benefit of this method is that an Elastic IP address can be reassigned to another instance and traffic will immediately flow to the new instance without any configuration changes on the 'old' or 'new' instance. They just get traffic via their private IP addresses without knowing that a public IP address was involved.