The resource server (your server-side application) accept only the access token from a client. This is because access tokens are intended for authorizing access to a resource. ID Tokens, on the other hand, are intended for authentication. This granted by the OpenID Provider that contains information about an End-User. source
Access tokens are the thing that applications use to make API requests on behalf of a user. The access token represents the authorization of a specific application to access specific parts of a user’s data. Access tokens must be kept confidential.
The idea of refresh tokens is that if an access token is compromised, because it is short-lived, the attacker has a limited window in which to abuse it. Refresh tokens, if compromised, are useless because the attacker requires the client id and secret in addition to the refresh token in order to gain an access token. source
The lifetime of a refresh token is up to the (AS) authorization server — they can expire, be revoked, etc. The difference between a refresh token and an access token is the audience: the refresh token only goes back to the authorization server, the access token goes to the (RS) resource server. source