Yes I guess you should. There has been lots of different opinions about that to clear the old refresh tokens or keep as much as it can stays fresh. Actually the primary intentions was to keep refresh token was you can re-use them later.
But still if people logout by hitting the button, then you can cleanup those refresh token.
But if you want to keep them on your site more longer without re-login, you can re-issue access token with refresh token. So the user's auth token will be more longer valid. But if user want to logout by themselves, then you should clean that up and store again on their next login.