Is the absence of any album really seen as an error? Assuming the albums are returned as a JSON array, the common response to such a situation would be a HTTP 200 with an empty array as the body.
Returning 404 signals that the resource doesn't exist, kind of saying that it isn't even possible to ask for the list of albums for this particular user. But in fact, it's possible to successfully return the list of albums, it's just that the list is empty. It doesn't seem at all exceptional to me. This is completely in contrast to retrieval of one specific album using an ID that doesn't exist (using your other endpoint); in such a situation a 404 is correct.
While a 204 seems better than a 404, because it at least tells the client that the request was successful but had no content, its intention is not really to be used to signal a "successful absence". Rather, it signals that the resource DOES exist but for some reason the server chose not to include the resource in the response body - for example the purpose of the request could have been to simply pass back some headers to the client.
A 204 can also be used as a response to a POST request where some action was carried out by the server without necessarily creating any new resource (which would have implied a 201 CREATED), or where it's for some other reason not relevant to return any resource.
I think it's clear that what you need is a
GET /user/xxx/albums
HTTP/1.1 200 OK
[]