I'm trying to understand the same thing, I will explain what I did understand so far (in the hope that if I'm wrong someone will correct me):
First of all RLPx is a protocol suite and not a single protocol.
On the one hand it maintains the p2p overlay network with the means of a modified version of the kademlia node discovery protocol (using UDP messages). We could name it RLPx node discovery protocol.
The first time two peers meet each other they should perform a two-phase handshake (i.e. two different handshakes).
In the first handshake they exchange some cryptographic secrets (the details are explained here (in the rlpx specification)) that are used for the subsequent communications between them. The subsequent messages are therefore encrypted and authenticated.
In the second handshake they exchange their capabilities, i.e. which devp2p subprotocols they support (the peers can support multiple protocols). This second handshake could be considered already part of devp2p wire protocol (the Hello message). The documentation says that:
the ÐΞVp2p nodes communicate by sending messages using RLPx
(this time using TCP messages).
I list some devp2p wire subprotocols that are widely used:
Therefore RLPx messages are encrypted and authenticated and on top of them you can communicate with different protocols.
We can say that RLPx is the transport layer protocol and devp2p and its subprotocols are the application level protocols.