There is no UI Tests for iMessage app extension in Xcode currently. But you can perform it by launching Messages by yourself and find elements in the Messages app. At first, you'll have to launch the Message app and open a conversation :
let messageApp = XCUIApplication(bundleIdentifier: "com.apple.MobileSMS")
messageApp.terminate()
messageApp.activate()
messageApp.cells.firstMatch.tap()
Then, you can access your iMessage app by doing so :
// Replace appIndex by the position of your app in the iMessage bottom bar
let appIndex = 2
messageApp.collectionViews.descendants(matching: .cell).element(boundBy: appIndex).tap()
When your iMessage app is opened in the expanded mode, you can access the close button :
let closeButton = messageApp.buttons.element(boundBy: 1)
If you want to test your iMessage app when the user send a message and then open it, you can do it this way :
// Send your message after it is inserted in the Messages app text field
let sendButton = messageApp.buttons["sendButton"]
waitForElementToExists(sendButton)
sendButton.tap()
// Tap on the iMessage first bubble
let firstBubble = messageApp.collectionViews["TranscriptCollectionView"].cells.element(boundBy: 2)
waitForElementToExists(firstBubble)
firstBubble.tap()
private func waitForElementToExists(_ element: XCUIElement) {
let exists = NSPredicate(format: "exists == 1")
expectation(for: exists, evaluatedWith: element, handler: nil)
waitForExpectations(timeout: 5, handler: nil)
}