How does Google Now (Now on Tap) search inside apps?

Question

So Google has just released a new feature for Google Now called Now on Tap. It enables a user to get contextual help from different apps (The Verge video example).

• Does it simply take a screenshot of the current app and use OCR to extract the text?
• Does it have "special Google permission" which allows it to read the values within the views?

Falconiform

Disclaimer: speaking only as myself... I don't know for certain, but Google generally avoids "special" capabilities in Android and typically makes the APIs it uses the same as those consumable by other apps. My guess is that it uses the accessibility APIs that exist in Android. With those APIs, it is possible to read the content of text views as well as the accessibility description of any image that is on the screen. – Naturopathy

Answer 1

Primarily, it should work using the assist API added to the M Developer Preview. However as "Now on Tap" is proprietary and unreleased, we can only really guess at this time.

Jeffersonjeffery

I wonder if google now on tap is using the accessibility api, what's your opinion? But how does it get web view content? – Ebba

@Jerome: Now On Tap uses the Assist API, as is covered in the answer. – Jeffersonjeffery