How can I tell Actions on Google to stream audio?

Question

I am writing an app to work with Google Actions. The only bummer is that I can't find any information about how to form my response so that Google will stream audio from a given URL. Does Google even support this yet?

I have written the same app on Alexa already, and on Alexa all you have to do is Return an audio item (token, URL, play command) and Alexa will start streaming it.

I should mention that I am NOT using API.AI, but am simply using the Actions SDK and am hosting my web service on Asure using C#.

So, bottom line... How can I format a response via the Actions SDK to stream an MP3 file to Google Home?

Waterish

Note that streaming audio (e.g. HLS) is not currently supported, but the actions platform can play an MP3 file as detailed in the accepted answer. – Meares

Answer 1

UPDATE: The first answer works only with the V1 of Dialogflow. As for the V2, you can create the mediaResponse this way (from Google's doc):

conv.ask(new MediaObject({
  name: 'Jazz in Paris',
  url: 'http://storage.googleapis.com/automotive-media/Jazz_In_Paris.mp3',
  description: 'A funky Jazz tune',
  icon: new Image({
    url: 'http://storage.googleapis.com/automotive-media/album_art.jpg',
    alt: 'Media icon',
  }),
}));

========================================================================

I posted an answer over here.

Basically you can create a mediaResponse object that will play your audio file. I can play a 50 mins audio file just fine.

A code example in Node.js could be (with the current documentation):

const richResponse = app.buildRichResponse()
 .addSimpleResponse("Here's song one.")
  .addMediaResponse(app.buildMediaResponse()
  .addMediaObjects([
    app.buildMediaObject("Song One", "https://....mp3")
      .setDescription("Song One with description and large image.") // Optional
      .setImage("https://....jpg", app.Media.ImageType.LARGE)
        // Optional. Use app.Media.ImageType.ICON if displaying icon.
  ])
)
.addSuggestions(["other songs"]);

Groping

@RémiC. How long does it take to Google Home to download a 50mins mp3 file ? The Google Home has a long latency ? Thank you – Bumkin

@JordanMontel To stream the 50 mins album, we need a mp3 version of it (as the doc says) and then we need to upload it somewhere we can get an https link. I chose google cloud storage. Finally, to play it, you need to give a link that plays the audio when you click on it, it's important. Thus about your question, the Google Home downloads nothing, but streams the audio it gets in the link you gave. – Hyposensitize

@RémiC. Google said mp3 files are 2mins max. But we can play mp3 more than 2mins and we don't have an answer of google about this limitation. Ours mp3 are located on ours servers with https link (not inside Google Cloud) and we can't move it. But Google Home took different times to stream the mp3 depending of the duration. It's for that we think Google Home download the mp3 on theirs servers before streaming (because ours mp3 are progressives links). Is for that I ask you about the latency ? And if the stream is very short, do you do something for reduce the latency ? Thank you – Bumkin

I tried that it seems only can play mp3. 😕 I want to play m3u8 streaming in my context. – Nadia

@JordanMontel I never tried with files stored outisde of Google's services, it might be because of that that I do not see this problem... And using Media responses allow us to bypass the 120s limitation of audio put by SSML. You might be right about google downloading your audio before playing it, I don't know. About the latency, I only put audio link of my google storage so I don't know how to reduce it otherwise, sorry – Hyposensitize

@Nadia Yes, in the documentation and every example made by Google they always use mp3 files, that's quite sad because the quality is not that great but it's probably a limitation on their end. – Hyposensitize

Answer 2

According to the documentation you can embed elements in SSML. https://developers.google.com/actions/reference/ssml includes the following example:

<speak>
  Here are <say-as interpet-as="characters">SSML</say-as> samples.
  I can pause <break time="3s"/>.
  I can play a sound
  <audio src="https://www.example.com/MY_MP3_FILE.mp3">didn't get your MP3 audio file</audio>.
  I can speak in cardinals. Your number is <say-as interpret-as="cardinal">10</say-as>.
  Or I can speak in ordinals. You are <say-as interpret-as="ordinal">10</say-as> in line.
  Or I can even speak in digits. The digits for ten are <say-as interpret-as="characters">10</say-as>.
  I can also substitute phrases, like the <sub alias="World Wide Web Consortium">W3C</sub>.
  Finally, I can speak a paragraph with two sentences.
  <p><s>This is sentence one.</s><s>This is sentence two.</s></p>
</speak>

EDIT

p/s : SSML in Documents has these limitations :

• Single channel is preferred, but stereo is acceptable.

• 120 seconds maximum duration. If you want to play audio with a longer duration, consider implementing a media response. 5 megabyte file size limit.

• Source URL must use HTTPS protocol.

• Our UserAgent when fetching the audio is "Google-Speech-Actions".

Ellipsis

Hey thanks for the answer, that's actually not what I am looking for due to the 120 second limitation. Our organization publishes weekly audio podcasts which are about an hour long. We want to be able to stream them to users upon request as we do on Alexa. Is there any way to stream an hour long mp3 file? – Waterish

Streaming is not supported yet. You can include multiple audio sources in a single SSML response. – Leffert

What about starting a cast session? Can anyone think of a way to initiate a cast session with the user action of speaking to the device. Currently it seems only Netflix and some approved partners are able to initiate cast session on chromecast devices. – Alvaroalveolar

@LeonNicholls Do you have any idea when streaming will be support in Actions on Google? – Hyposensitize

@LeonNicholls can we play live audio on google home device using ssml? – Quintal