Playing a wav file as a snippet?

#1

Does anyone have an example of playing a wav file as a code snippet rather than a tts replay to an intent. I looked at the documentation on the audio server and I just can’t make that transition in my mind.

#2

what sort of code?? python?

#3

Yes, I’ve created the TTS for the same answer in Python. I’d like to show a customer the difference. BTW, I’ve tried all the available voices on the RPi.

#4

two ways

just using MQTT in terminal

using python

fp = open('file.wav','rb')
f = fp.read()

#send over mqtt as a byte array
mqtt.publish('hermes/audioServer/<snips_device_Id>/playBytes/', bytearray(f))

Wave file must be 16k mono… if not you will have to re-encode the wave file before sending it. If not I think you just will not hear snips play it

#5

Ozie, I entered the code below into the snippet for my intent (modified for the audio I created and placed in the Pi directory). I’ve used aplay to play the wav file so I know it is 16 bit. I get the message “The session was ended because one of the component didn’t respond in a timely manner” when using Sam Watch. Not sure what component is failing; I suspect it is the Audio Server. Am I doing something obviously wrong?

fp = open(’./Audio2.wav’,‘rb’)
f = fp.read()

#send over mqtt as a byte array
mqtt.publish(‘hermes/audioServer/snips-base/playBytes/’, bytearray(f))

#6

Ozie,
I turned up verbosity in the skills-server to see if I can find where the component is stalling.
Appears to be the definition of mqtt

[01:06:04.348131] INFO : snips_skill_server_lib::runner : [rsheffield99-WhaleTypes-rsheffield99.Rod_Sheffield][err] mqtt.publish(‘hermes/audioServer/default/playBytes/’, bytearray(f))

[01:06:04.348175] INFO : snips_skill_server_lib::runner : [rsheffield99-WhaleTypes-rsheffield99.Rod_Sheffield][err] NameError: name ‘mqtt’ is not defined

Advice?

#7

does your snippet code also return a endSession message to inform snips you completed the intent?? https://docs.snips.ai/reference/dialogue#end-session

if you are not sending the endSession snips will wait for the timeout period and if it does not receive the endSession you will see the The session was ended because one of the component didn’t respond in a timely manner message.

If you do have the endSession code, then you have other problems where the code of the snippet is failing so that code after the failing point is never run

#8

ahh… well my example code was only an example of how to use a file to use… It was not in the format of what would be used in a snippet

mqtt is not an object in your code… as I don’t know what code you have or what you’re using using you will have to configure it to suit

If you are using hermes-python I do not believe it has any api to publish to the audio-server (which is why I don’t use hermes-python for my applications - it is too restrictive)

#9

Ozie,
I get it. The Dialog Manager API has a restricted number of interfaces that can be used by Code Snippets. The End Session only publishes in response to a specific Session ID with a string to TTS. There is no option for replacing the text string with a Byte Array created by the Audio Server even though the Audio Server has a Hermes-Python API. So … if I want to play a wav file in response to an intent, I will have to create lower level code as an action and post it to Git.

#10

Hey guys,

It may help,
There is a way (a bit hidden an limited tough :p)

We wanted to advertise it for the next release, but it’s already available in the latest one.

So here is the trick,

You first need to register a sound,

mosquitto_pub -h my-platform.local -p 1883 -t hermes/tts/registerSound/mysound -f my_wav_file.wav

This route is not yet documented but allows you to register a sound limited to 10 secs. (It is a design choice, the goal is not to play music but ponctuate tts with short sounds)

Then you can use it by passing the corresponding tag [[sound:mysound]] along a text you want to be uttered in a start, stop or continue session.

mosquitto_pub -h my-platform.local -p 1883 -t hermes/dialogue/[start|stop|continue]Session -m '{"siteId":"default", "lang":"en", "text": "Do you hear this sound? [[sound:mysound]] Thanks for listening. ", "id": "someramdomid", "sessionId": "somerandomsessionid"}

It is available using MQTT directly.

1 Like
#11

I was right


I just didn’t know how to use it once it was registered :slight_smile:

1 Like
#12

SakananSan Thanks for the update. I’ve tried the solution but I get a syntax error on the very first parameter (host name) of the registration. I’ve looked up mosquito pub in the man page, verified the syntax and put in some punctuation to no avail. The only changes to your example are to put in snips-base for “my-platform” and fully specify the path to my wav file. Do I need additional punctuation when specifying the path? That still doesn’t explain why I am getting syntax errors on the host parameter.

#13

@ozie yup, nice catch :slight_smile:

#14

@Rod

Can you provide the command you used and the corresponding output to ease the support?

#15

Sure, I went to the command line and input the commands first so as to get a clean process. Once I got that, I went back and entered it into the Snippet action code. Still getting a syntax error on the hostname parameter, see below (in a stretched out debug window, the “^” points to “snips-base.local”:

[01:19:05.668703] INFO :snips_skill_server_lib::runner            : [rsheffield99-WhaleTypes-rsheffield99.Rod_Sheffield][err]     mosquitto_pub -h snips-base.local -p 1883 -t hermes/tts/registerSound/mysound, -f /home/pi/Audio2.wav
[01:19:05.668818] INFO :snips_skill_server_lib::runner            : [rsheffield99-WhaleTypes-rsheffield99.Rod_Sheffield][err]                          ^
[01:19:05.668899] INFO :snips_skill_server_lib::runner            : [rsheffield99-WhaleTypes-rsheffield99.Rod_Sheffield][err] SyntaxError: invalid syntax
#16

Hi Rod,

The syntax error is due to the comma right after ‘mysound’
The command should be
mosquitto_pub -h snips-base.local -p 1883 -t hermes/tts/registerSound/mysound -f /home/pi/Audio2.wav

#17

oh, wait…

Are you using those commands in a python script?

The example provided are to be inputed in a terminal using the program mosquitto_pub (installed using sudo apt install mosquitto-clients on a debian like linux)

Using python, you have to rely on a mqtt library such as paho-mqtt.

Sorry if I was unclear about that.

#18

Eric, SakanaSan, Thanks to you both for the help. One more question. Will I need to import the paho-mqtt library in my Snippet code, or is that already done somewhere else in the set up since the rest of the support code for Snippets relies on the MQTT libraries?

#19

Unfortunately it is not possible from the console.

You can tinker with the scripts generated on your pi, but it is not really a good practice.

The best solution is to forge your own code repository to do so and add paho-mqtt in your project requirements

Take a look to some repository already existing if you need some inspiration to start with.

#20

SakanaSan, Thanks, I started by looking in the repository at apps like Music and Karaoke. Unfortunately, none of those really show how the audio files are played either. This is why I asked for an example. I would have thought there would be a straightforward way to play an audio file directly from the Console (Snippets) just as there is a way to do the TTS playback of a text string. The TTS server clearly calls the Audio Server to make this happen as I can see that in the output of the Sam Watch. Seems like this would be a way people could try out music files without going through an offline Python development.