Audio Server Spamming MQTT

I’ve just really started working with Snips. I have a Base running on Ubuntu 18.04 and a satellite on a Pi 0 W.

I’ve never gotten the base to properly respond to commands spoken to the satellite. The base recognizes the intent but doesn’t seem to send the audio. So after a few rebuilds, endless hours troubleshooting I had someone help me integrate Snips with Home Assistant and use HA to capture the intents and process them.

So last night I installed Node Red and started playing with it. I sub’d to # on my mqtt broker to figure out the topic and payload when Snips was being used and I discovered that Snips is spamming my MQTT broker.

So in Node Red I see that on the topic /hermes/audioServer/base/audioFrame there are thousands? of messages. I’ve attached a screenshot, it’s hard to describe but the time just keeps counting and the _msgid: " " numbers change so fast that they are honestly a blur.

Using mosquitto_sub on topic ‘#’ I see this constant lines of:

RIFF4WAVEfmt ▒>}tim▒▒
endata
RIFF4WAVEfmt ▒>}tim▒▒
endata
RIFF4WAVEfmt ▒>}tim▒▒
endata
RIFF4WAVEfmt ▒>}tim▒▒
endata
RIFF4WAVEfmt ▒>}timё
endata

When I stop ‘snips-audio-server’ the “RIFF4WAVEfmt ▒>}timё” messages disappear.

This can’t be normal can it?

There was so much traffic coming from the Base Audio server that it took nearly 5 minutes for the Node-Red buffer to clear.

When I checked jounalctl on the base I noticed this:

Nov 13 06:32:01 heimdall systemd[1]: Started Snips Audio Server.
Nov 13 06:32:02 heimdall snips-audio-server[15929]: INFO:snips_audio_server_cli: Starting audio server on mqtt for site base
Nov 13 06:32:02 heimdall snips-audio-server[15929]: INFO:snips_audio_alsa::capture: Starting ALSA capture on device "default"
Nov 13 06:32:02 heimdall snips-audio-server[15929]: INFO:snips_audio_server_hermes::player: started hermes audio player thread
Nov 13 06:32:02 heimdall snips-audio-server[15929]: INFO:snips_broadcast_mqtt_server      : Starting broadcasting audio on hermes for site "base"

Why is the base broadcasting audio when nothing is happening?

Normal, every message is a small wav file with audio.
This is the wat Snips streams audio from your microphone.

It is broadcasting audio for the dialogmanager to detect the hotword.

I’m sorry I don’t understand.

The messages are being sent by the snips-audio-server on the Snips base. There is no similar traffic coming from the satellite which is where the hotword detection is being performed.

The should be the same on your broker with:
/hermes/audioServer/<siteidsatellite>/audioFrame

If not, then that is also the reason why your base does not respond :wink:

How is your satellite configured?

Here is my Satellite toml file:

[snips-common]
 mqtt = "192.168.1.105:1883"

## MQTT authentication
 mqtt_username = "user"
 mqtt_password = "pass"


[snips-audio-server]
 bind = "Kitchen@mqtt"

[snips-hotword]
 audio = ["Kitchen@mqtt"]

And my Base toml file:

 [snips-common]
 mqtt = "192.168.1.105:1883"

## MQTT authentication
 mqtt_username = "user"
 mqtt_password = "pass"

[snips-audio-server]
 bind = "base@mqtt"

[snips-hotword]
 audio = ["Kitchen@mqtt"]

@Romkabouter It would be amazing if you could help sort out why the base isn’t responding properly.

If you use a mqtt program like MQTT fx, there is a button “scan”

When the satellite is producing audio, the topic hermes/audioServer/Kitchen/audioFrame should appear
Try that first and check if the audio is actually been transmitted to the broker.

Here’s a link: https://mqttfx.jensd.de/, but you can use other software as well.
You can also just add that topic to your NodeRed and see if the message broadcast is active :slight_smile:
Using hermes/audioServer/# will work for that as well, to capture all

@Romkabouter

Oh I know audio is being transmitted to the broker. Home Assistant will respond to intents that I setup in it.
Also from Snips-Watch -vvv I see this:

[18:27:28] [Asr] was asked to listen on site Kitchen
[18:27:32] [Asr] captured text "hello" in 3.0s
[18:27:32] [Asr] was asked to stop listening on site Kitchen
[18:27:32] [AudioServer] was asked to play a wav of 93.1 kB with id '4fe1f0cd-cf55-4a65-a74a-a6463da63480' on site Kitchen
[18:27:33] [AudioServer] finished playing wav with id '4fe1f0cd-cf55-4a65-a74a-a6463da63480'
[18:27:33] [Nlu] was asked to parse input "hello"
[18:27:34] [Nlu] detected intent SirBuildsALot7:hello with confidence score 1.000 for input "hello"
              Slots ->
                 hello -> hello (confidence: 1.000)
[18:27:34] [Dialogue] New intent detected SirBuildsALot7:hello with confidence 1.000
              Slots ->
                 hello -> hello (confidence: 1.000)
[18:27:39] [Dialogue] session with id '2e130ad6-4818-4d68-8c15-bd36313601fb' was ended on site Kitchen. The session timed out because its continuation or termination was not handled by any running action code.
[18:27:39] [Asr] was asked to stop listening on site Kitchen
[18:27:39] [Hotword] was asked to toggle itself 'on' on site Kitchen

O okay, sorry about that.

So now I am a bit confused about your setup :smiley:
You use home assistant, is that on the base? Or?..

@Romkabouter No need to apologize. Proper troubleshooting question. At least you didn’t tell me to reboot the modem :slight_smile:

@Romkabouter

I have HA running on a VM, someone was able to help me out to get some level of Snips working.
So I have the Snips module(?) setup in Home Assistant and a file called snips_intents.yaml where I manually tell HA what Intents I want to take action on. If the Intent isn’t in the file, nothing happens.

So here is an entry that turns on my table lamp when I say a command to snips that falls under the TurnOnLivingRoomLamp Intent. eg “Hey Snips, Turn on Table Lamp”

  TurnOnLivingRoomLamp:
    speech:
      type: plain
      text: 'OK, turning on the table lamp'
    action:
      - service: light.turn_on
        data:
          entity_id: light.living_room

This sends the text (as TTS) to my Snips Satellite “OK, turning on the table lamp” and then turns on the table lamp.

Ok, that seems to be fine then.

The Snips Addon in your Hassio serves as the base now and seems to be working.
Can you elaborate a bit more what you call “Base”?
If you have it in Hassio, then the Snips installation of the base is not needed.
It seems you use HassIO with the Snips Addon, is that correct?

So I have Home Assistant (I never got Hassio to work) running on a Ubuntu VM.
I have Snips running on a separate Ubuntu VM, this is my Base. I also have snips running on a Rpi 0 which is my Satellite

The only reason that Home Assistant is involved right now as I have been unable to get the Base to respond to spoken Intents in a year of trying.

ok, so the above problem is the issue correct?
You mention the base recognizing the intent, that implies that the configuration base <==> satellite work correctly.

What have you put in place in your base to reply to those intents? Running a skill-server or such?

@Romkabouter
I setup the Base as per the instructions on snips.ai. So I have snips-skill-server, snips-audio-server and others that I can’t recall at the moment.

I have Greetings by SirBuildALot7, Weather by snips, TimesTables_EN by snips, and Timers - EN by AcidFlow.
When I start snips-skill-server manually and watch the output there are no errors in the skills and they are all detected properly.

So, the base is responding well?

Does this mean the only problem is sending audio to base and/or satellite?

@Romkabouter

I don’t know, I’ve never gotten it to work. All I know is that I say an intent, the intent is recognized and then nothing happens and things time-out.

Ok, I think your configurations are all correct.

The only thing missing is a piece of software actually responding to the intents.
That is why your HomeAssitant works, HA is responding to the intents via the intent_script config.

I don’t work with the skillserver, but that should be the software I guess. I will try and see how that skillserver works :slight_smile:
I only use Home Assistant

I have done some investigations

The Weather and Greetings actually do not do anything by itself.
See this comment on the Weather app:


Timetables_EN should react, but I do not know if you have tried that.

When installing, I needed to run this command on the Pi before it worked:

sudo chmod +x /var/lib/snips/skills/snips-skill-times-tables-quiz/action-quizz.py

That is because I got this message (please ignore the read errors, that is about python 2.7):

After I had done that I could test it:

So I suggest you test with the TimeTables skill to see if you get audio output :slight_smile: