Integrating Snips with Home Assistant


#1

If you want to read the blogpost --> https://medium.com/snips-ai/integrating-snips-with-home-assistant-314723645c77

With the rise of Amazon Echo and Google Home, we are getting increasingly accustomed to controlling our home via voice. But our home is also our most intimate space, and we have to think twice before installing an always-listening, cloud-connected device streaming our data to some distant server.

On February 28th 2017, an Amazon S3 outage resulted in massive home IoT dysfunction.

Cloud-connected devices have already had disastrous consequences for parents in Germany. And in the US, where more and more devices depend on remote servers to function, we’ve seen potentially dangerous situations arise.

At Snips, we are challenging the idea that adding Artificial Intelligence to our everyday life means giving up Privacy. Some things are just not meant to happen online, especially when it involves things we are speaking. This belief is why we recently launched the Snips Voice Platform, a 100% on-device alternative to Amazon Alexa, allowing anyone to easily add powerful voice assistants to connected devices without compromising their Privacy.

In this article, we will show how to control housing lights by voice using Snips and Home Assistant, without requiring an Internet connection.

Snips integrates nicely with Home Assistant, a popular Open Source platform for home automation. If you have any questions or comments, please join our community on Discord.

The Snips Voice Platform

The Snips Voice Platform features Hotword Detection, Speech Recognition, Natural Language Understanding and Dialog Management.

Similar to Amazon Alexa or API.ai, Snips takes voice or text as input, and produces intents as output, which are explicit representations of an intention behind an utterance, and which can subsequently be used by the system to perform appropriate actions. But instead of running the ASR and NLU in the cloud, Snips runs entirely on the Raspberry Pi, with blasting performance!

From utterance to intent, all happening on the Raspberry Pi!

Setting up Snips and Home Assistant

Hardware

We will be using the following hardware:

  • Raspberry Pi 3 ($39 on Adafruit)
  • SanDisk 16GB Micro SDHC ($10 on Amazon)
  • A standard USB microphone ($15 on Amazon)
  • Philips Hue Starter Kit ($69 on Amazon)

Home Assistant

Setting up a Raspberry Pi with Home Assistant is straightforward. For a detailed tutorial, see the Getting Started Guide.

Philips Hue

In this post, we will use voice commands to trigger Philips Hue lights. They are well integrated into Home Assistant. We simply need to add the following entry to configuration.yaml :

light:  platform: hue  host: HUE_BRIDGE_IP_ADDRESS

Check the Philips Hue Getting Started Guide for help obtaining the IP address of your Hue bridge.

Snips

The Snips Voice Platform is equally straightforward to set up:

(pi) $ sudo apt-get update(pi) $ sudo apt-get install -y dirmngr(pi) $ 

sudo bash -c ‘echo “deb https://raspbian.snips.ai/$(lsb_release -cs) stable main” > /etc/apt/sources.list.d/snips.list’

(pi) $ sudo apt-key adv --keyserver pgp.mit.edu --recv-keys D4F50CDCA10A2849(pi) $ sudo apt-get update(pi) $ sudo apt-get install -y snips-platform-voice snips-watch

Bridging Snips and Home Assistant

Messages between Snips and Home Assistant are passed via MQTT. We must tell Home Assistant which MQTT broker to use by adding the following entry to configuration.yaml :

mqtt:
broker: MQTT_BROKER_IP
port: MQTT_BROKER_PORT

As a convenience, Snips can run an MQTT broker on port 9898. So if we wish to use this broker, the entry will look as follows:

mqtt:
broker: 127.0.0.1
port: 9898

If we wish to use an external MQTT broker, we must change these values accordingly. Furthermore, we must tell Snips not to start its own broker but rather use an external one. This is specified in the launch parameters, as explained in the “Running the assistant” section below.

Next step: creating our Snips Assistant!

Creating a Snips Assistant

Step-by-step guide to building an assistant and deploying it on a Raspberry Pi.

We head over to console.snips.ai to build our assistant. We choose Raspberry, English and Snips On-Device ASR.

We choose the default setup: Raspberry Pi, English, and Snips On-Device ASR.

Now, we could go on and create intents for light queries manually. This involves adding a few sample phrases such as “Turn on the lights” and tagging them. However, Snips already offers intent bundles, which are collections of intents pre-trained on larger datasets and thoroughly tested. For our purposes, we choose the “IoT” intent bundle, which includes intents to control the lights, but also the heating, television, speakers and more.

The IoT intent bundle packages a few intents to control home IoT devices.

We then click Save, and launch the re-training of the assistant. Once done, we can download the assistant model.

Installing the assistant on the Pi

An assistantproj_XXX.zip file will be downloaded. Unzip it, and copy the folder to our Pi (replace <PI_HOSTNAME> with the actual device hostname):

$ scp -r <PATH_TO_ASSISTANT_FOLDER> pi@<PI_HOSTNAME>.local:/home/pi/assistant

Now log in to your Pi, and move the folder to /usr/share/snips/assistant and restart the Snips platform:

$ ssh

 

pi@<PI_HOSTNAME>.local

(pi) $ sudo mv /home/pi/assistant /usr/share/snips/assistant

Running the assistant

Make sure a microphone is plugged to the Pi and correctly detected. If you are having trouble setting up audio, we have written a guide on Raspberry Pi Audio Configuration. Snips is restarted as follows:

(pi) $ 

sudo systemctl restart snips*

Snips is now ready to take voice commands from the microphone. To trigger the listening, simply say

“Hey Snips”

followed by a command, e.g.

“Turn the lights green”

We should see the transcribed phrase in the logs, as well as a properly parsed intent. Last step is now to have Home Assistant react to these intents.

Optional: specifying an external MQTT broker

As explained earlier, by default Snips runs its own MQTT broker. But we can also tell Snips to use an external broker by specifying this when launching Snips. In this case, edit the file /etc/snips.toml on your device, uncommenting the mqtt line in the [snips-common] section, replacing <BROKER_HOSTNAME> and <BROKER_PORT> with appropriate values:

[snips-common]
mqtt = " <BROKER_HOSTNAME> : <BROKER_PORT> "

For more details on launch options, check the documentation on Snips Platform Commands.

Adding Home Assistant triggers

As mentioned above, the output of Snips is an intent. For instance,

“Turn the kitchen lights blue”

will output the following intent:

{
“text”: “Turn the kitchen lights blue”,
“intent”: {
“intentName”: “ActivateLightColor”,
“probability”: 0.9521337
},
“slots”: [
{
“entity”: “objectLocation”,
“slotName”: “objectLocation”,
“value”: {
“kind”: “Custom”,
“value”: “kitchen”
}
},
{
“entity”: “objectColor”,
“slotName”: “objectColor”,
“value”: {
“kind”: “Custom”,
“value”: “blue”
}
}
]
}

We now have to tell Home Assistant how to handle such intents. This is done via actions, defined in configuration.yaml , using the Home Assistant Script Syntax. For instance, to handle the above intents and trigger a light color change in a given room, we add the following entry:

snips:
intents:
ActivateLightColor:
action:

  • service: light.turn_on
    data_template:
    entity_id:
light.{{ 

objectLocation

 | replace(" ","_") }}            

color_name:

{{ 

objectColor

 }}

What is happening here is that the Snips custom component will invoke the script as defined in the action field, passing the intent slots as parameters. The templating system used by Home Assistant allows us to perform more complex actions involving conditionals, loops, states, and more. For a detailed explanation, see the Templating Guide.

That’s it! We now have a voice-controlled Home Assistant setup running entirely on-device! A sample configuration file can be found on Github. We invite you to contribute and make it better!


#2

This is essentially what I’m looking to do, but I’m wondering how to do with VMs rather than RPi. I recently moved my Home Assistant from an RPi to a VM running on ESXI 6 and would prefer to run Snips there as well.

I can’t seem to find instructions for this - many of the documentation pages seem to be down.

HA is running as a Hass.io image. Can I install snips in the same VM or should I spin up a separate one?

If I want remote mike/speaker sets (satellites?), what the best practice for those? Any detailed instructions on building them? Maybe so,e thing like the spkr tutorial but not deprecated?

I may be using wrong terms, I’ve literally started reading about snips today.


#3

Hi guys. I am trying to get this to work on my homeassistant (hassio) for 3 days now (and maybe 15 hours of reading and testing).

When i added no assistant file I could turn off the light in the living room. The other stuff did not work.

Now i installed an app without changing anything which works for another home assistant user. I created and copied the assistant.zip into the share folder. Restarted the addon and hassio found it and extracted it.

I (of course) changed the intent_script because the names of my covers are different.

  Covers:
    speech:
      type: plain
      text: 'Allright, {{ actionType }}ing the {{ whichCover }} cover.'
    action:
      - service_template: cover.{{ actionType | lower }}_cover
        data_template:
          entity_id: cover.{% if whichCover == "back" %}oeq1850763{% elif whichCover == "front" %}oeq1850763{% elif whichCover == "right" %}peq0005525{% elif whichCover == "left" %}peq0005525{% endif %}

I know its two times the same cover but i wanted to change as little as possible.

and thats my error.

[18:29:19] [Asr] was asked to stop listening on site default
[18:29:19] [Hotword] was asked to toggle itself 'off' on site default
[18:29:19] [Dialogue] session with id 'f7d50d60-b6c4-4eac-84ec-52ad71f655d5' was started on site default
[18:29:19] [AudioServer] was asked to play a wav of 41.1 kB with id 'e61623a0-5017-4e65-b3ee-02aad1aaead1' on site default
INFO:snips_audio_portaudio: Playing "e61623a0-5017-4e65-b3ee-02aad1aaead1" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 22050, bits_per_sample: 16, sample_format: Int }
INFO:snips_audio_portaudio: Playing of "e61623a0-5017-4e65-b3ee-02aad1aaead1" finished
[18:29:20] [AudioServer] finished playing wav with id 'e61623a0-5017-4e65-b3ee-02aad1aaead1'
WARN:snips_tts_hermes: Did received a "play finished" message from audio server with an unknown id 'e61623a0-5017-4e65-b3ee-02aad1aaead1'. It could be normal if you have more than one audio server.
[18:29:20] [Asr] was asked to listen on site default
INFO:snips_asr_hermes: Listening
INFO:snips_asr_lib::asr: T0       entered AsrRunner::run
INFO:snips_asr_lib::asr: T0+0.000 capture started
INFO:snips_asr_lib::asr: T0+4.577 endpoint detected (rule:1) frame:274 samples:70144 signal_time:4.384 rtf:0.857
INFO:snips_asr_lib::asr: T0+4.579 capture ended
INFO:snips_asr_lib::asr: T0+4.748 decoder finalized
INFO:snips_asr_lib::asr: T0+4.749 lookup and post-processing done
INFO:snips_asr_hermes  : Cleanup
INFO:snips_asr_hermes  : Preparing decoder
[18:29:24] [Asr] captured text "" in 4.0s
[18:29:24] [Asr] was asked to stop listening on site default
[18:29:24] [AudioServer] was asked to play a wav of 93.1 kB with id '6884693b-2562-42fa-a79a-169b8b42945f' on site default
INFO:snips_audio_portaudio: Playing "6884693b-2562-42fa-a79a-169b8b42945f" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 22050, bits_per_sample: 16, sample_format: Int }
INFO:snips_asr_hermes  : Idle
INFO:snips_audio_portaudio: Playing of "6884693b-2562-42fa-a79a-169b8b42945f" finished
[18:29:26] [AudioServer] finished playing wav with id '6884693b-2562-42fa-a79a-169b8b42945f'
WARN:snips_tts_hermes: Did received a "play finished" message from audio server with an unknown id '6884693b-2562-42fa-a79a-169b8b42945f'. It could be normal if you have more than one audio server.
[18:29:26] [AudioServer] was asked to play a wav of 61.1 kB with id '39ab5d44-2f19-45ca-82ec-83e322718bc6' on site default
INFO:snips_audio_portaudio: Playing "39ab5d44-2f19-45ca-82ec-83e322718bc6" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 44100, bits_per_sample: 16, sample_format: Int }
INFO:snips_audio_portaudio: Playing of "39ab5d44-2f19-45ca-82ec-83e322718bc6" finished
[18:29:26] [AudioServer] finished playing wav with id '39ab5d44-2f19-45ca-82ec-83e322718bc6'
INFO:snips_dialogue_lib::coordinator::coordinator: Session [f7d50d60-b6c4-4eac-84ec-52ad71f655d5]: closing session which was active: IntentNotRecognized
INFO:snips_dialogue_lib::coordinator::coordinator: Site [default]: reset site -> turn off ASR and turn on Hotword
[18:29:26] [Dialogue] session with id 'f7d50d60-b6c4-4eac-84ec-52ad71f655d5' was ended on site default. The session was ended because the platform didn't understand the user
WARN:snips_tts_hermes: Did received a "play finished" message from audio server with an unknown id '39ab5d44-2f19-45ca-82ec-83e322718bc6'. It could be normal if you have more than one audio server.
[18:29:26] [Asr] was asked to stop listening on site default
[18:29:26] [Hotword] was asked to toggle itself 'on' on site default
INFO:snips_hotword_lib::features : Audio thread for site default running

edit: going to reinstall the addon

edit2: when i plug the headset into my computer and use it on snips to test the assistant, it recognizes the things i say perfectly!

edit3: When i delete the assistant.zip file and use the standard one it detects what I say.

[18:52:46] [Hotword] detected on site default, for model hey_snips
INFO:snips_dialogue_lib::coordinator::coordinator: Site [default]: reset site -> turn off ASR and turn off Hotword
INFO:snips_dialogue_lib::coordinator::coordinator: Session [f6f834e8-b832-402a-89fa-5ef36f650d09]: The session has been created because the site "default" is free
[18:52:46] [Asr] was asked to stop listening on site default
[18:52:46] [Hotword] was asked to toggle itself 'off' on site default
[18:52:46] [Dialogue] session with id 'f6f834e8-b832-402a-89fa-5ef36f650d09' was started on site default
[18:52:46] [AudioServer] was asked to play a wav of 41.1 kB with id 'ace67098-2e89-4145-bfa6-42607b7bce8f' on site default
INFO:snips_audio_portaudio: Playing "ace67098-2e89-4145-bfa6-42607b7bce8f" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 22050, bits_per_sample: 16, sample_format: Int }
INFO:snips_hotword_lib::features : Audio thread for site default idle.
INFO:snips_audio_portaudio: Playing of "ace67098-2e89-4145-bfa6-42607b7bce8f" finished
[18:52:47] [AudioServer] finished playing wav with id 'ace67098-2e89-4145-bfa6-42607b7bce8f'
WARN:snips_tts_hermes: Did received a "play finished" message from audio server with an unknown id 'ace67098-2e89-4145-bfa6-42607b7bce8f'. It could be normal if you have more than one audio server.
[18:52:47] [Asr] was asked to listen on site default
INFO:snips_asr_hermes  : Listening
INFO:snips_asr_lib::asr: T0       entered AsrRunner::run
INFO:snips_asr_lib::asr: T0+0.000 capture started
INFO:snips_asr_lib::asr: T0+4.187 endpoint detected (rule:2) frame:169 samples:43264 signal_time:2.704 rtf:1.407
INFO:snips_asr_lib::asr: T0+4.197 capture ended
INFO:snips_asr_lib::asr: T0+4.484 decoder finalized
INFO:snips_asr_lib::asr: T0+4.484 lookup and post-processing done
INFO:snips_asr_hermes  : Cleanup
INFO:snips_asr_hermes  : Preparing decoder
[18:52:51] [Asr] captured text "light on" in 4.0s
[18:52:51] [Asr] was asked to stop listening on site default
[18:52:51] [AudioServer] was asked to play a wav of 93.1 kB with id 'e55fe924-1da4-415d-bcc6-555b479cfe08' on site default
INFO:snips_audio_portaudio: Playing "e55fe924-1da4-415d-bcc6-555b479cfe08" using output "default", wav spec: WavSpec { channels: 2, sample_rate: 22050, bits_per_sample: 16, sample_format: Int }
INFO:snips_asr_hermes  : Idle
INFO:snips_audio_portaudio: Playing of "e55fe924-1da4-415d-bcc6-555b479cfe08" finished
WARN:snips_tts_hermes: Did received a "play finished" message from audio server with an unknown id 'e55fe924-1da4-415d-bcc6-555b479cfe08'. It could be normal if you have more than one audio server.
[18:52:52] [AudioServer] finished playing wav with id 'e55fe924-1da4-415d-bcc6-555b479cfe08'
[18:52:53] [Nlu] was asked to parse input "light on"
[18:52:53] [Nlu] detected intent hass:HassTurnOn with probability 0.896 for input "light on"
              Slots ->
                 name -> light
[18:52:53] [Dialogue] New intent detected hass:HassTurnOn with probability 0.896
              Slots ->
                 name -> light
[18:52:53] [Dialogue] was ask to end session with id f6f834e8-b832-402a-89fa-5ef36f650d09 by saying 'Turned Xiaomi Gateway Light on'
[18:52:53] [Tts] was asked to say "Turned Xiaomi Gateway Light on"
INFO:snips_audio_portaudio: Playing "3aff9bac-30d6-4b53-b0f7-b827dff8cc30" using output "default", wav spec: WavSpec { channels: 1, sample_rate: 16000, bits_per_sample: 16, sample_format: Int }
[18:52:53] [AudioServer] was asked to play a wav of 68.7 kB with id '3aff9bac-30d6-4b53-b0f7-b827dff8cc30' on site default

#4

You ever figure this out?


#5

Hi schmidty.

Hassio seems to use an old version which is not compatible. I switched to raspbian + HomeAssistant.


#6

I just updated it this weekend. The issue was more that snips had a bug in the ARM/PI version of the NLU and didn’t update the packages or docker image for it so I ended up having to rebuild docker images from scratch. Let me know if you need help with anythign HA related.


#7

I use Hassio addon and snips recognize well all my intents.

If homeassistant don’t trigger the intent, try to remove ' ' from text:

speech:
  type: plain
  text: Allright, {{ actionType }}ing the {{ whichCover }} cover.

For me it worked