Custom Google ASR component

Since Snips Google ASR is now deprecated I decided to create my own implementation:


It’s skill, so it’s easy to install. Generally it works, but when Snips is asking for missing intents and I give an answer by publishing result on hermes/asr/textCaptured topic I get an error like this one:

[21:31:08] [Dialogue] session with id 'c4d2d391-3c88-4a22-a362-70652e584732' was ended on site room. The session was ended because an error happened: Receives error from component Nlu: { error: Cannot use unknown intent 'kblachowicz:Answer' in intents filter , context: telewizor }

Here is the dump of the whole MQTT session:
mqtt

Can someone help me with that, please?

1 Like

Hi chris,
I think this component is amazing and i’m actually cloning it into my Snips app on raspberry pi. However, after I install your code into “skills” folder and run “sam watch”, it seems like it is still using the old Snips default ASR, which means Google ASR is not triggered. Am I missing something here? What’s the steps of triggering this skill component? Is there any other files I need to manually edit? Thanks so much! And your asr component would be a big help for me.

You need to remove Snips ASR manually by running:

apt-get remove snips-asr

Thx for your reply! I did that step but the Google asr is still not triggered. Since I remove snips-asr while google asr is not running, currently i don’t have any asr running.

snips-analytics .............. 0.64.0 (not running)

snips-asr .................... 0.64.0 (not running)

snips-audio-server ........... 0.64.0 (running)

snips-dialogue ............... 0.64.0 (running)

snips-hotword ................ 0.64.0 (running)

snips-nlu .................... 0.64.0 (running)

snips-skill-server ........... 0.64.0 (running)

snips-tts .................... 0.64.0 (running)

 

**pi@raspberrypi** : **~ $** sam watch

**[16:40:22]** Watching on localhost:1883 (MQTT)

[16:40:23] [Hotword] detected on site **default** , for model **hey_snips**

[16:40:23] [Asr] was asked to stop listening on site **default**

[16:40:23] [Hotword] was asked to toggle itself 'off' on site **default**

[16:40:23] [Dialogue] session with id 'ceed79bd-8115-4914-8ae0-a13fc848dfe4' was started on site **default**

[16:40:23] [AudioServer] was asked to play a wav of 41.1 kB with id ' **7a20312c-3fef-4a40-85f9-c56fa052169d** ' on site **default**

[16:40:23] [AudioServer] finished playing wav with id ' **7a20312c-3fef-4a40-85f9-c56fa052169d** '

[16:40:23] [Asr] was asked to listen on site **default**

[16:40:38] [Dialogue] session with id 'ceed79bd-8115-4914-8ae0-a13fc848dfe4' was ended on site **default** . The session timed out because the **ASR component** did not respond in a timely manner. Please ensure that the Asr is started and running correctly.

[16:40:38] [Asr] was asked to stop listening on site **default**

[16:40:38] [Hotword] was asked to toggle itself 'on' on site **default**

[16:40:49] [Hotword] detected on site **default** , for model **hey_snips**

[16:40:49] [Asr] was asked to stop listening on site **default**

[16:40:49] [Hotword] was asked to toggle itself 'off' on site **default**

[16:40:49] [Dialogue] session with id '1e4b9b42-8dc6-4b19-ab4e-cfc59f3ef0b3' was started on site **default**

[16:40:49] [AudioServer] was asked to play a wav of 41.1 kB with id ' **7369a19e-ea0d-4145-9f62-0f31ee325f9e** ' on site **default**

[16:40:50] [AudioServer] finished playing wav with id ' **7369a19e-ea0d-4145-9f62-0f31ee325f9e** '

[16:40:50] [Asr] was asked to listen on site **default**




sam service log snips-skill-server

-- Logs begin at Thu 2016-11-03 10:16:43 PDT. --

Oct 03 16:38:12 raspberrypi systemd[1]: Started Snips Skill Server.

Oct 03 16:38:13 raspberrypi snips-skill-server[459]: INFO:snips_skill_server_lib::runner: searching dir for actions: /var/lib/snips/skills/snips_google_asr

Oct 03 16:38:13 raspberrypi snips-skill-server[459]: INFO:snips_skill_server_lib::runner: found action "/var/lib/snips/skills/snips_google_asr/action-asr.py"

Oct 03 16:38:24 raspberrypi snips-skill-server[459]: INFO:snips_skill_server_lib::runner: [asr][out] Connected to MQTT

Oct 03 16:38:24 raspberrypi snips-skill-server[459]: INFO:snips_skill_server_lib::runner: [asr][out] Subscribing site "+"

Any reasons? And again really appreciate your help!

Please pull the change I made in repo (the site ID should be “default” instead “+”) and check again.

Oct 03 17:05:50 raspberrypi snips-skill-server[1651]: INFO:snips_skill_server_lib::runner: [asr][out] Connected to MQTT

Oct 03 17:05:50 raspberrypi snips-skill-server[1651]: INFO:snips_skill_server_lib::runner: [asr][out] Subscribing site "default"

still not working… I think the def text_captured(text, likelihood, seconds, siteId, sessionId):
is not called for some reason. Do I need to modify files like “/etc/snips.toml” or “assistant.json”? Do i need to do “apt-get snips-asr-google” ? I did that.

My component is a replacement for snips-asr and snips-asr-google, so both of them should be uninstalled.
Please paste your snips.toml config, I’ll take a look.

PS. Does googlecredentials.json file exists in the location given in config.ini?

Now I removed both snips-asr and snips-asr-google. yes I also included my google serviceaccount.json in “/usr/share/snips/googlecredentials.json”. Below is my snips.toml.
Thx so much for reviewing it!

[snips-common]

# bus = "mqtt"

# mqtt = "localhost:1883"

# audio = ["+@mqtt"]

# assistant = "/usr/share/snips/assistant"

# user_dir = "/var/lib/snips"

## MQTT authentication

# mqtt_username = ""

# mqtt_password = ""

## MQTT TLS configuration

# mqtt_tls_hostname = ""

# mqtt_tls_disable_root_store = false

# mqtt_tls_cafile = ""

# mqtt_tls_capath = ""

# mqtt_tls_client_cert = ""

# mqtt_tls_client_key = ""

[snips-analytics]

[snips-asr]

# no_fst_map = true

# beam_size = 8

# model = "/usr/share/snips/snips-asr-model-en-500MB"

# audio = ["+@mqtt"]

# partial = false

# partial_period_ms = 250

[snips-asr-google]

#credentials = "/usr/share/snips/googlecredentials.json"

#audio = ["+@mqtt"]

[snips-audio-server]

# frame = 256

# bind = "default@mqtt"

# mike = "Built-in Microphone"

# disable_playback = false

# disable_capture = false

[snips-dialogue]

# resources = "/usr/share/snips/dialogue"

# session_timeout = 15

# lambda_timeout = 5

# retry_count = 3

# sound_feedback_disabled_default = true

[snips-hotword]

# model = "/usr/share/snips/hotword"

# hotword_id = "default"

# sensitivity = "0.5"

# audio = ["+@mqtt"]

# no_vad_inhibitor = false

# vad_messages = false

[snips-injection]

#asr_model = "/usr/share/snips/asr"

#nlu_engine = "/usr/share/snips/nlu_engine"

#keep_nfirst = 3

[snips-nlu]

[snips-tts]

## Choose one tts provider (defaults to picotts)

# provider = "picotts"

# provider = "makerstts"

# provider = "customtts"

## customtts specific configuration (here configured to use picotts using the en-US language)

## available placeholder variables : %%OUTPUT_FILE%%, %%LANG%%, %%TEXT%%

# customtts = { command = ["pico2wave", "-w", "%%OUTPUT_FILE%%", "-l", "en-US", "%%TEXT%%"] }

[snips-pegasus]

# address = "0.0.0.0:8443"

## Pegasus TLS configuration

## Self signing certificates are generated with /usr/local/bin/snips-pegasus-tls-keygen.sh

# tls_certificate_chain = "/var/lib/snips/pegasus/tls/server.fullchain"

# tls_private_key = "/var/lib/snips/pegasus/tls/server.key"

BTW, this is my “sam watch” log:

    [13:31:58] Watching on localhost:1883 (MQTT)
    [13:32:12] [Dialogue] session with id '352de748-62fd-450f-baf2-0860077f0538' was ended on site default. The session timed out because the ASR component did not respond in a timely manner. Please ensure that the Asr is started and running correctly.
    [13:32:12] [Asr] was asked to stop listening on site default
    [13:32:12] [Hotword] was asked to toggle itself 'on' on site default
    [13:32:15] [Hotword] detected on site default, for model hey_snips
    [13:32:15] [Asr] was asked to stop listening on site default
    [13:32:15] [Hotword] was asked to toggle itself 'off' on site default
    [13:32:15] [Dialogue] session with id '9a72d964-6dba-4ddb-833b-90cdfa42f8c4' was started on site default
    [13:32:15] [AudioServer] was asked to play a wav of 41.1 kB with id 'b28b67a6-3dcb-46a4-8ba3-0e0377ee6862' on site default
    [13:32:16] [AudioServer] finished playing wav with id 'b28b67a6-3dcb-46a4-8ba3-0e0377ee6862'
    [13:32:16] [Asr] was asked to listen on site default
    [13:32:31] [Dialogue] session with id '9a72d964-6dba-4ddb-833b-90cdfa42f8c4' was ended on site default. The session timed out because the ASR component did not respond in a timely manner. Please ensure that the Asr is started and running correctly.
    [13:32:31] [Asr] was asked to stop listening on site default
    [13:32:31] [Hotword] was asked to toggle itself 'on' on site default

There is nothing odd in your configuration.

Is that all you have in log after session ends? Please run:

journalctl -u snips-skill-server

after timeout occurs.

Hi,
The output of journalctl -u snips-skill-server is nothing: -- no entries --.
I set couple breakpoints in the code as shown here:

def on_message(client, userdata, msg):
    if msg.topic.endswith("audioFrame"):
        capture_frame(msg)
    if msg.topic == startListeningTopic:
        print('168')
        payload = (json.loads(msg.payload))
        print('170')
        site_id = payload['siteId']
        session_id = payload['sessionId']
        print('172')
        start_listening(site_id, session_id)
    if msg.topic == stopListeningTopic:
        print('174')
        payload = (json.loads(msg.payload))
        site_id = payload['siteId']
        print('179')
        stop_listening(site_id)

However, when I looked at the snips-skill-server log, it seems that when “msg.topic == startListeningTopic” and “msg.topic == stopListeningTopic” are reached, start_listening and stop_listening function are never called. And the reason is due to the line “payload = (json.loads(msg.payoload))” is not called.

Oct 07 10:28:09 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: searching dir for actions: /var/lib/snips/skills/snips_google_asr

Oct 07 10:28:09 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: found action "/var/lib/snips/skills/snips_google_asr/action-asr.py"

Oct 07 10:28:12 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] Connected to MQTT

Oct 07 10:28:12 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] Subscribing site "default"

Oct 07 10:28:15 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

Oct 07 10:28:16 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 168

Oct 07 10:28:31 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

Oct 07 10:29:34 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

Oct 07 10:29:34 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 168

Oct 07 10:29:49 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

Oct 07 10:29:57 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

Oct 07 10:29:58 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 168

Oct 07 10:30:13 raspberrypi snips-skill-server[15858]: INFO:snips_skill_server_lib::runner: [asr][out] 174

you can see in the log that only “174” and “168” are printed, which only means msg.topic = startListeningTopic and stopListeningTopic.

Do you know how to solve this issue? Thanks so much!

I have just merged fix for that. Make a pull and try again.

Thanks for your update! I’ve tried the fix and it finally captures text after I fixed one bug. So I think the language=en-EN should be changed to language=en in config.ini. Otherwise, it will give me exception.

After I fixed this, the asr finally works and successfully captures text! So that’s great! But my log shows this error when record stops:

Oct 14 11:42:21 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Audio processing stopped on site default
Oct 14 11:43:05 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Listen start on site default
Oct 14 11:43:05 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Processing audio on site default...
Oct 14 11:43:05 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Record start on site default
Oct 14 11:43:08 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Captured text "what's your name" on site default with confidence 0.9876290559768677
Oct 14 11:43:08 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Listen stop on site default
Oct 14 11:43:08 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Record stop on site default
Oct 14 11:43:08 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] 'Transcoder' object has no attribute '_stopevent'
Oct 14 11:43:08 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Audio processing stopped on site default

The error is 'Transcoder' object has no attribute '_stopevent'

I looked into your code and found that there is only one occurrence of “_stopevent” in join() function. Have you encountered this? How to solve this bug?

And also, if I stop speaking for a while, the log will show this 400 Audio Timeout Error error after “record stop”…Is this normal? And does it affect the loop?

Listen timeout on site default
Oct 14 11:43:22 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Nothing captured on site default
Oct 14 11:43:22 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Listen stop on site default
Oct 14 11:43:22 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Record stop on site default
Oct 14 11:43:32 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] 400 Audio Timeout Error: Long duration elapsed without audio. Audio should be sent close to real time.
Oct 14 11:43:32 raspberrypi snips-skill-server[1684]: INFO:snips_skill_server_lib::runner: [asr][out] Audio processing stopped on site default

Thank you so much for keeping your github updated with new fixes~ And really appreciate your help with my bugs!