MQTT messages on Android / RASA integration

Hi,

I am working on a conversational voice assistant for a specific topic. I want integrate it into an Android app, thus Snips is perfect! However, due to its limited capabilities regarding conversations I would like to combine it with the NLU and dialogue capabilities of RASA instead and only use the ASR features.

Since the database backend is likely also in the cloud it is fine to query a remote server for chatbot interactions after the speech has been transcribed.

I figured out that I apparently can deactivate the snips-nlu daemon and just subscribe to ASR related messages on the Hermes bus protocol with a Hermes client. In this way, it should be easy to access the transcription and then forward it to a RASA backend in the cloud.

Though, here comes the hitch: the documentation says on Android only Dialogue API is implemented as yet and no further messaging regarding ASR or other Hermes-alike functionality is available!?

So, my questions are:

  • Can I use MQTT (Hermes) on Android with Snips, instead?
  • Or can I still use the Snips lib for my suggested app in some way or another in order to catch ASR transcription and forward it to my own backend (like RASA NLU and dialogue control)?

Thank you very much for your help in advance!

Cheers, Snipper

Hi again @DevelopmentTeam - is there no one able/willing to help out here? :-/

Hi @Snipper,

You can’t use MQTT to talk to the android platform as it using a different (inprocess) bus, and this bus is not exposed in the SDK.

You cannot deactivate the NLU without deactivating the ASR either.

NLU is usually no the part of the pipeline that takes time so you can imagine just leaving it on an retrieving the ASR generated text using the onTextCapturedListener on the SnipsPlatformClient. Note that you’ll however need to manage the sessions using a continue/endSession when you receive an intent. The easier may be to always end the session with no text and start a new one with the question of your dialogue manager

Oh, thanks for your help! I will look into this.

Cheers, Snipper

Hi @fredszaq again,

I looked into this, but I cannot find the referenced method onTextCapturedListener !? There is only a comparable one called onSnipsWatchListener but according to the documentation this should only be used for DEBUGGING!?

I just cloned the quickstart demo from https://github.com/snipsco-samples/quickstart-android – is it an outdated verson of the client library I use by accident then maybe?

Thanks for your help