Incorrect intent recognition

Hi!

I have created 2 intents that overlaps on some words :

onMusicPlay:
  - joue moi de la musique

onMusicPlayArtist:
  - joue moi de la musique de <snips/musicArtist>

When I request “joue moi de la musique de David Bowie”, I get the onMusicPlay intent which is not correct.

It’s seems that the NLU engine ignore the presence of the “…de <snips/musicArtist>” at the end of the utterance.

Is it due to the fact that the words “de la musique” are more present in the onMusicPlay intent’s training phrases?

Is the word “de” a stopword that gets ignored?

Thanks for your help :slight_smile:

how big and varied are the two intent training data?
do you have David Bowie in the slot data?? or have slot data extensibility enabled??

Both intent have 98 varied training phrases.

I did not put David Bowie in the utterances.

The onMusicPlay intent have “de la musique” in almost all of them whereas the onPlayArtist have more variations (de la musique de…, des chansons de…, des morceaux de…, etc.).

Seems that the “de la musique” term is more prominant in the onMusicPlay intent which cause this intent to be the detected although it does not match the end of the utterance (de la musique DE <…>). Is the word “de” marked as a stop word somewhere?

Maybe I can test with more variations in the intent onMusicPlay (joue moi des chansons, joue moi des morceaux de musique, etc.) to decrease the “de la musique” weight in the overall intent.

I’ll test it and report back.

Thanks @ozi for your help :wink:

one simple solution… just use ONE intent…
have loads of training samples… some with a slot and others without

then in the code behind that does al the work, just check if there is a slot returned or not… if not slot then it doesn’t include an artist name… if there is a slot then pull the artist name info from it

simple :slight_smile:

Thanks @ozie
It is indeed a possible solution. I will test it later today.
I would still like to know why the ASR is correct but the NLU does not match the correct intent though.
Word weight? Stopwords?
If anyone from Snips can shed some light on the matter it would be brilliant :slight_smile: ?

Hi @fastjack,
@ozie solutions seems reasonable, however your use case should be handle correctly.
It’s a bit hard to understand what’s going on without more details/data, do you allow me to access your assistant’s data so that I can debug ?
Clément

Hey @ClemDoum !

Please take a look at my assistant.

My assistant is proj_XaK3Kv3E4vQ
The onMusicPlay/onMusicPlayArtist intents works flawlessly if I remove the Playback app from the assistant.

I created an alternative assistant with a minimum test case: proj_8A11KlKGdxw
If you remove the custom test/bad slot (from the first app), everything works (“joue moi de la musique de david bowie” -> onMusicPlayArtist).
With this slot I get onMusicPlay almost everytime.

Looks like a strange slot / slot value / slot synonym / injected value / utterance weight issue.

Hope you can identify the cause of this strange behavior (can this be due to the latest release?).
Tell me if you need more info.

Thanks for your help.

Hi @fastjack,
Sorry for the late reply, I’ve finally had time to debug your case.

I don’t know if you’re familiar with the NLU parsing flow, but basically the parsing happens in 2 steps:

  1. we first run a DeterministicIntentParser on your query, this parser is regex based and is supposed to parse all utterances that match a pattern seen in your input dataset (with the current implementation it rather parses “most utterances”)
  2. if this parser fails we run a second parser that leverage machine learning to generalize to unseen queries

In your case we have a double failure:

  1. the queries you type is a patterns known from the dataset: "joue moi de la musique de <snips/musicArtist>", but the deterministic intent parser fails
  2. the probabilistic intent parser also fails

Why does the deterministic parser fail ?

With the current implementation at training time we see the following sentence pattern: "joue moi de la musique de <snips/musicArtist>" that we extract from your labelling in the console, we thus create a regex to match this patterns.
At inference time we get "joue moi de la musique de david bowie", the first thing the parser does it to extract entities in the utterance and replace the entities with placeholders in order to match the patterns.

However in our case "moi" matches an artist called "Moi", "la musique" matches an artist named "La musique populaire" (we allow partial matches) and "david bowie" matches "David Bowie", so we end up with the following pattern: "joue <snips/musicArtist> de <snips/musicArtist> de <snips/musicArtist>" which was not created as train time (at train time we followed your labelling and created "joue moi de la musique de <snips/musicArtist>").

So we fail here, that’s embarrassing but this should not be a problem since we get a second change with the ML-powered parser !

Now, why does the deterministic parser fail ?

Looking at the logs, I saw that the algorithm learnt the following weights in for the classification of the setence:

"fastjack:onMusicPlayArtist" -> (ngram:builtinentityfeaturesnipsmusicartist, 4.61)
"fastjack:onTest" -> (ngram:builtinentityfeaturesnipsmusicartist, -4.50)
"fastjack:onMusicPlay" -> (ngram:builtinentityfeaturesnipsmusicartist, -4.28)
"None" -> (ngram:builtinentityfeaturesnipsmusicartist, -3.63)
"fastjack:onMusicPlay" -> (ngram:musiqu, 3.42)
"fastjack:onMusicPlay" -> (ngram:entityfeaturetestbad, 3.10)
.
.
.

which basically translates in:

I see a <snips/musicArtist> entity then I add 4.61 to the score of "fastjack:onMusicPlayArtist"
I see a <snips/musicArtist> entity then I remove 4.50 to the score of "fastjack:onTest"
I see a <snips/musicArtist> entity then I remove 4.28 to the "fastjack:onMusicPlay"
I see a <snips/musicArtist> entity then I remove 3.62 to the None intent
I see the word "musique" then I add 3.42 to the score of "fastjack:onMusicPlay"
I see a <test/bad> entity then I add 3.10 to the score of "fastjack:onMusicPlay"
.
.
.

The first 5 lines seem very reasonable however the last one is responsible for the classifier wrong decision. Why did the classifier learned such a rule ? If it see a <test/bad> entity then it should point to you onTest intent.
The problem is that I see that the word “musique” is listed in the values of your test/bad entity, thus the classifier learn to give a strong weight towards the onMusicPlay intent. That’s normal the word “musique” appears a lot in the onMusicPlay intent and it’s tagged a test/bad entity.

What are your options ?

1. Wait for the next release

I the next release we’ll roll out a new implementation of the determistic intent parser that won’t have the same problems has the current one, I’ve tested it on your data and the "joue moi de la musique de david bowie" gets perfecly parsed (1.0 score for onMusciPlayArtist).

2. Remove "musique" from the test/bad entity

I’m not sure of the role of this entity and intent but tagging this word that is very very common intents of the same assistant can a LOT of negative side effects

I Hope this helps !
Clément

1 Like