[Solved] Intents are not recognized correctly anymore


#1

Hi!
I have lots of intent that worked flawlessly 2 weeks ago that does not work anymore (at least in the console).

Example

Utterance: joue de la musique salsa
Correct intent training example: joue de la musique [salsa](genre)
NLU match another intent with training example "joue de la musique DE [rihanna](artist)" with probability 0.08 (really low) and no slots

If I change the utterance to: joue de la musique salsa dans le salon
Correct intent training example: joue de la musique [salsa](genre) [dans le salon](zone)
NLU match the correct intent with probability 0.16 (really low) with both slots correctly extracted.

Can anyone tell me what’s wrong? Did the NLU recent update changed something?

Thanks


Platform Update 1.1.0 (0.61.1) - 25/02/2019
#2

Hi @fastjack, thanks for reporting, I’ll need more info to debug, will contact you in DM


#3

I think I’m having the same problem with another app. I’m using the german app “FHEM”
[https://console.snips.ai/store/de/skill_dAeD3Y37le97]. Until the update I could use the sentence “Licht an” (“lights on”) to switch on the lights. Since the update, the intent isn’t recognized anymore even though “an” is still in the slots


#4

I think NLU update might be the reason to not recognize the intents, irrespective the language on FinancePolice. That slotting issue has been faced with few other users to as I read in the some threads.

I have registered today was about to ask the same question before i saw your thread.


#5

I think I’m having the same problem with another app. I’m using the german app “FHEM”
[https://console.snips.ai/store/de/skill_dAeD3Y37le97 ]. Until the update I could use the sentence “Licht an” (“lights on”) to switch on the lights. Since the update, the intent isn’t recognized anymore even though “an” is still in the slots

Hi @jowe, can you precise the expected intent and slots names ?


#6

when entering “Licht an” the correct intent would be “SetOnOff”. But as you can see, the returned intent is “MediaControls”:
{
“input”: “licht an”,
“intent”: {
“intentName”: “Thyraz:MediaControls”,
“probability”: 0.57475424
},
“slots”: []
}

Then I forked the app and removed the MediaControls intent. As you can see, I don’t even get an intent even though there are some training examples with “an” in the “SetOnOff” intent:

{
“input”: “licht an”,
“intent”: {
“intentName”: null,
“probability”: 0.45701548
},
“slots”: []
}


#7

Do you need more information or are you investigating the problem?


#8

wow, same here, on 55 tests 8 unmatched !!!

will try to tune training but not a nice thing indeed.


Platform Update 1.1.0 (0.61.1) - 25/02/2019
#9

And lot of retrain error, please retrain …


#10

lance la musique -> trigger another intent that not have single sentence with lance
allume la télé -> trigger lightsetjeedom without any télé in sentences or slots !!!

Every single sentence with allume now trigger lightsetjeedom.

éteins la musique -> volumeMute which has no sentence with éteins

How can nlu miss intents that have exact sentence into with words in slots, and take another one without words or slots into ???

nlu totaly broken or what ?? SNIPS totally unusable here :zipper_mouth_face:


Platform Update 1.1.0 (0.61.1) - 25/02/2019
#11

Hi @KiboOst thanks for reporting, we’re investigating your issue as well as @jowe’s one


#12

@jowe @KiboOst thanks for reporting the issues.
We’ve identified a couple of intent classification bugs that have been fixed and will be release pretty soon.


#13

Great, thanks for the update


#14

That said, I’ve investigated with you assistant @jowe and I think that "licht an" is a very ambiguous query. In the past it might have been parsed correctly “by chance”.

Let’s me explain a bit:

  • licht appears nowhere in the training data so when doing prediction this words is ignored
  • so for the NLU "licht an" is simply seen as "an"
  • when the NLU sees an it detects 2 matches with entities seen in the training data: 1 match for the de.fhem.OnOffValue slot and one match for the de.fhem.OnOffStatus slot
  • so when you say licht an the NLU highly hesitates between GetOnOff and SetOnOff, when you retrain the assistant it could choose one or the other

To avoid this unstable setting, I recommend to fork the app and intent do one of the following modifications or both (and of course see if you get the intent right and with a better confidence):

  • add licht as a de.fhem.Device entity value (of course if you have a list with other devices, add them to the entity values, this will improve intent classification)
  • add this simple formulations to the SetOnOff intent with the following pattern: “[de.fhem.Device] [de.fhem.OnOffValue]” like “Standardgerät aus”. This will help the NLU to choose this intent rather than the GetOnOff intent which also contains the de.fhem.Device slot and utterances containing ein, an which can be both considered as de.fhem.OnOffValue or de.fhem.OnOffStatus

With these modifications you’ll help the NLU to choose SetOnOff when you input "licht an"


#15

Thanks for your reply. I will try this as soon as you rolled out the update and I will report the result


#16

Hi there,
We have rolled out the fix addressing the intent classification issues. You need to re-train your assistant on the console in order for the fix to apply.
cc @KiboOst @fastjack

Sorry for the inconvenience that it caused.

Cheers


#17

Woho !!!

It’s even better than before !!

pySnipsBatcher:
matched: 58 | unmatched: 0 | total: 58

Congrats and many thanks for the fix !!
:beers:

Now, fix code snippets to python skills :kissing_closed_eyes:


#18

HI!

The last update (fix) is indeed much better for intents classification. It seems also a bit faster.

Congrats and thanks.


#19

I can confirm that Snips is working again like before. Until now no ploblems with the “licht an” command, works like a charm. Thank you for the fix!


#20

I also confirm that Snips intent recognition is now back to normal/OK :smile:
Thanks a lot for the hard work guys !