Snips does not recognize hotword unless there is a connection to the RPI as the pi user

I have a working handler running via systemd as the pi user. If I disconnect from the raspberry pi the handler does not recognize hotwords. Just connecting to the PI as pi causes the handler to begin recognizing the hotword and performing my actions.