Shutdown app (in the app manager via gesture/hardware button) or Force Shutdown in the Settings->Apps->Spotify and "Force Shutdown".
I think only swiping away the Spotify in app (multitask) manager will do the trick. But again...it will maybe be confused when you open Spotify next day.
Maybe disabling notification permission for Spotify will help (perhaps even Spotify has it in it's own settings). But this will disable Spotify controls in notification drawer of course.
Hope it helps.