It's a bug in Dunst which causes colours not to be set when you change the urgency. Workaround by setting colours manually