- lowering the volume to minimum wouldn't mute
- muting was possible with the keyboard, unmuting was not
xfconf-query -c xfce4-mixer -l
to get a list of soundcards and then tell xfce4-mixer to use that
xfconf-query -c xfce4-mixer -p /active-card -s PlaybackInternalAudioAnalogStereoPulseAudioMixer
Now the volume controls seem to work proper.
Edit: A killall xfce4-volumed might be needed, as otherwise the setting will be instantly reverted.