Hello,
The callstacks you quoted exhibit signs of being compiled in Debug (i.e. no optimization). I'd recommend evaluating performance with the Profile or Release configuration. Well, Profile is best during development given you can connect Wwise with it.
Also, these performance numbers need more context: how many sounds are active? Check the Wwise Profiler for the "Physical Voices" counter. A reasonable number on Switch would be around 150. Use Virtual Voices and Limits to manage that. If you are running 1000 voices, sure, decoding cost will be major.
You can obviously use another Codec for very short files (ADPCM or PCM) if you'd like.