Cemu

I am using the XAudio2 audio API and initially I don't get any audio lag when I start a game up for the first time. But then after about 10 - 20 minutes of game time, there is a noticeable audio lag of about 200ms that gets progressively worse the longer I play the game. I have tested with Breath of the Wild and Smash WII U and it is particularly noticeable on Smash WII U. It is even worse with the DirectSound API, there is a 500 ms audio lag right from the start.

Using latest version of Windows 10
AMD Ryzen 2700x, Nvidia 1080 TI, 32 gb RAM
Tried with multicore and single core settings, latency set at 24 ms, Vulkan API, VSync off