Google Assistant randomly activating - eviltoast

For a while now I have had this problem that everytime I used a voice command for Google Assistant that a few minutes after the command got executed I get the sound of Google Assistant activating and waiting for input for like 2 seconds or so (also with the green microphone activated icon in the corner).

Does anyone else experience the same problem or has an idea how I can get rid of this?