I do this as well, but the smart TV bullshit on my Vizio constantly switches itself back on.

I have a button on my remote now that fixes my input by switching to the box in one fell swoop because this happens so much. There is no setting to stop this, as I already have it set to not change the input.