The American spelling “matte” probably comes from the spelling “mate” derived from French “mate”, and doubling the “t” to differentiate it from “mate”. The British spelling “matt” was probably primarily influenced by the German word “Matt” considering the UK tended to have more German influence.

Alternatively, either (or both) may be an etymological spelling from Latin “mattus” (which means “drunk” but likely became a word for “pale” in French).

While I am a linguist, I only deduced this from a bit of Googling and a lot of speculating, so don’t take my word for it…