Correct, it is a pejorative used in two ways, either people racist against asians claiming anyone who isn’t asian dating an asian, has yellow fever, or that person in fetishizing them. Just being attracted to a person is not the same thing.

this applies to other ethnic groups, hair colors, and many other things.