Yeast is an opportunistic pathogen that will secrete toxins. The reason that you don’t get sick when eating bread and stuff is because the yeast dies when you cook it, preventing infection. Although, I’m pretty sure you’ll be fine if you ate live yeast, because your stomach would kill them, also preventing an infection.