🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕕𝕚𝕝𝕖@lemm.ee to LocalLLaMA@sh.itjust.worksEnglish · 26 天前How much gpu do i need to run a 90b modelmessage-squaremessage-square15fedilinkarrow-up113arrow-down11file-text
arrow-up112arrow-down1message-squareHow much gpu do i need to run a 90b model🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕕𝕚𝕝𝕖@lemm.ee to LocalLLaMA@sh.itjust.worksEnglish · 26 天前message-square15fedilinkfile-text
minus-square🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕕𝕚𝕝𝕖@lemm.eeOPlinkfedilinkEnglisharrow-up1·26 天前That looks like exactly the sort of thing i want. Any existing solution to get it to behave like an ollama instance (i have a bunch of services pointed at an ollama run on docker).
minus-squareSylovik@lemmy.worldlinkfedilinkEnglisharrow-up2·26 天前You may try Harbor. The description claims to provide an OpenAI-compatible API.
That looks like exactly the sort of thing i want. Any existing solution to get it to behave like an ollama instance (i have a bunch of services pointed at an ollama run on docker).
You may try Harbor. The description claims to provide an OpenAI-compatible API.