Trying to set up LLama but i get a error saying CUDA has no path set - eviltoast

Hello y’all, i was using this guide to try and set up llama again on my machine, i was sure that i was following the instructions to the letter but when i get to the part where i need to run setup_cuda.py install i get this error

File "C:\Users\Mike\miniconda3\Lib\site-packages\torch\utils\cpp_extension.py", line 2419, in _join_cuda_home raise OSError('CUDA_HOME environment variable is not set. ' OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root. (base) PS C:\Users\Mike\text-generation-webui\repositories\GPTQ-for-LLaMa>

i’m not a huge coder yet so i tried to use setx to set CUDA_HOME to a few different places but each time doing echo %CUDA_HOME doesn’t come up with the address so i assume it failed, and i still can’t run setup_cuda.py

Anyone have any idea what i’m doing wrong?

  • remotelove@lemmy.caEnglish
    2·
    4 months ago

    I had much better success using WSL, but I haven’t used it or even updated it in a long while. (I have been meaning to see how AMD GPU support has evolved over the last few months. Back in January’ish, AMD support was still bad.)

    Anything that even is remotely Linux related is much easier to get working with WSL, btw. Almost all of my personal python stuff is running under it and it works great with VS Code

    • yokonzo@lemmy.worldOPEnglish
      1·
      4 months ago

      I mean Linux is an option but haven’t people been saying nvida drivers are a huge hassle to use on Linux?

      • Pumpkin Escobar@lemmy.worldEnglish
        2·
        4 months ago

        Nah. There are some nvidia issues with wayland (that are starting to get cleared up), and nvidia’s drivers not being open-source rubs some people the wrong way, but getting nvidia and cuda up and running on linux is pretty easy/reliable in my experience.

        WSL is a bit different but there are steps to get that up and running too.

      • remotelove@lemmy.caEnglish
        2·
        4 months ago

        They can be, I suppose. However, the AI libraries that I was tinkering with seemed to all be based around Ubuntu and Nvidia. With Docker, GPU passthrough is much better under Linux and Nvidia.

        WSL improved things a bit after I got an older GTX 1650. For my AMD GPU, ROCm support is (was?) garbage under Windows using either Docker or WSL. I don’t remember having much difficulty with Nvidia drivers though… I think there might have been some strange dependency problems I was able to work through though.

        AMD GPU passthrough on Windows to Docker containers was a no-go. I remember that fairly clear though.

        My apologies. It has been a few months since I messed with this stuff.