I was mainly referring to language models which have somewhat predictable scaling laws. It doesn’t make sense to continue scaling the parameters when you can scale the data instead.
Diffusion models are a completely different domain which is less established. Most advancements made in that space are related to the architecture and training methodology. In terms of scale they haven’t changed much.
Large models will always be trained in datacenters because the compute will always be exponentially greater and cheaper than what you could get as an individual. Local finetuning already happens but it’s expensive and limited.
I was mainly referring to language models which have somewhat predictable scaling laws. It doesn’t make sense to continue scaling the parameters when you can scale the data instead.
Diffusion models are a completely different domain which is less established. Most advancements made in that space are related to the architecture and training methodology. In terms of scale they haven’t changed much.
Large models will always be trained in datacenters because the compute will always be exponentially greater and cheaper than what you could get as an individual. Local finetuning already happens but it’s expensive and limited.