3/14/2024: I am GPU Middle-Class
Happy Pi Day, everyone. I hope you have some good pies today~
Lately, I have been tinkering with running open-source GenAI models. I am a big believer of open-source GenAI models. IMO, people running their own models with their own preferences/biases is the way to go to avoid individual players hoarding too much power. But it turns out this thing is not so easy to do. Most of the online tutorials start with Google Colab. I actually have a Google Colab Pro subscription. Doing GenAI inference with Colab is fine but fine tuning a model is basically impossible and I used up my monthly credit in like a day with like 10 inferences. I also looked up other providers like Lambda and Paperspace. The GPU availability in both places is shockingly low. I basically can’t get a machine on-demand and have to get some kind of approval to get a GPU machine for like $3+ per hour.
Fortunately, I have a GPU server at home with two RTX4090 cards, which cost $1800 each. Each card only has 25GB of GPU memory though. I consider myself GPU middle class as I am neither GPU poor like developers who have no GPU and have to deal with cloud services, nor GPU rich like developers who have H100s. With my GPU server, I can get a lot of work done but I have to manage my resources very carefully with various tricks. Honestly, being GPU middle-class is a bit painful as I can’t do everything I want for my models and for things I can do, it’s usually quite slow. But I am not going to pay $30K just for a H100 card. Hopefully, in the next few years the world will have GPU abundance so we can all do what we want to do and the GPU class divide will go away for a more equitable world.