(This post was written a week ago as I am flying back to Taiwan when this is published. )
As many of you know, I was a machine learning engineer back in 2000s at Yahoo! and Facebook, having built the initial versions of machine learned search/ad ranking systems respectively. I have a pretty good understanding of deep learning since its birth around 2012 when Nvidia published its CUDA toolkit and I completed both the Deep Learning and GAN certificates from deeplearning.ai two years ago. Back in 2021, GPT-3 was not super impressive. It was good but not good enough to reliably perform many tasks. But GPT-4’s performance really stunned me. As an early stage investor, I have seen a good number of “AI” deals recently. But it dawned on me that not many of these startups are long term sustainable bets as the landscape is changing rapidly and what we are seeing today will be drastically different 10 years from now. We must look ahead 10 years to inform what to invest today as many of the investment opportunities we are seeing right now are transitory instead of transformational.
So what will happen to AI in 10 years? I couldn’t predict the details but I believe what will absolutely happen is the democratization and commoditization of LLMs. In 10 years, the hardware is going to get a lot more powerful and a lot cheaper. It’s already possible to run GPT-3 inference (175B parameters) on one single powerful server today as shown above. It is reasonable to assume that 10 years from now, we will see at least 100X hardware compute improvement. (Moore’s law still applies for GPU. 2**(10/1.5) = 101.59) With some software optimization, it’s almost certain that we can run GPT-4 inference (170T parameters) on one single powerful server in 10 years. It might still be expensive for consumers but it will be a relatively small investment for businesses considering the productivity gained from a dedicated GPT-4 appliance that is fine tuned and optimized for the businesses’ proprietary data. I could be a bit conservative here. It’s possible that in 10 years, we can run GPT-4 inference on a laptop or a handheld device but I think you get my point.
It is unlikely for a small number of players to dominate the model training unless some idiotic government regulations are implemented. There are already open-source efforts like BLOOM and Eleuther which aim to train LLMs for public uses and such efforts align interests very well with hardware vendors like Nvidia and compute vendors like AWS. It is a matter of time for the open-source efforts to catch up with OpenAI, Google and Anthropic. I imagine these open-source models will be sold like software distributions that are bundled with hardware or cloud providers. In this regard, I worry about efforts which try to be competitive with OpenAI or Google. Instead, it’s probably more worthwhile to try to be RedHat for LLMs. Consumer oriented companies like OpenAI probably will still succeed as they have won the the mindshare of consumers. But other wannabes are probably going to be drowned in the sea of open-sourced LLMs.
I realized I speak in the tone as if AGI will arrive within 10 years. My understanding is that we are not too far away from AGI so we are either very close or already there 10 years from now. There are a few issues that need to be addressed like hallucinations but I personally believe those can be addressed sufficiently by grounding the response on sources. I already noticed that I can get GPT-4 to hallucinate less if I asked it to say “I don’t know” when there’s no sufficient context. I could also ask it to double check and verify the answers to reduce hallucinations. I realize if I treat GPT-4 as a 7-year old and ask it to pay attention and double check and to not lie if it doesn’t know the answer, GPT-4 can hallucinate a lot less. Obviously, I can’t get my 7-year old to do everything I want so it’s not always accurate. Now, the AI engineers’ job is to make GPT-4 a 30-year old genius with a perfect memory.
With the commoditization of LLMs, many industries will be reimagined. I almost think the new AI technology will do more to empower the existing businesses than to start new industries. We will have AI agents or co-pilots in industries across many settings ranging from education, healthcare, agriculture, entertainment, transportation, e-commerce, manufacturing, construction and many others. It is going to be a very very different world where machines can accomplish a lot more with few human interventions. I don’t know what we humans are gonna do in the end and what politics will be like in the new world. I only know things will be very different. There will be positive changes and there will be negative changes. The changes themselves will matter less than how we react to them.