Trends in AI

Larger models, more data and faster computing have been the driving forces in AI development since the GPT-era began. But now this simple three pillar cycle has reached the inevitable phase of diminishing returns. Have we already reached a plateau where, despite all the hype, AGI (Artificial General Intelligence) will be postponed to the unseen future?
Although Nvidia and its rivals push the limits of computing power further with each new hardware generation, the demand grows even faster. More and more people use AI in their everyday life, and companies start realizing that they will not survive without large scale AI adoption. Satisfying the computing energy needs already raises the question of private nuclear power plants for AI computing centers.
All easily available data has already been gathered and used for AI models’ training. Even the largest of base models don’t seem to be getting that much wiser anymore just by throwing more raw data at them. There might still be untapped data sources, but it is questionable whether they would help generate truly better models.
Resent developments, however, have shown that these issues can and will be circumvented. Totally new kinds of hardware technologies, model algorithms, training methods and system thinking together with data selection and generation techniques assure that the growing needs for AI assistance and quality will be met.
The hardware issue, both computing power and energy hungriness, while not immediately commercially available, will be mitigated by emerging technologies like computing with light and analog computing. Light chips, where computation is done with photons, instead of electrons, are at laboratory state but carry a huge promise for much faster computing and way less power consumption than what is possible with current silicon-based transistors.
Analog AI-processors are closer to realization. They cannot be used for model training with current algorithms because they are not precise enough. But with ready computed models the inference can be successfully carried out with much less precision. Analog chips offer less power consumption, faster execution of inferences and orders of magnitude cheaper hardware due to affordable circuit printing techniques.
The trend in model sizes is towards smaller ones which are much cheaper and faster to both train and then utilize. The two main techniquest being used are Mixture of Experts (MoE) -models, and training the models with higher quality data. In MoE, instead of one huge all purpose monolithic model, many smaller models are deployed, with each of them specializing in some narrower domain. Higher quality data is achieved by synthetic generation with the help of the large models. Surprisingly thus trained smaller models become smarter in that narrow domain than their much larger teachers.
The biggest wakness in a GPT-algorithm is its quadratic computation need for its context window. Doubling the length of the content (number of tokens) to be processed at one call e.g. to ChatGPT demands four times more compute and thus also energy. There are, however, new promising architectures proposed, whose computation needs only rise linearly according to the input length. Another technique to use less energy and to enable more intelligent models, is to use concepts as tokes instead of words or parts of words. Larger concepts reduce the number of tokens for the same text, while allowing more abstract thinking at the same time.
Finally, the problem with diminishing IQ-returns from bigger and bigger models has also been addressed by allowing them more thinking time. Instead of giving a relatively short answer to a reasoning task, the wisest models are now taught to write out very long sequencies of thoughts with different techniques and to look at the problem from many perspectives, until they make a synthesis of it all and give out the final answer. Such models, like OpenAI o1/o3 and the remarkable open-source DeepSeek-R1, are beginning to push machine intelligence way past average human intelligence in many fields.
After a seemingly slower period in AI development in the final quarter of 2024, the pace of innovation races forward at breath taking speed again.