The speedy tempo of innovation and the proliferation of latest fashions have raised issues about expertise lock-in. Lock-in happens when companies change into overly reliant on a selected mannequin with bespoke scaffolding that limits their potential to adapt to improvements. Upon its launch, GPT-4 was the identical value as GPT-3 regardless of being a superior mannequin with a lot increased efficiency. Because the GPT-4 launch in March 2023, OpenAI costs have fallen one other six occasions for enter knowledge and 4 occasions for output knowledge with GPT-4o, launched Could 13, 2024. After all, an evaluation of this kind assumes that technology is bought at value or a hard and fast revenue, which might be not true, and vital capital injections and adverse margins for capturing market share have doubtless backed a few of this. Nonetheless, we doubt these levers clarify all the development features and value reductions. Even Gemini 1.5 Flash, launched Could 24, 2024, provides efficiency close to GPT-4, costing about 85 occasions much less for enter knowledge and 57 occasions much less for output knowledge than the unique GPT-4. Though eliminating expertise lock-in is probably not attainable, companies can scale back their grip on expertise adoption through the use of industrial fashions within the quick run.
Avoiding lock-in dangers
In some respects, the bitter lesson is a part of this extra appreciable dialogue about lock-in dangers. We anticipate scaling to proceed, a minimum of for an additional couple of interactions. Except you could have a specific use case with apparent industrial potential, or function inside a high-risk and extremely regulated business, adopting the expertise earlier than the complete scaling potential is set and exhausted could also be hasty.
In the end, coaching a language mannequin or adopting an open-source mannequin is like swapping a leash for a ball and chain. Both method, you’re not strolling away with out leaving some pores and skin within the recreation. You might want to coach or tune a mannequin in a slender area with specialised language and tail information. Nonetheless, coaching language fashions entails substantial time, computational sources, and monetary funding. This will increase the danger for any technique. Coaching a language mannequin can value tons of of hundreds to hundreds of thousands of {dollars}, relying on the mannequin’s dimension and the quantity of coaching knowledge. The financial burden is exacerbated by the nonlinear scaling legal guidelines of mannequin coaching, wherein features in efficiency could require exponentially larger compute sources—highlighting the uncertainty and danger concerned in such endeavors. Bloomberg’s technique of together with a margin of error of 30 p.c of their computing price range underscores the unpredictable nature of coaching.
