Listen to This Blog
Technology developments tend to follow a typical pattern of improvement over time, known as an S-curve. Although it is a familiar pattern, it is worth unpacking its five phases and considering how they apply to language technology and forecasts about it.
Phase 1: The Long, Hard Slog
The earliest work on a technology is difficult. Solving even basic problems requires substantial investment of time and effort. Researchers are figuring out basic approaches as they work through a lot of trial and error. As a result, improvements happen relatively slowly and may not appear to justify the effort put into them, but small breakthroughs indicated to developers that they appear to be on the right track. Technology development in this early phase frequently occurs in organizations that can afford to bear the costs in the hope of a future reward – that typically means large corporations and government-funded research labs.
I should note here that many technologies never get past this initial phase. Either they do not work as expected, funding runs out, or they are superseded by other approaches. In other cases, researchers wrongly conclude that they are on a dead end when projects fail to enter the next phase in what they consider a reasonable amount of time.
Phase 2: Bend it Like Beckham
Next, the pattern shifts as the curve bends quickly over a relatively short time and heads for a steeper trajectory. Developers build on initial work and begin solving the first wave of problems. At this point, growth looks like it is exponential, and it is not clear what trajectory development will take. The technology becomes attractive to early funders looking for the next big thing. New entrants, typically small-scale ones, start to experiment with the concepts and try out new approaches. Most will fail, but some will succeed.
Phase 3: The Sky’s the Limit
The third phase is one in which the basic framework for development is clear, technology leaders begin to break away from the pack, and most players are able to improve their offerings rapidly. Growth now appears to be on a linear trajectory and headed for the sky. Investors start to see the field as the next sure thing and flood it with money.
Early adopters buy the technology and implement it in a production application. Often, much of the growth in capability and function derives from optimization of approaches and development in related fields that enables large-scale efficiency improvements. For example, the concept of neural MT had been around for many years, but it only took off after multi-core graphical processing units (GPUs) and generalized machine learning approaches matured enough for it to run at reasonable speed and cost. In this phase, the sector also starts to see consolidation as early implementers seek to cash out and as larger players seek an edge through mergers or with acquisitions of complementary and even competing approaches.
Phase 4: The Rude Awakening
The third phase can extend for an indeterminate amount of time, but eventually future development starts to miss milestones. These will initially be minor deviations, but over time they become more common and the curve starts bending downward. Improvements still come, but they no longer climb quite as high or steeply. Developers may realize that they are bumping into fundamental limitations, but public perception continues to grow and mainstream reporting on advances may continue to promise outsized advances.
Phase 5: The Final Plateau
As technologies enter this phase, they are close to reaching their peak performance where any gains require exponentially increasing effort. The same effort that might have increased performance by 500% in Phase 3 might now deliver just 5%. The challenge is that future development has become asymptotic. In other words, its results approach a theoretical target but never get there. As they near the maximum possible performance, each additional increment of development effort delivers a rapidly dwindling return. At this point a technology has reached peak maturity, unless a fundamental breakthrough elsewhere can overcome some central challenge that has held it back.
It's important to remember in the fourth and fifth phases that even if rapid improvements are no longer coming, the implementations of technologies can continue to mature and make them far more useful than they were.
Understanding S-curves is important when assessing claims about how technologies will evolve. Pundits in the tech and business press tend to make extrapolations at the peak of Phase 3, leading to wild predictions of future capability. This tendency is exacerbated because it is impossible to know in advance how far Phase 3 will extend. Even cautious developers may make similar claims based on their expectations and hopes or because they hope to attract investors.
Why S-Curves Are Important to the Language Industry
S-curves are nearly universal in technology, and the language industry is no exception. They account for much of the boom or bust cycle, particularly in machine translation. After some birthing pains, each generation of the technology starts off strong and garners fantastic predictions for how soon it will be before it crosses the threshold of human performance. Sometime before it exceeds what humans can do – at least in terms of quality – the advances start slowing down as the tech bumps into basic limitations. In other areas – such as speed and unit cost – it has already surpassed human translation.
Fortunately, this figure shows how machine translation has skirted some of the limitations imposed by S-curves. MT is not a single innovation – instead, it’s a series of technologies that together enable a longer-term trajectory of improvement.
What’s crucial is to understand that although each individual approach to MT eventually runs out of steam and fades away, each new one builds on the previous ones to ensure long-term improvement. As for the neural MT wave, we cannot yet say how far it will go before it becomes asymptotic, so the graphic shows some potential directions because it is still in the linear-type growth of Phase 3. So far predictions that it would exceed human quality have been exaggerated but the advances of responsive machine translation may create yet another S-curve that will bring MT closer to its maximum potential. Already efforts from companies such as Lilt, Microsoft, Translated (Modern MT), and Unbabel (among others) are exploring different approaches to reach that goal and are achieving promising results.
Although machine translation has been the poster child for advances in AI, language technology developers have started to apply machine learning to tasks such as vendor selection, translation memory repair, quality estimation, multilingual sentiment analysis, and terminology extraction. Many of these are in their early phases, but they will follow their own S-curves as they mature and deliver benefits.
The language industry has good reason for optimism about advanced natural language technologies – beyond MT – that are driven by machine learning. They have just started to make inroads in the market, but the trajectory taken by machine translation suggests that they have a lot of room to expand and become more useful. You can expect the traditional tech hype cycle to mirror the S-curve for each of them, so being aware of this trend will help you make a realistic assessment of current and future claims.