Arms races are endemic to much of modern software. AI applications are (mostly) not.
If competition is the great accelerant of technical innovation, then arms races are the cauldrons of infinite advancement. Arms races force continuous evolutionary progress, driving all participants to seize the disruptive initiative with acute alacrity. While arms races can be evenly matched for a time, one side is often dominant and must continually defend its territory and leadership against a constant fusillade of asymmetric attacks by others.
Development of artificial intelligence appears to be an obvious arms race. Hackers use AI to crack computer systems, and so we need better AI cybersecurity to fight back. Computational propagandists will generate endless synthetic deepfakes, forcing us to build resilient AI systems to identify this junk and flush it out of our social systems.
I argue though that AI applications by and large aren’t arms races, which has some extremely important implications for the development of the field in the years ahead.
Broadly, most software is used by “friendly” users, in the sense that the user is ultimately on the same side as the software. Word processors, photo editors, integrated developer environments, and even games: all of our apps are designed to help us, and ultimately, we want to work with them to finish our tasks or just play.
Almost all software developed prior to the internet was in this vein, but as software transitioned from single-player experiences installed on our own desktops to the wider web, it increasingly encountered the arms race dynamic. For instance, all networked software must now account for cybersecurity, constantly mutating and improving as hackers and state actors find new holes and flaws in code.
Arms races don’t just show up with cybersecurity though, but also in core functionality. Social networks like Facebook, Twitter, and YouTube must continuously adapt their algorithms to fight new forms of spam, traffic gaming and illegal content distribution. Wikipedia needs editorial and moderation systems that fight off the sulfurous vultures at PR agencies. It’s even harder at Google, where there are massive incentives for ranking higher on search results through search-engine optimization (SEO). One estimate places the annual SEO market at about $80 billion, all devoted to gaming rankings and improving a site’s performance on Google.
This was a jarring lesson for me. Back at Stanford, I took the graduate course on information retrieval, and it’s incredible how quickly you can build a modern search engine using off-the-shelf, open-source tools. Using Apache Lucene, I could design a search engine during a single academic quarter to peer into a corpus of millions of documents in multiple languages with pretty good results just on my personal computer.
A few weeks later, I joined the Google+ search team working with Frances Haugen (who coincidentally a decade later, would become the ‘Facebook Whistleblower’ and whose new book is arriving soon). Suddenly, I went from ranking millions of documents on my personal computer to working with a massive search team trying to highlight social content from hundreds of thousands of users, many of whom were actively trying to sabotage the rankings and boost their own profile.
I had, inadvertently, entered into a software arms race. While indexing and searching my own documents, I had no reason to fight my own software — it was a single-player experience, and I wanted my search engine to succeed because it was fundamentally serving me. There was also a point where my custom Lucene instance was “good enough” – it found what I wanted and it didn’t really need any further improvements. But online, Google+ search needed to outwit the smartest optimizers on the planet, and had to do that meticulously in real time.
Unsurprisingly, companies fighting an arms race have to invest aggressively in maintenance and innovation to stay competitive. Google’s search quality goes up and down over time as the company gets ahead of and falls behind the actors trying to thwart it. That focus keeps the company’s systems and management limber — get lazy, and a few weeks later the search results will quite literally be junk. But it also means that Google search takes on an outsized level of executive attention, and not just because it’s the major profit center of the company. It’s always possible to slip up and lose whatever lead you might have had against spammers.
Fast forward to the current battle of the AI chatbots like ChatGPT from OpenAI (plus by extension, Sydney from Microsoft) and Bard from Google. Are these AI chatbots and other AI applications in an arms race? And if so, where and with whom?
Given that they’re almost exclusively single-player interfaces right now, the chatbot experience harkens back to the software of yore. Users ask questions, and they get answers — no one else is involved. While many tech writers have pushed chatbots to do illegal activities or offer up confidential information, that doesn’t suddenly foist an arms race dynamic on AI. You can just as easily use Microsoft Word and Adobe Photoshop to commit illegal acts given their capabilities. Avoiding negative publicity is not the same as an arms race.
There is a potential arms race among these applications in trying to sway the corpus for the models that underpin these bots, in much the way an SEO firm might try to spin a web site for Google’s search engine. But with OpenAI and others sucking up in the entire written output of humanity in their quest for complete dominance, it seems hard to imagine that some edits on a site (or even extensive edits across thousands of sites) would radically change a bot’s output. This is even more true as the black box of these chatbots has become ever closer in shade to Vantablack. Without knowledge of how these models are constructed, it’s hard to glean precisely what it would take to influence them.
Here’s where it gets analytically challenging: there’s certainly competition between chatbots and other AI applications. OpenAI, Google and others want their chatbots to be better and more useful to users than their competitors’ products. It’s easy to look at this market structure and argue that AI right now is indeed a cauldron of infinite advancement.
Instead, we should look at the marginal utility for users of those infinite advancements. Will every additional improvement to an AI application ultimately lead to better user outcomes? No, since there’s often a plateau of capability where an AI app will surpass a threshold of competence — they’ll be “good enough” for users, and any further gains will be mostly superfluous. This isn’t about reaching that pinnacle of intelligence, AGI, but rather the reality that AI apps at some point soon will just do what we expect them to do.
That has huge implications for the long-term value that can be created by companies around AI and whether models will tend toward open or closed approaches. It’s very hard to compete in an arms race with open-source software, since the constant workload and investment required to sustain a competitive edge isn’t conducive to the decentralized development model of open software. But if the goal is to reach a threshold of competence, then open-source AI models absolutely have a chance to dominate the market in the years ahead.
The most obvious analogy is to Wikipedia, which is an open-sourced encyclopedia that also runs on open software. It’s eminently possible (and I dare say likely) that model building and tuning will happen in a fully open and democratized way like Wikipedia in the years ahead. This becomes even more possible in a multi-model world with domain specificity, where decentralized networks of experts could optimize the model of their own field.
There will still be categories where AI exhibits an arms race. Deepfake detection and other authenticity verification tools will be battling the purveyors of these media. Big companies could theoretically be built in such sectors, since their agility and steady investment will offer them a moat that allows them to capture value.
The bulk of AI use cases though don’t have that arms race dynamic, where “good enough” technology will be all that most users need. In these markets, it’s going to be much harder for a for-profit company to monopolize a market, since open-source solutions will likely be cheaper, more flexible to integrate via APIs, and more extensible than proprietary options. Companies may be the first to cut through the thicket of challenges to deliver apps to the marketplace, but as learnings seep out, it becomes easier for those behind them — including open applications — to catch up.
Investing in AI requires very precise attention on the competition dynamics with users and products as well as between products themselves. Where marginal outperformance leads to cumulative market advantages, there are outsized profits to be seized. But where there are diminishing marginal utilities, expect a plateau of capabilities and more openness of technology. The cauldron of infinite advancement requires a very specific alchemy, and it’s not one definitionally shared by all software. In fact, much like the alchemy of the medieval era, it’s actually relatively elusive and rare.
“Securities” Podcast: “It subverts the structure even of other stories that are told about creation”
Danny Crichton, Eliot Peper and Sam Arbesman on the "Securities" podcast
We’ve added it multiple times in the Lux Recommends section, but Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin has been a popular novel around the Lux offices (I glimpsed Josh Wolfe reading a copy recently). It’s also been a smash hit, becoming Amazon’s book of the year for 2022 while also securing a major film adaptation. At its core, the novel is a bildungsroman of two video game creators who tussle, make up, and learn from each other over two decades of designing and playing their worlds.
Zevin’s description of the creative process inspired us to do a podcast on some of the themes of the novel and how they relate to our own creative lives. Joining me in this week’s episode of the “Securities” podcast was novelist Eliot Peper (last seen on the podcast in Speculative fiction is a prism to understand people) and our own scientist-in-residence and multi-time book author Sam Arbesman.
We talk about the building of virtual worlds, the hero’s journey of creation, the uniqueness versus repetitiveness of producing art, whether video games are entering the literary zeitgeist, why the book garnered such popular success and finally, narratives of individuals versus groups.
Ina Deljkic is excited about the prospects of extraterrestrial intelligence from the news last week that a basic building block of life was discovered in outer space. “Scientists have discovered the chemical compound uracil, one of the building blocks of RNA, in just 10 milligrams of material from the asteroid Ryugu… The finding lends weight to a longstanding theory that life on Earth may have been seeded from outer space when asteroids crashed into our planet carrying fundamental elements.”
Bilal Zuberi is interested (as I was in today's column) in the future evolution of AI apps, pointing to a post by Paul Kedrosky and Eric Norlin of SK Ventures talking about “AI, Mass Evolution, and Weickian Loops.” "For example, and we can push this analogy too far, we know that in biology over-fast evolution leads to instabilities; we know that slow-evolving species tend to do better than fast-evolving ones, in part because the latter respond too readily to transient stimulus, rather than exploiting their ecological niche.”
I recommend an interview with … myself. After years of plugging Five Books interviews, I finally got to sit in the hot chair on a topic I know and love: industrial policy. Here are the 5 best books on industrial policy in a lengthy overview of the field.
Sam Arbesman recommends Peng Shepherd’s novel, The Cartographers, which has been wildly successful and is described as “an ode to art and science, history and magic.”
That’s it, folks. Have questions, comments, or ideas? This newsletter is sent from my email, so you can just click reply.
Forcing China’s AI researchers to strive for chip efficiency will ultimately shave America’s lead
In incididunt ad qui nostrud sint ullamco. Irure sint deserunt Lorem id officia dolore non. Anim dolor minim sit dolor et sint aliquip qui est. Ex in tempor laborum laboris dolor laboris ullamco quis. Enim est cupidatat consequat est culpa consequat. Fugiat officia in ea ea laborum sunt Lorem. Anim laborum labore duis ipsum mollit nisi do exercitation. Magna in pariatur anim aute.
In incididunt ad qui nostrud sint ullamco. Irure sint deserunt Lorem id officia dolore non. Anim dolor minim sit dolor et sint aliquip qui est. Ex in tempor laborum laboris dolor laboris ullamco quis. Enim est cupidatat consequat est culpa consequat. Fugiat officia in ea ea laborum sunt Lorem. Anim laborum labore duis ipsum mollit nisi do exercitation. Magna in pariatur anim aute.
Right now, pathbreaking AI foundation models follow an inverse Moore’s law (sometimes quipped “Eroom’s Law”). Each new generation is becoming more and more expensive to train as researchers exponentially increase the number of parameters used and overall model complexity. Sam Altman of OpenAI said that the cost of training GPT-4 was over $100 million, and some AI computational specialists believe that the first $1 billion model is currently or will shortly be developed.
As semiconductor chips rise in complexity, costs come down because transistors are packed more densely on silicon, cutting the cost per transistor during fabrication as well as lowering operational costs for energy and heat dissipation. That miracle of performance is the inverse with AI today. To increase the complexity (and therefore hopefully quality) of an AI model, researchers have attempted to pack in more and more parameters, each one of which demands more computation both for training and for usage. A 1 million parameter model can be trained for a few bucks and run on a $15 Raspberry Pi Zero 2 W, but Google’s PaLM with 540 billion parameters requires full-scale data centers to operate and is estimated to have cost millions of dollars to train.
Admittedly, simply having more parameters isn’t a magic recipe for better AI end performance. One recalls Steve Jobs’s marketing of the so-called “Megahertz Myth” to attempt to persuade the public that headline megahertz numbers weren't the right way to judge the performance of a personal computer. Performance in most fields is a complicated problem to judge, and just adding more inputs doesn't necessarily translate into a better output.
And indeed, there is an efficiency curve underway in AI outside of the leading-edge foundation models from OpenAI and Google. Researchers over the past two years have discovered better training techniques (as well as recipes to bundle these techniques together), developed best practices for spending on reinforcement learning from human feedback (RLHF), and curated better training data to improve model quality even while shaving parameter counts. Far from surpassing $1 billion, training new models that are equally performant might well cost only tens or hundreds of thousands of dollars.
This AI performance envelope between dollars invested and quality of model trained is a huge area of debate for the trajectory of the field (and was the most important theme to emanate from our AI Summit). And it’s absolutely vital to understand, since where the efficiency story ends up will determine the sustained market structure of the AI industry.
If foundation models cost billions of dollars to train, all the value and leverage of AI will accrue and centralize to the big tech companies like Microsoft (through OpenAI), Google and others who have the means and teams to lavish. But if the performance envelope reaches a significantly better dollar-to-quality ratio in the future, that means the whole field opens up to startups and novel experiments, while the leverage of the big tech companies would be much reduced.
The U.S. right now is parallelizing both approaches toward AI. Big tech is hurling billions of dollars on the field, while startups are exploring and developing more efficient models given their relatively meagre resources and limited access to Nvidia’s flagship chip, the H100. Talent — on balance — is heading as it typically does to big tech. Why work on efficiency when a big tech behemoth has money to burn on theoretical ideas emanating from university AI labs?
Without access to the highest-performance chips, China is limited in the work it can do on the cutting-edge frontiers of AI development. Without more chips (and in the future, the next generations of GPUs), it won’t have the competitive compute power to push the AI field to its limits like American companies. That leaves China with the only other path available, which is to follow the parallel course for improving AI through efficiency.
For those looking to prevent the decline of American economic power, this is an alarming development. Model efficiency is what will ultimately allow foundation models to be preloaded onto our devices and open up the consumer market to cheap and rapid AI interactions. Whoever builds an advantage in model efficiency will open up a range of applications that remain impractical or too expensive for the most complex AI models.
Given U.S. export controls, China is now (by assumption, and yes, it’s a big assumption) putting its entire weight behind building the AI models it can, which are focused on efficiency. Which means that its resources are arrayed for building the platforms to capture end-user applications — the exact opposite goal of American policymakers. It’s a classic result: restricting access to technology forces engineers to be more creative in building their products, the exact intensified creativity that typically leads to the next great startup or scientific breakthrough.
If America was serious about slowing the growth of China’s still-nascent semiconductor market, it really should have taken a page from the Chinese industrial policy handbook and just dumped chips on the market, just as China has done for years from solar panel manufacturing to electronics. Cheaper chips, faster chips, chips so competitive that no domestic manufacturer — even under Beijing direction — could have effectively competed. Instead we are attempting to decouple from the second largest chips market in the world, turning a competitive field where America is the clear leader into a bountiful green field of opportunity for domestic national champions to usurp market share and profits.
There were of course other goals outside of economic growth for restricting China’s access to chips. America is deeply concerned about the country’s AI integration into its military, and it wants to slow the evolution of its autonomous weaponry and intelligence gathering. Export controls do that, but they are likely to come at an extremely exorbitant long-term cost: the loss of leadership in the most important technological development so far this decade. It’s not a trade off I would have built trade policy on.
The life and death of air conditioning
Photo by Ashkan Forouzani on Unsplash
Across six years of working at TechCrunch, no article triggered an avalanche of readership or inbox vitriol quite like Air conditioning is one of the greatest inventions of the 20th Century. It’s also killing the 21st. It was an interview with Eric Dean Wilson, the author of After Cooling, about the complex feedback loops between global climate disruption and the increasing need for air conditioning to sustain life on Earth. The article was read by millions and millions of people, and hundreds of people wrote in with hot air about the importance of their cold air.
Demand for air conditioners is surging in markets where both incomes and temperatures are rising, populous places like India, China, Indonesia and the Philippines. By one estimate, the world will add 1 billion ACs before the end of the decade. The market is projected to before 2040. That’s good for measures of public health and economic productivity; it’s unquestionably bad for the climate, and a global agreement to phase out the most harmful coolants could keep the appliances out of reach of many of the people who need them most.
This is a classic feedback loop, where the increasing temperatures of the planet, particularly in South Asia, lead to increased demand for climate resilience tools like air conditioning and climate-adapted housing, leading to further climate change ad infinitum.
Josh Wolfe speaking at Nvidia Auditorium at Stanford. Photo by Chris Gates.
Josh Wolfe gave a talk at Stanford this week as part of the school’s long-running Entrepreneurial Thought Leaders series, talking all things Lux, defense tech and scientific innovation. The .
Lux Recommends
As Henry Kissinger turns 100, Grace Isford recommends “Henry Kissinger explains how to avoid world war three.” “In his view, the fate of humanity depends on whether America and China can get along. He believes the rapid progress of AI, in particular, leaves them only five-to-ten years to find a way.”
Our scientist-in-residence Sam Arbesman recommends Blindsight by Peter Watts, a first contact, hard science fiction novel that made quite a splash when it was published back in 2006.
Mohammed bin Rashid Al Maktoum, and just how far he has been willing to go to keep his daughter tranquilized and imprisoned. “When the yacht was located, off the Goa coast, Sheikh Mohammed spoke with the Indian Prime Minister, Narendra Modi, and agreed to extradite a Dubai-based arms dealer in exchange for his daughter’s capture. The Indian government deployed boats, helicopters, and a team of armed commandos to storm Nostromo and carry Latifa away.”
Sam recommends Ada Palmer’s article for Microsoft’s AI Anthology, “We are an information revolution species.” “If we pour a precious new elixir into a leaky cup and it leaks, we need to fix the cup, not fear the elixir.”
I love complex international security stories, and few areas are as complex or wild as the international trade in exotic animals. Tad Friend, who generally covers Silicon Valley for The New Yorker, has a great story about an NGO focused on infiltrating and exposing the networks that allow the trade to continue in “Earth League International Hunts the Hunters.” "At times, rhino horn has been worth more than gold—so South African rhinos are often killed with Czech-made rifles sold by Portuguese arms dealers to poachers from Mozambique, who send the horns by courier to Qatar or Vietnam, or have them bundled with elephant ivory in Maputo or Mombasa or Lagos or Luanda and delivered to China via Malaysia or Hong Kong.”
