The future of science in an age of spending cuts

Samuel Arbesman:
Welcome to The Orthogonal Bet, where we explore the weird and wonderful ideas that shape our world. It's a cabinet of curiosities for the ears and the mind. I'm your host, Samuel Arbesman, complexity scientist, author, and scientist-in-residence at Lux Capital. In this episode, I had the pleasure to speak with Kenneth Stanley. Ken is a renowned computer scientist and an AI researcher who has been a professor, company co-founder, researcher at OpenAI and Uber, and he's currently the Senior Vice President of Open-endedness at Lila Sciences. In addition, Ken is the co-author of the fantastic book, Why Greatness Cannot Be Planned: The Myth of the Objective.
Ken's work touches on evolution, creativity, open-endedness, and so much more. Given Ken's experience working in many different types of institutions, I wanted to speak with Ken about research across organizational structures. We explored these different organizational settings, their trade-offs, new types of research organizations, and even how to create a place within a larger organization for weirder and more open-ended research. We even discussed how research itself is a means of handling or managing disruption, and the current research world, in large, AI companies. And of course, we explored AI research into open-endedness. This was a really fun conversation, and I had a great time speaking with Ken. Let's jump in. Ken, great to be chatting with you, and welcome to The Orthogonal Bet.

Kenneth Stanley:
Thanks for having me. Yeah, it's great to be here.

Samuel Arbesman:
I want to talk about a lot of different things. But I think perhaps the place to start might be in terms of thinking about research in different kinds of institutions. I feel like you've had a number of different roles in different kinds of organizations like academia, corporate industry labs, startups. And presumably, these different types of institutions are better or worse at different kinds of research. And so I'd love to just get maybe a little bit of your own experience in these, and you can use that as a way of discussing a little bit of your background, but also, how to think about research in different institutional and organizational forms.

Kenneth Stanley:
Yeah, that's a good question. I have been in a lot of different places. I've been in academia, I've been in multiple big corporations, startups, I've been in all kinds of things. So I think that what I've learned is that there's just trade-offs is really the fundamental answer to the question. It's like there's nothing really perfect. And so you're moving around on this trade-off front to figure out which things actually matter the most to you. But so I could say a few things about what are good and bad about the different options that you have.
In academia, the really good thing is generally freedom. The freedom is really because you don't really have a normal kind of a manager. You might have a department head, but they're not really telling you what you should be researching. And you also don't have company objectives. So you're not actually trying to think about aligning with the objectives of anyone else. It's true though that you are trying to beg for money because that's what grants are about. But at least you can beg for money for the things that you actually want to do in theory. And to some extent, you can do some things even if you don't have support for them. You have some time to experiment, and it's easier to do that in that kind of a setting.
So while it's not perfect, it has that advantage, but then it has disadvantages. You generally have fewer resources than you have in industry, like compute resources in particular in computer science or AI. And you also have a lot of distractions like constantly begging for money, the grant money. That's a huge distraction. That system needs to be optimized a lot because, I mean, you think about how much time we're wasting. These are some of the most talented people in a field, and they're spending a huge chunk of their time just asking for money. Probably could work a little bit better and more efficiently so they could use their mind to actually come up with interesting research.
And you've got these other things that... Distraction might be too strong of a word because these are good things, but things like teaching, service activities, and sort of the accounting management of running a group, which is almost like an entrepreneurial activity, causes there to be a huge, huge portfolio of responsibilities. I think for a researcher in academia, if the main thing that we're talking about is just how well the research is going to go, that stuff is going to eat into it, of course.
One other good positive of there is the PhD students, I think, are a big positive. The thing that's interesting about PhD students compared to, say, employees in industry is that they tend to be more committed because they want to get their PhD. And so there's a sunk cost issue. If they're in there for two years, they don't want to just leave. Whereas in industry, you can just be opportunistic. If something better comes along, you can just leave, and there's not really that much you're losing.

Samuel Arbesman:
Related to the PhD students, and certainly, there's some positives. But I wonder, are there downsides in terms of having to train, and obviously, training is a good thing for PhD students. We want to create expertise. But does that create a drain on effort in terms of the principal investigator for a lab?

Kenneth Stanley:
Yeah, maybe. I mean, there is the difference when someone's not fully trained, which is what you expect with a PhD student, although generally, they come in with a range of capabilities. I mean, it's not like-

Samuel Arbesman:
Sure.

Kenneth Stanley:
... they just came out of high school. Some of them have masters, some of them don't. Some of them have industry experience, some of them don't. But it's true that there is a lot of training involved. There's no doubt there's a lot of training involved. But then that's another trade off, a lot of training involved. But also, in some ways, you have a blank slate, like somebody who doesn't yet have a dogmatic view or is not entrenched in a certain style of thinking. And so they're very mentally flexible about starting into a new area or taking risks and things like that and tend to be committed to doing that. In industry, if somebody takes a big risk, then if it starts looking like it's not going well, they might leave. But here, they're generally committed for a long time because they want to get the PhD, and it gives you a longer time horizon to think.
So I think PhD students do fantastic research. It just might take longer for it to come out because there is a training ramping up phase, some longer than others. But after that, they can pay off enormously because they're completely invested in the field that they're getting their PhD in. So just more trade-offs. There's some good things there, there's some bad things there.
And then industry's a little different that you can't really say some blanket statement about industry because every company is different. But some generalizations maybe would be that more resources is a big part. Generally speaking, these days in AI, at least, I mean, there tends to be more GPUs in industry than in academia. And so you can do bigger things, you can think bigger.
Another thing is less distraction because you're not constantly trying to ask for money, figure out how you're going to get your next grant. That's an enormous burden taken off. So you can spend more time on research in industry, which might be ironic in some ways. You hope academia people are spending a lot of time on research. You can also hire people with a lot of experience, of course. You're paying much better salaries. And so in theory, you can get some of the best people in the world in your team who already have experience. But again, there are trade-offs there. So it's like a more experience also means more entrenched in current thinking and having their own strong opinions already that are not going to change. And so PhD students don't have that aspect to them. And sometimes experience can be overrated, actually.
And then there's the issue of, the downside in industry is again, the other side of [inaudible 00:07:13] is the freedom issue. And that varies a lot though company to company, and even within a company from time to time. So it's not like it's always the same. But companies, I find, tend to go through phases where during some phases, they can support more research freedom. That tends to be good times or maybe early days. And they say, "Well, this is a basic research unit. This is what we want to do." And they can afford to let you do that. But during tighter times, maybe more mature stages of companies, they tend to start wanting to give their investors responses like, "Well, how is this going to make us money?" That will trickle down into research units. Research units get increasing pressure to justify their existence objectively. I'm not the biggest supporter of objectives in research, having written a whole book about this. So it can, I think, corrupt that basic research exploratory serendipitous function that, I think, that's why these units exist in industry.
In some way, I see in industry, why do you have research labs and innovation labs? I think one reason is to prevent disruption. The problem is if you just continue on with your very successful business, some big company with basically a money printing type of business, someday somebody might just pop up who does something really radically different and then completely undermine the business. And that happens all the time. I mean, there's lots of stories like this in industry. And so a research lab is a defense against that because they're able to think way outside the box or an innovation lab. But you undermine that mission if you then start to impose company OKRs and objectives on top of that, and it trickles into the lab.
And the thing about people in industries, they know what the real incentives are. And so there tends to be lip service and then there tends to be reality. And so the company will say something like, "Here in this particular part of the company, we're really about basic research. We want to support basic research. And don't get distracted by other things." But then at the end of the day, people see that, "Well, we need to justify our existence. We're evaluated at the end of every year. There's bonuses that kick in. If you do something that everybody thinks is really great for the company, you're going to get a bonus, going to get a promotion." And people respond to the implicit incentives, not the ones that the company is saying in words. And so it tends to reduce the ability to explore and the ability to think about blue sky stuff, which is the whole purpose of these units.
And so that's not a problem in academia as much. There's still some of it because you're asking for money from grants. And actually, the whole system of funding in academia is also messed up in some ways. It's also very objectively driven and based on deliverables and undermines the ability to explore for exploration's sake and so forth. But nevertheless, I think it's still not as problematic as what happens in industry when company incentives come from the top down and sort of rain down on top of fundamental researchers.
However, I still would say, these things tend to, like I said, change over time. So you go through oscillations, where there are periods of really nice basic research support and then periods of less. And it happens in many companies, and sometimes like the abolition of great labs like Bell Labs or something and after amazing periods of thriving. And so it tends to be inconsistent. It's not like it's always one way or another. It varies from one company to the next as well. And so it's hard to really completely generalize.
In industry, there are sweet spots you can find that are absolutely incredible next to paradise. Some people think and who are researchers, "What's the paradise for research?" If you just care about only research and that's all you care about, you just want to do your own thing, what should you do? Where should you be? I mean, it seems like the best thing, if you just think about it briefly, is like, well, we should find a billionaire who just gives you millions of dollars and you just do whatever you think is cool. But even that is not optimal, because there's trade-offs there, because how are you going to attract people to an institution that doesn't exist? Are you going to get the best people to work for you? Where's all the support infrastructure, the compute infrastructure?
And so it's not necessarily just easy just off the bat just because you have a huge amount of money just to be a great research enterprise. So there's no really quick solution to creating research paradise. And I think you just have to look at the trade-offs and where you are in life and what aligns the best with your interests.

Samuel Arbesman:
And this makes a lot of sense. And actually related to, I mean, you're saying finding the ideal thing, at least from a surface level pass is like, oh, you find the billionaire who can provide all the money that you need. But I've also seen certain instances where not only is there certain downsides in terms of attracting people to something that doesn't yet exist. There's also the idea of when you have a patron to a certain degree, you are dependent on the patron's whims and fortunes and interests. And those things can all change. And so being able to have a patron can be good while it lasts. And there's always that tendency. It's similar to what you're saying, these oscillations, even with the billionaire patron, there can be a lot of these downsides. And so related though to the oscillations, I'm wondering if there's ways of thinking about kinds of rules like finding when these kinds of places might be more amenable to certain types of research.
For example, you were talking about times when organizations in industry, when they allow for certain amounts of exploration, and I kind of think about this in terms of explore versus exploit, and I feel like startups on the one hand, sometimes when they're very early, there's a little bit more exploration. But there's also a very clear incentive towards exploiting as quickly as possible, because you're a small startup, while when you are more mature, on the one hand may move from exploration to exploitation. But you also presumably have a greater flexibility to create these kind of innovation labs where you can say, okay, we're going to exploit and everyone's going to be specialized, but then we're going to have one little pocket where people can do weird things that is not really understandable, but maybe it will be valuable to help avoid disruption.
So I can see lots of different arguments about why you can argue based on the size of an institution or organization or a company, why smaller or larger might be better in certain kinds of things. And does it really just come down to there is no kind of single rubric, it's much more we can't abstract away the details. We actually have to take all the details to try to figure out, okay, is this the right environment, or it's kind of thing where, yeah, it's the right environment right now and then in five years it might not be the right environment.

Kenneth Stanley:
Yeah, I think it is that there isn't really one blanket answer for industry what should be done or what can be done. And it does depend a lot on context and the details, like you say. I think the details matter a lot. But there's also a difference between protecting against disrupting and actually being the disruptor. The startup tends to be more of in the disruptor role. The startup may as a disruptor, be more willing to go out on a limb and do something you think is more kind of blue sky. But obviously it varies though, because you have these pressures, the short-term pressures at the startup, they don't yet have the protection of a business that's actually working really well. And so all those details have to come into play in the thinking of the leadership.
But I mean there's also another layer on top of that, which is just outside of just the current business context, there's also just thinking about how innovation should be run as an enterprise, just what is the right way to organize these things as a principled kind of analysis of what makes a successful innovation lab or research lab, which I think could enter into this as well. But it's something that I think tends to not be done that much at most companies. Like most companies, it's just somebody's gut instinct ultimately is what determined what's going on right now. And it often goes to the top, the very top, which is like the CEO, something like that. Because even though the CEO EO may delegate, eventually the CEO EO talks to the person they delegate to and says, "Well, where's the payoff here? What's going on?" So it ultimately ends up at the CEO what's going to happen. And CEOs don't have an enormous amount of time to think philosophically about how to run a research lab, and just have certain gut instincts.
And that makes it hard to deal with the nuances and the complexities of what's going on inside of those research organizations. But if we could, I think talk more about, I mean not necessarily on the show, but just in general, talk more in terms of with people who are involved in these things, about what makes these kinds of organizations actually successful and work well. I think there could be also change from that direction. So it's somewhat independent of the details.

Samuel Arbesman:
So do you think there's almost a need? I mean, because been talking about there's academia and corporate industry labs or kind of a portion of a larger organization, a larger company, or maybe some companies are a little bit more research oriented. But I wonder what your thoughts are on the need for new types of institutional forms. I mean, we've talked about this before, and this is one of the things I think a lot about it, which is it is great that we have startups and corporate industry labs and academia. But in my mind at least, these are just kind of three points in some weird high dimensional institutional space, and we should actually be exploring this high dimensional institutional space and whether or not it's for-profit research labs, nonprofit independent labs that are kind of different, or there's all different ways of structuring these kinds of things. Have you given much thought to what these kinds of things could look like? What other sorts of institutions that might be a positive good for the world of research?

Kenneth Stanley:
Yeah. I mean I think we should be exploring the space of research organizations and systems, that's for sure. There should be much more exploration. I know there are people who are trying to agitate for that, so it's just not a completely dead idea. But in terms of actual implementation, it takes a lot obviously of momentum to actually put together something significant like this, because basically it boils down to funding ultimately in most of these cases. And so to attract the funding under a different model, man, you really have to put a huge amount of effort and have a lot of good connections to do that, because the risk involved is very high. I mean, research is intrinsically about risk. So people are already worried about risk just because doing research, now we're adding a whole other layer of risk doing it in a new way. Nevertheless, I still think it's something we should be doing, because we want to get good at this.
And research as an enterprise is just completely rife with counterintuitive types of principles. A lot of the things that feel like you should be doing, you probably shouldn't be doing and vice versa. And then it makes it really amenable, I think, to institutional disruption, because there probably are some upside down type of models that would be really radically successful. But I would add though that even within the existing institutions, there's room for exploration. And so I don't want to fail to call that out, because that is also important to happen, both in academia and in industry. For example, the way that funding agencies work is just so boring and so stereotypical after so many years. Where's the innovation in the funding incentive system? The National Science Foundation for example, could try all kinds of creative and interesting ways of deciding who gets funding and how much funding. Where's the experimentation on this?
And of course there are answers to this. I mean, politics enters into it. There's all kinds of reasons. But whatever the reasons are, there still needs to be that there should be more exploration, and there's an opportunity there, because you do have this public institution that actually could try things because they don't have a bottom line the way a company does. Now in a company, obviously they do have a bottom line, so there's only so much room for experimentation. But there still is room, because like you said, and I totally agree, it makes sense to carve out a pocket that works differently. And as long as you could do that, you could try all kinds of crazy things. But you have to really believe what you're saying. If you're going to carve out that pocket, you're really carving out the pocket. But most people don't really mean what they're saying when they carve out the pocket.
It's like we're going to have this pocket, but I still really want it to do these things and satisfy these OKRs. But then it turns out it's not really a pocket after all. That's the challenge there. But once you do that, if you do that, I think there's a lot of different styles of incentive systems that you could try in industry as well. I think the problem in industry is there's another kind of politics, which is people being treated special tends to bother other people. And so if you create something that looks like a playground with just children playing with toys, and they don't have to answer to anybody, they do whatever they want. Everybody else in the company is like, "What the heck is going on there?" They contribute nothing to the bottom line and we're just giving them money. And so it takes a certain amount of insulation from leadership again, got to go all the way to the top to be able to experiment in that zone,. But people need to get better at articulating why this is actually essential.
And it's dangerous not to have it, both for the disruptor and the defender. I think it can happen. We can try different models. And some companies, there are always some exceptions out there are willing to try things. But yeah, generally there's a lot of inertia going in the other direction back towards the OKRs, hooking back into the company objectives in the short run. And so you don't get as much experimentation as I think would be healthy. But still back to the main point that you're making, I think ultimately new institutional structure is a great thing to try. And so if somebody is able to attract the funding and pool it, I think the fundamental thing is that has to be taken very, very seriously.
There's a lot of dialogue out there about new ways of thinking about funding organizations or institutions, but nothing much happens if you just put $100,000 on the table. I mean, you need a lot more money to do anything interesting in these kinds of areas, and that has to be taken seriously, or else it'll be just a toy experiment. It's not serious. And I think that's why we don't see that much of that, because who's going to put the money down for this kind of a thing. But it doesn't mean it's impossible. I think it could happen, and should happen.

Samuel Arbesman:
There's many wealthy individuals who made their money by taking risk in their professional lives, but as you kind of imply, they often seem a little bit more risk averse when it comes to their philanthropic endeavors, for example. Why do you think that's the case? In addition to that, how would you make the argument to people about being willing to truly run these kinds of experiments as opposed to just providing some sort of token amount which doesn't really allow these experiments to be run?

Kenneth Stanley:
That's a good question about why people who were actually willing to take significant risks end up wanting to control something that's really about risk taking. That's counterintuitive. Ultimately, everybody wants to know that they have control of where they're putting their money, to some extent. So I think there's very few people who would just say, "Well, I'll just throw this money at you, and you just figure out what to do." They want to get some reassurance like, "Well, what exactly am I throwing my money into?" And then radical experiments feel like loss of control. And so there's only so many people who are willing to do that. But I don't think there's zero though. I mean there are wealthy people who do probably buy into these kinds of arguments. So I think it can happen, but it's just not going to happen that often. So yeah, making the argument is complex, because it's a kind of thing where you're arguing against common sense, or what's considered to be common sense.
So my book with Joe Layman, Why Greatness Cannot Be Planned, is really an attempt to make that argument. So I've thought a lot about what the arguments should be. But why would we need to write a book to make this argument? Because it's a complicated argument. And so it's arguing against a very, very common sense ideas. The common sense ideas I'm talking about are things like you should set an objective before you go out and try to do something. That's like almost everybody just thinks that intuitively. Or consensus is a really good way of deciding what we should do. And so these are almost ubiquitous universal assumptions. Now, if you go to individuals, what's interesting, you can find all kinds of people who would argue against these things as an individual. But institutionally, almost nobody's willing to break these molds. So for example, when it comes to consensus, that's how funding agencies work.
They say, "Okay, you send out your proposal to a panel, hopefully they're experts in your field. And then the more people agree that it's an awesome idea that a lot more likely you are to get money." That's a consensus driven system. And so what does it take to convince people that actually there are systems that are not driven by consensus that might be better? And it seems like it takes a lot. It's just breaking intuitions. But there's good arguments for breaking out of consensus. For example, if you have consensus, then you're probably not at the cutting edge of knowledge, because everybody agrees when you're inside the status quo. People start to disagree when we're at the borderlands of knowledge. I mean that's where you start to get disagreements. So that's probably where we be exploring.
But then if we reward consensus, no one will be willing to be daring enough to propose something at the borderland, because they know they won't get consensus. And this leads to a vicious cycle. That is what happens, generally. There's a joke in academia, you don't actually propose what you want to do, you propose what you think people want you to do, and then you actually do what you want to do later.

Samuel Arbesman:
I was going to say, and conversely, one of the experiments that people talk about that needs to be run with funding is you don't necessarily fund the ones that everyone thinks are good grant proposals. You say, okay, here's the ones that are the most polarizing. They've passed some sort of threshold of reasonableness. And then beyond that, if half the people think it's great, half the people think it's a terrible idea, maybe those are the ideas that are worth exploring as well.

Kenneth Stanley:
Yeah, for sure. Those are really interesting ideas. That at minimum is something to consider as a slight break of the mold is like, go for polarizing. Let's see what would happen and if we did that. But there's all kinds of other things often really radical, interesting directions almost nobody agrees with at first. So how can we tease out those that are really interesting, those kinds of things, that it's not even a half-and-half situation. And there may be creative ways of setting up systems for doing that. For example, a system where the people who make the decisions, maybe they have some limited amount of votes. But if they make a vote, then it counts for a lot. So maybe one vote is enough, but they have very few votes. Or maybe there's a skin in the game argument I think also. Because something I think is a really interesting thermometer for whether something's a good idea is whether other people not just, they would say, "Okay, yeah, you should give money." But they're willing to make a sacrifice to see what you have go forward, because it's so exciting.
Can we create a system like that? Because I think that's a much better indicator. It's like where it's like, actually, I don't really care about my project if this one goes forward, or at least I'll give up something valuable to me, I'm willing to part with to see his thing go or her thing go forward. I think it's a really interesting possibility. And so there's different ways of detecting where there an interesting path forward. These are worth experimenting with. The other thing that I think is very counterintuitive is regarding objectives, and having to actually tell people where you're going before you go there. That's basically the main topic of our book. And there should certainly be institutions that are not objectively driven from a research perspective. And even granting agencies, we don't have to be, they tend to be implicitly, they want to know what your broader impacts are going to be. What are the benefits going to be from doing this?
And that's a deliverable request. And that the problem is when you don't know what's going to happen, you can't really answer that question. So we're not allowed to propose things where we don't know what's going to happen. But we should be, because things where we don't know what are going to happen are often some of the most interesting things we can do. It's like people venturing out west after coming to North America or something. It's like, well, what are you going to find? Well, I have no idea. No one's ever been there before. How am I supposed to tell you what I'm going to find? Don't know what's out west. But is that a reason not to go out west? Obviously, actually it's the opposite. I mean, we should go and see what's over there. And so the question is where should we go? It would be maybe I have less expectation of just going north into Antarctica or something than I do going west, because probably it's not going to be freezing. So there's probably more stuff I can find that's useful to me.
So you can have intuitions about which directions are actually promising. And those intuitions are very important and don't get discussed much. They're not objective. And so an institutional framework for actually talking about why something is interesting rather than what it's going to deliver would be another potentially radical departure that's worth exploring. And those are the kinds of arguments that I would go into in more depth, talking to serious potential funding sources about why we should create institutions that have these characteristics.

Samuel Arbesman:
Well, and I think some of that, those arguments can also be made. So if you are in a larger organization and you're carving out this little pocket of playful weirdness and kind of open-endedness and lack of objectives, you have to make that argument to the rest of the organization of why this thing exists, and why these things are interesting, why they're valuable. And yeah, I'm just trying to get a sense, also even internally within this maybe a pocket or a more open-ended research organization, creating that culture of interestingness and trying to optimize for novelty or whatever it is, you have to also balance certain things. It can't just be like, oh, you're going to play with anything. Maybe sometimes that can work. But I mean, I'm thinking back to the heyday of Xerox PARC.
I think they still had sort of the overarching goal of something around building the office of the future. It was kind of very nebulous and vague, but at least there was a sort of direction and similar to like, okay, we're going to go west versus we're just going to go just some deeply polar region or whatever. How do you cultivate that culture with the right balance of playfulness?

Kenneth Stanley:
Yeah, right. I mean, I totally agree. The answer here is not just anything goes. But that would be a straw man argument against what I'm saying. You could say, well, he's just crazy, because he's basically saying, just let people play with toys and do whatever they want. And then obviously nothing will ever pay off. It's probably true. So you do need something. And I would call that constraint. It's different from an objective. But there are constraints. So this is an AI research lab. We're not playing with finger paint. That's not actually something we can do, even though you could argue that there could be an interesting result, a nice painting could come out of this. But it probably has nothing to do. I'm not saying that there is no argument that anyone could ever make, but you'd have to at least make the argument.

Samuel Arbesman:
There's always counter examples that you can make. But yeah, you need to have at least maybe some constraints.

Kenneth Stanley:
But you should at least explain why would playing with finger paint have any connection to AI. That would help me to understand. But probably doesn't. So with constraints, for example, if you're in an AI lab, you probably don't want to support finger painting. And it's not because finger painting isn't necessarily going to bring you something interesting. It could bring you a good painting, possible. But it has nothing to do with the topic. So there should be a constraint. And we don't want to just encourage people to go do finger painting. And of course there's always exceptions. I mean, it could be the case that somebody could make a case. I don't have a good case, but somebody could make some case that somehow finger painting connects back to AI. But we'd need to hear the case. The case needs to be made. So constraints are totally okay, and I think a necessary. All open-ended systems have constraints of some sort. The only open-ended system with no constraints is random search, which is not a good algorithm for doing anything. So constraints are useful.
But within constraints though, yeah, you can carve these things out, like you say. Even in a corporate environment, it's totally possible to carve out out a place where there's not an objective incentive system, and it has just constraints. It just isn't generally done. But yeah, I mean maybe more is happening more often as the conversation expands about this.

Samuel Arbesman:
You mentioned AI research. And I know you have some thoughts on how to think about some of these frontier AI organizations and the ways in which some of these are doing things better, some are worse, the different approaches. I'd love to hear how you think about frontier AI research and organizations, and what do you think are the right ways to approach this kind of thing?

Kenneth Stanley:
Yeah, true. I mean, that's mostly where I've been as an AI research. That's why I use it as an example. So frontier AI research, it's a great example of these kinds of phenomena. AI just in general has a long history, of course, of decades of different fads coming in and out, and different levels of funding. And of course in recent years we've been in a, let's say an AI summer. There's been lots of funding, especially in industry. For a while what we were seeing was that I think frontier Research believed to some extent in diversity. There would be some kind of different teams. There'd be a statistical AI team. There'd be a deep learning team, or maybe called neural networks back in the day, neural networks team. And there'd be some types, some other areas represented. And it's kind of like there's a healthy balance of different bets that we're placing.
So that's sort of the way things where at some points in time. And then more recently we've seen a lot of convergence. And so it's for an understandable reason, because there's been a breakthrough, and happened to be in the deep learning area, and it was so captivating that it just sucked all the resources into one thing. And I don't just mean deep learning. I mean into a particular pocket in a way of doing deep learning, which is the LLMs and how the LLMs work. And I'm not saying that this is irrational. There clearly was a breakthrough. I mean, I think there's been real value created. And so when there's value created, it makes sense to actually invest. But now that we're in this situation, I think you're seeing probably over convergence. It's hard to say where the inflection point is or was, but there's some inflection point where it just turns into over optimization along a single dimension, or let's say single gradient.
And the problem is, and this is always the problem in research, but the problem is its possible it's leading to a local optimum. Now some people when they hear something like that, react very indignantly, and think somehow this is a commentary condemning all of AI, say we have to go back to square one. That wouldn't be my position. I don't think that everything needs to be thrown out and we just start all over again. When we're talking about a local optimum, we're talking about not starting over, but possibly a couple steps backward to move a few more steps forward. And so there's some circuitous aspects to the path that will lead to revolutionary change or new advances. That's completely reasonable. And that tends to be always how search spaces are structured. I mean, complex search spaces that are high dimensional essentially as far as I've ever seen in anything, always have circuitous structure.
In other words, it's not like a single end run up to the top of the hill. It's complex. And so a lot of the measurements that we use, which is how we decide what gradient we're on, which are benchmarks, we call them benchmarks, tests, they end up being deceptive at some point. It starts to be that we're over-optimizing towards the benchmarks, we end up on a local optimum, and somebody needs to do something radical at some point to get off that local optimum, which is not the same thing as saying start over again. So the challenge I think for frontier labs is to avoid that pitfall at this moment in time. That's a real danger right now, that it's just the same kind of disruptor dilemma. Some small entity is going to see an opportunity that they're blind to, because they're so over-invested on the current gradient, which is so compelling, because it's still paying off dividends. Even now as we move up that gradient, it still looks like things are going well.
But the looming future involves some kind of pivot that's different from that gradient. And that is probably true, because that's just the nature of complex spaces. That's something maybe that's confusing about my argument is that people often will say, "Well, give me the specific thing we need to do." I mean, the whole point here is that we're talking in generalities here about research. The whole point is we don't know what we don't know. So it's like my argument doesn't hinge on me telling you what we don't know. Even though I have plenty of speculations about what it might be, as do many people. Lots of people out there including famous names, people LeCun have their ideas about the right other direction we should go. But that's not my point. I'm making a much more general point.
It's just that complex problems tend to be circuitous. They tend to be deceptive. And so just as a general principle, you have to be defensive against that possibility. And it's especially dangerous for places that feel like they're on the golden road to perfection. That can be very deceptive. That road, that garden path is just so tempting, and it brings in so much investment, and it makes you feel so comfortable and so safe that it's be very hard to carve out this kind of contrarian entity internally in that situation, which is why there's hope for disruptors. They can come in and change the game. One thing that is indicative of this is the fact that a lot of frontier labs have the same kind of rallying cry as each other, when saying, "What's exciting in the future? What's going to happen? Where is the payoff going to be?"
They tend to sound like parrots. They all say sort of the same things. And this is a sign of convergence. Ideally, they should disagree on some fundamentals, but they don't seem to. It's true that there are some exceptions. I mean, like LeCun at Meta, I mean he'll actually literally say some contrarian things. But generally speaking, if you hear the CEOs of these organizations at the highest level, they tend to talk the same language and sort of say the same things and give the same justifications for investments. And so there's a sign of convergence. And so to what degree did they have the ability, I think it's a really interesting question to create other types of bets internally in an organization like that, where things are nominally going so well. It's a really interesting situation.

Samuel Arbesman:
Because on the one hand, if you were able to make that case within this large organization saying, "Okay, carve out this little space to make some other bets and try some of these things,. And we can be relatively agnostic as to which one is going to succeed. But we need to try other things." That if those bets were being tried in a large organization, these large labs, they would have the resources to succeed. On the other hand though, if the organizations become so monolithic, then a startup outside could maybe be better poised to disrupt. But it might be harder for that one to have the necessary resources to be successful. So how do you think about the balance there in terms of internal competition, external competition in terms of being willing to try these different paths?

Kenneth Stanley:
Yeah, it is true. I mean, there's a big difference between the startup disrupter and the internal attempt to facilitate disruption, or you might call it defense against disruption, self-disruption. Yeah. And there are trade-offs again. Yeah, because I totally agree, the big organizations do have an advantage if they could get their act together. That makes them formidable. So these frontier labs are formidable, because in theory, they have the resources to try any really radical bet and go far down that path, which often is what's required to see radical bets pay off. I mean, the field of neural networks is a story like that itself. I mean, for years people thought neural networks were dead. And in the 1980s, and in the 90s, it's just like the usual assumptions, neural networks through bad propagation get stuck in local optima. They're not really the hot thing right now, the kind of old news, but we've seen that they don't actually do very much.
So there were other statistical techniques, port vector missions were hot at different times. But then because some people were willing to just keep on doubling down and doubling down and doubling down, turns out actually that was the path that ultimately paid off big time. As I said, doubling down that ability to have conviction and to stay on a path that is going to be really challenging in an environment where everyone else is on a different path. But I mean, I do want to be realistic here. I want to acknowledge that I'm not necessarily saying, when we talk about radical contrarian different path, we're not necessarily saying that it's completely misaligned. It's like it's still using maybe a lot of the foundational tools and ideas of say, deep learning. It could still be true. It's just that it's using those to go in a pivot direction, which is with a different hypothesis, something different that we're assuming might actually pay off.
And so we're not necessarily talking about just dump everything and start up again. I just want to emphasize that, because that's often how this kind of stuff is caricatured. But it's still difficult to do that internally, because you're moving against enormous inertia. If you're that person who's like everything else is paying off big time, you're not paying off at all, and you're in the middle of this environment where everybody else is showing off and having all these great things. And I mean, there's all kinds of things that can screw that up, even if it's not intentional, like a bonus structure can cause problems. And it's very implicit, so you can't really tell this is happening. But just having bonuses alone could create a situation where the people in that contrarian bet organization, sub-organization just can't get the courage to do this thing, because they know they're not going to get a payoff. And so they won't get their bonus at the end of the year.
And so they don't necessarily explicitly make that connection, or say that, it's very implicit. But then it's very subtle because you're in an environment where other people are way, way outpacing you in terms of doing exciting things and giving payoff to the company. And so it takes, I think, an extreme initiative from the top down to actually protect something like that. Which is why on the other hand, even though they have so many resources, they're at a disadvantage, because at the startup that isn't the problem. I mean, basically the contrarian bet is the company, and so they can go full throttle. But how did they get to do that? Well, that's because there's an investor who also had an open-minded view and was willing to try a different... But I think that's more likely than inside the big frontier lab, because the investors, they have a portfolio.
It includes some big risks, some lesser risks. They don't have to actually pit those against each other. The way when you're in the big frontier lab, you may feel like you're pitted against your peers who are also in the same company. You're sitting down in the cafeteria next to these people every day with your thing that still doesn't work after a year. And then they've got all these, you're like, "Yeah, I really belong here." So that won't happen in the VC or startup world. I mean, you're just kind of on your own. The investor said, "All right, I'm willing to take this bet." So I think it does give you a little more headspace to really get serious and think about this. It's not that it's impossible at the frontier Lab, I think it is possible, and a good leader would facilitate that. It's just harder.
So yeah, clearly the startup has the advantage of being able to do this without that kind of implicit restriction or worry. But then they have the disadvantage of fewer resources. So there's a trade-off again. And so there's definitely fewer resources is a huge, huge weight on the startup that the frontier Lab doesn't have. They really have to be careful about running out of runway. One thing that frontier labs should be worried about is the possibility that resource constraints actually does lead to the disruptive innovation. So that's sort of a way to turn that trade off on its head. Within frontier labs, we see that, a lot of the current craze is about the idea of just getting more and more money for more and more scaling. And so if there was a reverse trend of any type where it's actually we can massively, massively outpace all the scaling, that would be a huge, huge problem, I think.
So to get a little more concrete about it's not just to say that we're going somehow make things cheaper. I mean, I know that people like Dario at Anthropic have argued that it's still to the advantage of the frontier labs if things get cheaper, because it just scaled from there. We just jumped up a few runs, but we're still going up. But I think what could be something is something that just makes obsolete, if somehow makes obsolete all of the effort and technical gone into being able to run these absolutely gigantic labs, because there's a fundamental change in architecture or something like that. And so huge amount, not necessarily all research, but huge amounts of technical infrastructure and things that have been built up suddenly become obsolete. That would be a weird situation. And that would be something that I think startups are incentivized to actually affect. They actually want that to happen. That's the way they would become a new frontier lab.
So I think there's definitely room for some crazy stuff to happen in this dynamic. But what I would like to see, just from a philosophical point of view is for the leadership at these frontier labs, big companies to take seriously the idea of taking these carve-outs. I think that that's good for everybody. It's good for them. It's good for everybody as society, because there'll be more innovation. It's good for the field, because there'll be more innovation. So the one downside is just it potentially means that there's more concentration of resources and fewer numbers of companies, which isn't necessarily a good thing. But nevertheless, I think we should see more of this. They should take this really seriously, and think innovatively out of the box. It's not innovatively about AI, it's innovatively about how to run the research organization. I still feel very disappointed in general across the industry about how little innovative thinking there is about actually protecting these organizations, like the simplest things like bonus structures, and I'm not saying no one's thought about these things.
But they can make a big difference. I'm saying simple things can make a big difference. So why not put some serious weight on your carve-outs and take them really seriously? They're not just an afterthought, and that's generally how they are treated.

Samuel Arbesman:
Right. Not only are they afterthought, but they require certain structural differences in order for them to be able to succeed, and building them in this very deliberate way, which is what you're arguing for. And too often there's not that. They're just like, "Oh yeah, we'll have a little area. People can do some weird things." But if they're still playing by the rules of the rest of the organization, that's not a recipe for allowing them to actually succeed according to their own rules. And just out of curiosity, you don't necessarily have to have a single answer for this, because I feel like there's some agnosticism here. But what are the kinds of more contrarian bets and kinds of things in AI that you think should be tried more? Whether it's the kinds of things you're thinking about yourself or just other things that you've been seeing? Because there has been a certain amount of convergence. Where should we be diverging and exploring?

Kenneth Stanley:
It is always a tricky question. If you ever ask researchers publicly what are the right contrarian bets, it's like if they have one, it's probably their best idea. So they're not going to tell you.

Samuel Arbesman:
And that is fine. You can be as vague or evasive as necessary.

Kenneth Stanley:
So I'll be slightly evasive just because I'm not going to give you all my best ideas, basically. But I hope that they'll come out. I mean, I'm trying to hide ideas, but I hope they'll come out and see the light of day. But what I could say is I do think that the field that I've been trying to encourage people to look more at is a potentially very disruptive area, which is open-endedness. Open-endedness for those who don't know, refers to systems that continually produce interesting artifacts on their own. And the longer they run, the more interesting it gets. And a very intrinsic and important property of open-ended systems is that they're divergent. So divergence is something that commonly discussed in machine learning circles, at least historically, because most systems we're interested in convergence. Convergence to the global optimum would be the ideal situation. So why would you want it to diverge? It's like a crazy proposition, but actually there's a lot of meaning behind the divergence.
There's a lot of reasons to divergence. When you think about it, divergence is often associated with creativity. Even people often say divergent thinkers are brilliant and things like this. So obviously divergence has a role in intelligence, but it's fundamental to open-ended systems, because open-ended systems diverge and diverge over time. So for an example, civilization is an open-ended system. Civilization is divergent.

Samuel Arbesman:
And I assume you're referring to real civilization, not civilization the computer game.

Kenneth Stanley:
Oh, yeah, yeah. Sorry. Yeah.

Samuel Arbesman:
Just wanted to confirm.

Kenneth Stanley:
Not the computer game for audience members who play that game. But yeah, the real civilization, because all of the inventions that we've ever had over all of human history, that's a divergent tree of ideas that is not converging to a final uber invention, which is the end of this thing. And that doesn't look like a normal machine learning algorithm. But it is built on top of human intelligence. So if you look at civilization, it's not like some independent phenomenon where it's not really related to AI. I mean, that's basically what we want AI to do. That's the greatest thing we've ever done. And by the way, we're not just talking about technology inventions, we're talking about music and art and humanities and social systems, everything we've ever thought of. And so that's civilization, it's an open-ended divergent system.
The longer it goes, the more stuff comes out of it, and the more diversity there is of that stuff, and the more complex that stuff becomes. And it's all because of human intelligence, that's the fuel underneath what's going on. And so clearly we need to capture whatever that phenomenon is if we're going to do what human intelligence does, if it's worthy of the name AGI or superhuman or whatever. And by the way, I just want to add in just for context, the other big open-ended system is natural evolution, just to make the point that it doesn't have to have human intelligence to be open-ended. So the tree of life starting with a single cell, or presumably something like a single cell is also completely open-ended in the same spirit as civilization and the tree of ideas that we've had. So it's like every a hundred million years, there's a whole other level of diversity and complexity and things like that as you go through.
And so what kind of algorithm is that now? So I'm saying though that these kinds of systems. Open-ended is actually getting much more popular, I guess in recent years. I know there was a keynote on it just a few days ago at ICLR. That was Tim Rocktäschel, his name I'm probably destroying. But it's great to see him up there talking about open-endedness. I hope to see more of that kind of stuff. He leads the open-ended team at DeepMind, actually. It shows that there's increasing investment in this area. I mean, I'm leading now to open and open-ended team at Lila Sciences. So there's definitely places investing in open-endedness. But what I'm saying is that it has the potential for disruption, because it tends not to be the underlying principle that guides the systems that we're building today. They don't exhibit open-endedness. Scoring really well in a test is almost antithetical to open-endedness.
It's a very objectively driven kind of one-off type of achievement, but there's nothing new happening. If you think about it in civilization terms, one of the things that we won't remember about people, about what happened in our past, our ancestors, is the scores that got on tests. There's all kinds of people who got really good scores on tests in previous generations that you'll never hear of, and nobody will ever remember, because it doesn't make any difference to anything. It's all of this stuff that we created that is actually the legacy that we leave behind that actually matters for what human intelligence produces. And so it's kind of like things are a little off-kilter that actually the thing that's really guiding everyone forward are these tests like a math Olympiad test, or something like that. The last test humanity's ever going to take or whatever. All these tests, they're just tests. That's not going to actually help you to actually find a gradient towards open-endedness.
And so there's room because of that, because we're so obsessed with these benchmark tests, climbing these ladders, there's room for disruption. Because somebody's going to say they don't care about the test, and going to do something that's completely orthogonal. Again, I'm not saying necessarily it's with no LLMs involved or without deep learning. It could involve that. But we have an ability to think in a different direction, even with these tools or this infrastructure at our disposal.

Samuel Arbesman:
I love that. That is a very optimistic and exciting and open-ended version of the way we can do AI. That might actually be a perfect place to end. Ken, thanks so much. It's been a real pleasure to chat with you about all this.

Kenneth Stanley:
Yeah, absolutely. A real pleasure to talk to you too. And thank you for having me on the show.

‍