Mga Subtitle (258)
0:05I'm probably most well-known
as one of the transformers authors.
0:10Transformers are, of course,
the T in ChatGPT,
0:15and are the architectures that run
0:17most of the state-of-the-art
artificial intelligence.
0:22If I think back to that time
0:24when we were working
on the transformers,
0:28I remember it as a very organic,
0:32bottom-up kind of project,
0:34where the idea came
from talking over lunch
0:39or scribbling randomly
on the whiteboards in the office.
0:47And importantly, when we felt
like we did actually have a good idea,
0:52we had the freedom
to actually spend the time
0:56and go and work on it.
0:58And even more importantly,
1:01we didn't have any pressure
that was coming down from management.
1:07No pressure to work
on any particular project,
1:12publish a number of papers,
1:15to push a certain number up.
1:18So that's the image I want you
to have in your mind, right?
1:22That is the kind of environment
1:24that allowed the transformer
to come into existence.
1:30An organic, open-ended
1:34and with a lot of freedom
to pursue the ideas
1:37that we thought
were interesting and important.
1:40And my deep concern
is that right now in the AI industry,
1:46we do not have this kind of environment.
1:50And I want to talk about why not
1:54and what can we do about it.
1:57So the main paradox that I see
in artificial intelligence research,
2:02or the industry in general right now,
2:05is that despite the fact
2:07that there's never been so much interest
2:10and resources and money and talent,
2:15this has somehow caused a narrowing
of the research that we're doing.
2:22And to me, I think
the reason is fairly obvious.
2:27It's because the immense amount
of pressure that comes with that, right?
2:31Pressure from investors
2:33that are going to ask
for a return on their investment
2:37and pressure that comes from individuals,
2:40because this is such
an overcrowded industry right now,
2:44where it is very difficult to stand out.
2:48And the researchers are really
feeling this pressure, right?
2:53If you're doing, let's say,
standard AI research right now,
2:58you kind of have to assume
3:00that there's maybe
three or four other groups
3:03doing something very similar
or maybe exactly the same.
3:07So you have to spend the time
checking to see if you've been scooped,
3:11to see if someone else
has put your idea out there.
3:15And even in academia,
3:16where you would hope
you would have more freedom,
3:19there's pressure to publish, right,
3:21and to have your papers published.
3:23So if you have an interesting idea
3:27that could create
something very interesting,
3:31or you have kind of a mediocre idea
3:33that it'll probably get a paper
and probably get accepted,
3:37the temptation is to go
for the low-hanging fruit.
3:44this pressure damages the science,
3:48because people are rushing out papers,
3:51and it's reducing the amount
of creativity that we have.
4:00So I want to take an analogy
from AI itself.
4:03So when we're designing
AI search algorithms,
4:06we have to trade off something called
the exploration-exploitation trade off.
4:14When you're searching for a solution,
4:16you can either spend your time
exploring or exploiting.
4:21If you spend all your time exploring,
4:23then you will probably only find
a large number of medical solutions.
4:29If you spend your time just exploiting,
4:34then you might lose out
4:35on finding other alternative solutions
4:38that you might be able
to exploit better and improve better.
4:43And we are almost certainly
in that situation right now
4:50So all I really want to ask you today
4:55is to consider just changing
that balance a little bit, right?
4:59Just turning up the dial
and exploring more.
5:04So I actually remember what it was like
just before the transformers,
5:07and I want to paint that picture
for you as well.
5:10Back then, my main memory was
there were a lot of papers coming out,
5:15and they were always permutating
the current architecture,
5:20which was recurrent
neural networks at the time,
5:23just endlessly trying different things,
5:26different gates, different layers,
5:28mostly for incremental gains.
5:33And then after the transformer came out,
5:37all of that work that was spent
on improving the recurrent neural network
5:42kind of felt a bit pointless.
5:45Maybe that's a bit too harsh,
but think of it like this.
5:49How much time do you think
those researchers would have spent
5:54trying to improve
the recurrent neural network
5:58if they knew something like transformers
was around the corner, right?
6:02It turned out we needed
a longer conceptual leap.
6:06We needed to throw away
recurrence completely.
6:11And again, I am worried
that we're in that situation right now
6:17where we're just concentrating
on one architecture
6:20and just permuting it
and trying different things
6:24where there might be a breakthrough
6:26just around the corner.
6:28And if there is, then
we should be acting like it.
6:33The next breakthrough,
almost by definition,
6:36has to come from this sort of open-ended,
6:40much more speculative research, right?
6:43And the only way to really hedge your bets
6:46against missing out on the next big thing
6:49is to invest in this kind of research.
6:53So if I came up here and did nothing
but just moan about the current situation,
6:58I don't think it would be a great talk.
7:00So I want to give you
a couple of suggestions.
7:07in my company, we champion
having nature-inspired.
7:14So there are still things,
7:16plenty of things
that the human brain can do,
7:18that current state-of-the-art AI can't do.
7:23So maybe if we take
some inspiration from nature,
7:28we can get some of those properties.
7:30But that's kind of my bias.
7:33You should follow
what's interesting to you?
7:38There's actually a quote I heard
two weeks ago and I thought,
7:42that's perfect,
I'm having that for my talk.
7:45And I think I'm stealing it
from a guy called Brian Chung.
7:48And it goes like this.
7:52“You should only do research
7:57that wouldn’t happen
if you weren’t working on it.”
8:01And I think that captures it perfectly.
8:03And if we all did that,
8:05we wouldn’t be stepping
on each other’s toes,
8:07and we'd be exploring
much more efficiently.
8:13So I want to give you a concrete example.
8:16There's a piece of research
that we put out recently
8:18called the Continuous Thought Machine.
8:21And all we did is we just took
a little bit of inspiration from nature.
8:25So in the human brain,
synchronization is very important.
8:30And we try to add
this kind of synchronization
8:34into artificial neural networks.
8:37I remember the employee
coming to me with the idea
8:44OK, work on it for a week,
and we’ll see what happens.
8:50That employee later confided in me
8:55that in his previous employment,
8:58or even in his academic
position before that,
9:03that he probably would have gotten
skepticism and told not to waste his time.
9:10he started to find much more
interesting properties of this model.
9:18And the project became a success.
9:21We actually announced that we got
a spotlight at NeurIPS this year.
9:27And I think there's a couple
of reasons for that.
9:29I think there's a hunger for this kind
of new and differentiated research,
9:35and more interestingly,
9:37at no point, when we were
working on this project,
9:42did we have to worry about being scooped.
9:45So we could take our time, right,
9:47to do the science properly
and run the benchmarks
9:49that we wanted to run.
9:52And I think that's the kind
of research we should be doing.
9:57So hopefully, from that you can tell
that I'm not just up here
10:02trying to make a talk that sounds good.
10:06I actually believe this, right?
10:08I am putting my money where my mouth is,
10:10and I am creating this kind
of environment,
10:13the kind of environment
that allow transformers
10:16to come into existence at my company.
10:20I'm not sure if I should tell you this,
10:22because it's a bit of an advantage
that the company has right now,
10:28but it's a really,
really good way of getting talent.
10:33Talented, intelligent people,
10:39will naturally seek out
this kind of environment
10:43And some of our best hires recently
10:47have been explicitly
because of this reason.
10:52it works better than just money.
10:58These superstars that are
apparently being snapped up
11:01for literally a million dollars
a year in some cases,
11:06do you think that when
they start their new position,
11:10they feel empowered
to try their mad ideas,
11:14their more speculative ideas?
11:17Or do they feel immense pressure
to prove their worth
11:21and will once again go
for the low-hanging fruit?
11:26So there's another reason, I think,
that maybe we're not exploring
11:31quite as efficiently as we should be.
11:33And that's because
transformers are too good.
11:42But seriously, I mean,
what can I mean by that?
11:47What I mean is, I think
the punchline is going to be
11:49that when we look back
at this point in AI history,
11:53the fact that the current technology
is so powerful and flexible
11:58that it stopped us
from looking for better.
12:02It makes sense, right?
12:03If the current technology was worse,
12:05more people would be looking for better.
12:10So there's two points
I would like to clarify.
12:17I'm not saying that there isn't already
12:21plenty of very interesting
research happening.
12:26I'm just saying that given the amount
of talent and resources
12:31that we have currently,
12:32we can afford to do a lot more, right?
12:37I and several other,
many other researchers
12:42believe we're not done
12:44and we should be looking for better.
12:47But I'm also not saying that we should
throw away the current technology.
12:52No, there's still plenty
of very important research to be done
12:57on the current technology
12:59and will bring a lot of value
in the coming years.
13:03I personally made the decision
at the beginning of this year
13:08that I'm going to drastically reduce
the amount of time
13:12that I spend on transformers.
13:16I'm explicitly now exploring
and looking for the next thing.
13:23Now it might sound
a little controversial, maybe,
13:26to hear one of the transformers
authors stand on stage
13:29and tell you that he's
absolutely sick of them,
13:32but it's kind of fair enough, right?
13:34I've been working on them
longer than anyone,
13:38with the possible exception
of seven other people.
13:56to spend more time on the ideas
14:00that you think are important
and interesting?
14:11to give the researchers some more freedom
14:13to pursue these ideas?
14:21Are you bold enough to create businesses
14:25that create these kind of environments
14:28that will allow the managers
to feel like they can afford
14:32to give the freedom to their researchers?
14:44to invest in these kind of businesses,
14:49where, in my opinion,
14:51these are the kind of businesses
14:53is where the next breakthrough
is going to come from.
14:58And I will leave you with this.
15:02A lot of the pressure, like I said,
comes from competition, right?
15:08Competition between companies,
between products,
15:13between researchers,
15:16fighting over the same idea.
15:19But genuinely, from my perspective,
15:23this is not a competition.
15:26We all have the same goal.
15:30We all want to see
this technology perfected
15:34so that we can all benefit from it.
15:39So if we can all, collectively,
15:44turn up the explore dial
15:46and then openly share what we find,
15:50we can get to our goal
