Home
Entrar
Registrar
Loading...
Praticar Escuta
Ouvir
/
Video
/
TED Talk
/
How Competition Is Stifling AI Breakthroughs | Llion Jones | TED
How Competition Is Stifling AI Breakthroughs | Llion Jones | TED
Selecionar modo de aprendizagem:
Ver legendas
Escolher palavra
Reescrever palavra
Highlight:
3000 Oxford Words
4000 IELTS Words
5000 Oxford Words
3000 Common Words
1000 TOEIC Words
5000 TOEFL Words
Legendas (258)
0:04
So, as mentioned,
0:05
I'm probably most well-known as one of the transformers authors.
0:10
Transformers are, of course, the T in ChatGPT,
0:15
and are the architectures that run
0:17
most of the state-of-the-art artificial intelligence.
0:22
If I think back to that time
0:24
when we were working on the transformers,
0:28
I remember it as a very organic,
0:32
bottom-up kind of project,
0:34
where the idea came from talking over lunch
0:39
or scribbling randomly on the whiteboards in the office.
0:47
And importantly, when we felt like we did actually have a good idea,
0:52
we had the freedom to actually spend the time
0:56
and go and work on it.
0:58
And even more importantly,
1:01
we didn't have any pressure that was coming down from management.
1:07
No pressure to work on any particular project,
1:12
publish a number of papers,
1:15
to push a certain number up.
1:18
So that's the image I want you to have in your mind, right?
1:22
That is the kind of environment
1:24
that allowed the transformer to come into existence.
1:30
An organic, open-ended
1:34
and with a lot of freedom to pursue the ideas
1:37
that we thought were interesting and important.
1:40
And my deep concern is that right now in the AI industry,
1:46
we do not have this kind of environment.
1:50
And I want to talk about why not
1:54
and what can we do about it.
1:57
So the main paradox that I see in artificial intelligence research,
2:02
or the industry in general right now,
2:05
is that despite the fact
2:07
that there's never been so much interest
2:10
and resources and money and talent,
2:15
this has somehow caused a narrowing of the research that we're doing.
2:22
And to me, I think the reason is fairly obvious.
2:27
It's because the immense amount of pressure that comes with that, right?
2:31
Pressure from investors
2:33
that are going to ask for a return on their investment
2:37
and pressure that comes from individuals,
2:40
because this is such an overcrowded industry right now,
2:44
where it is very difficult to stand out.
2:48
And the researchers are really feeling this pressure, right?
2:53
If you're doing, let's say, standard AI research right now,
2:58
you kind of have to assume
3:00
that there's maybe three or four other groups
3:03
doing something very similar or maybe exactly the same.
3:07
So you have to spend the time checking to see if you've been scooped,
3:11
to see if someone else has put your idea out there.
3:15
And even in academia,
3:16
where you would hope you would have more freedom,
3:19
there's pressure to publish, right,
3:21
and to have your papers published.
3:23
So if you have an interesting idea
3:27
that could create something very interesting,
3:31
or you have kind of a mediocre idea
3:33
that it'll probably get a paper and probably get accepted,
3:37
the temptation is to go for the low-hanging fruit.
3:42
So unfortunately,
3:44
this pressure damages the science,
3:48
because people are rushing out papers,
3:51
and it's reducing the amount of creativity that we have.
4:00
So I want to take an analogy from AI itself.
4:03
So when we're designing AI search algorithms,
4:06
we have to trade off something called the exploration-exploitation trade off.
4:14
When you're searching for a solution,
4:16
you can either spend your time exploring or exploiting.
4:21
If you spend all your time exploring,
4:23
then you will probably only find a large number of medical solutions.
4:29
If you spend your time just exploiting,
4:34
then you might lose out
4:35
on finding other alternative solutions
4:38
that you might be able to exploit better and improve better.
4:43
And we are almost certainly in that situation right now
4:48
in the AI industry.
4:50
So all I really want to ask you today
4:55
is to consider just changing that balance a little bit, right?
4:59
Just turning up the dial and exploring more.
5:04
So I actually remember what it was like just before the transformers,
5:07
and I want to paint that picture for you as well.
5:10
Back then, my main memory was there were a lot of papers coming out,
5:15
and they were always permutating the current architecture,
5:20
which was recurrent neural networks at the time,
5:23
just endlessly trying different things,
5:26
different gates, different layers,
5:28
mostly for incremental gains.
5:33
And then after the transformer came out,
5:37
all of that work that was spent on improving the recurrent neural network
5:42
kind of felt a bit pointless.
5:45
Maybe that's a bit too harsh, but think of it like this.
5:49
How much time do you think those researchers would have spent
5:54
trying to improve the recurrent neural network
5:58
if they knew something like transformers was around the corner, right?
6:02
It turned out we needed a longer conceptual leap.
6:06
We needed to throw away recurrence completely.
6:11
And again, I am worried that we're in that situation right now
6:17
where we're just concentrating on one architecture
6:20
and just permuting it and trying different things
6:24
where there might be a breakthrough
6:26
just around the corner.
6:28
And if there is, then we should be acting like it.
6:33
The next breakthrough, almost by definition,
6:36
has to come from this sort of open-ended,
6:40
much more speculative research, right?
6:43
And the only way to really hedge your bets
6:46
against missing out on the next big thing
6:49
is to invest in this kind of research.
6:53
So if I came up here and did nothing but just moan about the current situation,
6:58
I don't think it would be a great talk.
7:00
So I want to give you a couple of suggestions.
7:05
First of all,
7:07
in my company, we champion having nature-inspired.
7:14
So there are still things,
7:16
plenty of things that the human brain can do,
7:18
that current state-of-the-art AI can't do.
7:23
So maybe if we take some inspiration from nature,
7:28
we can get some of those properties.
7:30
But that's kind of my bias.
7:33
You should follow what's interesting to you?
7:38
There's actually a quote I heard two weeks ago and I thought,
7:42
that's perfect, I'm having that for my talk.
7:45
And I think I'm stealing it from a guy called Brian Chung.
7:48
And it goes like this.
7:52
“You should only do research
7:57
that wouldn’t happen if you weren’t working on it.”
8:01
And I think that captures it perfectly.
8:03
And if we all did that,
8:05
we wouldn’t be stepping on each other’s toes,
8:07
and we'd be exploring much more efficiently.
8:13
So I want to give you a concrete example.
8:16
There's a piece of research that we put out recently
8:18
called the Continuous Thought Machine.
8:21
And all we did is we just took a little bit of inspiration from nature.
8:25
So in the human brain, synchronization is very important.
8:30
And we try to add this kind of synchronization
8:34
into artificial neural networks.
8:37
I remember the employee coming to me with the idea
8:42
and I said,
8:44
OK, work on it for a week, and we’ll see what happens.
8:50
That employee later confided in me
8:55
that in his previous employment,
8:58
or even in his academic position before that,
9:03
that he probably would have gotten skepticism and told not to waste his time.
9:08
But after that week,
9:10
he started to find much more interesting properties of this model.
9:18
And the project became a success.
9:21
We actually announced that we got a spotlight at NeurIPS this year.
9:27
And I think there's a couple of reasons for that.
9:29
I think there's a hunger for this kind of new and differentiated research,
9:35
and more interestingly,
9:37
at no point, when we were working on this project,
9:42
did we have to worry about being scooped.
9:45
So we could take our time, right,
9:47
to do the science properly and run the benchmarks
9:49
that we wanted to run.
9:52
And I think that's the kind of research we should be doing.
9:57
So hopefully, from that you can tell that I'm not just up here
10:02
trying to make a talk that sounds good.
10:06
I actually believe this, right?
10:08
I am putting my money where my mouth is,
10:10
and I am creating this kind of environment,
10:13
the kind of environment that allow transformers
10:16
to come into existence at my company.
10:20
I'm not sure if I should tell you this,
10:22
because it's a bit of an advantage that the company has right now,
10:28
but it's a really, really good way of getting talent.
10:32
Think about it.
10:33
Talented, intelligent people,
10:37
ambitious people,
10:39
will naturally seek out this kind of environment
10:42
with high autonomy.
10:43
And some of our best hires recently
10:47
have been explicitly because of this reason.
10:50
And by the way,
10:52
it works better than just money.
10:56
Think about it.
10:58
These superstars that are apparently being snapped up
11:01
for literally a million dollars a year in some cases,
11:06
do you think that when they start their new position,
11:10
they feel empowered to try their mad ideas,
11:14
their more speculative ideas?
11:17
Or do they feel immense pressure to prove their worth
11:21
and will once again go for the low-hanging fruit?
11:26
So there's another reason, I think, that maybe we're not exploring
11:31
quite as efficiently as we should be.
11:33
And that's because transformers are too good.
11:39
I know, modesty.
11:41
(Laughter)
11:42
But seriously, I mean, what can I mean by that?
11:47
What I mean is, I think the punchline is going to be
11:49
that when we look back at this point in AI history,
11:53
the fact that the current technology is so powerful and flexible
11:58
that it stopped us from looking for better.
12:02
It makes sense, right?
12:03
If the current technology was worse,
12:05
more people would be looking for better.
12:10
So there's two points I would like to clarify.
12:14
First of all,
12:17
I'm not saying that there isn't already
12:21
plenty of very interesting research happening.
12:26
I'm just saying that given the amount of talent and resources
12:31
that we have currently,
12:32
we can afford to do a lot more, right?
12:37
I and several other, many other researchers
12:42
believe we're not done
12:44
and we should be looking for better.
12:47
But I'm also not saying that we should throw away the current technology.
12:52
No, there's still plenty of very important research to be done
12:57
on the current technology
12:59
and will bring a lot of value in the coming years.
13:03
I personally made the decision at the beginning of this year
13:08
that I'm going to drastically reduce the amount of time
13:12
that I spend on transformers.
13:16
I'm explicitly now exploring and looking for the next thing.
13:23
Now it might sound a little controversial, maybe,
13:26
to hear one of the transformers authors stand on stage
13:29
and tell you that he's absolutely sick of them,
13:32
but it's kind of fair enough, right?
13:34
I've been working on them longer than anyone,
13:38
with the possible exception of seven other people.
13:42
So ...
13:46
Are we bold enough?
13:52
Researchers,
13:55
are you bold enough
13:56
to spend more time on the ideas
14:00
that you think are important and interesting?
14:04
Managers.
14:08
Are you bold enough
14:11
to give the researchers some more freedom
14:13
to pursue these ideas?
14:18
Business leaders.
14:21
Are you bold enough to create businesses
14:25
that create these kind of environments
14:28
that will allow the managers to feel like they can afford
14:32
to give the freedom to their researchers?
14:37
And investors.
14:40
Are you bold enough
14:44
to invest in these kind of businesses,
14:49
where, in my opinion,
14:51
these are the kind of businesses
14:53
is where the next breakthrough is going to come from.
14:58
And I will leave you with this.
15:02
A lot of the pressure, like I said, comes from competition, right?
15:08
Competition between companies, between products,
15:13
between researchers,
15:16
fighting over the same idea.
15:19
But genuinely, from my perspective,
15:23
this is not a competition.
15:26
We all have the same goal.
15:30
We all want to see this technology perfected
15:34
so that we can all benefit from it.
15:39
So if we can all, collectively,
15:44
turn up the explore dial
15:46
and then openly share what we find,
15:50
we can get to our goal
15:53
much faster.
15:56
Thank you.
15:58
(Applause)
