Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

GPTs are generating sequences in a left-to-right order. Is there another way? With François Fleuret and @evanncourdier, in partnership with SkySoft-ATM, we developed σ-GPT, capable of generating sequences in any order chosen dynamically at inference time. 1/6

ArnaudPannatier's profile picture

Arnaud Pannatier

1,104 subscribers

379,912 Aufrufe • vor 2 Jahren •via X (Twitter)

BildungWissenschaft & TechnologieNachrichten & Politik
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

We only need to shuffle the input sequence and to use a double positional encoding, so that the model is informed of what token to generate at each position of the output sequence. 2/6

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

This opens a multitude of possibilities. Such as: (A) Conditioning on any subset of tokens from a sequence or (B) Estimating the probability of the remaining tokens at once. 3/6

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

(A) is done by putting the conditioning tokens “first”, and (B) is done by putting all the remaining tokens as “the next”. The latter requires carving the attention matrix. 4/6

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

We combine these two functionalities into a burst-sampling scheme with rejection. It generates all the remaining tokens in one shot, and keeps only the ones that are self-consistent. This allows sublinear sequence generation similar to diffusion. 5/6

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

With the advantages that simple tasks can be fully completed in a few steps. 6/6 (with a diffusion comparison)

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

More detail about this work in our paper, that was just accepted at @ECMLPKDD. Paper: Demo: Code: in preparation.

Profilbild von Jason Stockwell
Jason Stockwellvor 2 Jahren

@francoisfleuret @EvannCourdier @SkysoftATM so the first words that come out are the most important and limit the following possibilities. cool.

Profilbild von Arnaud Pannatier
Arnaud Pannatiervor 2 Jahren

@francoisfleuret @EvannCourdier @SkysoftATM I agree that it can be limiting in a way, but it can also be useful for the model: fixing some tokens might give the overall structure of the text it want to generate, and then it only has to fill in between.

Profilbild von Rob Leclerc
Rob Leclercvor 2 Jahren

@francoisfleuret @EvannCourdier @SkysoftATM We think multidimensionally, but we communicate serially. Intuitively this feels like it has legs.

Profilbild von liu grey
liu greyvor 2 Jahren

@francoisfleuret @EvannCourdier @SkysoftATM I'm sorry what? Half an hour ago I was tweeting about an architecture just like this combined with type checking/linting information that is tokenized and injected to improve code generation.. And you just post a paper on the difficult part of that project? Just.. right now? O.o

Ähnliche Videos

PM Mark Carney: "The new world order will be built from Europe. Canada is the most European of non-European countries, and we are transforming our cooperation with the EU ... In a more dangerous and divided world, Canada has chosen to build and work in partnership with Europe."
1:00

PM Mark Carney: "The new world order will be built from Europe. Canada is the most European of non-European countries, and we are transforming our cooperation with the EU ... In a more dangerous and divided world, Canada has chosen to build and work in partnership with Europe."

WORLD NEWS

20,566 Aufrufe • vor 19 Tagen

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.
1:07

We are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life.

Leandro von Werra

404,474 Aufrufe • vor 1 Monat

Like many on the left, we've taken a lot of inspiration from Jeremy Corbyn . With the formation of the new left party, we must work together in a collaborative way in order to defeat the far right!
1:02

Like many on the left, we've taken a lot of inspiration from Jeremy Corbyn . With the formation of the new left party, we must work together in a collaborative way in order to defeat the far right!

Mothin Ali

14,666 Aufrufe • vor 11 Monaten

At the Munich Security Conference earlier this month, the post-1945 world order was pronounced dead by most leaders. German Chancellor Friedrich Merz said, “The world order as it has stood for decades no longer exists,” and that we are in a period “great power politics.” French President Emmanuel Macron echoed Merz’s assessment and said that Europe’s old security structures tied to the previous world order don't exist and that Europe must prepare for war. U.S. Secretary of State Marco Rubio said that we are in a “new geopolitics era” because the “old world” is gone. The way I see it, we are in the Stage 6 part of the Big Cycle in which there is great disorder arising from being in a period in which there are no rules, might is right, and there is a clash of great powers. How Stage 6 works is explained in detail in Chapter 6, “The Big Cycle of External Order and Disorder,” in my book Principles for Dealing with the Changing World Order. Last week, I shared all of Chapter 6 online for people to understand how this dynamic works, and it proved very popular, with more than 75 million views. You can find here: Given the now nearly universal agreement that the post-1945 world order has broken down and that we are entering a new world order, I think it would be worth your time to read.
1:00

At the Munich Security Conference earlier this month, the post-1945 world order was pronounced dead by most leaders. German Chancellor Friedrich Merz said, “The world order as it has stood for decades no longer exists,” and that we are in a period “great power politics.” French President Emmanuel Macron echoed Merz’s assessment and said that Europe’s old security structures tied to the previous world order don't exist and that Europe must prepare for war. U.S. Secretary of State Marco Rubio said that we are in a “new geopolitics era” because the “old world” is gone. The way I see it, we are in the Stage 6 part of the Big Cycle in which there is great disorder arising from being in a period in which there are no rules, might is right, and there is a clash of great powers. How Stage 6 works is explained in detail in Chapter 6, “The Big Cycle of External Order and Disorder,” in my book Principles for Dealing with the Changing World Order. Last week, I shared all of Chapter 6 online for people to understand how this dynamic works, and it proved very popular, with more than 75 million views. You can find here: Given the now nearly universal agreement that the post-1945 world order has broken down and that we are entering a new world order, I think it would be worth your time to read.

Ray Dalio

150,022 Aufrufe • vor 4 Monaten

Jacob Frey: "This is not sustainable. This is an impossible situation that our city is presently being put in, and at the same time we are trying to find a way forward to keep people safe, to protect our neighbors, to maintain order. We're in a position right now where we have residents that are asking the very limited number of police officers to fight ICE agents on the street, to stand by their neighbors. We cannot be at a place right now in America where we have two governmental agencies that are literally fighting one another. We are put in this position because we have approximately 600 police officers in Minneapolis -- far fewer that are able to work at any given time. And there are approximately 3,000 ICE agents in the area."
1:48

Jacob Frey: "This is not sustainable. This is an impossible situation that our city is presently being put in, and at the same time we are trying to find a way forward to keep people safe, to protect our neighbors, to maintain order. We're in a position right now where we have residents that are asking the very limited number of police officers to fight ICE agents on the street, to stand by their neighbors. We cannot be at a place right now in America where we have two governmental agencies that are literally fighting one another. We are put in this position because we have approximately 600 police officers in Minneapolis -- far fewer that are able to work at any given time. And there are approximately 3,000 ICE agents in the area."

Aaron Rupar

516,481 Aufrufe • vor 6 Monaten

Robbyant released LingBot-World 2.0 (Infinity), a new open-source World model capable of generating hour-long experiences. > No quality decay, stable 720p / 60 fps real-time output. > Users can interact with the world via Actions. > The world is dynamic, with built-in agents generating new events in real time. > Multiplayer is supported within the same world. Testing time 👀
0:47

Robbyant released LingBot-World 2.0 (Infinity), a new open-source World model capable of generating hour-long experiences. > No quality decay, stable 720p / 60 fps real-time output. > Users can interact with the world via Actions. > The world is dynamic, with built-in agents generating new events in real time. > Multiplayer is supported within the same world. Testing time 👀

🚨 AI News | TestingCatalog

15,272 Aufrufe • vor 5 Tagen

Yea once you put the chain of events in order, there is nothing left to see here. We will find out we were right in like 40 years
1:42

Yea once you put the chain of events in order, there is nothing left to see here. We will find out we were right in like 40 years

The Neighborhood Publicist

44,889 Aufrufe • vor 10 Monaten

POILIEVRE: "We cannot sacrifice our relationship with our biggest customer in favour of a tenuous and risky partnership for a new world order with the regime in Beijing."
1:36

POILIEVRE: "We cannot sacrifice our relationship with our biggest customer in favour of a tenuous and risky partnership for a new world order with the regime in Beijing."

Scott Robertson

15,359 Aufrufe • vor 4 Monaten

Runway has unveiled a massive breakthrough in AI video generation at NVIDIA's GTC event, showcasing a new model capable of generating high-definition video in genuine real-time. The new model boasts a "time-to-first-frame" of under 100 milliseconds. This means HD video begins generating and playing almost the instant a command is given. This research preview was co-developed with NVIDIA and runs on their newly announced Vera Rubin architecture. This real-time capability is a foundational step for Runway's General World Model (GWM-1). It pushes the technology beyond simple video generation and into the realm of live, interactive environments.
0:54

Runway has unveiled a massive breakthrough in AI video generation at NVIDIA's GTC event, showcasing a new model capable of generating high-definition video in genuine real-time. The new model boasts a "time-to-first-frame" of under 100 milliseconds. This means HD video begins generating and playing almost the instant a command is given. This research preview was co-developed with NVIDIA and runs on their newly announced Vera Rubin architecture. This real-time capability is a foundational step for Runway's General World Model (GWM-1). It pushes the technology beyond simple video generation and into the realm of live, interactive environments.

Wes Roth

82,506 Aufrufe • vor 3 Monaten

‘We are the party of law and order.’ The Conservatives left us with a prison system in crisis. We are fixing it. The opposition would do well to remember that.
0:23

‘We are the party of law and order.’ The Conservatives left us with a prison system in crisis. We are fixing it. The opposition would do well to remember that.

Alex Davies-Jones MP

87,044 Aufrufe • vor 1 Jahr

This is literally one of the most beautiful sequences in the movie. Gets me feeling emotional every time seeing Gary go through his family's home and seeing the lost history while his friends are just in awe and are right there to support him in his journey
2:24

This is literally one of the most beautiful sequences in the movie. Gets me feeling emotional every time seeing Gary go through his family's home and seeing the lost history while his friends are just in awe and are right there to support him in his journey

bit#FreePalestine🇵🇸

38,925 Aufrufe • vor 5 Monaten

The UN will use the environment and any other groups they create, to usher in laws that you will have to comply with in order to usher in the NWO, right now Canada, UK and Australia are in the grooming stages of its citizens in order to brainwash you into complying with all these laws.
2:59

The UN will use the environment and any other groups they create, to usher in laws that you will have to comply with in order to usher in the NWO, right now Canada, UK and Australia are in the grooming stages of its citizens in order to brainwash you into complying with all these laws.

TruthSeeker01011

67,519 Aufrufe • vor 8 Monaten

Frontier models that use reasoning to "think" during inference are generating 5X more AI tokens per year. ✨ "Inference is now a thinking process. And in order to teach AI how to think, reinforcement learning and very significant computation was introduced into post-training," said NVIDIA CEO Jensen Huang in a recent keynote at #CES2026. Reinforcement learning is increasing computation demands across all AI scaling laws: pre-training, post-training, and test-time scaling. Learn more about reinforcement learning and AI scaling ➡️
0:58

Frontier models that use reasoning to "think" during inference are generating 5X more AI tokens per year. ✨ "Inference is now a thinking process. And in order to teach AI how to think, reinforcement learning and very significant computation was introduced into post-training," said NVIDIA CEO Jensen Huang in a recent keynote at #CES2026. Reinforcement learning is increasing computation demands across all AI scaling laws: pre-training, post-training, and test-time scaling. Learn more about reinforcement learning and AI scaling ➡️

NVIDIA AI Infrastructure

13,329 Aufrufe • vor 5 Monaten

Whatever relationship you have attracted into your life at any giving time, are the relationship you need to be in at that time . When you are ready to do a new thing , in a new way ,you will do it, with new people. There are people waiting for who you are becoming. chances are, you are not ready to meet them today. At any given moment., we are each in the process of being and becoming. We are taking in as much as we can. The road is curved, and we are beginning preparing to handle what is waiting around the corner.
1:20

Whatever relationship you have attracted into your life at any giving time, are the relationship you need to be in at that time . When you are ready to do a new thing , in a new way ,you will do it, with new people. There are people waiting for who you are becoming. chances are, you are not ready to meet them today. At any given moment., we are each in the process of being and becoming. We are taking in as much as we can. The road is curved, and we are beginning preparing to handle what is waiting around the corner.

LaStarya 💜✨💫

28,101 Aufrufe • vor 9 Monaten

Friends, the preoccupation with “left” vs. “right” is a distraction from what matters: a pope capable of declaring Christ’s Resurrection in a compelling way.
1:36

Friends, the preoccupation with “left” vs. “right” is a distraction from what matters: a pope capable of declaring Christ’s Resurrection in a compelling way.

Bishop Robert Barron

103,715 Aufrufe • vor 1 Jahr

.Secretary Marco Rubio on Venezuelan oil: "They are not generating any revenue from their oil right now. They can't move it unless we allow it to move because we have sanctions, because we're enforcing those sanctions. This is tremendous leverage. We are exercising it in a positive way."
0:34

.Secretary Marco Rubio on Venezuelan oil: "They are not generating any revenue from their oil right now. They can't move it unless we allow it to move because we have sanctions, because we're enforcing those sanctions. This is tremendous leverage. We are exercising it in a positive way."

Rapid Response 47

81,868 Aufrufe • vor 6 Monaten

Sec. Rubio on Venezuelan oil: "They are not generating any revenue from their oil right now. They can't move it unless we allow it to move because we have sanctions, because we're enforcing those sanctions. This is tremendous leverage. We are exercising it in a positive way."
0:34

Sec. Rubio on Venezuelan oil: "They are not generating any revenue from their oil right now. They can't move it unless we allow it to move because we have sanctions, because we're enforcing those sanctions. This is tremendous leverage. We are exercising it in a positive way."

Breaking911

61,144 Aufrufe • vor 6 Monaten

This is one of my fav sequences of play. Ramos Elbows Messi in the face. Messi doesn’t make any fuss, gets up and plays a quick one-two with Xavi and BANG
1:25

This is one of my fav sequences of play. Ramos Elbows Messi in the face. Messi doesn’t make any fuss, gets up and plays a quick one-two with Xavi and BANG

TK

414,808 Aufrufe • vor 10 Monaten

POILIEVRE: "Canada's prosperity and security are inseparable from a stable relationship with the US. And that is why we should not declare a permanent rupture with our biggest customer and closest neighbour in favour of a strategic partnership for a new world order with Beijing."
1:44

POILIEVRE: "Canada's prosperity and security are inseparable from a stable relationship with the US. And that is why we should not declare a permanent rupture with our biggest customer and closest neighbour in favour of a strategic partnership for a new world order with Beijing."

Scott Robertson

76,797 Aufrufe • vor 4 Monaten