正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

we're turning University of Waterloo into a supercomputer! arceus is a cross-device distributed compute network for training large models, using model/tensor/pipeline parallelism. you can train anything, from deep neural networks to language models on the network. oss & deployment soon!

rajan agarwal

12,125 subscribers

35,720 次观看 • 1 年前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

11 条评论

rajan agarwal 的头像

rajan agarwal1 年前

it was a lot of fun working with @ishaandey_, @josh1yan and @simerusm on this project :) Technical Case Study: Design Case Study:

rajan agarwal 的头像

rajan agarwal1 年前

we are thankful for the opportunity to demo to & receive feedback from engineers at @openai @anthropic @telsa @apple, VCs at @a16z @sequoia @ThriveCapital @CommaCapital & C-suite at @tryramp @ollama!

UserInterface 的头像

UserInterface7 年前

The UIX Network is Live! Business & Social Networking 2.0. All the tools you need to get your $ right. By Us For Us. -It's for the people. #getoncode pls rt

andrew 的头像

andrew1 年前

@UWaterloo bro i was lied to i thought waterloo was already a super computer???

rajan agarwal 的头像

rajan agarwal1 年前

@UWaterloo LMAO

parshant 的头像

parshant1 年前

@UWaterloo looks really cool

Mayank Jain 的头像

Mayank Jain1 年前

@UWaterloo The design is so clean

Mike Bird (Hiring) 的头像

Mike Bird (Hiring)1 年前

@UWaterloo Man, when I was there, all we had was dc++

Jibraan ☾⁺ 的头像

Jibraan ☾⁺1 年前

@UWaterloo the waterloo fomo is very real

Roshan K 的头像

Roshan K1 年前

@UWaterloo that forward pass / backprop animation is sick

Wenitte Apiou 的头像

Wenitte Apiou1 年前

@UWaterloo Here for the Pokemon reference

相关视频

“Our compute nodes don’t run a full copy of the models we train on IOTA. They actually run a small sliver of the model. This means you can train really large frontier sized models using very small building blocks”. CTO crux unpacks how IOTA ・ SN9’s model parallel architecture allows us to train at scale by splitting our models across multiple machines, and then sewing them together. This is core to SN9’s architecture. See the full Eye On A.I. podcast below.

“Our compute nodes don’t run a full copy of the models we train on IOTA. They actually run a small sliver of the model. This means you can train really large frontier sized models using very small building blocks”. CTO crux unpacks how IOTA ・ SN9’s model parallel architecture allows us to train at scale by splitting our models across multiple machines, and then sewing them together. This is core to SN9’s architecture. See the full Eye On A.I. podcast below.

IOTA ・ SN9

15,465 次观看 • 2 个月前

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Alex Ker 🔭

1,367,097 次观看 • 2 年前

At #RenderCon Jules Urbach shared a view for the future of AI and decentralized compute, discussing how the industry is shifting from focusing on pre-training models towards inference time AI applications and agents that can run on The Render Network high performance consumer GPUs.

At #RenderCon Jules Urbach shared a view for the future of AI and decentralized compute, discussing how the industry is shifting from focusing on pre-training models towards inference time AI applications and agents that can run on The Render Network high performance consumer GPUs.

The Render Network

10,904 次观看 • 1 年前

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

Haider.

552,416 次观看 • 2 年前

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 次观看 • 2 年前

Going to #CVPR2023 ? Checkout our tutorial on Efficient Neural Networks: From Algorithm Design to Practical Mobile Deployment. We’ll do a deep dive into mobile transformers, mobile neural rendering and SnapFusion--the fastest mobile text-to-image model. Our content in 32 seconds

Going to #CVPR2023 ? Checkout our tutorial on Efficient Neural Networks: From Algorithm Design to Practical Mobile Deployment. We’ll do a deep dive into mobile transformers, mobile neural rendering and SnapFusion--the fastest mobile text-to-image model. Our content in 32 seconds

Sergey Tulyakov

23,365 次观看 • 3 年前

New research from the team officially drops tomorrow: scaling decentralized training from 72B to frontier models. SparseLoCo solved gradient sync. Heterogenous SparseLoCo solves activation transmission between pipeline stages. What's it mean? The internet can be your datacenter. Inter-datacenter training that connects datacenters with consumer GPUs, university clusters, and distributed infrastructure worldwide. Join us for a TGIF deep dive with Distributed State and the research team.

New research from the team officially drops tomorrow: scaling decentralized training from 72B to frontier models. SparseLoCo solved gradient sync. Heterogenous SparseLoCo solves activation transmission between pipeline stages. What's it mean? The internet can be your datacenter. Inter-datacenter training that connects datacenters with consumer GPUs, university clusters, and distributed infrastructure worldwide. Join us for a TGIF deep dive with Distributed State and the research team.

templar

10,047 次观看 • 7 个月前

🕹️ Is this the smallest language model in the world? I just managed to squeeze JAM, real artificial intelligence into 30 kilobytes, running on a 1979 Atari 800. Just A Model. Fully generative, deterministic language model, powered by a neural network and built to run on 8‑bit hardware. Atari Forever.

🕹️ Is this the smallest language model in the world? I just managed to squeeze JAM, real artificial intelligence into 30 kilobytes, running on a 1979 Atari 800. Just A Model. Fully generative, deterministic language model, powered by a neural network and built to run on 8‑bit hardware. Atari Forever.

Marek Spanel

113,732 次观看 • 4 个月前

🤔 Can lie detectors make AI more honest? Or will they become sneakier liars? We tested what happens when you add deception detectors into the training loop of large language models. Will training against probe-detected lies encourage honesty? Depends on how you train it!

🤔 Can lie detectors make AI more honest? Or will they become sneakier liars? We tested what happens when you add deception detectors into the training loop of large language models. Will training against probe-detected lies encourage honesty? Depends on how you train it!

FAR.AI

22,067 次观看 • 1 年前

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Been experimenting with Apple's new Core AI framework over the past few days Built a mobile app that runs local AI models directly on device, exploring Core AI model deployment (.aimodel), model specialization, caching, and inference across Apple Silicon Also spent some time integrating with Apple's Foundation Models framework and understanding how custom models can coexist with Apple's on-device AI stack Still early, but the combination of Core AI, Foundation Models, and the Apple Neural Engine feels like a pretty compelling foundation for local first AI applications Still in alpha and working towards a more stable version, but it's been a fun way to explore where Apple's on device AI ecosystem is headed !!

Sagar

12,828 次观看 • 1 个月前

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 次观看 • 3 年前

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

Andrew Ng

85,695 次观看 • 2 年前

Dive into how gensyn solves global compute access challenges. Our co-founder Ben Fielding and TechFlow 深潮｜APP 已上线 shared insights on a recent interview. 🧵 Clips from their conversation: 1. Imagine a network of devices worldwide forming a vast resource pool for training AI models.

Dive into how gensyn solves global compute access challenges. Our co-founder Ben Fielding and TechFlow 深潮｜APP 已上线 shared insights on a recent interview. 🧵 Clips from their conversation: 1. Imagine a network of devices worldwide forming a vast resource pool for training AI models.

gensyn

16,422 次观看 • 2 年前

$A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.$

A crypto project actually trained a 72B parameter AI model from scratch using decentralized GPU compute. Not fine-tuned, not a wrapper: trained from zero. The model benchmarks competitively against Meta's LLaMA 3 on reasoning tasks, and the entire training run cost a fraction of what centralized labs spend. If decentralized compute can produce frontier-class models, the moat around OpenAI and Anthropic is thinner than people think.

VirtualBacon

40,647 次观看 • 4 个月前

Destra Network | $DSYNC is the first platform to bring all your cloud needs together in a truly decentralized way. You can buy domain names in a decentralized manner ( powered by Decentralized DNS ) and host them on distributed networks ( powered by one click deployment ) , cutting out any central control. The decentralized ENS gateway ensures that everyone can easily access websites hosted on distributed networks through popular browsers ( Chrome, safari ), as these browsers don’t support Web3. At Destra network privacy, security and decentralisation is at forefront.

Destra Network | $DSYNC is the first platform to bring all your cloud needs together in a truly decentralized way. You can buy domain names in a decentralized manner ( powered by Decentralized DNS ) and host them on distributed networks ( powered by one click deployment ) , cutting out any central control. The decentralized ENS gateway ensures that everyone can easily access websites hosted on distributed networks through popular browsers ( Chrome, safari ), as these browsers don’t support Web3. At Destra network privacy, security and decentralisation is at forefront.

Destra Network

136,787 次观看 • 2 年前

"In the future, you won't need any software at all. The neural network will completely replace it." - stephen balaban, co-founder of Lambda "Here's the idea: instead of generating a program, just go to ChatGPT and say, 'Hi, please behave like a calculator or a spreadsheet.'" "'Generate an ASCII user interface for me. And I want you to essentially just respond, implement the logic of that program in your mind.'" "This is what I call neural software. Normal software is really brittle. 'Neural software' can't have bugs, only misunderstandings." "Large language models will be the software you interact with." From Stephen's appearance on the show in May.

"In the future, you won't need any software at all. The neural network will completely replace it." - stephen balaban, co-founder of Lambda "Here's the idea: instead of generating a program, just go to ChatGPT and say, 'Hi, please behave like a calculator or a spreadsheet.'" "'Generate an ASCII user interface for me. And I want you to essentially just respond, implement the logic of that program in your mind.'" "This is what I call neural software. Normal software is really brittle. 'Neural software' can't have bugs, only misunderstandings." "Large language models will be the software you interact with." From Stephen's appearance on the show in May.

TBPN

28,665 次观看 • 9 个月前

Today at Rowan, we're releasing Egret-1, a family of open-source neural network potentials. On lots of benchmarks, the Egret-1 models match or exceed the accuracy of quantum chemistry for organic & biomolecular simulation. Here's me and Eli Mann with a brief video intro:

Today at Rowan, we're releasing Egret-1, a family of open-source neural network potentials. On lots of benchmarks, the Egret-1 models match or exceed the accuracy of quantum chemistry for organic & biomolecular simulation. Here's me and Eli Mann with a brief video intro:

Corin Wagen

25,426 次观看 • 1 年前

we live in a $TAO simulation in a few months we went from a whitepaper, a concept, a dream (kinda) to now generating 3d simulations based on our own neural network (that we’re training using Celium btw) this is what Score - Subnet 44 is about: > collecting visual data > creating visual models > generating a ‘real time’ positional value function aren’t you entertained? I love this game!

we live in a $TAO simulation in a few months we went from a whitepaper, a concept, a dream (kinda) to now generating 3d simulations based on our own neural network (that we’re training using Celium btw) this is what Score - Subnet 44 is about: > collecting visual data > creating visual models > generating a ‘real time’ positional value function aren’t you entertained? I love this game!

Max

19,378 次观看 • 1 年前