Загрузка видео...

Не удалось загрузить видео

На главную

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model...

_akhaliq's profile picture

AK

523,564 subscribers

124,085 просмотров • 2 лет назад •via X (Twitter)

ОбразованиеНаука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

New Generation Model! 🚨 We're introducing the Mistral model to our expanding lineup of generation models. Mistral brings efficient performance and strong language understanding capabilities to our platform. Initial testing shows promising results in code comprehension and generation tasks, making it a valuable addition to development workflows. While we continue to optimize its implementation, early benchmarks demonstrate consistent and reliable outputs across various programming tasks.
0:17

New Generation Model! 🚨 We're introducing the Mistral model to our expanding lineup of generation models. Mistral brings efficient performance and strong language understanding capabilities to our platform. Initial testing shows promising results in code comprehension and generation tasks, making it a valuable addition to development workflows. While we continue to optimize its implementation, early benchmarks demonstrate consistent and reliable outputs across various programming tasks.

ALCHEMIST AI 🔮

16,438 просмотров • 1 год назад

Switti -- a new scale-wise transformer for text-to-image generation 🦾 🔥 Improved generation of fine-grained details. Outperforms existing T2I AR models and competes with state-of-the-art T2I diffusion models while being up to 7x faster.
0:16

Switti -- a new scale-wise transformer for text-to-image generation 🦾 🔥 Improved generation of fine-grained details. Outperforms existing T2I AR models and competes with state-of-the-art T2I diffusion models while being up to 7x faster.

Gradio

29,314 просмотров • 1 год назад

Introducing MiniMax H3, our next-gen open-weight multimodal video model, built for general intelligence beyond single-task generation. H3 understands text, images, video, and audio together, interpreting motion, sound, emotion, and cinematography as one unified creative language, delivering generation with fine-grained precision and cinematic quality. → Native Multimodal Understanding & Generation → Precision Editing & Control → Commercial-Grade Output: film, ads, MVs, UI, game CG → Cinematic Audio-Visual Quality: native stereo audio, up to 2K/24FPS Multi-asset reference for characters, motion, camera movement + voice cloning. Hailuo AI (MiniMax)'s H3 takes ideas from concept to production in one flow. Try it now: #MiniMaxH3
0:15

Introducing MiniMax H3, our next-gen open-weight multimodal video model, built for general intelligence beyond single-task generation. H3 understands text, images, video, and audio together, interpreting motion, sound, emotion, and cinematography as one unified creative language, delivering generation with fine-grained precision and cinematic quality. → Native Multimodal Understanding & Generation → Precision Editing & Control → Commercial-Grade Output: film, ads, MVs, UI, game CG → Cinematic Audio-Visual Quality: native stereo audio, up to 2K/24FPS Multi-asset reference for characters, motion, camera movement + voice cloning. Hailuo AI (MiniMax)'s H3 takes ideas from concept to production in one flow. Try it now: #MiniMaxH3

SARAH

97,618 просмотров • 2 месяцев назад

A new approach to AI video generation is here. MiniMax H3 combines multimodal understanding with generation, allowing it to work with text, images, video, and audio as part of the same creative process. Creators can use multiple references for characters, motion and camera movement, while also controlling edits and voices. Cinematic output reaches up to 2K at 24FPS with native stereo audio. Explore it on MiniMax Design (H3): #MiniMaxH3
0:15

A new approach to AI video generation is here. MiniMax H3 combines multimodal understanding with generation, allowing it to work with text, images, video, and audio as part of the same creative process. Creators can use multiple references for characters, motion and camera movement, while also controlling edits and voices. Cinematic output reaches up to 2K at 24FPS with native stereo audio. Explore it on MiniMax Design (H3): #MiniMaxH3

AGAFE

66,612 просмотров • 1 месяц назад

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵
0:18

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

24,296 просмотров • 3 месяцев назад

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.
0:12

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.

AK

50,995 просмотров • 2 лет назад

With Hunyuan3D World Model 1.0 now released and open-sourced, we're excited to showcase the technical highlights behind this impressive innovation: ✅360° Panoramic Generation: Creates complete, immersive “world scenes”, far beyond localized views. ✅Explorable 3D Scene Generation: Generates diverse, spatially consistent 3D worlds from text/image for truly immersive exploration. ✅Interactive/Editable: Achieves separation of foreground objects, background terrain, ground, and sky, for seamless secondary editing. ✅Exportable Mesh: Generated scenes can be exported as 3D meshes for direct import into mainstream game engines and modeling software. ✅Industry-Leading SOTA Evaluation: Surpasses state-of-the-art open-source models in generation quality. As the industry's first open-source model for physical simulation and explorable world generation, Hunyuan3D World Model 1.0 aims to foster a collaborative community ecosystem with developers and enthusiasts. ✨ Try it now: 🤗 Hugging Face:
0:07

With Hunyuan3D World Model 1.0 now released and open-sourced, we're excited to showcase the technical highlights behind this impressive innovation: ✅360° Panoramic Generation: Creates complete, immersive “world scenes”, far beyond localized views. ✅Explorable 3D Scene Generation: Generates diverse, spatially consistent 3D worlds from text/image for truly immersive exploration. ✅Interactive/Editable: Achieves separation of foreground objects, background terrain, ground, and sky, for seamless secondary editing. ✅Exportable Mesh: Generated scenes can be exported as 3D meshes for direct import into mainstream game engines and modeling software. ✅Industry-Leading SOTA Evaluation: Surpasses state-of-the-art open-source models in generation quality. As the industry's first open-source model for physical simulation and explorable world generation, Hunyuan3D World Model 1.0 aims to foster a collaborative community ecosystem with developers and enthusiasts. ✨ Try it now: 🤗 Hugging Face:

Tencent Hy

23,240 просмотров • 1 год назад

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,062 просмотров • 1 год назад

Qwen just released Qwen-Image-2.1, and this is exactly why open models matter. I loaded it onto my NVIDIA DGX Spark and built Spark Image Lab, a local image generation and editing workspace designed for the DGX Spark ecosystem. QWEN-IMAGE-2.1 • Text-to-image generation • Image editing • Native transparent/RGBA workflows • Up to 10 reference images • Strong identity and product preservation • Improved typography, lighting, textures and detail SPARK IMAGE LAB • Clean Gradio interface • Width and height controls • Steps, seed and batch controls • Persistent generation history • Prompts and settings saved with every result • Reference images saved and restored • Docker setup for DGX Spark • Measured DGX Spark performance benchmarks Everything runs locally. Spark Image Lab is now open source under the MIT License. This is the first public alpha, so clone it, test it on your DGX Spark, open an issue and show me what you create. MODEL REPO
0:11

Qwen just released Qwen-Image-2.1, and this is exactly why open models matter. I loaded it onto my NVIDIA DGX Spark and built Spark Image Lab, a local image generation and editing workspace designed for the DGX Spark ecosystem. QWEN-IMAGE-2.1 • Text-to-image generation • Image editing • Native transparent/RGBA workflows • Up to 10 reference images • Strong identity and product preservation • Improved typography, lighting, textures and detail SPARK IMAGE LAB • Clean Gradio interface • Width and height controls • Steps, seed and batch controls • Persistent generation history • Prompts and settings saved with every result • Reference images saved and restored • Docker setup for DGX Spark • Measured DGX Spark performance benchmarks Everything runs locally. Spark Image Lab is now open source under the MIT License. This is the first public alpha, so clone it, test it on your DGX Spark, open an issue and show me what you create. MODEL REPO

Joey

23,578 просмотров • 18 дней назад

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.
0:14

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Akshay 🚀

64,011 просмотров • 10 месяцев назад

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation
0:19

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,830 просмотров • 3 лет назад

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.
0:19

MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers paper page: Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

AK

25,465 просмотров • 2 лет назад

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵
0:12

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,258 просмотров • 1 год назад

🚨 Wait… I think I just discovered something HUGE about AI-generated 3D models. I found Hyper3D’s new Agentic Mode that can BUILD, EDIT & ANIMATE 3D models!! And there’s one thing that really surprised me. Unlike regular 3D generation, Agentic Mode can automatically filter and organize more informative input. You can give it different views, individual parts, or even a single reference image of separate components… And it can figure out how they fit together and generate a complete 3D model — with surprisingly accurate structure and details. That’s something I haven’t seen regular 3D generation handle this well. Check out this 3D windmill animation 👇 Hyper3D Agentic Mode lets you control 3D models with PARAMETERS !! Instead of rebuilding the sofa from scratch, you can change things like: → Parametric Control → Animation Pressets → Enhance Materials So you can generate the model once... and keep editing its geometry without starting from scratch
0:17

🚨 Wait… I think I just discovered something HUGE about AI-generated 3D models. I found Hyper3D’s new Agentic Mode that can BUILD, EDIT & ANIMATE 3D models!! And there’s one thing that really surprised me. Unlike regular 3D generation, Agentic Mode can automatically filter and organize more informative input. You can give it different views, individual parts, or even a single reference image of separate components… And it can figure out how they fit together and generate a complete 3D model — with surprisingly accurate structure and details. That’s something I haven’t seen regular 3D generation handle this well. Check out this 3D windmill animation 👇 Hyper3D Agentic Mode lets you control 3D models with PARAMETERS !! Instead of rebuilding the sofa from scratch, you can change things like: → Parametric Control → Animation Pressets → Enhance Materials So you can generate the model once... and keep editing its geometry without starting from scratch

SARAH

90,925 просмотров • 9 дней назад

Most video models do one thing well. MiniMax H3 does everything, together. Our next-gen open-weight multimodal model reads text, image, video, and audio as a single creative language: motion, sound, emotion, cinematography, all connected. The result: precision-level generation with real cinematic quality. → Commercial-grade: film, ads, MVs, UI, game CG → 2K/24FPS, native stereo audio → Voice cloning + multi-asset reference for characters, motion & camera Concept to production. One seamless flow. | #MiniMaxH3 Hailuo AI (MiniMax)
0:15

Most video models do one thing well. MiniMax H3 does everything, together. Our next-gen open-weight multimodal model reads text, image, video, and audio as a single creative language: motion, sound, emotion, cinematography, all connected. The result: precision-level generation with real cinematic quality. → Commercial-grade: film, ads, MVs, UI, game CG → 2K/24FPS, native stereo audio → Voice cloning + multi-asset reference for characters, motion & camera Concept to production. One seamless flow. | #MiniMaxH3 Hailuo AI (MiniMax)

KOLOVESKI

71,118 просмотров • 2 месяцев назад

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.
0:18

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 просмотров • 2 лет назад

A New Era with V3🪄 V3's new engine introduces significant advancements in output generation. Unlike V2, where the multi-model system processed prompts to produce a single output, V3 is designed to generate multiple outputs and logically link them together. This enhancement effectively removes limitations on output size, enabling more complex and expansive results. Key Features Seamless Multi-Output Generation: V3 has been trained to generate separate outputs and connect them logically. This advancement ensures that there are no longer any limitations on output size. Intelligent Image Creation: V3 improves image generation with better tools, allowing the AI to create as many images as needed and place them within the project’s context. It supports various formats like PNG, JPEG, SVG, and GLB. Web-Integrated Intelligence: V3 can now search the web for documentation and data, providing real-time context and up-to-date references. For example, if you run a restaurant and want to update your website, simply ask Alchemist AI to “generate this website in a more modern style,” and it will update all content accordingly. Improved Creativity and Output Quality: V3's creative capacity has significantly increased. Simple prompts now generate more complete and refined results, with the system efficiently combining multiple elements into cohesive outputs.
0:17

A New Era with V3🪄 V3's new engine introduces significant advancements in output generation. Unlike V2, where the multi-model system processed prompts to produce a single output, V3 is designed to generate multiple outputs and logically link them together. This enhancement effectively removes limitations on output size, enabling more complex and expansive results. Key Features Seamless Multi-Output Generation: V3 has been trained to generate separate outputs and connect them logically. This advancement ensures that there are no longer any limitations on output size. Intelligent Image Creation: V3 improves image generation with better tools, allowing the AI to create as many images as needed and place them within the project’s context. It supports various formats like PNG, JPEG, SVG, and GLB. Web-Integrated Intelligence: V3 can now search the web for documentation and data, providing real-time context and up-to-date references. For example, if you run a restaurant and want to update your website, simply ask Alchemist AI to “generate this website in a more modern style,” and it will update all content accordingly. Improved Creativity and Output Quality: V3's creative capacity has significantly increased. Simple prompts now generate more complete and refined results, with the system efficiently combining multiple elements into cohesive outputs.

ALCHEMIST AI 🔮

48,748 просмотров • 1 год назад