Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Here are more results from #RigidFormer: predicting physical dynamics with purely neural simulators — an attempt to learn physical dynamics in a scalable manner. 🤖 1) Controllable Articulated Body Simulation — More Results Additional Unitree G1 humanoid rollouts under controlled motion. Each sample uses a different initial state and...

20,955 Aufrufe • vor 3 Monaten •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 Aufrufe • vor 7 Monaten

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 Aufrufe • vor 1 Jahr

[LSTM] by Hand ✍️ LSTMs have been the most effective architecture to process long sequences of data, until our world was taken over by the Transformers. LSTMs belong to the broader family of recurrent neural network (RNNs) that process data sequentially in a recurrent manner. Transformers, on the other hand, abandon recurrence and use self-attention instead to process data concurrently in parallel. Recently, there is renewed interest in recurrence as people realized self-attention doesn’t scale to extremely long sequences, like hundreds of thousands of tokens. Mamba is a good example to bring back recurrence. All of a sudden, it is cool to study LSTMs. How do LSTMs work? [1] Given ↳ 🟨 Input sequence X1, X2, X3 (d = 3) ↳ 🟩 Hidden state h (d = 2) ↳ 🟦 Memory C (d = 2) ↳ Weight matrices Wf, Wc, Wi, Wo Process t = 1 [2] Initialize ↳ Randomly set the previous hidden state h0 to [1, 1] and memory cells C0 to [0.3, -0.5] [3] Linear Transform ↳ Multiply the four weight matrices with the concatenation of current input (X1) and the previous hidden state (h0). ↳ The results are feature values, each is a linear combination of the current input and hidden state. [4] Non-linear Transform ↳ Apply sigmoid σ to obtain gate values (between 0 and 1). • Forget gate (f1): [-4, -6] → [0, 0] • Input gate (i1): [6, 4] → [1, 1] • Output gate (o1): [4, -5] → [1, 0] ↳ Apply tanh to obtain candidate memory values (between -1 and 1) • Candidate memory (C’1): [1, -6] → [0.8, -1] [5] Update Memory ↳ Forget (C0 .* f1): Element-wise multiply the current memory with forget gate values. ↳ Input (C’1 .* o1): Element-wise multiply the “candidate” memory with input gate values. ↳ Update the memory to C1 by adding the two terms above: C0 .* f1 + C’1 .* o1 = C1 [6] Candiate Output ↳ Apply tanh to the new memory C1 to obtain candidate output o’1. [0.8, -1] → [0.7, -0.8] [7] Update Hidden State ↳ Output (o’1 .* o1 → h1): Element-wise multiply the candidate output with the output gate. ↳ The result is updated hidden state h1 ↳ Also, it is the first output. Process t = 2 [8] Initialize ↳ Copy previous hidden state h1 and memory C1 [9] Linear Transform ↳ Repeat [3] [10] Update Memory (C2) ↳ Repeat [4] and [5] [11] Update Hidden State (h2) ↳ Repeat [6] and [7] Process t = 3 [12] Initialize ↳ Copy previous hidden state h2 and memory C2 [13] Linear Transform ↳ Repeat [3] [14] Update Memory (C3) ↳ Repeat [4] and [5] [15] Update Hidden State (h3) ↳ Repeat [6] and [7]

Tom Yeh

72,966 Aufrufe • vor 2 Jahren

Yesterday at Brown University ICERM's workshop on “Agentic Scientific Computing and Scientific Machine Learning” I spoke about “Adaptive Swarms Across Scales”, making the case for scientific AI as systems that can create representations, stress them, fracture them, and enlarge the category in which future representations live. The category here is a composable and breakable working universe of science: data, hypotheses, simulations, measurements, tools, failures, figures, papers, provenance, and the transformations that connect them. Discovery happens when those transformations become executable, inspectable, composable, and capable of changing the world model they operate within. Atomistic modeling gives one category - states, forces, trajectories, observables, boundary conditions, conservation laws. Neural surrogates learn fast morphisms inside or between such categories. But discovery is higher-order: it changes which objects and morphisms are available in the first place: what variables exist, what operations are allowed, what evidence counts, what scale is active, what invariant is being preserved, and what kind of explanation the system is even capable of forming. This is scientific method as adaptive architecture: compression, stress, fracture, recomposition. Fracture matters here because it makes the logic physical: a non-commuting diagram realized in matter. The imposed load, material hierarchy, defect field, and assumed continuum description no longer map cleanly into the observed outcome. The crack is the obstruction and it identifies where the old morphism failed and where a new representation must be introduced. The physical crack and the categorical obstruction are the same event viewed in different substrates. ScienceClaw × Infinite is a machine for constructing and transforming a category of scientific artifacts. Each artifact is typed. Each operation has lineage. Each failed branch remains in the category as reusable structure. The “paper” is no longer the terminal object of science; it is one projection of a larger compositional trace, and it can be generated at any time for consumption by a human or an AI. With that the unit of scientific labor is changing. For most of the twentieth century the unit was the result (a measurement, a theorem, a synthesized molecule). It is now becoming the algorithm that produces results, and after that, the substrate of discovery itself. The static PDF is the wrong terminal object for this regime, and the role of the scientist with it. We now design algorithms that build algorithms, and eventually substrates in which such algorithms compose themselves. At that point, the scientist is no longer outside the discovery system. The scientist becomes one of the representations the system can transform. In that sense, the systems will eventually do science to us, and that is the structural consequence of the principle they are built on.

Markus J. Buehler

10,095 Aufrufe • vor 3 Monaten

Trained on zero real-world data. Learned to walk, pick up boxes, and follow multi-step instructions... in the REAL world. ( 📌 Paper below) Researchers from Amazon FAR, Berkeley, Stanford, and CMU scanned real rooms with an iPhone, rebuilt them as 3D Gaussian Splatting scenes, then generated 48,000 synthetic trajectories of a Unitree G1 walking, grasping, and placing objects inside those virtual replicas. They rendered the robot's first-person camera view from each run and paired it with the matching language instruction and motion data. That's the dataset every humanoid team needs and nobody has: synced egocentric video + language + kinematics, at scale. Instead of collecting it in the real world, they manufactured it. They trained a vision-language-kinematics policy on that synthetic data alone, then deployed it on the physical G1 across five task types: navigation to a named object, lifting boxes of three different sizes with no per-size tuning, chained multi-step tasks, robustness to mid-task layout changes and flickering lights, and multi-minute long-horizon runs. No real-world fine-tuning at any point. Real-world interaction data has been the hard limit on humanoid learning... slow, expensive, and small. If scanning a room once and synthesizing thousands of labeled interactions holds up as a general recipe, that limit moves. Data stops being the bottleneck robotics teams have to solve for. 📌 Paper: Project: ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

12,950 Aufrufe • vor 1 Monat

Agility Robotics and Boston Dynamics to Unitree: “Go back to China!” Leading US robotics companies have expressed support for the restrictions. Agility CEO Peggy Johnson issued a statement endorsing the GUARD Act, saying it “sends an important market signal as US robotics companies continue investing in resilient supply chains, domestic manufacturing capabilities, and trusted technologies.” In a recent interview with futurist Peter Diamandis, she said the Oregon-based robotics firm has been working with the Trump Administration. “We have an administration that’s very focused on the humanoid robots not going the way drones did,” she said. “We used to actually have a drone industry in the US and it’s largely gone offshore. They’re trying to prevent that same thing from happening to humanoids.” Earlier this year, Boston Dynamics sent its VP of software, Matthew Malcano, to Washington to testify before Congress about its biggest overseas rival. “Unitree is rapidly capturing market share in the US university, research, and law enforcement markets,” he said. “Researchers have reported crucial Unitree cybersecurity vulnerabilities. Unitree’s robots have also been seen in PLA military exercises and its ties to Chinese military have been noted by the House Select Committee on China as well as the Department of War.” Back in 2021, Unitree joined Boston Dynamics, Agility, ANYbotics and others in signing a pledge to not weaponize its autonomous machines. The company insists that its robots, including its new $650,000 rideable Mecha, are for civilian use only.

Mike Kalil

12,931 Aufrufe • vor 2 Monaten

Mel Gibson's physical depiction of evil in The Passion of the Christ mirrors an organism in a state of biological entropy, devoid of warmth and light. While likely an unconscious choice, Gibson’s stated goal is to represent evil as a distortion of good and order. Visually, the devil he portrays mimics the structural collapse of a biologically stressed organism. The devil's lack of eyebrows and hair is a manifestation of hypothyroidism. When the thyroid is underactive, overall energy production drops, forcing the body to forgo energy-intensive processes like hair growth and prioritize its remaining fuel for vital organs. The gaunt face reflects cortisol dominance. During periods of chronic stress, the body releases cortisol, which breaks down skin, muscle, bone, and connective tissues to provide emergency energy. This catabolic process dissolves structural proteins, leading to thinning skin and sagging, a sign of tissue wasting. Also, elevated cortisol levels forcibly redirect fat away from supportive areas like the face and deposit it in the abdomen. The devil’s pale, bloodless skin results from reduced peripheral blood flow. A hypothyroid body produces heat at a low rate, causing blood sugar levels to drop. To prevent hypothermia, the body secretes massive amounts of adrenaline, sometimes up to 30 to 40 times the normal amount (Peat, 1992). This adrenaline surge forcibly constricts blood vessels in the skin, conserving the little heat left around internal organs and leaving the skin cold and pale. The devil's androgyny is an expression of what happens to a developing organism under severe stress. A healthy, fully developed body relies on protective, differentiating hormones like progesterone and specific androgens to build structure, while a stressed organism relies on primitive emergency hormones like estrogen and cortisol (Peat, 2015). This stressed state halts proper cellular specialization and destroys the delicate hormonal ratios required for proper sexual dimorphism, pulling the organism toward an undifferentiated, amorphous, or "unisex" phenotype. The devil's behavior, especially during the manic state after Christ's resurrection, shows a reversion to the "lizard brain," or serpentine/snake-like behavior. In chronic stress, higher-order traits, such as love, creativity, empathy, cooperation, altruism, etc., are seen as expensive luxuries. When the brain lacks the ATP required to sustain these higher functions, it forcibly shuts them down to conserve fuel for basic survival. What remains is only the primitive brain stem, operating entirely on reactive survival instincts. In this low-energy, fat-oxidizing state, the devil, as shown here, becomes withdrawn, cold, angry, cancerous, and perceives the demands of the world, and the light of creation, as an active threat rather than an opportunity for expansion.

Dang

101,276 Aufrufe • vor 4 Monaten

If you think OpenAI Sora is a creative toy like DALLE, ... think again. Sora is a data-driven physics engine. It is a simulation of many worlds, real or fantastical. The simulator learns intricate rendering, "intuitive" physics, long-horizon reasoning, and semantic grounding, all by some denoising and gradient maths. I won't be surprised if Sora is trained on lots of synthetic data using Unreal Engine 5. It has to be! Let's breakdown the following video. Prompt: "Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee." - The simulator instantiates two exquisite 3D assets: pirate ships with different decorations. Sora has to solve text-to-3D implicitly in its latent space. - The 3D objects are consistently animated as they sail and avoid each other's paths. - Fluid dynamics of the coffee, even the foams that form around the ships. Fluid simulation is an entire sub-field of computer graphics, which traditionally requires very complex algorithms and equations. - Photorealism, almost like rendering with raytracing. - The simulator takes into account the small size of the cup compared to oceans, and applies tilt-shift photography to give a "minuscule" vibe. - The semantics of the scene does not exist in the real world, but the engine still implements the correct physical rules that we expect. Next up: add more modalities and conditioning, then we have a full data-driven UE that will replace all the hand-engineered graphics pipelines.

Jim Fan

6,183,111 Aufrufe • vor 2 Jahren

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,308 Aufrufe • vor 2 Jahren

UPDATE FROM THE SAPS: AN ADDITIONAL 340 ILLEGAL MINERS RESURFACE IN ORKNEY The VALA UMGODI task teams led by the SAPS and SANDF in North West are intensifying their operations and ensuring that illegal mining activities and operations are dealt a blow. From last night, an additional 340 illegal miners have resurfaced have been placed under arrest. As of 12:00 (midday) on Sunday, 03 November 2024, at least 565 illegal miners workers have resurfaced. The Acting National Commissioner of the SAPS, Lieutenant General Shadrack Sibiya has commended the teams on the ground and encouraged them not to back down and ensure that the rule of law is restored. Earlier… 225 ILLEGAL MINERS RESURFACE AS A RESULT OF PRESSURE EXERTED BY VALA UMGODI TEAMS IN NORTH WEST — ALL 225 ARRESTED North West, 02 November 2024; The Acting National Commissioner of the South African Police Service, Lieutenant General Shadrack Sibiya has commended the Vala Umgodi task teams in the North West province for stamping the authority of the state. This is as 225 illegal miners resurfaced from underground in Orkney as a result of starvation and dehydration. These 225 illegal miners are part of others believed to be hundreds if not a thousand illegal miners who are stuck underground with no food, water and necessities because the Vala Umgodi teams led by the SAPS and SANDF are blocking routes used to deliver food and necessities to these illegal miners. Just earlier this week, SAPS and members of the SANDF blocked communities in and around these abandoned mining shifts in Orkney from delivering food parcels, water and necessities to these illegal miners. This act of stamping the authority of the state eventually forced these illegal miners to resurface. This operation is ongoing and the SAPS and the SANDF are still monitoring these old abandoned mine shafts as more and more illegal miners resurface. Lieutenant General Shadrack Sibiya says Operation Vala Umgodi is yielding positive results across the country. “We are closely monitoring the situation that is unfolding in the North West province, we are not backing down until all those illegal miners resurface and are arrested . Since its inception in December 2023 to date, more than 13 691 suspects have been arrested in the seven provinces that are hotspots for illegal mining. We have seized R5million in cash and uncut diamonds worth R32 million through Operation Vala Umgodi”, said Lt Gen Sibiya. The majority of those that have been arrested are inclusive of South Africans, Mozambicans, and Basotho nationals. The SAPS will update on other nationalities as and when more illegal miners resurface.

Yusuf Abramjee

151,042 Aufrufe • vor 1 Jahr