Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Day 9/90 of Inference Engineering I built a visualization tool for CUDA kernels and C code to view the buffers for each step through. Check out Yesterday, I was terramogged by a simple implementation of a basic matmul between three matrices in 1D vector form. There were essentially three...

12,564 Aufrufe • vor 22 Tagen •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)

Lovis Odin

58,769 Aufrufe • vor 10 Monaten

Pi was built when there were already agent harnesses around. Here’s why Mario Zechner(Mario Zechner), found them suboptimal and built Pi, a minimalist self-modifying agent: #1 - Mario initially was a believer in Claude Code: "I was a believer in Claude code because they were the first that packaged agentic search up in a really compelling package. And at the time that fit my workflow really well. Everything around the LLM was kind of nice and tidy and easy to understand. I was super happy. I was proselytising Claude code." #2 - Reverse engineering Claude Code highlighted the degradation that Mario felt as a user: "I personally like simple tools that are stable and that I can rely on. Even if they have non-deterministic parts, all the deterministic parts should be as stable as possible. That was just not the experience with Claude Code around summer 2025. They would take away your control of the context. They would inject stuff behind your back, which is bad. Then, your workflows stopped working because there's now a system reminder that you don't even see in the UI that would modify the behaviour of the model. They would also do this to the system prompt. I built a little service where I can track the progression or evolution of the system, prompt and tool definitions and, with every release, it was messing with stuff. That just messed with my workflows and I don't appreciate that." #3 - PI was built with an appreciation for simple and reliable tools: "If I commit to a development tool, I want it to be a stable, reliable thing like a hammer. I don't want my hammer to break a different spot every day. That's terrible. We need somebody who goes the full velocity kind of way. But I don't want to work with a tool like that."

The Pragmatic Engineer

62,825 Aufrufe • vor 3 Monaten

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,858 Aufrufe • vor 2 Jahren

Scottie Scheffler explains why playing too many events in a row can mentally drain players and referred to Tiger Woods always playing a limited schedule: “I think when you look at this sport it's unique in a sense of like the amount of time we spend in front of people throughout a week. Like for me to play four days of tournament golf, from the moment I step out on to the driving range if we're playing at a four hour pace that's over five hours, five and a half hours if you include this part of my day, where we're in front of people. And doing that four days in a row in a mental sport I think can take a toll on you. “I think that's why you see a guy like Tiger can only play so many events because I think mentally the challenge of just being in front of people for that long and being on and what it takes in toward to compete week in week out takes a lot out of you mentally. So for me playing more than three weeks in a row is extraordinarily difficult. I can't handle much more than that. “When you look at the season now I think it, for family time it's great to have an off-season as well, just for us for me to be able to be at home with my family is important. As far as the cadence goes, yeah, I definitely have a cadence that I prefer when it comes to scheduling and there are definitely risks cramming a bunch of tournaments into little windows just because it's hard to play that many tournaments in a row. It's hard to be on for that many times. It's hard to get yelled at that many times in a row throughout a round of golf. It just is. That's why for me three weeks in a row is kind of my max.” PGA TOUR TravelersChamp

Flushing It

447,887 Aufrufe • vor 1 Monat

Eric Rohmer on the use of Colour in "La Collectionneuse" (1967) and "Claire's Knee" (1970): "I didn't use color as a dramatic element, as some filmmakers have done. For me it's something inherent in the film as a whole. I think that in 'La Collectionneuse' (1967) color above all heightens the sense of reality and increases the immediacy of the settings. In this film color acts in an indirect way; it's not direct and there aren't any color effects, as there are for example in Bergman's most recent film, his second one in color, where the color is very deliberately worked out and he gets his effects mainly by the way he uses red. I've never tried for dramatic effects of this kind, but. for example, the sense of time-evening, morning, and so on-can be rendered in a much more precise way through color. Color can also give a stronger sense of warmth, of heat, for when the film is in black-and-white you get less of a feeling of the different moments of the day, and there is less of what you might call a tactile impression about it. In 'Claire's Knee' (1970), I think it works in the same way: the presence of the lake and the mountains is stronger in color than in black-and-white. It's a film I couldn't imagine in black-and-white. The color green seems to me essential in that film, I couldn't imagine it without the green in it. And the blue too-the cold color as a whole. This film mould have no value for me in black-and-white. It's a very difficult thing to explain. It's more a feeling I have that can't be reasoned out logically." (Eric Rohmer's interview with Graham Petrie, Film Quarterly, 1971)

DepressedBergman

61,555 Aufrufe • vor 1 Jahr

YOKO ONO: ONOCHORD, VENICE, 2004 Yoko: The world is divided in two industries. One is the War Industry and the other is the Peace Industry. The people in the War Industry are totally together. They don't have to talk to each other, even. They know exactly what they want to do. They want to go out there, kill and make money. But the people in the Peace Industry, which are us - we are so idealistic that each one of us criticises the other Peace Person in the Peace Industry. And we are always just arguing and we are wasting our energies doing that. So let's just forgive each other and see that we are in the Peace Industry and that's all that counts. Even if you are not marching for peace, just be yourself, being a florist, being a merchant, being a talior, anything. That way you're contributing to the Peace Industry. People are just concentrating on fear, confusion and anger. And therefore just for a moment, I'd like us to think about Love. In a very magical, straight way, John and I met in London and from then on we stood for Peace and Love. And when I do this kind of event. Well it is... I was inspired to do it, but I still think that I'm still with John in spirit. John and I created the country called Nutopia. Not Utopia, because there was Utopia as a concept already. And we wanted to create a new concept, so we just added N on it - Nutopia - and as a country. Well, that is the concept of a country. And we all are citizens of that country. And in my apartment in the Dakota Building, we put a little plaque on the back door, the kitchen door. It says 'Nutopian Embassy' and even now we have that. (laughs). Nutopia exists in our minds. And because of that, some people want to rebel against it. The reason some want to rebel against it is a good proof that it exists. I think that it was a terrible thing that happened in Chechnya. But we have to still keep our hopes up. And instead of giving up, we have to keep on sending the message of Love to each other. You say that I am the Ambassador of Peace. We are all Ambassadors of Peace. You are too. Everybody in this room are Ambassadors of Peace. Just the fact that we are not participating in War. The fact that we are here, and we are what we are, means that we are in the Peace Industry. All of us. John and I used to say that our apartment in the Dakota is a conceptual monastry, just for the two of us. And when we go out of the Dakota, we get so many people communicating with us, so it's very important that we had silence and quietness. And my apartment is a very small space compared to the world. And I need that for my peace of mind. You should be kind to each other. You should come together, hug each other, love each other, express our love to each other and we should make it work. We should finally create a world that is a totally an Earth for Us. So let's do it. Yoko Ono, OpenAsia Press Conference, whilst exhibiting Onochord, 2004 by Yoko Ono (Nutopia) at the Venice Biennale: OpenAsia 2004, Lido Di Venezia, Venice, Italy, 9 September 2004.

Yoko Ono

35,208 Aufrufe • vor 2 Jahren

AI is changing the software engineering craft. Anders Hejlsberg (Anders Hejlsberg) - creator of C#, TypeScript and industry legend - on why code review needs to get more enjoyable in response: #1 - AI is shifting the craft from writing code, to reviewing code: "In a sense, we're all turning into project managers. We can have an army of junior programmers, called agents, that will just spit out reams of code but someone's got to have the big picture and review all of that. And so, increasingly, our craft is going from one of writing the code, to one of reviewing the code and building the architecture of the code and overseeing the work. It's a different kind of craft. It's a different kind of enjoyment. I've always liked writing the code. To me that was the fulfilling part, seeing it work. In a way, AI robs a little bit of that, because I am less interested in reviewing code." #2 - The code review experience should be improved: "I think we could also make the process of reviewing code much more interesting than it is today. I mean, today, you see a list of diffs in alphabetical order and now it's up to you to make heads or tails of it. There are more pedagogical ways of presenting that. And you could have commentary generated by the AI that tells you what the changes are and whatever, and then tries to guide you along. So that symbiotic relationship, I think we need to work on that more and to keep the enjoyment in there."

The Pragmatic Engineer

39,011 Aufrufe • vor 2 Monaten

On this day, 52 years ago, John Cassavetes' "The Ki!!ing of a Chinese Bookie" (1974) was released in the USA. John Cassavetes explaining why he made the movie: "'A Woman Under the Influence' (1974) was the first picture I’ve had anything to do with that wasn’t made out of plain, simple feeling, but rather out of a real desire to do something in my profession. It was extremely frightening for me not to come to work out of enthusiasm and instead put myself up as something of a craftsman. Earlier films such as 'Shadows' (1958) and 'Husbands' (1970) grew out of personal experiences reaching all the way back to my childhood days. They were expressions of my innermost feelings, and now that I’ve dealt with all that, I feel obligated to view life in other terms. I want to explore other areas of human and artistic experience. I made 'The Ki!!ing of a Chinese Bookie' (1976) as an intellectual experiment– not because I am in love with it. I enjoy a more intellectual and less emotionally demanding view than in my previous work. If I can make, out of certain intellectual ideas, films that are complex in their nature, then I’m entering into new ground. And that is certainly something I look forward to. It is a film that has little to do with me and with how I feel about life. It’s interesting to me to see how other people live in our society, to look at them and ask myself, ‘Why do they do it? And how do they do it?’ Without trying to explain. The fun and challenge of the film was to imagine a self-contained world different from the one I live in: to move into it and live in it." ("Cassavetes on Cassavetes", edited by Ray Carney, 2001)

DepressedBergman

86,531 Aufrufe • vor 5 Monaten