Loading video...

Video Failed to Load

Go Home

In Kafka, we have topics. Producers send data to a topic, and consumers pull data from it. Each topic has multiple partitions. You might have heard your team lead say that we can simply increase the partition count to make our cluster more scalable. There is no ordering guarantee...

0xlelouch_'s profile picture

Abhishek Singh

20,318 subscribers

16,942 views • 4 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵
0:52

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

Michelle Lam

38,067 views • 2 years ago

The potential benefits of hormone replacement therapy for women have been in the bucket of things that modern medicine has underappreciated, underrecognized, and underfunded. Women need to know the real data on this topic – not dogma.
1:23

The potential benefits of hormone replacement therapy for women have been in the bucket of things that modern medicine has underappreciated, underrecognized, and underfunded. Women need to know the real data on this topic – not dogma.

Dr. Marty Makary

748,072 views • 11 months ago

Which doctor do you choose and why? First Impressions do count. There is data on this topic.
0:28

Which doctor do you choose and why? First Impressions do count. There is data on this topic.

Oren Gottfried, MD

27,978 views • 8 months ago

On the topic of Eurovision, can we have a song in full Hebrew language? Let's represent Israel in its full glory the way it used to be:
5:29

On the topic of Eurovision, can we have a song in full Hebrew language? Let's represent Israel in its full glory the way it used to be:

Mossad Commentary

102,456 views • 1 year ago

get a load of Trump's brain flitting from topic to topic as the troops stand there silently
1:09

get a load of Trump's brain flitting from topic to topic as the troops stand there silently

Aaron Rupar

62,939 views • 7 months ago

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!
1:00

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,925 views • 1 year ago

X is working on topic-specific timelines that can be pinned to the top of your homepage
0:44

X is working on topic-specific timelines that can be pinned to the top of your homepage

Aaron

14,386 views • 2 months ago

Mate, you offered to "educate" me on the topic and then proceeded to run from literally every single fact when you realized how unbelievably out of your depth you were on the topic. Adding "Palestinian" to your profile doesn't actually confer any knowledge on the topic.
1:09

Mate, you offered to "educate" me on the topic and then proceeded to run from literally every single fact when you realized how unbelievably out of your depth you were on the topic. Adding "Palestinian" to your profile doesn't actually confer any knowledge on the topic.

Destiny | Steven Bonnell II

460,353 views • 2 years ago

1. Active Recall for Mastery Prompt: "I want to master [insert topic]. Act as a teacher and quiz me on this topic using active recall. Start with beginner-level questions, then gradually increase the difficulty to advanced concepts. After each question, provide the correct answer and a brief explanation. Ensure the questions cover the most important aspects of the topic."
0:31

1. Active Recall for Mastery Prompt: "I want to master [insert topic]. Act as a teacher and quiz me on this topic using active recall. Start with beginner-level questions, then gradually increase the difficulty to advanced concepts. After each question, provide the correct answer and a brief explanation. Ensure the questions cover the most important aspects of the topic."

Priyank Ahuja

48,331 views • 11 months ago

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:
1:00

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 views • 6 months ago

Secretary of State Rubio: NATO expansion hasn't been a prominent topic in recent talks with allies amid the multiple global crises but Kosovo's NATO membership cannot be ruled out in the future. The last thing we want to see is any sort of partition, division in BiH.
2:47

Secretary of State Rubio: NATO expansion hasn't been a prominent topic in recent talks with allies amid the multiple global crises but Kosovo's NATO membership cannot be ruled out in the future. The last thing we want to see is any sort of partition, division in BiH.

Admirim

21,622 views • 15 days ago

Giorgia Meloni returns to the topic of immigration, saying that uncontrolled immigration devastates societies. Fine words, but we prefer the data. From October 2022 to today, 400,000 illegal immigrants have invaded Italy. @Grok how many have been deported in the same period?
1:50

Giorgia Meloni returns to the topic of immigration, saying that uncontrolled immigration devastates societies. Fine words, but we prefer the data. From October 2022 to today, 400,000 illegal immigrants have invaded Italy. @Grok how many have been deported in the same period?

RadioGenoa

98,963 views • 1 month ago

My old friend has long wanted to draw a mini comic with me on the topic of tentacles, but he never managed to. My friend Captain Tanuki made art on this topic, in a dark-metal style. I simply couldn't make a T-shirt with this art. ) Artist - Captain 'T' Tanuki
0:47

My old friend has long wanted to draw a mini comic with me on the topic of tentacles, but he never managed to. My friend Captain Tanuki made art on this topic, in a dark-metal style. I simply couldn't make a T-shirt with this art. ) Artist - Captain 'T' Tanuki

Swollen-Ksenia 💣

14,026 views • 1 year ago

It’s awesome to have the Flyers back as a topic of conversation in Philly The Union had passed them for a while but this playoff run has made it close again My thoughts on the playoff push:
6:56

It’s awesome to have the Flyers back as a topic of conversation in Philly The Union had passed them for a while but this playoff run has made it close again My thoughts on the playoff push:

Eliot Shorr-Parks

35,034 views • 2 months ago

New Learn Anything version is out 🎇 Just a start but already you can track any topic you like. There is 1,000+ topics already available with topics including Rust, ML, Nix and more. Track your learning journey fully. Any topic. Any link.
0:25

New Learn Anything version is out 🎇 Just a start but already you can track any topic you like. There is 1,000+ topics already available with topics including Rust, ML, Nix and more. Track your learning journey fully. Any topic. Any link.

nikiv.dev

42,016 views • 2 years ago

So…this is wild. In five seconds, you can create a fully linked wikipedia page on any topic in the entire universe. Perplexity just dropped a new feature to Pro users called “Perplexity Pages”. Type in a topic, it automatically creates a webpage for it, with links and sources, you can publish it, share the link around, other people can access it, and other people can ASK FOLLOW UP QUESTIONS to the page. View count and followup count are public. Bit strange that these models are trained on wikipedia though… Here’s the one I made, check it out:
0:30

So…this is wild. In five seconds, you can create a fully linked wikipedia page on any topic in the entire universe. Perplexity just dropped a new feature to Pro users called “Perplexity Pages”. Type in a topic, it automatically creates a webpage for it, with links and sources, you can publish it, share the link around, other people can access it, and other people can ASK FOLLOW UP QUESTIONS to the page. View count and followup count are public. Bit strange that these models are trained on wikipedia though… Here’s the one I made, check it out:

Allie K. Miller

58,898 views • 2 years ago

Something amazing is coming to Apache Kafka… Consumer Groups v2! If you’ve ever used consumer groups in production at any non-trivial scale, you probably know all the problems with it: - ⛔️ Group-wide synchronization barrier acts as a cap on scalability A single misbehaving consumer can disturb the whole group. Even if you have cooperative rebalancing and static membership enabled, you will still have rebalances happen. 🤷‍♂️ It’s a fact of life I've heard - death, taxes & consumer group rebalances. And the problem is that even with cooperative rebalancing (which helped a lot!), you’re bound on waiting for the slowest member of the group to complete the rebalance(s). 🐌 The problem is that no consumer can commit offsets while a rebalance is in progress. ❌ Another subtle thing is that with cooperative rebalancing, a rebalance will take longer than usual. Why? Because consumers are allowed to process partitions during rebalances. They will call the poll() method more infrequently -- they're busy processing records after all. Thus, the overall rebalance time will increase. This makes it pretty hard to scale to 1000s of members. - 🤯 Complexity There’s a reason you’re reading this! The current protocol is pretty complex and hard to understand. It's used for a bunch of stuff, including metadata propagation in Kafka Streams. This compexity results in more: - 🐛 bugs The harder to reason about and the more moving parts you have - the greater chance for bugs. There have been quite a few in the protocol. And due to the fact that a lot of the protocol’s logic lives on the clients, that results in: - 🐌 slow fixes Bugs require client-side fixes, which are slow to be adopted. If you run a Kafka ops team, you know how hard it is to get all of your clients' teams to upgrade! If you're using a cloud service, you need to wait for a new Kafka release to go out. Can't have the cloud provider handle it for you behind the scenes! - 🔍 hard to debug Debugging is harder because you need client logs. In the cloud, that's hard to do again. On-prem, it requires reading through a lot of logs and collecting a lot of files. - ⚙️ very extendable There’s a reusable embedded protocol within the rebalance protocol, where clients are free to attach raw bytes that only they can then parse themselves. It's challenging to build compatible software for this cross-client protocol, as well as near-impossible to inspect from the broker side. - 😢 inconsistent metadata Clients are responsible for triggering rebalances based on the metadata, but different clients can have different views of the metadata. - 😵‍💫 interoperability Different implementations of the clients (i.e. anything besides the Java client) may have bugs. The complex logic needs to be re-implemented quite a few times. This usually means more bugs and slower time to ship features in your favorite client. A combination nobody likes. ... So what should an open source community do? Move the logic to the broker! Then? Simplify it. The new protocol is very elegant - it streamlines all of the regular Kafka consumer logic inside a new heartbeat API. It has the broker decide what partition assignments the consumers should have, and totally omits the notion of a Group Leader client. Another major change is that the notion of a group-wide rebalance is removed now. The rebalance is more fine-grained now. 👌 When you think about it, a rebalance is simply a reassignment of some partitions from some consumers to others. 💡 Why does the whole group need to stop and know about this? It had to before because the logic was on the client. It doesn’t now. 🎂 The new protocol is fine-grained in its assignments. It maintains per-member epochs, as well as separate epochs for the general group membership and the global assignment. The goal is simple - get all of those epoch numbers to be the same. The order is the following: 1. the group-wide epoch is bumped. 2. the target assignment epoch is bumped. 3. individual consumers catch up to the epoch via the heartbeat request, individually. (fine-grained) In general, what you have is a simple state machine inside the Group Coordinator broker that’s running a constant reconciliation loop. 💥 Because every member converges to the target state independently, the coordinator is free to simplify that convergence member by member. 👍 It has the logic to resolve dependencies between members - the act of: 1. revoking one member’s partition. 2. confirming the success of that. 3. bumping that member's epoch before assigning that partition to another consumer. Here is an example visual of what happens when two members join a consumer group one by one:
1:37

Something amazing is coming to Apache Kafka… Consumer Groups v2! If you’ve ever used consumer groups in production at any non-trivial scale, you probably know all the problems with it: - ⛔️ Group-wide synchronization barrier acts as a cap on scalability A single misbehaving consumer can disturb the whole group. Even if you have cooperative rebalancing and static membership enabled, you will still have rebalances happen. 🤷‍♂️ It’s a fact of life I've heard - death, taxes & consumer group rebalances. And the problem is that even with cooperative rebalancing (which helped a lot!), you’re bound on waiting for the slowest member of the group to complete the rebalance(s). 🐌 The problem is that no consumer can commit offsets while a rebalance is in progress. ❌ Another subtle thing is that with cooperative rebalancing, a rebalance will take longer than usual. Why? Because consumers are allowed to process partitions during rebalances. They will call the poll() method more infrequently -- they're busy processing records after all. Thus, the overall rebalance time will increase. This makes it pretty hard to scale to 1000s of members. - 🤯 Complexity There’s a reason you’re reading this! The current protocol is pretty complex and hard to understand. It's used for a bunch of stuff, including metadata propagation in Kafka Streams. This compexity results in more: - 🐛 bugs The harder to reason about and the more moving parts you have - the greater chance for bugs. There have been quite a few in the protocol. And due to the fact that a lot of the protocol’s logic lives on the clients, that results in: - 🐌 slow fixes Bugs require client-side fixes, which are slow to be adopted. If you run a Kafka ops team, you know how hard it is to get all of your clients' teams to upgrade! If you're using a cloud service, you need to wait for a new Kafka release to go out. Can't have the cloud provider handle it for you behind the scenes! - 🔍 hard to debug Debugging is harder because you need client logs. In the cloud, that's hard to do again. On-prem, it requires reading through a lot of logs and collecting a lot of files. - ⚙️ very extendable There’s a reusable embedded protocol within the rebalance protocol, where clients are free to attach raw bytes that only they can then parse themselves. It's challenging to build compatible software for this cross-client protocol, as well as near-impossible to inspect from the broker side. - 😢 inconsistent metadata Clients are responsible for triggering rebalances based on the metadata, but different clients can have different views of the metadata. - 😵‍💫 interoperability Different implementations of the clients (i.e. anything besides the Java client) may have bugs. The complex logic needs to be re-implemented quite a few times. This usually means more bugs and slower time to ship features in your favorite client. A combination nobody likes. ... So what should an open source community do? Move the logic to the broker! Then? Simplify it. The new protocol is very elegant - it streamlines all of the regular Kafka consumer logic inside a new heartbeat API. It has the broker decide what partition assignments the consumers should have, and totally omits the notion of a Group Leader client. Another major change is that the notion of a group-wide rebalance is removed now. The rebalance is more fine-grained now. 👌 When you think about it, a rebalance is simply a reassignment of some partitions from some consumers to others. 💡 Why does the whole group need to stop and know about this? It had to before because the logic was on the client. It doesn’t now. 🎂 The new protocol is fine-grained in its assignments. It maintains per-member epochs, as well as separate epochs for the general group membership and the global assignment. The goal is simple - get all of those epoch numbers to be the same. The order is the following: 1. the group-wide epoch is bumped. 2. the target assignment epoch is bumped. 3. individual consumers catch up to the epoch via the heartbeat request, individually. (fine-grained) In general, what you have is a simple state machine inside the Group Coordinator broker that’s running a constant reconciliation loop. 💥 Because every member converges to the target state independently, the coordinator is free to simplify that convergence member by member. 👍 It has the logic to resolve dependencies between members - the act of: 1. revoking one member’s partition. 2. confirming the success of that. 3. bumping that member's epoch before assigning that partition to another consumer. Here is an example visual of what happens when two members join a consumer group one by one:

Stanislav Kozlovski

52,814 views • 2 years ago

No matter who we are, sometimes we all have moments where we struggle with our mental health. This Movember we can all support each other’s wellbeing. Not sure how to approach the topic? Try this free practice tool at Check out this video from our community and share your own tips with the hashtag #MovemberSuperfest Clash Royale Clash of Clans Brawl Stars
2:28

No matter who we are, sometimes we all have moments where we struggle with our mental health. This Movember we can all support each other’s wellbeing. Not sure how to approach the topic? Try this free practice tool at Check out this video from our community and share your own tips with the hashtag #MovemberSuperfest Clash Royale Clash of Clans Brawl Stars

Supercell

618,923 views • 1 year ago

Putin: 'Denmark has ALWAYS treated Greenland as a COLONY' 'Has treated it quite HARSHLY, but that is a whole other topic' Confirms: 'this is CLEARLY none of our business — I think they'll be able to deal with this on their own'
0:43

Putin: 'Denmark has ALWAYS treated Greenland as a COLONY' 'Has treated it quite HARSHLY, but that is a whole other topic' Confirms: 'this is CLEARLY none of our business — I think they'll be able to deal with this on their own'

RT

26,950 views • 4 months ago

Our latest video is from Guest Editor alex wurz, former F1 driver and 2x Le Mans winner! Alex’s topic is what to do when your team-mate might have a better car than you, a particularly important topic for young drivers in spec series. FULL VIDEO 📺
0:25

Our latest video is from Guest Editor alex wurz, former F1 driver and 2x Le Mans winner! Alex’s topic is what to do when your team-mate might have a better car than you, a particularly important topic for young drivers in spec series. FULL VIDEO 📺

SAFEisFAST

10,914 views • 1 year ago