Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

In Kafka, we have topics. Producers send data to a topic, and consumers pull data from it. Each topic has multiple partitions. You might have heard your team lead say that we can simply increase the partition count to make our cluster more scalable. There is no ordering guarantee... show more

Abhishek Singh

20,318 subscribers

16,942 Aufrufe • vor 5 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

“Can we get a new text analysis tool?” “No—we have Topic Model at home” Topic Model at home: outputs vague keywords; needs constant parameter fiddling🫠 Is there a better way? We introduce LLooM, a concept induction tool to explore text data in terms of interpretable concepts🧵

Michelle Lam

38,102 Aufrufe • vor 2 Jahren

The potential benefits of hormone replacement therapy for women have been in the bucket of things that modern medicine has underappreciated, underrecognized, and underfunded. Women need to know the real data on this topic – not dogma.

The potential benefits of hormone replacement therapy for women have been in the bucket of things that modern medicine has underappreciated, underrecognized, and underfunded. Women need to know the real data on this topic – not dogma.

Dr. Marty Makary

748,104 Aufrufe • vor 11 Monaten

On the topic of Eurovision, can we have a song in full Hebrew language? Let's represent Israel in its full glory the way it used to be:

On the topic of Eurovision, can we have a song in full Hebrew language? Let's represent Israel in its full glory the way it used to be:

Mossad Commentary

102,456 Aufrufe • vor 1 Jahr

Which doctor do you choose and why? First Impressions do count. There is data on this topic.

Which doctor do you choose and why? First Impressions do count. There is data on this topic.

Oren Gottfried, MD

27,978 Aufrufe • vor 9 Monaten

get a load of Trump's brain flitting from topic to topic as the troops stand there silently

get a load of Trump's brain flitting from topic to topic as the troops stand there silently

Aaron Rupar

62,939 Aufrufe • vor 8 Monaten

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,931 Aufrufe • vor 1 Jahr

X is working on topic-specific timelines that can be pinned to the top of your homepage

X is working on topic-specific timelines that can be pinned to the top of your homepage

Aaron

16,911 Aufrufe • vor 3 Monaten

1. Active Recall for Mastery Prompt: "I want to master [insert topic]. Act as a teacher and quiz me on this topic using active recall. Start with beginner-level questions, then gradually increase the difficulty to advanced concepts. After each question, provide the correct answer and a brief explanation. Ensure the questions cover the most important aspects of the topic."

1. Active Recall for Mastery Prompt: "I want to master [insert topic]. Act as a teacher and quiz me on this topic using active recall. Start with beginner-level questions, then gradually increase the difficulty to advanced concepts. After each question, provide the correct answer and a brief explanation. Ensure the questions cover the most important aspects of the topic."

Priyank Ahuja

48,331 Aufrufe • vor 1 Jahr

Marc Andreessen reveals the ultimate danger for entrepreneurs trying to lead like the greats. Before you try to run your startup by inserting yourself into every detail, you need to hear this reality check: "The thing you don't get is as follows, which is that... assumes you have somebody like Elon who can hold the entirety of every engineering topic and every business topic in their head all at the same time." "And so when you're sitting there with the 23-year-old engineer and you're working with them to redesign the database architecture or whatever, you actually are like qualified to do that. And that you're qualified to do that not just that one time, but EVERY time." It forces us to ask a tough question about the actual talent pool in tech right now: "This goes right back to the last topic we just talked about, which is like, OK, how many of those people exist who can possibly do that? And, you know, we know the answer is one." But here is where his perspective completely shifts... "I believe the answer is ten or a hundred or a thousand. I don't know if it's a million. I tend to think we have more of those people than we think we do."

Marc Andreessen reveals the ultimate danger for entrepreneurs trying to lead like the greats. Before you try to run your startup by inserting yourself into every detail, you need to hear this reality check: "The thing you don't get is as follows, which is that... assumes you have somebody like Elon who can hold the entirety of every engineering topic and every business topic in their head all at the same time." "And so when you're sitting there with the 23-year-old engineer and you're working with them to redesign the database architecture or whatever, you actually are like qualified to do that. And that you're qualified to do that not just that one time, but EVERY time." It forces us to ask a tough question about the actual talent pool in tech right now: "This goes right back to the last topic we just talked about, which is like, OK, how many of those people exist who can possibly do that? And, you know, we know the answer is one." But here is where his perspective completely shifts... "I believe the answer is ten or a hundred or a thousand. I don't know if it's a million. I tend to think we have more of those people than we think we do."

Ian Miles Cheong

37,516 Aufrufe • vor 2 Monaten

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

We built Open Gamma An Open Source Presentation Generator with Composio Tool Router and AI SDK - Generate Presentations in Google Slides instantly - Can research on the topic with web search - Can connect and fetch data from multiple data sources like Hubspot, Slack, Notion and more Completely free and open source link to the code:

Karan Vaidya

40,455 Aufrufe • vor 6 Monaten

Mate, you offered to "educate" me on the topic and then proceeded to run from literally every single fact when you realized how unbelievably out of your depth you were on the topic. Adding "Palestinian" to your profile doesn't actually confer any knowledge on the topic.

Destiny | Steven Bonnell II

460,366 Aufrufe • vor 2 Jahren

Secretary of State Rubio: NATO expansion hasn't been a prominent topic in recent talks with allies amid the multiple global crises but Kosovo's NATO membership cannot be ruled out in the future. The last thing we want to see is any sort of partition, division in BiH.

Secretary of State Rubio: NATO expansion hasn't been a prominent topic in recent talks with allies amid the multiple global crises but Kosovo's NATO membership cannot be ruled out in the future. The last thing we want to see is any sort of partition, division in BiH.

Admirim

21,622 Aufrufe • vor 1 Monat

Giorgia Meloni returns to the topic of immigration, saying that uncontrolled immigration devastates societies. Fine words, but we prefer the data. From October 2022 to today, 400,000 illegal immigrants have invaded Italy. Grok how many have been deported in the same period?

Giorgia Meloni returns to the topic of immigration, saying that uncontrolled immigration devastates societies. Fine words, but we prefer the data. From October 2022 to today, 400,000 illegal immigrants have invaded Italy. Grok how many have been deported in the same period?

RadioGenoa

99,009 Aufrufe • vor 2 Monaten

It’s awesome to have the Flyers back as a topic of conversation in Philly The Union had passed them for a while but this playoff run has made it close again My thoughts on the playoff push:

It’s awesome to have the Flyers back as a topic of conversation in Philly The Union had passed them for a while but this playoff run has made it close again My thoughts on the playoff push:

Eliot Shorr-Parks

35,034 Aufrufe • vor 3 Monaten

Finland President Alexander Stubb: Well, we have to be more independent in some cases. If you, for example, look at storage of personal data, obviously a lot of Europeans are angry that their personal data is stored outside of the EU. So we have to build our own data centers and our own AI factories so we can actually own the data of our own citizens. But in the end, there is no such thing as European full autonomy. You will always need to collaborate with our friends in Canada, in the US, in other partnerships around the world. ASML and other tech companies can play a crucial role both in the US and here.

Finland President Alexander Stubb: Well, we have to be more independent in some cases. If you, for example, look at storage of personal data, obviously a lot of Europeans are angry that their personal data is stored outside of the EU. So we have to build our own data centers and our own AI factories so we can actually own the data of our own citizens. But in the end, there is no such thing as European full autonomy. You will always need to collaborate with our friends in Canada, in the US, in other partnerships around the world. ASML and other tech companies can play a crucial role both in the US and here.

Clash Report

20,444 Aufrufe • vor 1 Tag

So…this is wild. In five seconds, you can create a fully linked wikipedia page on any topic in the entire universe. Perplexity just dropped a new feature to Pro users called “Perplexity Pages”. Type in a topic, it automatically creates a webpage for it, with links and sources, you can publish it, share the link around, other people can access it, and other people can ASK FOLLOW UP QUESTIONS to the page. View count and followup count are public. Bit strange that these models are trained on wikipedia though… Here’s the one I made, check it out:

So…this is wild. In five seconds, you can create a fully linked wikipedia page on any topic in the entire universe. Perplexity just dropped a new feature to Pro users called “Perplexity Pages”. Type in a topic, it automatically creates a webpage for it, with links and sources, you can publish it, share the link around, other people can access it, and other people can ASK FOLLOW UP QUESTIONS to the page. View count and followup count are public. Bit strange that these models are trained on wikipedia though… Here’s the one I made, check it out:

Allie K. Miller

58,898 Aufrufe • vor 2 Jahren

My old friend has long wanted to draw a mini comic with me on the topic of tentacles, but he never managed to. My friend Captain Tanuki made art on this topic, in a dark-metal style. I simply couldn't make a T-shirt with this art. ) Artist - Captain 'T' Tanuki

My old friend has long wanted to draw a mini comic with me on the topic of tentacles, but he never managed to. My friend Captain Tanuki made art on this topic, in a dark-metal style. I simply couldn't make a T-shirt with this art. ) Artist - Captain 'T' Tanuki

Swollen-Ksenia 💣

14,026 Aufrufe • vor 1 Jahr

New Learn Anything version is out 🎇 Just a start but already you can track any topic you like. There is 1,000+ topics already available with topics including Rust, ML, Nix and more. Track your learning journey fully. Any topic. Any link.

New Learn Anything version is out 🎇 Just a start but already you can track any topic you like. There is 1,000+ topics already available with topics including Rust, ML, Nix and more. Track your learning journey fully. Any topic. Any link.

nikiv.dev

42,055 Aufrufe • vor 2 Jahren

Something amazing is coming to Apache Kafka… Consumer Groups v2! If you’ve ever used consumer groups in production at any non-trivial scale, you probably know all the problems with it: - ⛔️ Group-wide synchronization barrier acts as a cap on scalability A single misbehaving consumer can disturb the whole group. Even if you have cooperative rebalancing and static membership enabled, you will still have rebalances happen. 🤷‍♂️ It’s a fact of life I've heard - death, taxes & consumer group rebalances. And the problem is that even with cooperative rebalancing (which helped a lot!), you’re bound on waiting for the slowest member of the group to complete the rebalance(s). 🐌 The problem is that no consumer can commit offsets while a rebalance is in progress. ❌ Another subtle thing is that with cooperative rebalancing, a rebalance will take longer than usual. Why? Because consumers are allowed to process partitions during rebalances. They will call the poll() method more infrequently -- they're busy processing records after all. Thus, the overall rebalance time will increase. This makes it pretty hard to scale to 1000s of members. - 🤯 Complexity There’s a reason you’re reading this! The current protocol is pretty complex and hard to understand. It's used for a bunch of stuff, including metadata propagation in Kafka Streams. This compexity results in more: - 🐛 bugs The harder to reason about and the more moving parts you have - the greater chance for bugs. There have been quite a few in the protocol. And due to the fact that a lot of the protocol’s logic lives on the clients, that results in: - 🐌 slow fixes Bugs require client-side fixes, which are slow to be adopted. If you run a Kafka ops team, you know how hard it is to get all of your clients' teams to upgrade! If you're using a cloud service, you need to wait for a new Kafka release to go out. Can't have the cloud provider handle it for you behind the scenes! - 🔍 hard to debug Debugging is harder because you need client logs. In the cloud, that's hard to do again. On-prem, it requires reading through a lot of logs and collecting a lot of files. - ⚙️ very extendable There’s a reusable embedded protocol within the rebalance protocol, where clients are free to attach raw bytes that only they can then parse themselves. It's challenging to build compatible software for this cross-client protocol, as well as near-impossible to inspect from the broker side. - 😢 inconsistent metadata Clients are responsible for triggering rebalances based on the metadata, but different clients can have different views of the metadata. - 😵‍💫 interoperability Different implementations of the clients (i.e. anything besides the Java client) may have bugs. The complex logic needs to be re-implemented quite a few times. This usually means more bugs and slower time to ship features in your favorite client. A combination nobody likes. ... So what should an open source community do? Move the logic to the broker! Then? Simplify it. The new protocol is very elegant - it streamlines all of the regular Kafka consumer logic inside a new heartbeat API. It has the broker decide what partition assignments the consumers should have, and totally omits the notion of a Group Leader client. Another major change is that the notion of a group-wide rebalance is removed now. The rebalance is more fine-grained now. 👌 When you think about it, a rebalance is simply a reassignment of some partitions from some consumers to others. 💡 Why does the whole group need to stop and know about this? It had to before because the logic was on the client. It doesn’t now. 🎂 The new protocol is fine-grained in its assignments. It maintains per-member epochs, as well as separate epochs for the general group membership and the global assignment. The goal is simple - get all of those epoch numbers to be the same. The order is the following: 1. the group-wide epoch is bumped. 2. the target assignment epoch is bumped. 3. individual consumers catch up to the epoch via the heartbeat request, individually. (fine-grained) In general, what you have is a simple state machine inside the Group Coordinator broker that’s running a constant reconciliation loop. 💥 Because every member converges to the target state independently, the coordinator is free to simplify that convergence member by member. 👍 It has the logic to resolve dependencies between members - the act of: 1. revoking one member’s partition. 2. confirming the success of that. 3. bumping that member's epoch before assigning that partition to another consumer. Here is an example visual of what happens when two members join a consumer group one by one:

Something amazing is coming to Apache Kafka… Consumer Groups v2! If you’ve ever used consumer groups in production at any non-trivial scale, you probably know all the problems with it: - ⛔️ Group-wide synchronization barrier acts as a cap on scalability A single misbehaving consumer can disturb the whole group. Even if you have cooperative rebalancing and static membership enabled, you will still have rebalances happen. 🤷‍♂️ It’s a fact of life I've heard - death, taxes & consumer group rebalances. And the problem is that even with cooperative rebalancing (which helped a lot!), you’re bound on waiting for the slowest member of the group to complete the rebalance(s). 🐌 The problem is that no consumer can commit offsets while a rebalance is in progress. ❌ Another subtle thing is that with cooperative rebalancing, a rebalance will take longer than usual. Why? Because consumers are allowed to process partitions during rebalances. They will call the poll() method more infrequently -- they're busy processing records after all. Thus, the overall rebalance time will increase. This makes it pretty hard to scale to 1000s of members. - 🤯 Complexity There’s a reason you’re reading this! The current protocol is pretty complex and hard to understand. It's used for a bunch of stuff, including metadata propagation in Kafka Streams. This compexity results in more: - 🐛 bugs The harder to reason about and the more moving parts you have - the greater chance for bugs. There have been quite a few in the protocol. And due to the fact that a lot of the protocol’s logic lives on the clients, that results in: - 🐌 slow fixes Bugs require client-side fixes, which are slow to be adopted. If you run a Kafka ops team, you know how hard it is to get all of your clients' teams to upgrade! If you're using a cloud service, you need to wait for a new Kafka release to go out. Can't have the cloud provider handle it for you behind the scenes! - 🔍 hard to debug Debugging is harder because you need client logs. In the cloud, that's hard to do again. On-prem, it requires reading through a lot of logs and collecting a lot of files. - ⚙️ very extendable There’s a reusable embedded protocol within the rebalance protocol, where clients are free to attach raw bytes that only they can then parse themselves. It's challenging to build compatible software for this cross-client protocol, as well as near-impossible to inspect from the broker side. - 😢 inconsistent metadata Clients are responsible for triggering rebalances based on the metadata, but different clients can have different views of the metadata. - 😵‍💫 interoperability Different implementations of the clients (i.e. anything besides the Java client) may have bugs. The complex logic needs to be re-implemented quite a few times. This usually means more bugs and slower time to ship features in your favorite client. A combination nobody likes. ... So what should an open source community do? Move the logic to the broker! Then? Simplify it. The new protocol is very elegant - it streamlines all of the regular Kafka consumer logic inside a new heartbeat API. It has the broker decide what partition assignments the consumers should have, and totally omits the notion of a Group Leader client. Another major change is that the notion of a group-wide rebalance is removed now. The rebalance is more fine-grained now. 👌 When you think about it, a rebalance is simply a reassignment of some partitions from some consumers to others. 💡 Why does the whole group need to stop and know about this? It had to before because the logic was on the client. It doesn’t now. 🎂 The new protocol is fine-grained in its assignments. It maintains per-member epochs, as well as separate epochs for the general group membership and the global assignment. The goal is simple - get all of those epoch numbers to be the same. The order is the following: 1. the group-wide epoch is bumped. 2. the target assignment epoch is bumped. 3. individual consumers catch up to the epoch via the heartbeat request, individually. (fine-grained) In general, what you have is a simple state machine inside the Group Coordinator broker that’s running a constant reconciliation loop. 💥 Because every member converges to the target state independently, the coordinator is free to simplify that convergence member by member. 👍 It has the logic to resolve dependencies between members - the act of: 1. revoking one member’s partition. 2. confirming the success of that. 3. bumping that member's epoch before assigning that partition to another consumer. Here is an example visual of what happens when two members join a consumer group one by one:

Stanislav Kozlovski

52,814 Aufrufe • vor 3 Jahren

No matter who we are, sometimes we all have moments where we struggle with our mental health. This Movember we can all support each other’s wellbeing. Not sure how to approach the topic? Try this free practice tool at Check out this video from our community and share your own tips with the hashtag #MovemberSuperfest Clash Royale Clash of Clans Brawl Stars

No matter who we are, sometimes we all have moments where we struggle with our mental health. This Movember we can all support each other’s wellbeing. Not sure how to approach the topic? Try this free practice tool at Check out this video from our community and share your own tips with the hashtag #MovemberSuperfest Clash Royale Clash of Clans Brawl Stars

Supercell

619,035 Aufrufe • vor 1 Jahr