正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Consistent Hashing Explained Simply Let's break down consistent hashing in a way that's easy to get. It's a handy technique for distributing data evenly across servers. And if you're more of a visual learner, don't miss the attached video which brings this concept to life. Here's how it works:... We don't just hash our data keys; we also hash the server names using the same method. This puts everything on a scale, known as the hash space, which we think of as a big circular ring. So, imagine we've got a few servers. We hash them and place them on this ring. We do the same with our data, hashing it by key. To figure out where to store a piece of data, we travel clockwise on the ring from where the data lands until we bump into a server. Like, data key 0 might end up on server 0, and key 1 on server 1. Adding a new server? Only the data near the new server needs to shift. And if a server has to go? Only the data it was holding needs a new spot. But there's a little challenge: sometimes, data doesn't spread out evenly. The fix? Virtual nodes. They're like mini-replicas of a server placed around the ring. The more virtual nodes we have, the better the balance. But there's a catch: more virtual nodes mean more memory usage for tracking them, and it can make the whole system a bit more complex to manage. – Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages):show more

Sahn Lam

48,860 subscribers

36,787 次观看 • 2 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

4 条评论

Riccardo Causo 的头像

Riccardo Causo2 年前

the mother of all animations! 👍

Nirmalya Sengupta 的头像

Nirmalya Sengupta2 年前

The animation makes this even more retention-ready! Well done! 👍👍👍

Simform 的头像

Simform2 年前

I would add that consistent hashing is a fundamental concept in distributed systems and is used in many real-world applications. For instance, it’s used in load balancing, where requests are distributed across multiple servers, and in data sharding, where data is partitioned across multiple databases.

Manoj Khangaonkar 的头像

Manoj Khangaonkar2 年前

"Consistent" is a poor name choice for this technique.

相关视频

Sen. Husted blames his constituents for record number of data centers in Ohio: "We just have to realize we're in this world in which we've created the demand. We the people have created the demand for these data centers because we use all of these digital products that it processes, and if we don't want them, then don't use them. I mean, you can go back to the bunny ear antenna on your TV, you don't need a data center. But if you're connected to a wire, you're using the data center. So, understand, when we look in the mirror, we are the problem."

Sen. Husted blames his constituents for record number of data centers in Ohio: "We just have to realize we're in this world in which we've created the demand. We the people have created the demand for these data centers because we use all of these digital products that it processes, and if we don't want them, then don't use them. I mean, you can go back to the bunny ear antenna on your TV, you don't need a data center. But if you're connected to a wire, you're using the data center. So, understand, when we look in the mirror, we are the problem."

Aaron Rupar

68,323 次观看 • 1 个月前

#WATCH | Delhi | Principal Secretary to PM, Dr PK Mishra says, "The Indian statistical system, which has a glorious past, saw some decline for a few years, but the system itself and others have been able to revive it. Also, the Prime Minister's office played a catalyst and a facilitator. The way we have now improved our system, I'm sure it will again become very well established and predominant globally." "A lot of data is generated by our departments, ministries and digital activities also. So, how to use the potential of the data, how to standardise it, how to make it compatible with each other, how to derive inferences from the data, how to ensure that the data is as trusted as comprehensive and as compatible as the earlier survey data or maybe it is expanding the scope of our statistics."

#WATCH | Delhi | Principal Secretary to PM, Dr PK Mishra says, "The Indian statistical system, which has a glorious past, saw some decline for a few years, but the system itself and others have been able to revive it. Also, the Prime Minister's office played a catalyst and a facilitator. The way we have now improved our system, I'm sure it will again become very well established and predominant globally." "A lot of data is generated by our departments, ministries and digital activities also. So, how to use the potential of the data, how to standardise it, how to make it compatible with each other, how to derive inferences from the data, how to ensure that the data is as trusted as comprehensive and as compatible as the earlier survey data or maybe it is expanding the scope of our statistics."

ANI

87,255 次观看 • 2 个月前

Right now, when you send a query to an LLM, it gets decrypted on the server. The LLM sees your data in plain text. Prof. Ajay Joshi (BU, CipherSonic AI ) on fully homomorphic encryption, which may be key for the future of AI privacy: how we can compute on data without ever decrypting it. The catch: it's a brutally memory-bound workload. Exactly the bottleneck wafer-scale was built to solve.

Right now, when you send a query to an LLM, it gets decrypted on the server. The LLM sees your data in plain text. Prof. Ajay Joshi (BU, CipherSonic AI ) on fully homomorphic encryption, which may be key for the future of AI privacy: how we can compute on data without ever decrypting it. The catch: it's a brutally memory-bound workload. Exactly the bottleneck wafer-scale was built to solve.

Cerebras

293,561 次观看 • 2 个月前

Tony Blair and Oracle co-founder Larry Ellison plan to use digital ID to "unify" all data on each country's citizens "so it can be consumed and used by" their AI models. "We have to take all of this data... and move it into a single... unified data platform." "When we want to ask a question, we've provided that AI model with all the data they need to understand our country." "We need to unify all of the national data, put it into a database where it's easily consumable by the AI model, and then ask whatever question you like."

Tony Blair and Oracle co-founder Larry Ellison plan to use digital ID to "unify" all data on each country's citizens "so it can be consumed and used by" their AI models. "We have to take all of this data... and move it into a single... unified data platform." "When we want to ask a question, we've provided that AI model with all the data they need to understand our country." "We need to unify all of the national data, put it into a database where it's easily consumable by the AI model, and then ask whatever question you like."

Wide Awake Media

54,156 次观看 • 10 个月前

MCP is mind-blowing. Here is a genius setup that will earn you a promotion with your manager. I have multiple data sources: data in S3, MySQL, and local files. Using Spice (an open-source project), I can co-locate all of that data behind a single SQL endpoint and augment it using MCP. Before, accessing data was slow, and I had to make multiple requests to get anything. After using Spice, I can query the data much faster (since it's mirrored locally), and I don't have to worry about where the data is coming from. Spice acts as an MCP server, so I can connect to it from my IDE or the Claude Desktop app and have access to all the data. But even more cool than that: Spice also acts as an MCP client, so it can connect to other MCP servers. This is where things get really interesting! Watch the video. I'm querying multiple data sources from my IDE and using the Google Maps MCP server for driving directions.

MCP is mind-blowing. Here is a genius setup that will earn you a promotion with your manager. I have multiple data sources: data in S3, MySQL, and local files. Using Spice (an open-source project), I can co-locate all of that data behind a single SQL endpoint and augment it using MCP. Before, accessing data was slow, and I had to make multiple requests to get anything. After using Spice, I can query the data much faster (since it's mirrored locally), and I don't have to worry about where the data is coming from. Spice acts as an MCP server, so I can connect to it from my IDE or the Claude Desktop app and have access to all the data. But even more cool than that: Spice also acts as an MCP client, so it can connect to other MCP servers. This is where things get really interesting! Watch the video. I'm querying multiple data sources from my IDE and using the Google Maps MCP server for driving directions.

Santiago

142,696 次观看 • 1 年前

#WATCH | Delhi: On National Statistics Day, Saurabh Garg, Secretary, Ministry of Statistics and Programme Implementation, says, "The theme for National Statistics Day celebration is ' Unlocking the potential of administrative data'. Over the past decade, we have a lot of administrative data, especially because of the growth of Digital India. How we can leverage this administrative data for decision-making is the theme for today's discussions. The PM has been focusing on data-driven decision-making. This is in the context that if we are talking of a Viksit Bharat by 2047, the policies that we take, the decisions that we make, have to be so much more focused and more data-driven. Ministry of Statistics is helping to ensure that the data is more harmonised to make it interoperable."

#WATCH | Delhi: On National Statistics Day, Saurabh Garg, Secretary, Ministry of Statistics and Programme Implementation, says, "The theme for National Statistics Day celebration is ' Unlocking the potential of administrative data'. Over the past decade, we have a lot of administrative data, especially because of the growth of Digital India. How we can leverage this administrative data for decision-making is the theme for today's discussions. The PM has been focusing on data-driven decision-making. This is in the context that if we are talking of a Viksit Bharat by 2047, the policies that we take, the decisions that we make, have to be so much more focused and more data-driven. Ministry of Statistics is helping to ensure that the data is more harmonised to make it interoperable."

ANI

30,758 次观看 • 2 个月前

Larry Ellison—owner of Oracle, CBS, and now TikTok—tells Tony Blair about his plan to use digital ID to "unify" all data on each country's citizens "so it can be consumed and used by" his AI models. "We have to take all of this data... and move it into a single, if you will, unified data platform." "When we want to ask a question, we've provided that AI model with all the data they need to understand our country." "We need to unify all of the national data, put it into a database where it's easily consumable by the AI model, and then ask whatever question you like."

Larry Ellison—owner of Oracle, CBS, and now TikTok—tells Tony Blair about his plan to use digital ID to "unify" all data on each country's citizens "so it can be consumed and used by" his AI models. "We have to take all of this data... and move it into a single, if you will, unified data platform." "When we want to ask a question, we've provided that AI model with all the data they need to understand our country." "We need to unify all of the national data, put it into a database where it's easily consumable by the AI model, and then ask whatever question you like."

Wide Awake Media

144,618 次观看 • 7 个月前

Which one is our priority for securing a digital future for Zimbabwe? The Honourable Speaker's position while providing a clear, logical, and data-informed counter-argument we appreciate his efforts and submission. Let us have a conversation on our priorities in the current Zimbabwean context which we presented yesterday: This is a critical national conversation on our digital future. The proposition of a Google data centre is undoubtedly appealing. Building a centre of Technological Excellence that houses multiple Tech companies and a Start-up Ecosystem is also prudent, though we are calling it a Technology Park. However, I was of the following view which I am willing to be criticised and guided on. The strategic prioritisation of a national ICT Park ecosystem currently is the foundational path to sustainable, sovereign growth and capacity for Zimbabwe. We are reaching out to every province through the Digital Centres, Innovation Hubs at Universities and Colleges. Infrastructure which we need to capacitate better. This is a decentralised system utilising and expanding the infrastructure we have. The central question is not if we want major players like Google; that is not questionable, but when, how and on what terms. Why are we building our own foundation, house, techno park and mini ICT Parks in the provinces and districts? We absolutely need at least a one hyperscale data centre, but centralisation in the current and future environment can be restrictive. This was our plan: 1.The Foundation Before the Skyscraper: Capacity and Traffic We need to increase our national data volumes and internet traffic . Our urgent priority should be upgrading our national backbone, specifically the Optical Fibre on Power Transmission Lines (e.g., Powertel's network from Insukamini to Johannesburg), to peer efficiently with global giants. This builds the foundational "digital highway" we manage. 2.Generating Traffic Through Digitalization A data centre is a response to demand, not a creator of it. We must first drive digitalization aggressively through digital payments, process automation, and e-governance to generate the significant local traffic that would make a data centre viable. Furthermore, we must incorporate Edge Computing strategies to process data closer to the source, a more efficient model for our current needs and in compliance to our Cyber and Data Protection Act which we can amend if need be. Being a regional internet and cybersecurity hub is crucial as the Speaker alluded to me mentioning it to SADC Parliamentary Forum, this is the ultimate goal. 3.PPPs are key to lessen the burden to the fiscus and we are ready to accommodate organisations willing to assist government.(Cont)

Which one is our priority for securing a digital future for Zimbabwe? The Honourable Speaker's position while providing a clear, logical, and data-informed counter-argument we appreciate his efforts and submission. Let us have a conversation on our priorities in the current Zimbabwean context which we presented yesterday: This is a critical national conversation on our digital future. The proposition of a Google data centre is undoubtedly appealing. Building a centre of Technological Excellence that houses multiple Tech companies and a Start-up Ecosystem is also prudent, though we are calling it a Technology Park. However, I was of the following view which I am willing to be criticised and guided on. The strategic prioritisation of a national ICT Park ecosystem currently is the foundational path to sustainable, sovereign growth and capacity for Zimbabwe. We are reaching out to every province through the Digital Centres, Innovation Hubs at Universities and Colleges. Infrastructure which we need to capacitate better. This is a decentralised system utilising and expanding the infrastructure we have. The central question is not if we want major players like Google; that is not questionable, but when, how and on what terms. Why are we building our own foundation, house, techno park and mini ICT Parks in the provinces and districts? We absolutely need at least a one hyperscale data centre, but centralisation in the current and future environment can be restrictive. This was our plan: 1.The Foundation Before the Skyscraper: Capacity and Traffic We need to increase our national data volumes and internet traffic . Our urgent priority should be upgrading our national backbone, specifically the Optical Fibre on Power Transmission Lines (e.g., Powertel's network from Insukamini to Johannesburg), to peer efficiently with global giants. This builds the foundational "digital highway" we manage. 2.Generating Traffic Through Digitalization A data centre is a response to demand, not a creator of it. We must first drive digitalization aggressively through digital payments, process automation, and e-governance to generate the significant local traffic that would make a data centre viable. Furthermore, we must incorporate Edge Computing strategies to process data closer to the source, a more efficient model for our current needs and in compliance to our Cyber and Data Protection Act which we can amend if need be. Being a regional internet and cybersecurity hub is crucial as the Speaker alluded to me mentioning it to SADC Parliamentary Forum, this is the ultimate goal. 3.PPPs are key to lessen the burden to the fiscus and we are ready to accommodate organisations willing to assist government.(Cont)

Hon Tatenda.A. Mavetera

490,466 次观看 • 10 个月前

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

Twelve years ago we asked the question: how can we put public data "on wheels"? Datawheel was born from vision that data alone doesn't get far. It needs vehicles to reach hearts and minds. Vehicles that make data easy to find, explore, and understand. Today, Datawheel turns twelve. In these twelve years, we have built dozens of solutions that are helping define the state of the art in public data distribution. Today we celebrate this journey. A journey with a great team and amazing clients. The journey to make public data all it can be. are some of the tools defining all that public data can be. Thank you!!!

César A. Hidalgo

12,931 次观看 • 1 年前

what have I been working on for the last 2 weeks at OpenCode? we are bringing workspaces in as a first-class concept: agents will be able to run anywhere: local directory/container/worktree, remote sandbox, or anything. instead of having a single server that only knows about local directories, the server will act as a "control plane" that routes prompts to where they should go it will manage all of the sessions across these environments and provide a unified layer for all clients to interact with them we will be smart about this: it's not *only* about routing. we will sync all data from remote environments into the local control plane server. this means the remote env can be destroyed, and we will be able to later recreate a new env with exactly the same state. it's reproducible note: this is very early and experimental, the UI you see the video is just a prototype and not guaranteed to make it to prod. We hope to start shipping something soon-ish though!

what have I been working on for the last 2 weeks at OpenCode? we are bringing workspaces in as a first-class concept: agents will be able to run anywhere: local directory/container/worktree, remote sandbox, or anything. instead of having a single server that only knows about local directories, the server will act as a "control plane" that routes prompts to where they should go it will manage all of the sessions across these environments and provide a unified layer for all clients to interact with them we will be smart about this: it's not only about routing. we will sync all data from remote environments into the local control plane server. this means the remote env can be destroyed, and we will be able to later recreate a new env with exactly the same state. it's reproducible note: this is very early and experimental, the UI you see the video is just a prototype and not guaranteed to make it to prod. We hope to start shipping something soon-ish though!

James Long

170,819 次观看 • 6 个月前

Our retired data center hardware has a second life. The Amazon Web Services reverse logistics hubs use circular economy principles to test, repair, and reuse demanufactured server racks, all while maintaining the highest level of data security. One of our reverse logistics operations managers says it well, “The components we repair are small, but they have the potential to make a big impact for our company, our customers, and our planet.” #Circularity23 #AWS

Our retired data center hardware has a second life. The Amazon Web Services reverse logistics hubs use circular economy principles to test, repair, and reuse demanufactured server racks, all while maintaining the highest level of data security. One of our reverse logistics operations managers says it well, “The components we repair are small, but they have the potential to make a big impact for our company, our customers, and our planet.” #Circularity23 #AWS

Adam Selipsky

25,368 次观看 • 3 年前

When it comes to capex/capacity buildout conversations, I'm always reminded of the legendary James Hamilton's 2016 re:Invent talk (the whole talk is worth watching – it's fantastic). The main slide says: "AWS adds the capacity equivalent of a Fortune 500 enterprise daily." Lots of people over the past couple of years had been saying that AWS had missed the boat on AI. It turns out that a lot of 'the boat' has to do with the ability to massively build out capacity quickly (and, over time, capital-efficiently). Keep in mind when watching this clip that this describes AWS's data center buildout capabilities ten years ago – AWS has been extremely good at this for a very long time. "I'm in love with this slide. We update it, but it doesn't matter. If we if we never updated it if we left it the way it was three years ago, it'd be wonderful. This is a crazy number. Think what this means. Just imagine – forget about all the innovation. Forget about all the things I'm going to talk about today. Just imagine figuring out how to do this. What you need to do is you need to get all the component manufacturers delivering to the server manufacturer. The server manufacturer has to build boards, assemble servers, package them up, test them, push them into the supply chain. The logistic channel has to deliver them to their data centers. Trucks have to back up to the right data center, deliver racks. There has to be enough data center space built. There has to be networking. There has to be power. The servers has to be installed by technicians and it all has to to be brought up. And then we do it again tomorrow."

When it comes to capex/capacity buildout conversations, I'm always reminded of the legendary James Hamilton's 2016 re:Invent talk (the whole talk is worth watching – it's fantastic). The main slide says: "AWS adds the capacity equivalent of a Fortune 500 enterprise daily." Lots of people over the past couple of years had been saying that AWS had missed the boat on AI. It turns out that a lot of 'the boat' has to do with the ability to massively build out capacity quickly (and, over time, capital-efficiently). Keep in mind when watching this clip that this describes AWS's data center buildout capabilities ten years ago – AWS has been extremely good at this for a very long time. "I'm in love with this slide. We update it, but it doesn't matter. If we if we never updated it if we left it the way it was three years ago, it'd be wonderful. This is a crazy number. Think what this means. Just imagine – forget about all the innovation. Forget about all the things I'm going to talk about today. Just imagine figuring out how to do this. What you need to do is you need to get all the component manufacturers delivering to the server manufacturer. The server manufacturer has to build boards, assemble servers, package them up, test them, push them into the supply chain. The logistic channel has to deliver them to their data centers. Trucks have to back up to the right data center, deliver racks. There has to be enough data center space built. There has to be networking. There has to be power. The servers has to be installed by technicians and it all has to to be brought up. And then we do it again tomorrow."

Zack Kanter

38,760 次观看 • 1 个月前

Doug Ford on opposition to data centres: "Honestly, I think people have to learn a lot more about data centres ... This is the way of the future. Either we do it or Donald Trump's going to do it."

Doug Ford on opposition to data centres: "Honestly, I think people have to learn a lot more about data centres ... This is the way of the future. Either we do it or Donald Trump's going to do it."

Scott Robertson

20,773 次观看 • 27 天前

Introducing the #WELL3Ring: Elevating Health Data to On-Chain Assets🔘 At #WELL3, we are leading a transformative shift in how health data is valued and monetized. We cordially invite our #WELLID holders to be the first to experience the benefits of this DePIN wearable. By donning the WELL3 Ring and sharing your health data, you will gain access to unlimited and exclusive rewards🧵

Introducing the #WELL3Ring: Elevating Health Data to On-Chain Assets🔘 At #WELL3, we are leading a transformative shift in how health data is valued and monetized. We cordially invite our #WELLID holders to be the first to experience the benefits of this DePIN wearable. By donning the WELL3 Ring and sharing your health data, you will gain access to unlimited and exclusive rewards🧵

WELL3

223,045 次观看 • 2 年前

Robots enter data centers! 💽 SeRo developed by NAVER LABS, is built for one of the most overlooked but critical tasks in a data center: managing servers in the IT warehouse. It can autonomously load and unload servers, identify their asset numbers, and log their lifecycle data. Physically, SeRo is designed tall to handle racks up to three meters high, maximizing storage density. SeRo also works as part of a larger system. It hands assets to GaRo, an autonomous transport robot, for seamless movement between warehouse and server room. More AI ops, more data centers, more data centers ops... Perfect market for robotics! 🦾

Robots enter data centers! 💽 SeRo developed by NAVER LABS, is built for one of the most overlooked but critical tasks in a data center: managing servers in the IT warehouse. It can autonomously load and unload servers, identify their asset numbers, and log their lifecycle data. Physically, SeRo is designed tall to handle racks up to three meters high, maximizing storage density. SeRo also works as part of a larger system. It hands assets to GaRo, an autonomous transport robot, for seamless movement between warehouse and server room. More AI ops, more data centers, more data centers ops... Perfect market for robotics! 🦾

Lukas Ziegler

30,334 次观看 • 1 年前

You can connect your Supabase database via MCP server to Cursor! The Cursor agent can get the schema and any data from your database through the official Postgres MCP server! This helps Cursor understand your database better when it needs to modify it to add a new feature!

You can connect your Supabase database via MCP server to Cursor! The Cursor agent can get the schema and any data from your database through the official Postgres MCP server! This helps Cursor understand your database better when it needs to modify it to add a new feature!

Tyler Shukert

203,451 次观看 • 1 年前

#BudgetSession2025 | Lok Sabha LoP and Congress MP Rahul Gandhi says "People talk about AI, but it's important to understand that AI on its own is absolutely meaningless because AI operates on top of data. Without data, AI means nothing. And if we look at data today, there is one thing which is very clear. Every single piece of data that comes out of the production system in the world. The data that was used to make this phone, the data that is used to make electric cars. The data that is used to make basically all electronics on the planet today is owned by China. And the consumption data is owned by the United States... China has at least a 10 year lead on India in this space. China has been working on batteries, robots, motors, optics for the last 10 years and we are behind...We would ensure that our banking system is not captured by 2-3 companies that basically do not allow you to build a production system. But our banking system is open, dynamic and accessible to small and medium businesses and to millions and millions of entrepreneurs who want to take part in this revolution. Our foreign policy would take into account this revolution. When we talk to the United States, we would not send our foreign minister to invite our Prime Minister to his coronation...Because if we had a production system and if we were working on these technologies, the American President would come here and invite the Prime Minister."

#BudgetSession2025 | Lok Sabha LoP and Congress MP Rahul Gandhi says "People talk about AI, but it's important to understand that AI on its own is absolutely meaningless because AI operates on top of data. Without data, AI means nothing. And if we look at data today, there is one thing which is very clear. Every single piece of data that comes out of the production system in the world. The data that was used to make this phone, the data that is used to make electric cars. The data that is used to make basically all electronics on the planet today is owned by China. And the consumption data is owned by the United States... China has at least a 10 year lead on India in this space. China has been working on batteries, robots, motors, optics for the last 10 years and we are behind...We would ensure that our banking system is not captured by 2-3 companies that basically do not allow you to build a production system. But our banking system is open, dynamic and accessible to small and medium businesses and to millions and millions of entrepreneurs who want to take part in this revolution. Our foreign policy would take into account this revolution. When we talk to the United States, we would not send our foreign minister to invite our Prime Minister to his coronation...Because if we had a production system and if we were working on these technologies, the American President would come here and invite the Prime Minister."

ANI

163,581 次观看 • 1 年前

We have data in abundance now. The real value isn't within the RAW Data, but is this data validated and verified? More value in data refinement. We have a solution for this which is validated by 6 years of research and development with the Department of Defense Now securing and validating commercial data from partners like Common Crawl and Panasonic

We have data in abundance now. The real value isn't within the RAW Data, but is this data validated and verified? More value in data refinement. We have a solution for this which is validated by 6 years of research and development with the Department of Defense Now securing and validating commercial data from partners like Common Crawl and Panasonic

Constellation²

10,567 次观看 • 1 年前