Loading video...

Video Failed to Load

Go Home

What's the difference between retrieving a fact and truly reasoning? Prof. Kambhampati Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) begins by noting that human reasoning is tricky to define. Yet, since the Greeks, we’ve relied on formal logic (like syllogisms) to guide sound reasoning. A thread 🧵👇

19,132 views • 1 year ago •via X (Twitter)

9 Comments

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

2/7 He points out that we lack a neat definition of “human reasoning,” but that hasn’t stopped our civilization from moving forward. We built entire disciplines—from Aristotle’s logic to modern computer science—on the foundation of formal, structured reasoning.

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

3/7 Kambhampati contrasts retrieval (pulling stored info) with reasoning (connecting ideas with logical rigor). Just because we string ideas together doesn’t guarantee we’re actually reasoning—there must be standards for correctness and validity.

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

4/7 He humorously references Monty Python’s witch trial scene: the argument is “If she floats like wood, she must be a witch.” It looks like reasoning—there’s a chain of statements—but it’s clearly not sound. It’s a playful reminder that not all stepwise arguments are valid.

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

5/7 Between raw retrieval (“She’s a witch!") and truly logical inferences, there’s a vast middle ground of fallacies and “Monty Python logic” that mimics reasoning but fails basic tests of correctness.

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

6/7 Hence, in AI (and broader AGI pursuits), we can’t just replicate the surface features of human reasoning. We need rigorous definitions and methods—logic, probability, evidence—to separate mere associations from true inference.

Machine Learning Street Talk's profile picture
Machine Learning Street Talk1 year ago

7/7 By preserving “sound reasoning” standards, Kambhampati argues we uphold the legacy of centuries of philosophical and mathematical thought, aiming for AI that doesn’t just retrieve but reasons in a formally robust way.

OnlineBookClub.org's profile picture
OnlineBookClub.org1 year ago

What is the nature of an existence that is experienced entirely outside of time itself? Can a single decision that is made in a state of timelessness simultaneously affect EVERY point in time and space? Groundbreaking reconciliation of creationism with natural science.

Bahaeddin ERAVCI's profile picture
Bahaeddin ERAVCI1 year ago

@rao2z I touched on the same concept on my last substack. LLMs are statement generators with populist vote based on training data and hallucinations are not bug but feature. We need formal systems like logic for verifying truth among these statements.

Eray Özkural, PhD - OG AI ⏭️🌟's profile picture
Eray Özkural, PhD - OG AI ⏭️🌟1 year ago

@rao2z Come on who needs logic? 🤪

Related Videos

🔥 Battle for the top reasoning LLM intensifies! The QwQ-32B-Preview is a very good reasoning LLM. Full video of my tests here: Summary of my findings and thoughts: It was able to solve a couple of hard math problems so it looks very promising for maths. It didn’t do so well on my coding task (generating bash script). By the results reported on the LiveCodeBench it has room for improvement. One thing that’s become very clear to me is that the reasoning capabilities of these LLMs are significantly closing the gap between the open and closed-sourced models. The competition is now going to be on a different level and it's going to be focused on which model produces the most efficient, optimized, accurate, and fastest reasoning steps beyond just accurate responses. That's what developers will care about. Traditional benchmarks are not going to be good enough for this. On that note, it's getting harder to assess these models, especially the consistency, efficiency, and quality of reasoning steps. After experimenting with this model, I realized that the reasoning paths are not fully optimized and there is a lot more optimization that needs to happen before these models are used in production settings. There might be a need to build some type of native and efficient self-assessment or self-reflection capability that prevents these reasoning LLMs to go in loops or produce unnecessary lengthy sequences. I also noticed that this model, at least from the HF demo, doesn’t separate the reasoning from the response. I think that actually hurts the performance of the model. On the other hand, o1 and R1 do that really well. In addition to that, I believe the training on reasoning is hurting the performance of the LLM in other areas such as helpfulness (check the code example in the video). Something that’s necessary at the moment is validating or evaluating the quality of the reasoning chains and figuring out a better strategy to optimize them. Current methods are probably not sufficient to solve this problem but that's where innovation will comes next. I recognize that this is a first effort so kudos to the Qwen team on this release. These issues highlight the importance of transparency with reasoning LLMs. We need to know how it was trained and with exact data or optimization strategy. Understanding that will enable researchers and developers to build better intuition and improve the reasoning capabilities and components at a faster rate. There is an opportunity for someone or a company to build a truly open-reasoning LLM. The race is on! I will continue to track the state-of-the-art in reasoning LLMs and report my takes and observations here. Stay tuned for more.

elvis

14,740 views • 1 year ago