Video wird geladen...
Video konnte nicht geladen werden
You can now run SQLCoder with a GUI on Apple Silicon or any NVIDIA GPU-enabled device! On Apple Silicon, just run CMAKE_ARGS="-DLLAMA_METAL=on" pip install "sqlcoder[llama-cpp]" sqlcoder launch The Apple Silicon version is not super accurate, but works great for simple questions – and everything happens (fairly fast) on just... show more
10,405 Aufrufe • vor 2 Jahren •via X (Twitter)
6 Kommentare
Cris Lentavor 2 Jahren
This looks great! Have you tried also running it with .cpp models? Curious of the performance
Rishabh Srivastavavor 2 Jahren
Thank you! The demo above is for a llama.cpp model (with q5 km quantization). Haven't done a rigorous eval on those yet though – will do in a few hours!
Nerdynavvor 2 Jahren
hey great work, what's the license for this? Are code llama fine tunes allowed in their Meta's licensing?
Gautam Padiyarvor 2 Jahren
GPU with 20gb memory not required?
Rishabh Srivastavavor 2 Jahren
7B model, so you only need ~16GB or so of memory to make it run without beam search
Harsh Patelvor 2 Jahren
On Apple silicon how do I make it use the 7b-2 model? It is downloading the v1 I guess
![The latest MLX has a CUDA back-end! To get started: pip install "mlx[cuda]" With the same codebase you can develop locally, run your model on Apple silicon, or in the cloud on Nvidia GPUs. MLX is designed around Apple silicon - which has a unified memory architecture. It uses the same design with CUDA. So there's no need to move arrays around from CPU memory to GPU memory. Note, this is early days - some operations are missing and performance is still being optimized. But it's already quite fast for Transformer training, text generation, and more! Here's a demo using mlx-lm to generate text with Llama 3 8B (bf16) on an A100:](https://image.24vids.com/tw-1948878861795819662/amplify_video_thumb/1948878782372478981/img/k7q6fzLo3fGrR5WI.jpg)
![We made a thing! Very happy to announce sqlcoder-pro and the Defog Alignment Platform. Available to use immediately without a wait-list, weights will be open-sourced very soon. The video does a quick show and tell comparison against ChatGPT (with gpt-4o). Read on for more details! TLDR 💪 equal (or better) performance on text-to-SQL as the most capable Claude-3.5 or GPT-4 models 🤝 You can use it today on a free plan/free trial, without a waitlist 🪽 self-hostable on a single RTX4090, with 2 second median generation times for SQL queries 🔁 exactly the same output every time, give the same prompt 👨🏻🏫 teachable and steerable: show the model what you want it to do 🛞 debuggable – you can understand WTF is going on inside the model, instead of treating it like a black box Let's dig into each of these one-by-one! Performance SQLCoder-8b-pro significantly exceeds the performance of our previous sqlcoder-8b model on Postgres text-to-SQL (from 88.2% to 90.2% accuracy - gpt-4o is at 87.6%, for reference). It is also better at following instructions. This was done via self-merges, hand crafted fine-tuning data, and adapting the training data to fit our tokenizer. Cost You can host this on the model on a single $3,500 RTX4090, and support ~5 requests/second via VLLM. If you're looking to host on the cloud instead, you can run it on a single L4 GPU that costs $300/mo on GCP Repeatability We have a dense 8b model with no MoE shenanigans. For the same prompt with temperature=0, you'll always get the same answer – which is critical in BI. Teachable In our alignment and feedback modes, you can give the model feedback on how it answered certain questions, and it will automatically adapt to the feedback. Debuggable You can use logprobs and attention scores to determine where, exactly is the model paying attention to inside a prompt + what it's getting confused by when generating outputs. Available today You can use Defog on the cloud today by going to docs[dot]defog[dot]ai, and getting an API key. Excited to hear what you think!](https://image.24vids.com/tw-1825943270935113842/ext_tw_video_thumb/1825942610525220864/pu/img/qmpqqvd_3KNdJDGd.jpg)