Fully local Code Assistant running on NVIDIA GPU! In this tutorial, I'll show you how to run Llama3 using TensorRT and Nvidia's Triton Inference Server to use it as a Code Assistant in VSCode In this thread 🧵, I'll walk you through the integration process, explaining each step simply... and clearly 👇 NVIDIA AI NVIDIA AIDev AI at Metashow more

To get started, we need a @nvidia GPU 🤩 In this case, we will use the following hardware 💻

We need to have Docker and CUDA installed Follow the guides below for installing both tools Docker: CUDA: and then run the following commands to confirm everything is set up correctly

Download the llama3-8B model from @huggingface

Now, Run TensorRT to compile the model using the Docker container Clone the TensorRT repository and move the model folder

You should now be able to test the compiled model

Perfect! We have the model now, let's deploy it on Triton Inference Server

The server is up and ready to connect with CodeGPT via the custom connection Open CodeGPT in VSCode, select Custom as the provider, and enter "ensemble" for the model

That's all! I'm sharing the link to the full article with all the details of the tutorial

INTRODUCING: Agentic Security - LLM Security Scanner! 🔍 🔑 Features: Scans for prompt injections, jailbreaking & more. Provides detailed reports & options to customize attack rules. 🔗access the GitHub Link ↓

Or just use Continue and Ollama with whatever brand GPU 🤷‍♂️ that’s open source

you can also use CodeGPT with Ollama Check this link: