vLLM Server Using OpenAI API on Gaudi 3 | AI with Guy

Published on 5 Jun 2025, 15:06

In this episode of AI with Guy, learn how to build a real-world retrieval-augmented generation (RAG) system using vLLM, the OpenAI API, and Intel Gaudi 3 across multiple AWS instances. This demo shows how to deploy a scalable, production-ready Gen-AI application using the OPEA framework.

Whether you're a developer, ML engineer, or AI architect, this walkthrough covers:
- Setting up a vLLM server
- Connecting to the OpenAI API for inference
- Deploying across AWS EC2 using Intel Gaudi 3
- Coordinating workloads using OPEA for high performance and cost efficiency

Resources:

OPEA Documentation: opea.dev
ChatQnA Gen-AI Example: opea-project.github.io/latest/...
Intel Cloud Dev Tools: cloud.intel.com

Tech Stack:
-vLLM
-OpenAI API
-Intel Gaudi 3 (AWS DL1/DL2 instances)
- OPEA
- AWS EC2

About Intel Software:
Intel® Developer Zone is committed to empowering and assisting software developers in creating applications for Intel hardware and software products. The Intel Software YouTube channel is an excellent resource for those seeking to enhance their knowledge. Our channel provides the latest news, helpful tips, and engaging product demos from Intel and our numerous industry partners. Our videos cover various topics; you can explore them further by following the links.

Connect with Intel Software:
INTEL SOFTWARE WEBSITE: intel.ly/2KeP1hD
INTEL SOFTWARE on FACEBOOK: bit.ly/2z8MPFF
INTEL SOFTWARE on TWITTER: bit.ly/2zahGSn
INTEL SOFTWARE GITHUB: bit.ly/2zaih6z
INTEL DEVELOPER ZONE LINKEDIN: bit.ly/2z979qs
INTEL DEVELOPER ZONE INSTAGRAM: bit.ly/2z9Xsby
INTEL GAME DEV TWITCH: bit.ly/2BkNshu

#intelsoftware

vLLM Server Using OpenAI API on Gaudi 3 | AI with Guy | Intel Software