LLMOps at Scale: Deploying and Monitoring LLMs With HPE MLIS

AI Connect webinarSimplifying LLMOps at Scale for Data ScientistsWatch the full session · 52 min

In a 2024 AI Connect webinar, "Simplifying LLMOps at Scale for Data Scientists," two HPE speakers explained why deploying large language models, not training them, has become the hard part of enterprise AI. Derek, the product manager for HPE's Machine Learning Inference Software (MLIS), and Jimmy Whitaker described the challenges and demonstrated how the product handles deployment, scaling, canary rollouts and monitoring. A host relayed audience questions. The webinar description does not give the speakers' full names or titles beyond those they stated.

What is LLMOps, and why is deployment the bottleneck?

Derek defined LLMOps as "the practice of streamlining the deployment, monitoring and optimization of large language models in a production environment." The lifecycle starts with data preparation, prompt engineering and fine-tuning, he said, but this session focused on deployment, monitoring, logging and governance, where he saw gaps when LLMs arrived.

Since ChatGPT launched in 2022, he said, the emphasis has shifted. There used to be a lot of focus on training and evaluation, and now the challenges are on the deployment side. After all the work of preparing data and evaluating a model, "it's not just as simple as clicking a button," he said, and companies employ entire teams to handle what comes next.

What challenges do teams face deploying LLMs?

Derek grouped them in three buckets.

• Technical: Scaling across GPUs in real time as users grow, and using GPU hardware efficiently. GPUs, he said, are probably the most expensive AI cost today, so teams need to scale up and down to free them for other uses.
• Operational: Tracking performance metrics such as latency, monitoring drift against training and evaluation data, detecting issues quickly, and rolling out updates without users noticing.
• Security and compliance: Protecting data under rules such as HIPAA and GDPR, detecting vulnerabilities such as unauthorized access or data leakage, and meeting each company's compliance standards. He noted LLMs can unintentionally expose sensitive user data in answers.

His summary: data scientists should spend their time improving models, not managing the risks of deployment.

How does HPE's MLIS approach these problems?

Whitaker said HPE released MLIS in June 2024. The goal, he said, was something data scientists can use without deep infrastructure skills: "You don't need to be a Kubernetes-level expert," but you still get the benefits of running on Kubernetes.

He described support for NVIDIA NIM containers, which he characterized as prebuilt inference microservices bundled with optimizations such as Triton and TensorRT-LLM and with enterprise support. MLIS also supports Hugging Face models through tools such as OpenLLM and vLLM, and custom containers. It adds a GUI and command-line interface, autoscaling, canary rollouts, API-key authentication, and monitoring and logging. It can run on any recent stable Kubernetes environment, on premises or in the cloud, including fully disconnected from the internet.

What did the demo show?

Whitaker ran MLIS in Google Cloud and showed a registry connected to NVIDIA's NGC, a packaged Llama 3 8B Instruct model, and resource templates for CPUs, memory and GPUs. He configured a deployment, optionally secured with an API token, and set scaling rules. In one example, he said, an instance could scale from zero to eight, adding a GPU when requests per second exceed 10 for an instance.

The deployed endpoint exposed an OpenAI-compatible API, so standard tools and chat interfaces could connect. To update the model, he created a new version and started a canary rollout sending 25% of traffic to it, while the rest continued to use the original. He later moved to 100% on the same endpoint, so other software did not need to change. A Grafana dashboard showed request and token throughput.

What did the audience ask?

Asked about models tuned in HPE's Machine Learning Development Environment, Whitaker said they can be served by pointing a registry at storage such as an S3 bucket. Derek added that the upcoming MLIS 1.2.0 release would connect directly to HPE's MLDM data and model management tool, which can then serve as a model registry.

On deployment, Whitaker said MLIS is not a SaaS-only product and needs only a recent Kubernetes install. Asked about open source, he said MLIS was not open source, though it is built on open source components, and NIMs carry an NVIDIA license. These were the speakers' statements at the time.

Where can you find more AI Connect webinars?

AI Connect hosts 24 free live webinars on its 2027 schedule, in four series: AI x Enterprise Panel Series, AI Demo Days, Agentic AI Showcases and Plug & Play Series. Details are on the webinars page, and registration is coming soon.

Frequently asked questions

What is LLMOps?

Derek defined it as streamlining the deployment, monitoring and optimization of large language models in production.

What challenges come with deploying LLMs?

The speakers named technical (scaling, GPU use), operational (latency, drift, updates) and security and compliance challenges.

What is a canary rollout?

A way to send a percentage of traffic to a new model version while the rest continues to use the current one, so issues surface before full rollout.

Is HPE's MLIS open source?

Not at the time, the speakers said, though it builds on open source components and NVIDIA NIMs carry an NVIDIA license.

This article is based on remarks by Derek (last name not given), Product Manager, HPE Machine Learning Inference Software; Jimmy Whitaker, HPE (title not given) at AI Connect webinar on October 31, 2024. More coverage: Webinars.

Filed under: Data & AI Infrastructure, Observability & Data Quality, AI Governance & Trust