{"id":2485,"date":"2026-06-08T19:23:02","date_gmt":"2026-06-08T19:23:02","guid":{"rendered":"https:\/\/ai-box.eu\/?p=2485"},"modified":"2026-06-09T03:17:52","modified_gmt":"2026-06-09T03:17:52","slug":"llama-benchy-llama-bench-style-llm-benchmarks-for-any-openai-compatible-endpoint","status":"publish","type":"post","link":"https:\/\/ai-box.eu\/en\/news\/llama-benchy-llama-bench-style-llm-benchmarks-for-any-openai-compatible-endpoint\/2485\/","title":{"rendered":"llama-benchy: llama-bench-style LLM benchmarks for any OpenAI-compatible endpoint"},"content":{"rendered":"

If, like me, you run your models locally and sovereignly, you know the problem: I want to know how fast a model really is on my<\/strong> hardware and not as a theoretical figure, but the way it actually reaches me as an end user. For exactly that, the llama.cpp<\/strong> world has the popular tool llama-bench<\/strong>. The catch: it works exclusively with llama.cpp. The moment I bring vLLM<\/strong>, SGLang<\/strong> or any other inference server into play, I’m left without a comparable tool.<\/p>\n

In this post I introduce you to llama-benchy<\/strong>. It’s a tool that closes exactly this gap: the great thing is that I now get measurements at different context depths, but for any OpenAI-compatible endpoint<\/strong>. No matter whether you run Ollama, vLLM or llama.cpp.<\/p>\n

You’ll find llama-benchy here on GitHub: https:\/\/github.com\/eugr\/llama-benchy<\/a><\/p>\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n