Hebrew LLM Eval Suite
Trusted88/100Benchmark and compare LLMs on Hebrew reasoning, comprehension, sentiment, translation, and Israeli cultural knowledge. Wraps the HuggingFace Open Hebrew LLM Leaderboard tasks (HeQ, HebrewSentiment, Hebrew Winograd, translation) plus DictaLM 3.0 benchmark tasks (Summarization, Nikud, Israeli Trivia) into a reproducible evaluation harness. Runs evals against Claude, GPT, Gemini, AI21 Jamba, DictaLM, Llama, and local HuggingFace models. Produces comparison scorecards in JSON and markdown. Use when choosing an LLM for a Hebrew product, answering procurement questions about Hebrew performance, validating a fine-tuned Hebrew model, or tracking Hebrew regressions after a model upgrade. Do NOT use for Arabic NLP, ASR benchmarking, or general English benchmarks.
Trust score 88/100 (Trusted) · 7+ installs · 3 GitHub contributors · MIT license
Israeli product teams pick LLMs blind. There is no standardized Hebrew benchmark that a PM can run in an afternoon to compare Claude against GPT against DictaLM against AI21 Jamba on their actual use case. The HuggingFace Open Hebrew LLM Leaderboard is built for base models and few-shot prompts, not for API-hosted chat models. DictaLM publishes benchmark results but only for its own suite. Teams end up guessing, testing informally, or trusting marketing claims.
skills-ilnpx skills-il add skills-il/developer-tools --skill hebrew-llm-eval-suite -a claude-codeInstall on Claude.ai, Claude Desktop, ChatGPT, Manus, or other platforms
- 1. Click "Download ZIP" to download the skill files.
- 2. Open Claude Desktop and go to Customize > Skills.
- 3. Click "+" and select "Upload a skill", then upload the ZIP file.
- 4. Start a new conversation. The skill will activate automatically when relevant.
When to Apply
- When choosing an LLM for a new Hebrew product and needing to justify the choice to leadership
- When answering enterprise procurement questions about Hebrew performance
- When validating whether a provider upgrade improved or regressed Hebrew quality
- When validating a fine-tuned Hebrew model against a baseline
- When comparing providers on a specific task: comprehension, translation, summarization, or diacritization
Try These Prompts
We are building a Hebrew news summarization feature and need to pick between Claude Sonnet, GPT-5, and DictaLM-3.0-24B. Run the relevant benchmarks (HeQ, DictaLM Summarization, Winograd) with 1000 samples and 3 runs, and recommend a model with reasoning.
Anthropic released a new version of claude-sonnet. Run the hebrew-core suite on the new and previous versions and tell me if there was any regression over 2 points on any benchmark.
I am building a Hebrew chatbot and deciding between Claude Haiku and AI21 Jamba 1.5 Mini. Compare them on HeQ, HebrewSentiment, and HebNLI with 500 samples and 3 runs, and provide a scorecard with a recommendation.
We have a data residency constraint requiring a local model. Run Hebrew benchmarks on DictaLM-3.0-Nemotron-12B-Instruct and compare to Claude Sonnet quality. How much quality am I giving up?
Frequently Asked Questions
Related Skills
Author: skills-ilBest practices for programmatic video creation using HyperFrames, plain HTML compositions with GSAP animations rendered to MP4, with full Hebrew and RTL support. Covers composition authoring, data-* timing attributes, GSAP timeline contract, layout-before-animation methodology, visual identity gate, Hebrew fonts via Google Fonts auto-fetch (Heebo, Rubik, Assistant), RTL text with dir="rtl", Hebrew captions via Whisper, Hebrew voiceover via external TTS (Kokoro doesn't support Hebrew), audio-reactive visuals, scene transitions, and bidirectional text with <bdi>. Use when building HTML-based video content or Hebrew social/marketing videos without React. Do NOT use for Remotion or general React video work.
Author: skills-ilBuild Zapier Zaps that connect Israeli business apps (Morning/Green Invoice, Cardcom, Tranzila, iCount, Grow by Meshulam) with global services for billing, payment, and workflow automation. Covers Hebrew text handling, ILS formatting, bimonthly VAT logic, Israel Invoice Reform 2026, and Zapier AI features (Copilot, Agents, MCP Server). Includes WhatsApp Business via third-party providers (Twilio, WATI), Zapier Tables as Google Sheets alternative, and comparison with Make.com/n8n. Do NOT use for n8n workflows (use n8n-hebrew-workflows), Make.com scenarios (use make-com-israeli-automations), or custom code automation without Zapier.
Author: skills-ilManage JFrog Artifactory repositories, artifacts, Docker registry, build info, and Xray security scanning for DevOps workflows. Use when user asks about JFrog, Artifactory, Xray, artifact management, "deploy artifact", Docker registry with Artifactory, build promotion, vulnerability scanning with Xray, or DevOps artifact pipeline. Covers REST API operations, JFrog CLI usage, Docker registry configuration, and security scanning patterns. Do NOT use for general Docker or CI/CD questions unrelated to JFrog.
Use at your own risk. Terms of Use · Security
Want to build your own skill? Try the Skill Creator · Submit a Skill