Promptfoo vs Speech To Markdown: Features, Pricing & Which Is Better (2026)
A side-by-side comparison of Promptfoo and Speech To Markdown — features, pricing, and ideal use cases — to help you decide which AI tool fits your workflow.
Promptfoo
Promptfoo
CLI and web tool for testing, evaluating, red‑teaming, and monitoring LLM prompts and outputs to catch regressions and vulnerabilities.
Key features
- Red-Teaming & Vulnerability Scanning: Declarative red‑team tests and automated scans to surface prompt injections, unsafe completions, and other model security risks across providers.
- Evaluations & Regression Detection: Run reproducible eval suites and compare outputs before/after changes to detect regressions, with CI/CD and GitHub Action integration for automated checks on PRs.
- Multi-Provider Model Comparison: Execute the same tests across multiple model providers and families (e.g., OpenAI, Claude, Gemini, Llama) to compare quality and safety consistently.
- CLI and Web UI: Command‑line tools for running tests and a web 'view' UI to inspect prompts, final rendered prompts, outputs, and structured results in tabular form.
- Declarative Configs & Templating: Use promptfooconfig.yaml with Nunjucks templating and custom filter plugins to generate complex prompts and test permutations programmatically.
- Extensible Provider & Plugin System: Add or customize providers, local execution, or custom filters (JS/Python) to adapt tests to specific stacks or private model endpoints.
- Docker Distribution & Local Execution: Official container images and local execution modes enable isolated, reproducible runs and CI friendliness.
- GitHub & CI Integrations: Official GitHub Action and CI-friendly tooling to automatically post evaluations on PRs and enforce prompt quality gates.
- Command‑line interface and library for running declarative evals and tests
- Red‑teaming and vulnerability scanning for LLM outputs
- Declarative configuration via promptfooconfig.yaml (prompts, providers, filters, tests)
- Support for templated prompts using Nunjucks and custom filter modules
- Providers for multiple model backends (OpenAI and others; compare GPT, Claude, Gemini, Llama, etc.)
- Docker images published to GHCR (multi‑arch support: linux/amd64, linux/arm64, etc.)
- Web UI (src/app) that integrates with `promptfoo view` for inspecting outputs and final prompts
- CI/CD integrations including an official GitHub Action for evals on PRs
- Developer productivity features: live reload, caching, npm scripts for local dev
- Configurable Python executable (PROMPTFOO_PYTHON) and language‑agnostic test data (supports Python, JavaScript, others)
Best for
- Red‑teaming LLM integrations to find prompt injections, unsafe outputs, and info‑leakage before release.
- Regression testing in CI to automatically detect when a prompt or model update degrades output quality or safety on pull requests.
- Comparing model performance across providers and model families to choose the best model for a given task or guardrail requirements.
- Building test-driven prompt development workflows where prompts are versioned, evaluated, and iterated using reproducible eval suites.
- Adding automated before/after eval diffs on GitHub PRs to give reviewers quantitative and qualitative signal about prompt edits.
- Validating agents, RAG pipelines, and LLM apps end‑to‑end by running scenario-based tests and inspecting final rendered prompts and outputs.
- Test‑driven prompt engineering and automated evaluation of model outputs
- Red‑teaming and security testing of language model behavior
- Regression testing of prompts and model changes via CI/CD and GitHub Actions
- Comparing performance across multiple model providers
- RAG (retrieval augmented generation) and agent testing in local/dev environments
- Integrating automated evals into PR workflows to produce before/after views of prompt edits
Speech To Markdown
xajik
Free, 100% local macOS menu-bar app that turns speech into structured markdown using whisper.cpp and any local LLM.
Key features
- 100% Local Pipeline: Runs whisper.cpp for speech-to-text and any local LLM server for structuring — no cloud calls and no API keys required.
- Global Dictation Hotkey: Press ⌘⌥] in any app to have the transcript typed straight at your cursor, works in Terminal, browser, Slack, and more.
- Agent Mode Live Structuring: A floating capsule streams your voice through the LLM into a real-time Markdown, plain text, or HTML document.
- One-Line Install: A single curl-piped script installs xcodegen, whisper-cpp, and ffmpeg via Homebrew, then builds the app from source into /Applications.
- iOS Companion: A fully offline iPhone/iPad app that uses Apple Intelligence on iOS 26+ (iPhone 15 Pro and up).
- Multiple Output Formats: Format, edit, or append the LLM output as Markdown, plain text, or HTML from a single control panel.
- Send-Now Flush: The Send (⏎) control flushes the current buffer to the LLM immediately instead of waiting for the pause/word-count threshold.
- Model Picker: Download and swap Whisper models from Settings — Base (~150 MB) is a good starting point.
Best for
- Private Meeting Notes: Dictate meeting recaps on a Mac with sensitive content that must never leave the device.
- Voice-Driven Coding Comments: Speak function docstrings or PR descriptions into your editor at the cursor via Global Dictation.
- Structured Journaling: Use Agent Mode to ramble freely and get a clean, headed Markdown document out in real time.
- Offline Field Notes on iOS: Capture voice notes on an iPhone with no signal, structured into markdown using on-device Apple Intelligence.
- Slack / Email Long-Form: Dictate long replies straight into Slack or Mail without opening a separate transcription tool.