Skip to content

Speech Recognition Providers in Hedy

What are Speech Recognition Providers?

Hedy supports multiple speech recognition options, giving you flexibility to choose between complete privacy with local processing or cloud-based alternatives. You can switch providers anytime based on your current needs - use local for offline sessions and cloud services when you prefer their specific features.

Getting Started

  1. Open the Hedy app

  2. Open Settings and go to the Speech & AI tab

  3. Look at the list of speech models. Each model is one row. The row you use now says In use. The cloud services are in a separate section, Cloud services · uses your own API key

  4. Tap the row you want. If the model isn’t on your device yet, choose Download on its row first. For a cloud service, choose Set up to add your API key

  5. Your choice takes effect in the next session

A Recommended tag with the line Best match for your meeting language marks the row that suits your meeting language best on your device. Each row shows the model name with its maker (NVIDIA or OpenAI) underneath, a few facts such as the download size (for example 2.4 GB download), the languages it covers and whether it gives instant captions, and a status on the right: In use, Downloaded, Included with Hedy or Ready. While a model downloads, its row shows the progress and a Cancel button. If a download fails, choose Retry. If an update is available for a model, its row offers only Update.

Rows are named after the model behind them:

RowMakerEngine
Nemotron EnglishNVIDIANemotron English Only (local)
Nemotron MultilingualNVIDIANemotron Multilingual (local)
Whisper Base (Mac, Windows, iPhone, iPad and Android)OpenAIWhisper (local), Base model
Whisper Small (Mac, Windows, iPhone, iPad and Android)OpenAIWhisper (local), Small model
Whisper Large v3 Turbo (Mac and Windows)OpenAIWhisper (local), Large v3 Turbo model
Deepgram, OpenAI, xAICloud services with your own API key

Whisper Large v3 Turbo is a one-time download (about 900 MB on a Mac, about 550 MB on Windows). Hedy keeps using your current model until the download has finished and checked out, and only then switches. If you cancel the download, nothing changes.

Improved transcription on Mac and Windows

If you have never chosen a speech engine on an Apple Silicon Mac or a Windows PC, Hedy offers Improved transcription. It downloads the on-device engine that suits your meeting language in the background, shows the progress, and switches to it once the download is complete. If you cancel the download, or it can’t finish, you can start it again from that model’s row in Settings → Speech & AI.

  • Starting a session pauses the download, and it continues after the session.
  • If the download doesn’t finish, Hedy keeps working as before.
  • If you chose an engine yourself, Hedy never changes it.

Available Providers

Hedy offers five speech recognition options, each with unique characteristics:

  • Whisper (local): Default option - 100% private, works offline, no usage costs. Your audio never leaves your device for transcription. Can add speaker labels after the session ends on native apps, except 32-bit Android. Whisper is not available in the web app.

  • Nemotron (local): A newer on-device streaming engine with live transcripts and on-device speaker labels. You choose between an English-only mode (the fastest option) and a multilingual mode that covers a broad set of major languages. Available on every platform Hedy ships a native app for: Apple Silicon Macs on macOS 15 or later, iPhone XS (or newer), iPads with a Neural Engine (iPad Pro 2018, iPad Air 3, iPad mini 5, iPad 8th generation, or newer), Windows, and Android. On Apple hardware it runs on the Neural Engine and labels speakers live; on Windows and Android the labels are added at the end of the session. Requires a one-time model download (about 0.6 GB for English-only, 0.7 GB for multilingual).

  • Deepgram: Cloud-based service with real-time streaming and smart formatting features. Uses Nova-3, which supports dozens of languages. Hedy exposes every language Nova-3 offers, so you can transcribe meetings in any supported language without switching providers, including Swiss German, Armenian, Gujarati, Nepali and Punjabi, plus regional variants of Dutch, French, Portuguese, Spanish and Chinese. Requires your own API key.

  • OpenAI: Cloud transcription with Voice Activity Detection and automatic language detection. Live sessions run on OpenAI’s current generation, gpt-live-transcribe, and imported files on gpt-transcribe; the older models stay selectable if you had one saved. Hedy automatically continues long sessions past OpenAI’s 60-minute per-connection cap by rotating connections behind the scenes, so hour-plus meetings keep going without interruption. Requires your own API key.

  • xAI: Cloud transcription on xAI’s grok-voice-transcribe, with live streaming for sessions and batch transcription for imported files. It covers 25 languages, and your custom vocabulary is sent along as hints, so names and product terms come back spelled the way you want them. Requires your own API key.

Configuring Whisper (local)

Whisper appears as several rows, so you pick the size that fits your device:

On a Mac or Windows PC:

  • Whisper Base: the lightest and fastest. Hedy recommends it on Intel Macs

  • Whisper Small: balanced speed and accuracy. Hedy recommends it on Apple Silicon Macs running macOS 14

  • Whisper Large v3 Turbo: the most accurate choice for many non-English languages (a one-time download of about 900 MB on a Mac, about 550 MB on Windows)

On iPhone, iPad and Android:

  • Whisper Base: the standard choice

  • Whisper Small: more accurate, especially in non-English languages (recommended only for iPhone 12 or newer and Android phones from 2024 or later)

Voice Activity Detection (VAD):

VAD automatically filters out silence and background noise to improve transcription quality. This feature is enabled by default for Whisper.

  • Enable/Disable: Toggle VAD on or off based on your recording environment

  • Sensitivity: Adjust from “High Sensitivity” (captures more speech, including quieter sounds) to “Maximum Filtering” (only captures clear speech, filters more background noise)

Transcript Frequency:

  • Slower: Updates the transcript about twice a minute (saves battery)

  • Normal: Updates the transcript every 5 seconds (recommended for most users)

  • Faster: Updates the transcript every second (uses more power)

Configuring Nemotron (local)

It transcribes entirely on-device and shows live transcripts as you talk. It’s available on every platform Hedy ships a native app for: iOS, iPadOS, macOS, Windows, and Android. On Apple hardware it runs on the Neural Engine.

Device requirements:

  • Apple Silicon Mac (M1 or newer) running macOS 15 or later, or

  • iPhone XS or newer, or an iPad with a Neural Engine (iPad Pro 2018, iPad Air 3, iPad mini 5, iPad 8th generation, or newer), or

  • Windows, or

  • A 64-bit Android phone

English-only or multilingual:

Nemotron appears as two rows, so you can pick the one that matches your meetings:

  • Nemotron English (Nemotron English Only): streaming English transcription, the fastest option.

  • Nemotron Multilingual (Nemotron Multilingual): on-device streaming across a broad set of major languages, for when you need more than English.

Both run fully on-device. English Only always transcribes in English and ignores your meeting language setting. Multilingual uses your Meeting/Class Language unless you turn on Auto-detect language (Settings > Speech & AI), and then it identifies the language from the audio instead.

First-time setup:

  1. In Settings → Speech & AI, find the Nemotron English or Nemotron Multilingual row

  2. Choose Download on the row (about 0.6 GB for English-only, 0.7 GB for multilingual) - we recommend Wi-Fi

  3. Once the download finishes, tap the row to switch to it. Nemotron is used from your next session

If you start a session while a Nemotron download is waiting, the row says the download is paused during your session, and it continues once the session ends. If the files of your chosen Nemotron model go missing on an iPhone, iPad, Mac or Windows PC, Hedy downloads them again on its own while you’re not in a session and on Wi-Fi, and uses Whisper in the meantime.

Speaker labels and the temporary audio cache:

Both on-device engines label who’s speaking. With Nemotron on Apple hardware the labels appear live as you talk; with Nemotron on Windows and Android, and with Whisper on supported devices, they are added at the end of the session. To work out the speakers, Hedy keeps each session’s audio in a temporary on-device cache while it processes, then deletes it. This audio stays on your device. The setting, Temporary audio cache, is on by default; you can turn it off in Hedy’s settings, though leaving it on gives the best speaker attribution. Whisper needs either this cache or Save Session Audio, plus Voice Activity Detection, to add speaker labels. Whisper speaker labels are not available in the web app or on 32-bit Android.

Setting Up Cloud Providers

Deepgram Setup:

  1. Create an account at console.deepgram.com

  2. Generate an API key from your dashboard

  3. In Settings → Speech & AI, choose Set up on the Deepgram row

  4. Paste your API key and tap “Test” to verify

  5. Choose your model and language preferences

  6. Set maximum session duration to control costs

OpenAI Setup:

  1. Get your API key from platform.openai.com/api-keys

  2. In Settings → Speech & AI, choose Set up on the OpenAI row

  3. Enter your API key and test the connection

  4. Choose your preferred model. Live sessions default to gpt-live-transcribe, and imported files use gpt-transcribe

  5. Optionally enable Voice Activity Detection with adjustable sensitivity

  6. Set maximum session duration for cost control

xAI Setup:

  1. Get your API key from console.x.ai

  2. In Settings → Speech & AI, choose Set up on the xAI row

  3. Paste your API key

  4. Set maximum session duration for cost control

Choosing the Right Provider

Select based on your priorities and use case:

  • Privacy First: Use a local engine (Whisper or Nemotron) - audio never leaves your device for transcription

  • Offline Use: All local engines work without internet

  • Cloud Features: Deepgram, OpenAI and xAI offer cloud-based processing

  • Voice Detection: Whisper and OpenAI include Voice Activity Detection features

  • Smart Formatting: Deepgram offers automatic formatting options

  • No Usage Costs: Local engines (Whisper, Nemotron) have no per-minute charges

  • Faster On-Device Transcription: Nemotron typically delivers a lower-latency transcript than Whisper

  • Multilingual On-Device Streaming: Nemotron Multilingual gives you on-device transcription across a broad set of languages

  • Best On-Device Engine for Your Language: See the table below

  • Fully Private Analysis: On Apple Silicon Macs, Windows PCs, iPhone 15 Pro or newer, M-series or A17 Pro iPads and supported Android phones, you can pair local speech recognition with Local AI Processing to keep both transcription and AI analysis fully on-device.

Which On-Device Engine to Use for Your Language

Whisper and Nemotron are not equally accurate in every language. Find your Meeting/Class Language below to see which one to pick. Where both engines handle a language well, we recommend Nemotron, because it is lighter and faster.

A few notes that apply to the whole table:

  • Nemotron means Nemotron Multilingual (local). For English, pick Nemotron English Only (local) (the Nemotron English row) instead.

  • Whisper Large v3 Turbo means the Whisper Large v3 Turbo row on a Mac or Windows PC. It is the most accurate on-device option for several languages, but it uses more memory and runs slower than Nemotron. If it is too slow on your computer, Nemotron is the next best choice for those languages.

  • Whisper Large v3 Turbo isn’t available on phones and tablets. Where the table says Whisper, pick Whisper Small if your device supports it.

  • If your device can’t run Nemotron (an Intel Mac or an older iPhone, for example), use Whisper.

  • The Recommended tag in Settings follows the same advice where Hedy can tell what fits your device. On Android, and on phones for Arabic, Hindi and Thai, no row is marked as recommended, so use this table.

  • Accents, dialects, people talking over each other and distant microphones lower accuracy in every language.

Meeting languageMac or Windows PCiPhone, iPad or Android
EnglishNemotron English OnlyNemotron English Only
ArabicNemotronNot available on mobile
DutchNemotronNemotron
FrenchNemotronNemotron
GermanNemotronNemotron
HindiNemotronNot available on mobile
ItalianNemotronNemotron
JapaneseNemotronNemotron
KoreanNemotronNemotron
PortugueseNemotronNemotron
RussianNemotronNemotron
SpanishNemotronNemotron
TurkishNemotronNemotron
UkrainianNemotronNemotron
VietnameseNemotronNemotron
Chinese (Mandarin)Whisper Large v3 TurboNemotron
CroatianWhisper Large v3 TurboNemotron
CzechWhisper Large v3 TurboNemotron
DanishWhisper Large v3 TurboNemotron
FinnishWhisper Large v3 TurboNemotron
HungarianWhisper Large v3 TurboNemotron
NorwegianWhisper Large v3 TurboNemotron
PolishWhisper Large v3 TurboNemotron
RomanianWhisper Large v3 TurboNemotron
SlovakWhisper Large v3 TurboNemotron
SwedishWhisper Large v3 TurboNemotron
AfrikaansWhisper Large v3 TurboWhisper
CantoneseWhisper Large v3 TurboWhisper
CatalanWhisper Large v3 TurboWhisper
GreekWhisper Large v3 TurboWhisper
HebrewWhisper Large v3 TurboWhisper
IndonesianWhisper Large v3 TurboWhisper
MalayWhisper Large v3 TurboWhisper
SerbianWhisper Large v3 TurboWhisper
SlovenianWhisper Large v3 TurboWhisper
TagalogWhisper Large v3 TurboWhisper
ThaiWhisper Large v3 TurboNot available on mobile

Expect noticeably lower accuracy in Afrikaans, Cantonese, Hebrew and Serbian than in the other languages, whichever engine you use.

Cost Considerations

Understanding the cost implications of each provider:

  • Whisper (local): Free - no usage charges

  • Nemotron (local): Free - no usage charges (one-time model download, about 0.6-0.7 GB)

  • Deepgram: Pay-per-minute pricing (check current rates on their dashboard)

  • OpenAI: Usage-based pricing (check current rates on their platform)

  • xAI: Hourly pricing, charged separately for live sessions and imported files (check current rates on their platform)

The maximum session duration setting helps prevent accidental overnight recordings and manage API costs. When a live session reaches the limit, Hedy keeps the session going and transcribes the rest of it on your device instead.

Best Practices

  • Start with Whisper (local) to familiarize yourself with the feature, then try Nemotron if your device is supported

  • Test cloud providers with short recordings before important sessions

  • Monitor your API usage on provider dashboards to track costs

  • Use different providers for different scenarios based on your needs

  • Switch to local when traveling or in areas with limited internet

  • Set the maximum session duration a little longer than your longest meetings (the default is 120 minutes), so a long meeting stays on cloud transcription from start to finish

Troubleshooting

API Key Not Working

  • Ensure you copied the complete key without spaces

  • Verify your account has available credits

  • Check the API key has necessary permissions

  • Try regenerating the key from provider dashboard

Connection Test Failed

  • Check your internet connection stability

  • Verify firewall isn’t blocking WebSocket connections

  • Ensure API key is active with sufficient quota

  • Wait a moment and try again (temporary service issues)

Transcription Issues

  • For Whisper: Try a different row (Whisper Base, Whisper Small or Whisper Large v3 Turbo)

  • For Whisper on Windows: If transcription lags far behind the conversation, check slow transcription GPU settings

  • For specialized terms, names, and acronyms: Add them via the custom vocabulary feature

  • For Nemotron: Use the English Only mode for English meetings; for other languages, use the Multilingual mode or switch to Whisper with the language set explicitly

  • For Cloud: Check internet connection stability

  • Ensure microphone is properly configured

  • Minimize background noise during recording

Settings Not Saving

  • Wait for the “Saved” indicator to appear

  • Don’t switch screens while saving

  • Restart the app if issues persist

  • Ensure you have a stable internet connection

Your API keys are stored securely in your device’s encrypted keychain and never transmitted to Hedy’s servers. For maximum privacy with sensitive conversations, always use a local engine (Whisper or Nemotron).