🎧 dsh-audiogen

AI audio generation for DeepSeek Harness (DSH) — turn your DSH web GUI into an audio studio: text-to-speech, music, sound effects and voice design, from the sidebar panel or straight from the Agent.

English | 简体中文

✨ Features

Four generation modes: text-to-speech, music, sound effects, and voice design
Multi-vendor channels in one place: OpenAI-compatible TTS, MiniMax, ElevenLabs, Stability AI, or any custom OpenAI-compatible / generic POST endpoint
Per-channel model & voice catalogs with one-click discovery, display aliases, capability categories, and per-model advanced fields (duration, seed, steps, cfg_scale, loop, prompt influence, and ElevenLabs SFX output format split into format/sample rate/bitrate, combined host-side into the single output_format, …)
Model comparison: run the same prompt across 2–4 models at once with per-model parameter overrides — results are grouped side by side
Prompt enhancement: rewrite a rough idea into a ready-to-generate description with an LLM (pick any model from Settings → Models; falls back to the agent default model)
History with one-click restore: prompt, config, model set and the original audio come back into the panel — no regeneration, no extra cost
Resource library: auto-save generated audio (or opt in per run), organized by type — voices / music / SFX / TTS — with search, tags, rename, category moves, and full provenance (channel, model, voice id, prompt, params snapshot). Reuse a voice or music bed instead of regenerating
Agent tools: generate_audio and search_audio_library, manage_audio_voices (vendor voice browsing/deletion + prompt-based voice recommendation + role voice casting), plus bundled session skills — the Agent can generate and find audio on demand
Role voice casting: assign a primary voice (+ backups) to each character of a novel/game — manage_audio_voices action=cast takes character profiles (JSON array/object or a text description structured first) and applies deterministic hard filters (gender/age/use_case strict; accent is a preference relaxed only when the strict pool is empty) per character; the Agent picks voices globally (no primary reuse across lead/major roles) and action=save_cast validates membership, auto-fills backups, flags reuse and persists the plan to ~/.dsh/dsh-audiogen/cast-selections.json; then TTS with the chosen voice_id (or design a custom voice first via generate_audio(mode=voice_design))
Panel voice management: a «音色» entry in the studio's left mode row (next to TTS/music/SFX/voice-design) — browse/filter vendor voices (language/keyword/source + official ElevenLabs shared-voice filters), ask the agent default model to recommend voices for a natural-language requirement (e.g. «清亮甜美的少女音»), preview, delete account-owned voices (confirmed) and backfill the chosen voice_id into the TTS form; every AI recommendation is recorded automatically (last 50, shared by panel and Agent) so you can revisit requirements/channels/reasons and reuse a voice later
Keys stay local: API keys live in the local DSH settings document and generation is proxied by the local host; the browser and the Agent never touch plaintext credentials

📸 Screenshots

Generation panel	Resource library

Library — full provenance drawer	Channels settings

Channel editor (model catalog & auto capabilities)	LLM models (Settings → Models)

📦 Installation

The plugin is published on npm. DSH host (Node ≥ 20) required.

▼
bash
复制代码
dsh plugin --profile web add dsh-audiogen

Local development install:

▼
bash
复制代码
dsh plugin --profile web add /path/to/dsh-audiogen

Restart dsh web after install — the sidebar will show the AI Audio entry.

🚀 Quick start

Open Settings → Plugins → AI Audio
Add a channel: pick a preset provider (+ Add provider) or a custom endpoint (+ Add custom provider)
Fill in the API URL, API key, and the model/voice catalog (use Fetch available models to import them)
Save, then open the AI Audio sidebar panel:
- choose a mode (Speech / Music / Sound effects / Voice design)
- type your text or prompt (optional: ✨ Enhance prompt)
- pick a model — or tick Model comparison for 2–4 models at once
- press Start generation and play the results, download them, or add them to the resource library

🎛 Modes supported by each vendor

Mode	MiniMax	ElevenLabs	Stability AI	OpenAI-compatible / custom
TTS	✅ (8 voices)	✅ (voices + streams)	—	✅
Music	✅ (`music-3.0` / `music-2.6` / `music-cover`)	✅ (`music_v2`)	✅ (`stable-audio-*`)	✅ (generic POST)
Sound effects	—	✅ (`eleven_text_to_sound_v2`, loop / prompt influence / output format as codec+sample rate+bitrate → `output_format`)	✅ (`stable-audio-*` — same text-to-audio protocol; auto-detected in both Music and SFX)	✅ (generic POST)
Voice design	✅ (`/v1/voice_design`)	✅ (`/v1/text-to-voice/design`)	—	—

🤖 Agent usage

Tool	Purpose
`generate_audio`	Submit a TTS / music / SFX / voice-design task; waits for completion and returns same-origin audio URLs. Optional `enhance_prompt`, `save_to_library`, per-vendor params.
`manage_audio_voices`	Browse/filter the vendor voice libraries (MiniMax, ElevenLabs) with language/keyword/source filters; recommend top-k voices for a natural-language requirement (`action=recommend`, uses the agent default model, ids validated against the pool); role casting (`action=cast` prepares per-character filtered candidate pools from character profiles; `action=save_cast` validates + persists the plan); delete account-owned voices (official/shared/system voices are read-only and refused). Then use the returned `voice_id` with `generate_audio` (mode=tts).
`search_audio_library`	Search the local resource library (type / category / keyword) and reuse an existing voice, music bed or effect.

Typical session commands (skills bundled with the plugin):

▼
text
复制代码
/audio:tts      Read this sentence with a warm voice
/audio:music    Generate a 30-second lo-fi background track
/audio:sfx      Create a sci-fi UI cue
/audio:design   Craft a warm retro synth voice

🔐 Security & data notes

API keys are stored in the local DSH settings document; requests are proxied by the local host (/api/dsh-audiogen/*, loopback-only routes)
Generation consumes your upstream provider quota; audio content is produced by the upstream model
History & library persist under ~/.dsh/dsh-audiogen/
Prompt enhancement calls the LLM model you choose (default: agent default model) — no extra API key

🛠 Development

▼
bash
复制代码
pnpm install
pnpm run typecheck
pnpm run build      # outputs lib/ (host + client bundles)

📄 License

Apache-2.0

✨ Features

Four generation modes: text-to-speech, music, sound effects, and voice design

Multi-vendor channels in one place: OpenAI-compatible TTS, MiniMax, ElevenLabs, Stability AI, or any custom OpenAI-compatible / generic POST endpoint

Per-channel model & voice catalogs with one-click discovery, display aliases, capability categories, and per-model advanced fields (duration, seed, steps, cfg_scale, loop, prompt influence, and ElevenLabs SFX output format split into format/sample rate/bitrate, combined host-side into the single output_format, …)

Model comparison: run the same prompt across 2–4 models at once with per-model parameter overrides — results are grouped side by side

Prompt enhancement: rewrite a rough idea into a ready-to-generate description with an LLM (pick any model from Settings → Models; falls back to the agent default model)

History with one-click restore: prompt, config, model set and the original audio come back into the panel — no regeneration, no extra cost

Resource library: auto-save generated audio (or opt in per run), organized by type — voices / music / SFX / TTS — with search, tags, rename, category moves, and full provenance (channel, model, voice id, prompt, params snapshot). Reuse a voice or music bed instead of regenerating

Agent tools: generate_audio and search_audio_library, manage_audio_voices (vendor voice browsing/deletion + prompt-based voice recommendation + role voice casting), plus bundled session skills — the Agent can generate and find audio on demand

Role voice casting: assign a primary voice (+ backups) to each character of a novel/game — manage_audio_voices action=cast takes character profiles (JSON array/object or a text description structured first) and applies deterministic hard filters (gender/age/use_case strict; accent is a preference relaxed only when the strict pool is empty) per character; the Agent picks voices globally (no primary reuse across lead/major roles) and action=save_cast validates membership, auto-fills backups, flags reuse and persists the plan to ~/.dsh/dsh-audiogen/cast-selections.json; then TTS with the chosen voice_id (or design a custom voice first via generate_audio(mode=voice_design))

Panel voice management: a «音色» entry in the studio's left mode row (next to TTS/music/SFX/voice-design) — browse/filter vendor voices (language/keyword/source + official ElevenLabs shared-voice filters), ask the agent default model to recommend voices for a natural-language requirement (e.g. «清亮甜美的少女音»), preview, delete account-owned voices (confirmed) and backfill the chosen voice_id into the TTS form; every AI recommendation is recorded automatically (last 50, shared by panel and Agent) so you can revisit requirements/channels/reasons and reuse a voice later

Keys stay local: API keys live in the local DSH settings document and generation is proxied by the local host; the browser and the Agent never touch plaintext credentials

Generation panel

Resource library

Library — full provenance drawer

Channels settings

Channel editor (model catalog & auto capabilities)

LLM models (Settings → Models)

🚀 Quick start

Open Settings → Plugins → AI Audio

Add a channel: pick a preset provider (+ Add provider) or a custom endpoint (+ Add custom provider)

Fill in the API URL, API key, and the model/voice catalog (use Fetch available models to import them)

Save, then open the AI Audio sidebar panel:

choose a mode (Speech / Music / Sound effects / Voice design)
type your text or prompt (optional: ✨ Enhance prompt)
pick a model — or tick Model comparison for 2–4 models at once
press Start generation and play the results, download them, or add them to the resource library

🎛 Modes supported by each vendor

Mode

MiniMax

ElevenLabs

Stability AI

OpenAI-compatible / custom

TTS

✅ (8 voices)

✅ (voices + streams)

—

✅

Music

✅ (music-3.0 / music-2.6 / music-cover)

✅ (music_v2)

✅ (stable-audio-*)

✅ (generic POST)

Sound effects

—

✅ (eleven_text_to_sound_v2, loop / prompt influence / output format as codec+sample rate+bitrate → output_format)

✅ (stable-audio-* — same text-to-audio protocol; auto-detected in both Music and SFX)

✅ (generic POST)

Voice design

✅ (/v1/voice_design)

✅ (/v1/text-to-voice/design)

—

🤖 Agent usage

Tool

Purpose

generate_audio

Submit a TTS / music / SFX / voice-design task; waits for completion and returns same-origin audio URLs. Optional enhance_prompt, save_to_library, per-vendor params.

manage_audio_voices

Browse/filter the vendor voice libraries (MiniMax, ElevenLabs) with language/keyword/source filters; recommend top-k voices for a natural-language requirement (action=recommend, uses the agent default model, ids validated against the pool); role casting (action=cast prepares per-character filtered candidate pools from character profiles; action=save_cast validates + persists the plan); delete account-owned voices (official/shared/system voices are read-only and refused). Then use the returned voice_id with generate_audio (mode=tts).

search_audio_library

Search the local resource library (type / category / keyword) and reuse an existing voice, music bed or effect.

Typical session commands (skills bundled with the plugin):

▼
text
复制代码

/audio:tts Read this sentence with a warm voice /audio:music Generate a 30-second lo-fi background track /audio:sfx Create a sci-fi UI cue /audio:design Craft a warm retro synth voice

🔐 Security & data notes

API keys are stored in the local DSH settings document; requests are proxied by the local host (/api/dsh-audiogen/*, loopback-only routes)

Generation consumes your upstream provider quota; audio content is produced by the upstream model

History & library persist under ~/.dsh/dsh-audiogen/

Prompt enhancement calls the LLM model you choose (default: agent default model) — no extra API key