HeartMuLa AI Music Generator
AIOpen-source music foundation models and generator that create full songs (melody, vocals, and lyrics) from text prompts and tags.
-(0 Reviews)
Free Available
Starting from Free
HeartMuLa AI Music Generator
AI
Open-source music foundation models and generator that create full songs (melody, vocals, and lyrics) from text prompts and tags.
-(0 Reviews)
Free Available
Starting from Free
About HeartMuLa AI Music Generator
HeartMuLa is a family of open-source music foundation models and tooling that generate complete songs from text or lyrics, producing melody, instrumental arrangement, and synthesized vocals. The system separates a transformer-based generation backbone (HeartMuLa) from an audio codec (HeartCodec) and includes a transcription model (HeartTranscriptor) for lyrics extraction. It supports prompt conditioning via lyrics and tags, section-level control, and multiple model variants (including 3B and RL-tuned versions) for different fidelity and performance trade-offs. HeartMuLa is designed for local inference and integration with UI projects (Gradio, ComfyUI, Next.js studios), and can download model weights from HuggingFace/ModelScope for offline use.
Screenshots
+
Key Features
End-to-End Song Generation: Produces full songs (melody, arrangement, and vocal synthesis) from plain text prompts or lyrics and user-provided tags, exporting audio (e.g., MP3) for immediate use.
Modular Architecture: Separates a transformer-based generation model (HeartMuLa) from an audio codec (HeartCodec) so users can swap or update components independently for fidelity or speed trade-offs.
Multiple Model Variants: Offers model checkpoints including standard 3B, 'happy-new-year' variants, and RL-tuned models to balance audio quality, lyric clarity, and inference resource requirements.
Lyrics Transcription: Includes a transcription component (HeartTranscriptor, Whisper-based) to convert input audio into text, enabling lyric extraction and alignment workflows.
Local Inference & Downloadable Weights: Official support for downloading model weights from HuggingFace or ModelScope and running locally; examples and scripts provided for offline generation.
Developer & UI Integrations: Ready-made examples and community plugins for ComfyUI, Gradio, and web studio projects to enable interactive generation, low-VRAM modes, and one-click installs.
Low-VRAM & Performance Optimizations: Community tooling and ComfyUI nodes implement low-VRAM modes and smart device loading to allow 3B-class models to run on consumer GPUs (e.g., 12GB VRAM) by moving components between CPU/GPU during inference.
Post-Processing & DSP Utilities: Audio post-processing utilities (e.g., mastering tools) and codec decoders included to convert model tokens into high-fidelity playable audio.
Text-to-song generation: generate complete songs (melody + vocals) from lyrics and tags
Lyrics transcription: Whisper-based model to transcribe lyrics from audio
Modular architecture: separate model loaders (LLM backbone), codec loader (HeartCodec), generator, and audio decoder
Low VRAM mode: intelligent device management keeps models on CPU and moves needed components to GPU at inference time
Automatic model download: optional automatic fetching of checkpoints from Hugging Face or ModelScope
Device loading options: load_device flag to choose CPU or CUDA (supports mixed-device workflows)
HeartCodec audio decoder: audio decoding in fp32 for maximum fidelity
Torch optimizations: support for torch.compile / inductor / default execution modes
ComfyUI custom nodes: prebuilt loader/generator/transcriptor nodes for visual workflows
CLI examples and Python API usage (examples/run_music_generation.py) with configurable model_path and version
Use Cases
Rapid Song Prototyping: Convert lyrics or short text prompts into full demo tracks (melody + vocals) to iterate on song ideas quickly without a studio.
Local/Private Music Production: Run models and codecs locally with downloaded weights for privacy-sensitive projects or on-premises production pipelines.
Integration into Music Studios and Web UIs: Embed HeartMuLa backends into Gradio, ComfyUI, or Next.js-based studios to provide interactive generation, section control, and history/tagging features for creators.
Lyric Transcription and Editing: Transcribe vocals from recordings into editable lyric text using HeartTranscriptor, enabling correction, alignment, and re-generation workflows.
Custom Model Fine-Tuning: Use open-source checkpoints and repo examples to fine-tune models or create RL-tuned variants for specific genres, voices, or production styles.
Automated Content Generation Pipelines: Automate creation of short songs for content channels (e.g., social, explainer videos) by combining HeartMuLa generation with tagging and programmatic post-processing.
Low-Resource Deployment: Deploy on consumer-grade GPUs using low-VRAM modes and community-optimized builds to make high-fidelity music generation accessible outside large cloud providers.
Generate full songs from user-provided lyrics and tags for demos or content creation
Local-first music production workflows on consumer GPUs (12GB+ VRAM with low VRAM optimizations)
Batch or scripted music generation via CLI/python examples for prototyping or automated pipelines
Integrate into web frontends (Gradio, Next.js + FastAPI) or custom UIs for interactive music studios
Transcribe vocals/lyrics from recorded audio for metadata generation or lyric editing
