A.X K2 Raon-Speech: Introducing Korea’s First Independently Developed SpeechLM

How We Built A.X K2 Raon-Speech

A.X K2 Raon-Speech is designed to understand speech input and generate both text and speech responses. For speech understanding, the speech encoder converts input speech into embeddings used as language model input. For speech generation, the speech generation module predicts discrete speech tokens from the language model’s hidden states, and the codec decoder reconstructs the final waveform. The system includes a 300M speech encoder, a 90M speech codec, and the A.X K2 Light base language model. With the speech input and output modules, the complete A.X K2 Raon-Speech model has 21B total and 3B active parameters. The following sections describe how we trained the speech encoder and codec, extended A.X K2 Light into a SpeechLM, and conducted post-training.

1. Training the Speech Modules In-House

Rather than propose a new architecture, this project tested whether a competitive model could be trained from scratch using independently collected and curated data and an in-house training pipeline. Changing both the data and architecture would make it difficult to isolate the source of performance differences.

Speech Encoder

The A.X K2 Raon-Speech speech encoder uses the AuT architecture from Qwen3-ASR-1.7B and was trained from scratch with a small decoder on STT and QA tasks^[6]. Table 2 compares this encoder with Qwen AuT on the speech recognition (STT), SpokenQA, and SpeechQA benchmarks used to evaluate Raon-Speech. Speech recognition uses English word error rate (WER) and Korean character error rate (CER); SpokenQA and SpeechQA use accuracy.

Qwen3-ASR-1.7B AuT was trained on approximately 40 million hours of audio. Our model achieved similar performance using 1 million hours, about one-fortieth as much data.

Table 3 shows the input and output formats and examples for each benchmark task.

Speech Codec

The A.X K2 Raon-Speech speech codec uses the Mimi codec architecture from Moshi and was trained from scratch^[7]. We applied WavLM-based semantic distillation so that the discrete speech tokens would encode semantic information suitable for language model processing^[8]. Moshi introduced the Mimi architecture but did not disclose its training data or detailed recipe. We therefore designed and tested several codec training recipes and selected a stable two-stage setup.

• Stage 1: Train the codec to stable convergence using reconstruction loss and semantic distillation.
• Stage 2: Freeze the encoder trained in Stage 1, add a discriminator, and apply adversarial training loss to improve perceptual quality, primarily through the decoder.

We separated the stages because introducing discriminator-based adversarial training at the start, or applying regression and adversarial losses together, prevented stable convergence.

Table 4 compares Mimi with our codec. Moshi used approximately 7 million hours of audio, while our model approached its performance using 500,000 hours, about one-fourteenth as much data.

These results show that speech encoders and codecs trained in-house can support competitive SpeechLM performance at a relatively small data scale. The next step is to improve the architecture rather than only reproduce training recipes. The encoder and codec could expand from speech to general audio, including sound effects, environmental sounds, and music, enabling the model to understand and generate non-speech audio cues. Mimi-style semantic distillation could also be made less dependent on language and domain, while the SpeechLM input/output formats and encoder/decoder architecture could be redesigned for efficiency. FSQ-based acoustic tokenization, as used in Voxtral TTS, is another direction beyond RVQ-based methods^[17].

2. Extending the Language Model into a SpeechLM

Our training strategy began with the SpeechLM architecture and recipe validated in Raon-Speech and focused on connecting speech input and output while preserving A.X K2 Light's text capabilities. We first froze the language model and trained only the speech modules, aligning speech input with A.X K2 Light hidden states and its output with discrete speech-token generation. We then trained the language model jointly to improve speech processing. Half of the training mix remained text and existing task data to prevent degradation of text capabilities. We also used data generated by A.X K2 Light and subsequently refined to preserve its instruction-following and reasoning behavior. For text-only data, we distilled from A.X K2 Light outputs. For speech input, we used the model's response to the equivalent text input as the target for cross-modal distillation.

This completed the pretraining stage for extending A.X K2 Light into a SpeechLM. We then conducted separate post-training stages for tool use, emotion and speaker recognition, and stable speech generation. The following examples show the effect of each stage.

Tool Use

A practical SpeechLM needs tool-use capability. We extended text-based tool-use data with generated speech for training. In the news briefing example below, the model receives a spoken question, selects and invokes a tool, and integrates the result into its response. It handled tool calls and tool responses reliably in a single turn. Maintaining context and state and combining tool results across multiple turns remains future work.

The example is based on actual model output, with some tool inputs and outputs edited for clarity.

{
  "query": "latest sports news",
  "num_results": 5
}

{
  "query": "latest sports news",
  "summary": "Mexico beat Ecuador 2-0 in the round of 32 at the 2026 World Cup, earning its first World Cup knockout-stage win in 40 years and advancing to the round of 16. Lee Kang-in reportedly completed personal terms for a move to Atletico Madrid, with an official announcement expected soon. The Netherlands lost to Morocco on penalties in the round of 32, after which head coach Ronald Koeman resigned. [yna.co.kr](https://www.yna.co.kr/view/AKR20260701105900007), [starnewskorea.com](https://www.starnewskorea.com/sports/2026/07/01/2026070114191566948), [starnewskorea.com](https://www.starnewskorea.com/sports/2026/07/01/2026070109534230105)",
  "sources": [
    {
      "title": "[World Cup] Mexico beats Ecuador 2-0 and advances to round of 16 | Yonhap News",
      "url": "https://www.yna.co.kr/view/AKR20260701105900007"
    },
    {
      "title": "Lee Kang-in nears Atletico Madrid move; announcement expected this week | Star News",
      "url": "https://www.starnewskorea.com/sports/2026/07/01/2026070114191566948"
    },
    {
      "title": "Netherlands coach Ronald Koeman resigns after loss to Morocco | Star News",
      "url": "https://www.starnewskorea.com/sports/2026/07/01/2026070109534230105"
    },
    {
      "title": "Kim Sun-bin passes Lee Jong-beom's 1,797-hit mark | Money Today",
      "url": "https://www.mt.co.kr/sports/2026/07/01/2026070113152957009"
    },
    {
      "title": "Manager Lee Kang-chul emphasizes the period before the All-Star Game | Sports Kyunghyang",
      "url": "https://sports.khan.co.kr/article/202607011130003"
    }
  ]
}

Emotion and Speaker Recognition

Emotion and speaker cues are easily lost when speech is converted to text before response generation. A transcript does not fully preserve delivery, pitch range, timbre, or emotion. These cues matter when interpreting a user's state or adapting a response to the voice.

To address this, we applied self-distillation to emotion-related datasets^[9]. The model reuses emotion and speaker cues inferred from speech as training signals, helping it learn nonverbal information not retained in text. In the song recommendation example below, the same request produces different rationales and results based on perceived gender, pitch range, and timbre. The model uses cues from the speech signal as well as the transcript.

The example is based on actual model output, with some tool inputs and outputs edited for clarity.

{
  "vocal_profile": {
    "gender": "female",
    "pitch": "high",
    "timbre": [
      "bright",
      "clear"
    ]
  },
  "music_criteria": {
    "genre": "pop",
    "mood": "happy"
  }
}

{
  "recommendations": [
    {
      "artist": "NMIXX",
      "song": "DICE",
      "range": "mid",
      "difficulty": "hard",
      "vocal_demand": [
        "powerful",
        "rhythmic"
      ],
      "reason": "Matches a mid range and a powerful, bright, clear timbre"
    },
    {
      "artist": "Lee Young-hyun",
      "song": "Yeon (捐)",
      "range": "high",
      "difficulty": "hard",
      "vocal_demand": [
        "powerful",
        "sustained"
      ],
      "reason": "Matches a high range and a powerful, clear timbre"
    },
    {
      "artist": "SAAY",
      "song": "Summer In Love",
      "range": "mid",
      "difficulty": "mid",
      "vocal_demand": [
        "gentle",
        "falsetto",
        "rhythmic"
      ],
      "reason": "Matches a mid range, moderate difficulty, and clear timbre"
    },
    {
      "artist": "TAEYEON",
      "song": "I",
      "range": "high",
      "difficulty": "hard",
      "vocal_demand": [
        "powerful",
        "sustained",
        "belting"
      ],
      "reason": "Matches a high range and a powerful, bright, clear timbre"
    }
  ],
  "applied_filters": [
    "genre",
    "mood"
  ],
  "relaxed": [
    "genre",
    "mood"
  ]
}

{
  "vocal_profile": {
    "gender": "male",
    "pitch": "low",
    "timbre": [
      "deep",
      "powerful"
    ]
  },
  "music_criteria": {
    "genre": "rock",
    "mood": "energetic"
  }
}

{
  "recommendations": [
    {
      "artist": "Lee Seung-hwan",
      "song": "Dunk Shot",
      "range": "mid",
      "difficulty": "mid",
      "vocal_demand": [
        "rhythmic",
        "powerful"
      ],
      "reason": "Matches a mid range, moderate difficulty, and powerful timbre"
    },
    {
      "artist": "Songgolmae",
      "song": "The Moment I First Saw You",
      "range": "mid",
      "difficulty": "mid",
      "vocal_demand": [
        "powerful",
        "rhythmic"
      ],
      "reason": "Matches a mid range, moderate difficulty, and powerful timbre"
    },
    {
      "artist": "THE KOXX",
      "song": "Trouble Maker",
      "range": "mid",
      "difficulty": "mid",
      "vocal_demand": [
        "belting",
        "powerful",
        "rhythmic"
      ],
      "reason": "Matches a mid range, moderate difficulty, and powerful timbre"
    },
    {
      "artist": "Cho Yong-pil",
      "song": "Let's Go on a Trip",
      "range": "mid",
      "difficulty": "mid",
      "vocal_demand": [
        "rhythmic",
        "powerful"
      ],
      "reason": "Matches a mid range, moderate difficulty, and powerful timbre"
    }
  ],
  "applied_filters": [
    "genre",
    "mood"
  ]
}

Speech Generation Stability

Speech generation sometimes prolonged syllables abnormally or failed to complete sentences. We applied DPO to address these issues^[10]. Training used approximately 5,000 samples from INTP, an English preference dataset^[11]. For Korean, we selected a similarly sized set from a model-generated pool and adapted it to the language. The examples below compare output before and after DPO.

After this stabilization, A.X K2 Raon-Speech can go beyond clear, read-aloud delivery to generate Korean speech that more closely resembles natural conversation. The examples below demonstrate synthesized informal dialogue with fillers and hesitations such as “uh,” “like,” and “how should I put it,” as well as natural conversational pacing and intonation.

Performance Gains by Stage

Each post-training stage improved its target capability over the preceding checkpoint. Tool-use and emotion/speaker scores are accuracy measurements on internal benchmarks; speech generation uses average word error rate (WER). Tool-use accuracy increased from 0.54 to 0.74, emotion and speaker accuracy from 0.57 to 0.61, and average speech-generation WER decreased from 3.6% to 3.1%. Performance on untargeted capabilities, including speech recognition, SpokenQA, and general SpeechQA, remained at 105% of the initial post-training model on average (0.71 to 0.74). We observed no regression from specialized fine-tuning.

Figure 3 summarizes the effect of each post-training stage.

Final Performance Analysis

The final evaluation compares public SpeechLMs across speech understanding, speech generation, tool use, and emotion and speaker recognition in English and Korean. For consistency, all reported scores were measured in-house using the same benchmarks and evaluation criteria. A.X K2 Raon-Speech ranked first in Korean and third in English.

Comparison Set

We retained the comparison set used for Raon-Speech^[5]. Because A.X K2 Raon-Speech differs from Raon-Speech in architecture and scale, we added five models: comparable MoE models and SpeechLMs released after Raon-Speech^{[12][13][14][15][16]}. The comparison set includes Audio Flamingo 3 and AF-Next from NVIDIA and the University of Maryland; Qwen3-Omni, Qwen2.5-Omni, and Fun-Audio-Chat from Alibaba Cloud; Kimi-Audio from Moonshot AI; MOSS-Audio from OpenMOSS; and HyperCLOVA X 8B Omni from NAVER. Table 5 reports our model and 14 comparison models, for 15 rows in total.

Results Summary

• First in Korean — highest among all comparison models (0.72).
• Third in English — 0.75, behind Raon-Speech (0.79) and Qwen3-Omni (0.78).

Korean Rank	Model	Korean	English	English Rank
1	A.X K2 Raon-Speech	0.72	0.75	3
2	Raon-Speech	0.70	0.79	1
3	Qwen3-Omni	0.65	0.78	2
4	HyperCLOVA X 8B Omni	0.60	0.68	8
5	MOSS-Audio	0.58	0.62	13
6	Qwen2.5-Omni	0.55	0.73	5
7	Fun-Audio-Chat	0.53	0.69	7
8	Step-Audio 2 mini	0.32	0.67	10
9	Interactive-Omni	0.27	0.71	6
10	Ming-Lite-Omni v1.5	0.21	0.68	8
10	AF-Next	0.21	0.65	11
12	COVO-Audio	0.18	0.41	15
—	MiniCPM-o 4.5	—	0.74	4
—	Kimi-Audio	—	0.65	11
—	Audio Flamingo 3	—	0.60	14

Performance by Language

A.X K2 Raon-Speech performs particularly well in Korean, scoring 0.72, the highest among all comparison models. Its English score of 0.75 ranks third behind Raon-Speech and Qwen3-Omni. Raon-Speech leads in English (0.79 versus 0.75), while A.X K2 Raon-Speech has a small lead in Korean (0.72 versus 0.70). Qwen3-Omni also scores higher in English (0.78) but lower in Korean (0.65).

Conclusion

A.X K2 Raon-Speech combines SKT's independently developed A.X K2 Light language model with an in-house speech encoder and codec, using the SpeechLM architecture and training recipe validated in Raon-Speech. We trained an AuT-based speech encoder and Mimi-based codec with our data and recipes, then connected them to A.X K2 Light to support both speech understanding and generation. Post-training improved tool use, emotion and speaker recognition, and speech-generation stability. The model ranked third in English and first in Korean in the final evaluation, showing that the in-house speech module stack can translate into SpeechLM performance.

The significance of this project extends beyond benchmark rankings. Adapting and controlling a SpeechLM for Korean and English users requires the ability to train, integrate, and evaluate the full pipeline, including the language model, speech encoder, and speech codec. A.X K2 Raon-Speech is Korea's first public speech foundation model built through independent training and demonstrates this capability.

Citation

Keon Lee†, Beomsoo Kim, Dohyun Kim, Dongwon Kim, Jihwan Moon, Joonghyun Bae, Sehun Lee, Seungjun Chung, Sungwoo Cho‡, Jaewoong Cho*

KRAFTON AI

* Corresponding author. † Project lead. ‡ Contributed during an internship. All other authors are listed in alphabetical order.

@misc{axk2raonspeech2026,
  title  = {A.X K2 Raon-Speech: Introducing Korea's First Independently Developed SpeechLM},
  author = {Keon Lee and Beomsoo Kim and Dohyun Kim and Dongwon Kim and Jihwan Moon and Joonghyun Bae and Sehun Lee and Seungjun Chung and Sungwoo Cho and Jaewoong Cho},
  year   = {2026},
  note   = {KRAFTON AI}
}

References

• [1] Amazon's Alexa+. Amazon. https://www.aboutamazon.com/what-we-do/devices-services/alexa-plus
• [2] Google Assistant, your own personal Google. Google. https://assistant.google.com/
• [3] AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head. arXiv:2304.12995. https://arxiv.org/abs/2304.12995
• [4] Multimodal Audio-textual Architecture for Robust Spoken Language Understanding. arXiv:2306.06819. https://arxiv.org/abs/2306.06819
• [5] Raon-Speech Technical Report. arXiv:2605.23912. https://arxiv.org/abs/2605.23912
• [6] Qwen3-ASR Technical Report. arXiv:2601.21337. https://arxiv.org/abs/2601.21337
• [7] Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037. https://arxiv.org/abs/2410.00037
• [8] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. arXiv:2110.13900. https://arxiv.org/abs/2110.13900
• [9] ParaBridge: Bridging Paralinguistic Perception and Dialogue Behavior in Speech Language Models. arXiv:2606.10581. https://arxiv.org/abs/2606.10581
• [10] Preference Alignment Improves Language Model-Based TTS. arXiv:2409.12403. https://arxiv.org/abs/2409.12403
• [11] Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment. arXiv:2505.04113. https://arxiv.org/abs/2505.04113
• [12] Qwen3-Omni Technical Report. arXiv:2509.17765. https://arxiv.org/abs/2509.17765
• [13] Ming-Omni: A Unified Multimodal Model for Perception and Generation. arXiv:2506.09344. https://arxiv.org/abs/2506.09344
• [14] Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music. arXiv:2604.10905. https://arxiv.org/abs/2604.10905
• [15] MOSS-Audio Technical Report. arXiv:2606.01802. https://arxiv.org/abs/2606.01802
• [16] Covo-Audio Technical Report. arXiv:2602.09823. https://arxiv.org/abs/2602.09823
• [17] Voxtral TTS. arXiv:2603.25551. https://arxiv.org/abs/2603.25551