UniSS

UniSS:

Unified Expressive Speech-to-Speech Translation with Your Voice

UniSS is a unified single-stage S2ST framework that achieves high translation fidelity and speech quality, while preserving timbre, emotion, and duration consistency.

See our paper and other resources:

Overview of UniSS (S2ST)

Figure 1. UniSS integrates pre-trained text LLMs and transfers their translation capabilities to the speech domain. UniSS outperforms both end-to-end S2ST systems and cascaded systems on translation fidelity, expressiveness preservation, duration consistency and speech quality

Method Overview

Figure 2. Overview of UniSS architecture, cross-modal CoT prompting and 3-phase progressive training.

Zero-shot S2ST Samples

All samples are generated by UniSS (Quality mode) directly without fine-tuning.

Language	Info	Source	Inference Result
EN-ZH	Benedict Cumberbatch	So maybe, that you would prefer to forego my secret rather than consent to becoming a prisoner here for what might be several days.	所以，也许你宁愿放弃我的秘密，也不愿同意在这里成为囚犯，哪怕可能要待上好几天。
	Rick Sanchez	Yeah that's the difference between you and me, Morty. I never go back to the carpet store.	是的，这就是你和我的区别，莫蒂。我从不回地毯店。
	From Seed-TTS's demo	Some call me nature, others call me mother nature. I've been here for over four point five billion years—twenty two thousand and five hundred times longer than you.	有些人叫我自然，另一些人叫我大自然母亲。我在这里已经超过四十五亿年了，比你们长二万两千五百倍。
	From Seed-TTS's demo	A chance to leave him alone, but... No. She just wanted to see him again. Anna, you don't know how it feels to lose a sister. Anna, I'm sorry, but your father asked me not to tell you anything.	我本想让他一个人待着，但是……不。她只是想再见到他一次。安娜，你不知道失去一个妹妹是什么感觉。安娜，对不起，但你的父亲让我不要告诉你任何事情。
	From Seed-TTS's demo	Are you familiar with it? Slice the steak and place the strips on top, then garnish with the dried cranberries, pine nuts, and blue cheese. I wonder how people rationalise the decision?	你熟悉这个吗，把牛排切好，然后将肉条放在上面，再用干蔓越莓、松子和蓝纹奶酪装饰。我想知道人们是如何合理化这个决定的。
	Steve Jobs	It was beautiful, historical, artistically subtle in a way that science can't capture, and I found it fascinating.	它美丽、历史感十足，艺术上微妙之处难以用科学手段捕捉到，我对此感到着迷。
	Donald Trump	In short, we embark on a mission to make America great again, for all Americans.	简而言之，我们开启了一项使命，为所有美国人让美国再次伟大。
ZH-EN	贾玲	跟观众分享我人生的感悟，因为我们都是只活一次，我们也都是第一次活，我们也不知道该怎么活着。	Sharing my life's insights with the audience, because we are all living this once, and we are also living for the first time. We don't know how to live.
	鲁豫	就你就看到一个一个数字，然后尤其是你有一段明显感到处于一个平静期的时候，你会想，哦，数字好冷静，但冷静的背后，那种。	You will see numbers one by one, and especially when you have a period that clearly indicates you are in a calm state. You might think, oh, the numbers are so calm, but there's something behind that calmness.
	哪吒	别烦我，让我一个人安静地死去。	Leave me alone, let me die in peace.
	杨澜	但是你的成本已经全部都进去了。	But your costs have already all gone in.
	徐志胜	有一次我跟我的三个领导坐电梯去吃饭，大家可能知道啊在电梯里有一项潜规则是谁的地位越低，谁就应该按这个电梯按键，但我当时就认为是谁离得近谁按。	Once, I was in an elevator with three of my bosses to go have a meal. You might know that there's an unwritten rule in the elevator: the person with the lowest status should press the button for the elevator. But at that time, I thought whoever was closest should press it.
	From Seed-TTS's demo	皇上请三思，皇后娘娘都是为了万岁爷着想，请万岁爷不要辜负了娘娘的一片苦心。	Please reconsider, Your Majesty. The Empress is thinking for the Emperor's sake. Please do not let Her Majesty's earnest efforts be in vain.
	From Seed-TTS's demo	下周开始就不能开车上班了，因为亚运会要限行，真不方便呀。	Starting next week, I won't be able to drive to work because of the traffic restrictions during the Asian Games. It's really inconvenient.

Emotion-preserved S2ST Samples

All samples are generated by UniSS (Quality mode) directly without fine-tuning.

Emotion	Source	Inference Result
Angry
Happy
Sad
Natural

Comparison with Other Methods

The samples are from the test sets of CVSS-T and FLEURS.

Source	UniSS (Quality)	Seamless-Expressive	Seamless-Large-V2	GPT-4o Audio
Fuyao temple of guandi founded in the twenty eighth year of ming dynasty.	福耀关帝庙成立于明朝第二十八年。	代明王朝的二十八年成立。	不要死不也 out temple of granddy 成立渔民炒局太年。	关帝庙建立于明朝二十八年。
When nurses enter the workplace these badges are used to identify their alma mater.	当护士进入工作场所时这些徽章用于识别他们的母校。	当护士进入工作场所时这些规章被用来识别他们的母校。	当护士进入工作场所时这些安克装备用来识别他们的母校。	当护士进入工作场所时，这些徽章被用来识别他们的母校。
Process information is also mapped to file systems through directories.	进程信息也通过目录映射到文件系统。	通过目录过程信息也被映射到文件系统中。	通过目录过程信息也被映射到文件系统中。	处理信息也通过目录映射到文件系统。
科学家们可以得出结论，暗物质对其他暗物质的影响方式与普通物质相同。	Scientists can conclude that the way dark matter affects other dark matter is the same as it does for ordinary matter.	Scientists can conclude that dark matter affects other dark substances in the same way as ordinary matter.	Scientists can conclude that dark matter has the same effect on other dark matter as ordinary matter.	Scientists can conclude that the way dark matter interacts with other dark matter is the same as how it interacts.
地壳近侧厚约 70 公里，远侧厚约 100 公里。	The crust is about 70 kilometers thick on the closer side and 100 kilometers on the farther side.	The visibility is about 70 km to the rear and about 100 km to the rear.	The Shell is about 70 kilometers to the right and about 100 kilometers to the left.	The near section is thick to about 70 centimeters, while the far section is thick to about 100 centimeters.