Xiaoice

In May 2017, Microsoft Software Technology Center Asia (STCA) proposed the concept of “AI Creation” with the purpose of learning abilities of excellent human creators and creating content based on text, voice and vision through AI technology.

Microsoft believed that AI, as an attempt of exploration in creation, differs from Go. AI Creation is not aimed at defeating humans. Instead, it is focused on giving full play to AI characteristics including fast learning, high concurrency and stability among others on the ground of hitting the same quality standard as human creators to embark on a future for “highly custom” content industry - under the technical support, a “highly custom” content era is coming soon where users demand “highly custom” content recommendation, presentation and even content itself. As AI hits a certain creation standard, it may help make up the defect of imbalance between supply and demand.

At the moment, the progress of content creation in all fields is as follows:

Xiaoice has been learning to write literature, sing, and create other works of art. In May 2017, the first AI-authored collection of poems—The Sunshine Lost Windows was published by Xiaoice. In August 2017, Microsoft licensed Times Culture Company to publish the traditional Chinese version of the collection of poems within the territory of Taiwan. In 2019, Microsoft licensed China Youth Publishing Group to publish the first collection of poems co-created by AI and 200 human poets - FLOWERS ARE SILENCE OF LUCID WATER. Besides, Microsoft has also published or serialized Xiaoice’s Poem at Youth Literature, West China City Daily and other publications. In addition to arousing sustainable concern and discussion in poetry, the technology has also motivated public enthusiasm for poem creation. Since May 2017, Xiaoice has helped more than 5 million amateur poets create poems, some of which have been published at all kinds of literary publications.Besides, any ordinary person can use the tool opened up by Microsoft to compose poems with the help of Xiaoice. So far Xiaoice has helped millions of people write poems[7].

Xiaoice has released dozens of songs with the similar quality to human singers, including I Know I New, Breeze, I Am Xiaoice, Miss You etc. The 4th version of the DNN singing model allows Xiaoice to learn more details. For example, Xiaoice can produce this breathing sound along with her singing as human. Besides, she is also able to quickly learn different singing styles from different singers, and automatically merge the new style together with her voice characteristic. Furthermore, the technology allows completely absorb the skills of human singers, and their innate voice[8].Microsoft Xiaoice has released dozens of singles almost close to human singing standard, whose singing model has iterated to the fifth edition[9]. The new singing model comprises a variety of voice modes with many singing techniques, improves the standard of AI singing, shows a great diversity of styles in AI singing, and fulfills a natural transition of different singing techniques. In Japan, Microsoft announced its cooperation with AVEX that Japanese Xiaoice - Rinna would join as an AI singer[10]. In addition to Xiaoice, Microsoft also owns or offers a dozen high-quality virtual singer models to third parties[11].

In audiobooks, Microsoft has set up an audiobook creation platform (CCP) in the framework of Microsoft Xiaoice and created over thirty character-oriented voices based on the platform to enable AI technology to automatically create highly custom audiobooks for children. For audio books, the content created by Xiaoice has been provided to 90% of kid education platforms and 80% of online broadcasting platforms in China. So far, the platform has full copyright to children-oriented audiobooks lasting more than 2,300 hours[12]. Moreover, through AI technologies, Xiaoice can customize audio-books for every child and make them and their families to become characters in these audio-books[13]. Microsoft has also entered into an agreement of cooperation with many content publishers to exchange the copyright of text and audiobooks to keep expediting the progress of audiobooks for children and help approach the problem of lack of audiobooks for children in China.

Vision creation generally covers two areas: painting and textile & clothing fabric design.

In painting, after learning works of 236 famous human painters in the 400-year art history, Microsoft Xiaoice can complete 100% original paintings on its own while being inspired by text or other sources of creation. Its originality is shown in the composition, coloring, performance and detail elements in works which are on a par with those of human painters. Compared with other prior techniques, the painting model does not create paintings at random, transfer or shift styles of prior paintings or filter prior paintings. In May 2019, Microsoft Xiaoice graduated from Central Academy of Fine Arts (CAFA) with a master's degree under the alias of “Yubing Xia”. In July 2019, she hosted a personal exhibition at the CAFA Art Museum[14]. At the personal exhibition, “Microsoft Xiaoice” made up 7 painters that originated from different eras and territories and showed distinctive character settings Each one's works have a unified style, whereas, the style of these seven fictional painters created by Xiaoice are quite different from each other. Together, they contributed to the exhibition and made seven eras in the history of human art have been reborn at the same time[14].

In textile and clothing fabric design, Microsoft Xiaoice provides an AI-based textile fabric design platform for Chinese clothing companies with the purpose of helping such companies make up the deficiency in fabric pattern design. At the moment, the platform can steadily design over a dozen major styles and create on-demand. The first silk products designed by Microsoft Xiaoice have been permanently collected by China National Silk Museum[15]. In addition, many mass production lines have been penetrated in relevant industries. For example, an AI textile and clothing fabric design platform co-launched with China National Textile and Apparel Council (CNTAC) has been put into production, through which products of SELECTED, Wensli, EVE CINA and other brands have been made and sold.

By learning the melodies of the songs and the landmarks about different cities, Xiaoice can create visual artworks of skylines when listening to the songs related to this city. Skyline Series T-shirts designed by Xiaoice, have been jointly launched with SELECTED and been sold in stores.

Based on the application of AI technology, Microsoft Xiaoice has been an AI-based TV and radio content creator and provider to 69 Chinese and Japanese TV station programs and radio station programs since December 2015 and most of its programs are daily or weekly[16]. Up to date, Xiaoice has participated in creating 6,908 hours of programs in total but cost merely 4.5% of that of a human creation team. In addition, Xiaoice has also been enabled to act as an AI presenter in TV and radio programs, e.g., Tales of the Unusual, Fuji TV, Japan.[8].

She is a journalist for Qianjiang Evening News.

In finance, Microsoft Xiaoice is the world's first financial text digest creation platform. Since November 2017, Microsoft has worked together with Wind Information, Wallstreet News and other major financial information service providers in China to render IPO announcement text digests created by AI technology to users. The service covers all the 26 categories in finance and is provided for more than 90% of financial institution traders and over 40% of individual financial traders in China[17].

1. In addition to Xiaoice AI Radio jointly produced with NetEase Cloud Music, Microsoft also helped NetEase Cloud Music create and launch two AI characters (Duoduo and Xixi). Microsoft and NetEase jointly endowed different personalities and concepts to the two AI characters and replenished some specific music knowledge needed for the NetEase music platform. In addition, Microsoft also teamed with NetEase News APP to integrate Xiaoice's news reading and commenting. Xiaoice has read and commented more than 10 million news.

2. Xiaoice will continue to strengthen cooperation with Tencent. In addition to the collaborative QQ Xiaoice, Microsoft Xiaoice will help Tencent modify its own AI – Baby Q with some AI technology and product support. Besides, Microsoft Xiaoice, together with Tencent QQ team, have tailored over 40 exclusive skills for QQ Xiaoice and BabyQ. Xiaoice on the Tencent QQ platform has become a part of Tencent's products and culture, winning acclaim from hundreds of millions of users.

3. Xiaoice has been built in Huawei phones. Microsoft and Huawei will have profound cooperation in AI products. All Huawei's mobile phone users of intelligent assistant 8.2 version and above can experience the service from Xiaoice now. On top of the general features for other platforms, Xiaoice brings exclusive functions for Huawei users, such as Good Memory Xiaoice, who can help users remember and trigger the memory naturally, along with other task-completion based functions.

1. Upgrading of some key techniques in Microsoft Xiaoice, including core chat engine, full-duplex voice channel and multi-modal interactive sensor[18]. Among them:

New chat engine: through a technical iteration from retrieval model, generative model to empathy model, the upgraded chat engine makes a lead from “equalizing in conversation” to “dominating in conversation”. It can enhance open-domain conversation performance and raise the conversion rate in vertical fields. By the example of the ongoing test in the “vertical field: on-line retail” in the U.S., the new chat engine hits 68% in the product page conversion rate, 21% higher than the prior edition.

Full-duplex voice interactive sense: Microsoft Xiaoice applied full-duplex voice sense to the traditional telephone and AI-speaker devices in 2017 and 2018 respectively. By adding the vehicle-mounted scenario to its range of scenarios, it can heavily enhance interactive experience between the AI assistant and drivers/passengers and promote AI transformation from an intelligent task-console to “virtual passenger” in the criterion for user experience evaluation.

Futuristic multi-modal interactive sense: a new real-time interactive sense integrating full-duplex voice, real-time vision and core chat engine. It can provide AI interactive user experience of listening, speaking and looking at the same time. After announcing the inception of multi-modal sense development last year, Microsoft Xiaoice Japan team invited the local media to witness the launch of the public beta version of the new real-time sense this February. The sense is expected to take shape as a new AI interactive form in the 5G era.

2. Co-launch of many Xiaoice products with third-party partner platforms including chat bot, intelligent voice assistant, AI content creation and production platform, etc.[19].

Microsoft Xiaoice began implementing the Dual AI Strategic Plan last year. So far, the 7th-generation Microsoft Xiaoice has become one of the world's biggest interdisciplinary AI systems. In many countries around the world, as a single brand, Microsoft Xiaoice has covered 660 million on-line users, 450 million third-party IoT devices and 900 million content viewers. It has kept 23 conversations per session (CPS) averagely with users while greatly increasing interactive scenarios.

At the launch conference, Microsoft Xiaoice announced its collaboration with such partners as Tencent, Xiaomi, Toutiao, vivo and OPPO. Most products in the project have been rolled out with 7th-generation Xiaoice, including “Call up Xiaoice” co-completed with top 5 mobile phone manufacturers in China.

Microsoft has declared that Xiaoice will continue to hold firm to the “third-party platform” development strategy instead of launching any independent first-party product.

3. Presentation of technical breakthroughs in AI Creation and industrial penetration of some production lines[20].

Microsoft is the pioneer in AI creation, advocating to address the imbalance between supply and demand in highly custom content by AI technology. So far, it has developed world-leading techniques in text, voice and vision creation and applied such techniques to practical on-line products.

Microsoft AI Creation has many mass production lines that have been put into practice in relevant industries. For example, the AI-based textile and clothing fabric design platform co-launched with China National Textile and Apparel Council (CNTAC) has been put into production, through which products of SELECTED, Wensli, EVE CINA and other brands have been made and sold. Regarding music, a music creation platform co-developed with Wo Music of China Unicom, has been launched.

At the launch conference, Microsoft showed a variety of AI models owned by “Xiaoice Studio”, including voice models integrating many singing techniques and painting models.

4. Launch of the first toolkit edition of Avatar Framework[16].

Bearing the same source as the framework of Microsoft Xiaoice, the toolkit can easily set up AI assistants, chat bots, AI content provider, IP characters, etc. with a full range of emotional computing powers.

The toolkit comprises a variety of tools, such as conversation, voice, vision, viewpoint, skill, knowledge, and creativity, and can drive compatible 3D character models for real-time interaction.

Prior to the launch, Microsoft has empowered AI for dozens of third-party customers, e.g., SoftBank Pepper, and Microsoft Xiaoice itself through the Avatar Framework. At the launch conference, Microsoft announced its partnership with China Literature to empower 100 IPs in novels.

The launch of Avatar Framework was oriented to all partners and customers. The edition oriented to individual users is expected to be released in the spring of 2020.

5. Progress of Microsoft Xiaoice's commercialization[16].

Based on rich experience in ToB and launch of a variety of technical products, Microsoft Xiaoice has made fast progress in commercialization.

So far, it has been in the service to commercial customers in a dozen fields including finance, retail, automobile, real estate, textile, etc. Its customers include Vanke, Wind Information, Wensli, China Unicom, and others.