悟道 (Wu Dao)
Original author(s)Beijing Academy of Artificial Intelligence
Initial releaseJanuary 11, 2021
Websitewww.baai.ac.cn/english.html

Wu Dao (Chinese: 悟道; pinyin: wùdào; lit. 'road to awareness')[1] is a multimodal artificial intelligence developed by the Beijing Academy of Artificial Intelligence (BAAI).[2][3][4] Wu Dao 1.0 was first announced on January 11, 2021;[1][5] an improved version, Wu Dao 2.0, was announced on May 31.[6][5] It has been compared to GPT-3,[7] and is built on a similar architecture; in comparison, GPT-3 has 175 billion parameters[8][9] — variables and inputs within the machine learning model — while Wu Dao has 1.75 trillion parameters.[6][10] Wu Dao was trained on 4.9 terabytes of images and texts (which included 1.2 terabytes of Chinese text and 1.2 terabytes of English text),[6][11] while GPT-3 was trained on 45 terabytes of text data.[12] Yet, a growing body of work highlights the importance of increasing both data and parameters.[13] The chairman of BAAI said that Wu Dao was an attempt to "create the biggest, most powerful AI model possible";[8] although direct comparisons between models based on parameter count (i.e. between Wu Dao and GPT-3) do not directly correlate to quality.[9] Wu Dao 2.0, was called "the biggest language A.I. system yet".[4] It was interpreted by commenters as an attempt to "compete with the United States".[14][15]. Notably, the type of architecture used for Wu Dao 2.0 is a mixture-of-experts (MoE) model,[5] unlike GPT-3, which is a "dense" model:[16] while MoE models require much less computational power to train than dense models with the same numbers of parameters,[16] trillion-parameter MoE models have shown comparable performance to models that are hundreds of times smaller.[16]

Wu Dao's creators demonstrated its ability to perform natural language processing and image recognition, in addition to generation of text and images.[5] The model can not only write essays, poems and couplets in traditional Chinese, it can both generate alt text based on a static image and generate nearly photorealistic images based on natural language descriptions. Wu Dao also showed off its ability to power virtual idols (with a little help from Microsoft-spinoff Xiaoice) and predict the 3D structures of proteins like AlphaFold.[5]

History

Wu Dao's development began in October 2020, several months after the May 2020 release of GPT-3.[1] The first iteration of the model, Wu Dao 1.0, "initiated large-scale research projects"[17] via four related models.[18][17]

  • Wu Dao – Wen Yuan, a 2.6-billion-parameter pretrained language model, was designed for tasks like open-domain answering, sentiment analysis, and grammar correction.[17]
  • Wu Dao – Wen Lan, a 1-billion-parameter multimodal graphic model, was trained on 50 million image pairs to perform image captioning.[17]
  • Wu Dao – Wen Hui, an 11.3-billion-parameter generative language model, was designed for "essential problems in general artificial intelligence from a cognitive perspective"; Synced says that it can "generate poetry, make videos, draw pictures, retrieve text, perform complex reasoning, etc".[17]
  • Wu Dao – Wen Su, based on Google's BERT language model and trained on the 100-gigabyte UNIPARC database (as well as thousands of gene sequences), was designed for biomolecular structure prediction and protein folding tasks.[17]

WuDao Corpora

WuDao Corpora (also written as WuDaoCorpora), as of version 2.0, was a large dataset constructed for training Wu Dao 2.0. It contains 3 terabytes of text scraped from web data, 90 terabytes of graphical data (incorporating 630 million text/image pairs), and 181 gigabytes of Chinese dialogue (incorporating 1.4 billion dialogue rounds).[19] Wu Dao 2.0 was trained using FastMoE, a variant of the mixture of experts architecture published by Google.[18] TheNextWeb said in June 2021 that "details as to exactly how Wu Dao was trained, what was in its various datasets, and what practical applications it can be used for remain scarce".[8] OpenAI's policy director called Wu Dao an example of "model diffusion", a neologism describing a situation in which multiple entities develop models similar to OpenAI's.[20]

References

  1. 1 2 3 智源研究院 (January 11, 2021). "面向认知，智源研究院联合多家单位发布超大规模新型预训练模型"悟道·文汇"".
  2. ↑ Bailey, Ronald (June 11, 2021). "Amazing New Chinese A.I.-Powered Language Model Wu Dao 2.0 Unveiled". Reason.
  3. ↑ "Record-breaking AI model WuDao 2.0 unveiled in Beijing". China.org.cn. June 3, 2021.
  4. 1 2 "Can A.I. help investors find the next hot technology? A number of companies are betting on it". Fortune.
  5. 1 2 3 4 5 Tarantola, A. (June 2, 2021). "China's gigantic multi-modal AI is no one-trick pony". Engadget.
  6. 1 2 3 Feng, Coco (June 2, 2021). "Beijing-funded AI language model tops Google and OpenAI in raw numbers". South China Morning Post.
  7. ↑ "WuDao 2.0, l'intelligenza artificiale cinese più potente di OpenAI e Google". lantidiplomatico.it.
  8. 1 2 3 Greene, Tristan (June 3, 2021). "China's 'Wu Dao' AI is 10X bigger than GPT-3, and it can sing". TNW | Neural.
  9. 1 2 Mott, Nathaniel (June 3, 2021). "China Says WuDao 2.0 AI Is an Even Better Conversationalist than OpenAI, Google". Tom's Hardware.
  10. ↑ "Wu Dao 2.0, el modelo de IA multimodal de aprendizaje profundo más grande del mundo". El Comercio Perú. June 3, 2021.
  11. ↑ Heikkilä, Melissa (June 9, 2021). "Meet Wu Dao 2.0, the Chinese AI model making the West sweat". Politico.
  12. ↑ B. Brown, Tom (2020). "Language Models are Few-Shot Learners". arXiv:2005.14165 [cs.CL].
  13. ↑ Hoffmann, Jordan (2022). "Training Compute-Optimal Large Language Models". arXiv:2203.15556 [cs.CL].
  14. ↑ "Китайская нейросеть WuDao 2.0 с 1,75 трлн параметров превосходит аналоги от Google и OpenAI". 3DNews - Daily Digital Digest.
  15. ↑ Mertens, Jennifer (June 5, 2021). "La Chine développe une IA pour concurrencer les États-Unis".
  16. 1 2 3 Artetxe, Mikel (2021). "Efficient Large Scale Language Modeling with Mixtures of Experts". arXiv:2112.10684 [cs.CL].
  17. 1 2 3 4 5 6 "China's GPT-3? BAAI Introduces Superscale Intelligence Model 'Wu Dao 1.0' | Synced". syncedreview.com. March 23, 2021.
  18. 1 2 Schreiner, Maximilian (13 June 2021). "Sprach-KI aus China: Wu Dao 2.0 ist zehnmal größer als GPT-3".
  19. ↑ "resource". resource.wudaoai.cn.
  20. ↑ Wiggers, Kyle (June 4, 2021). "AI Weekly: China's massive multimodal model highlights AI research gap". VentureBeat.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.