During this period, an artificial intelligence chat system called ChatGPT was like a spring thunder on the ground and suddenly burst the network. "From the current general feeling of dialogue, the biggest highlight of ChatGPT is its excellent intention recognition and language understanding ability, which makes people realize that the interactive ability of AI has achieved leapfrog progress." Chen Chen, senior analyst of Analysys Intelligence Institute, commented. Answering every question and responding with eloquence. This amazing human-computer interaction experience has aroused people's enthusiasm for asking questions to ChatGPT, and brought it an amazing number of users: the system was launched by the American AI research company OpenAI more than two months ago, and has reached 100 million active users in January this year, with an unprecedented growth rate of users. This product is very popular, and its important behind the scenes is the technology known as the pre-training model in the industry. In the face of the explosion of ChatGPT, domestic scientific and technological institutions and enterprises have "lighted their swords". According to the latest news, the Chinese version of ChatGPT will meet the public next month. The scale of parameters has soared from millions to hundreds of billions. "At present, the pre-training technology is an important breakthrough in AI research." Su Zhong, the head of the Future Technology Center of Alibaba Research Institute, explained that in traditional research methods, the cost of labeling has been an obstacle to the promotion of AI algorithms to larger data sets, and the pre-training technology can train a large-scale in-depth learning model without relying on data labeling. The reporter of Science and Technology Daily learned that because there is no need for data annotation, the pre-training model can often use a larger data set, so it can choose a larger model size - which has led to the emergence of a large pre-training model. "Compared with the previous artificial neural network, the biggest difference of the large model of pre-training is that it is large enough, and the deep learning network has many layers, connections, and parameters." Zhang Jiaxing, a lecturer of the Cognitive Computing and Natural Language Research Center of IDEA Research Institute (Guangdong-Hong Kong-Macao Greater Bay Area Digital Economy Research Institute), told reporters in an interview that the deep learning network around 2012 had only a few million parameters; Before and after 2018, the mainstream pre-training model reached 100 million parameters; At present, the large-scale pre-training model that has been proved to be very effective has hundreds of billions of parameters, which has increased several thousand times in just a few years. Quantitative change causes qualitative change. Su Zhong analyzed that in the various technical evaluation of the pre-training model, the algorithm performance showed a rule: the larger the data size, the more the parameters of the pre-training model, the higher the output accuracy of the algorithm. In addition to its large scale, Zhang Jiaxing said that the current large pre-training model is very suitable for parallel training due to its new structure model. At the same time, in terms of training methods, the pre-training task of the artificial neural network is similar to "problem solving". For example, remove a word or the first half of a sentence, and let the pre-training model recover the sentence using massive big data. In this way, the "problem solving" ability of the pre-training model will become better and better. Moreover, on the basis of the large language model, the introduction of instruction learning can far exceed the benefits of model fine-tuning. On this basis, human feedback can help the model better understand human intentions and receive information, and human-computer dialogue will become more and more