Abstract
The potential of large-scale models to enhance industrial productivity and catalyze societal progress is undeniable. However, inherent challenges-such as lengthy training cycles and the demand for advanced computational resources-remain daunting. Given recent advancements in computational adaptability, this paper introduces a systematic approach to effectively fine-tune these models for domain-specific tasks. Our method encompasses three key phases: (1) a thorough analysis of domain-specific business needs and data acquisition; (2) precise task segmentation, designing standardized instruction formats to construct a fine-tuning dataset, and subsequently fine-tuning the large-scale models; (3) rigorous model validation using a test dataset. Through these steps, we effectively fine-tuned our training using 5,000 data instances and validated our results with an additional 1,000 test instances. To complement our study, we provide a comparative analysis of different training techniques and assess the fine-tuning results on four prominent open-source models. The conclusions drawn offer valuable insights for the future application of large-scale models in specialized domains and pave the way for further research and applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Radford, A., Narasimhan, K., Salimans, T., et al.: Improving language understanding by generative pre-training, pp. 8–25 (2018)
Liu, Y., Ott, M., Goyal, N., et al.: Roberta: A robustly optimized Bert pretraining approach, pp. 24 ArXiv preprint arXiv:1907.11692 (2019)
Narayanan, D., Shoeybi, M., Casper, J., et al.: Efficient large-scale language model training on GPU clusters using Megatron-lm. Proceedings of the International Conference for High-Performance Computing, Networking, Storage, and Analysis, pp. 35–39 (2021)
ChatGPT and Open-AI Models: A Preliminary Review. 15, 6, 192. https://doi.org/10.3390/fi15060192.ChatGPT.(n.d.). Will ChatGPT replace search engines?, p. 3
ChatGPT. (n.d.). Will ChatGPT replace search engines?
ChatGPT. (n.d.). Explaining some common misconceptions about large language models, pp. 8–25
Mitchell, E., Lee, Y., Khazatsky, A., et al.: DetectGPT: ZeroShot Machine-Generated Text Detection using Probability Curvature, p. 19. ArXiv preprint, abs/2301.11305 (2023)
Amatriain, X.: Transformer models: an introduction and catalog, pp. 8–41 (2023)
Korthikanti, V., Casper, J., Lym, S., et al.: Reducing activation recomputation in large transformer models, p. 39 ArXiv preprint arXiv:2205.05198 (2022)
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization, and Huffman coding, p. 45. ArXiv preprint, abs/1510.00149 (2015)
GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. http://arxiv.org/abs/2303.10130 Accessed 05 Sep 2023
Yuan, S., Zhao, H., Du, Z., et al.: WuDaoCorpora: a super large-scale Chinese corpora for pre-training language models. AI Open 2, 65–68 (2021)
Wei, J., Bosma, M., Zhao, V., et al.: Fine-tuned Language Models are Zero-Shot Learners. In: Proceedngs of ICLR, p. 54 (2022)
Sanh, V., Webson, A., Raffel, C., et al.: Multitask prompted training en- ables zero-shot task generalization. In: Proceedings of ICLR, pp. 54 (2022)
Hu, E.J., et al.: Lora: low-rank adaptation of large language models, p. 4. arXiv preprint hyperimagehttp://arxiv.org/abs/2106.09685arXiv:2106.09685 (2021)
Liu, X., et al.: P-tuning v2: Prompt tuning can be comparable to fine- tuning universally across scales and tasks, p. 2. arXiv preprint arXiv:2110.07602 (2021)
Liu, Y., Agarwal, S., Venkataraman, S.: Autofreeze: automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXiv:2102.01386. 18. (pp. 5) (2021)
Sun, X., Ji, Y., Ma, B., Li, X.: A Comparative Study between Full-Parameter and LoRA-basedFine-Tuning on Chinese Instruction Data for Instruction Following LargeLanguage Model, p. 2 (2023). https://arxiv.org/pdf/2304.08109.pdf
Ziqingyang/Chinese-llama-2-7b \(\cdot \) hugging face. ziqingyang/chinese-llama-2-7b \(\cdot \) Hugging Face. (n.d.). https://huggingface.co/ziqingyang/chinese-llama-2-7b
THUDM/CHATGLM2-6B \(\cdot \) hugging face. THUDM/chatglm2-6b \(\cdot \) Hugging Face. (n.d.). https://huggingface.co/THUDM/chatglm2-6b
Raven RWKV 7B - a hugging face space by blinkdl. Raven RWKV 7B - a Hugging Face Space by BlinkDL. (n.d.). https://huggingface.co/spaces/BlinkDL/RWKV-World-7B
Decapoda-research/llama-7b-HF \(\cdot \) hugging face. decapoda-research/llama-7b-hf \(\cdot \) Hugging Face. (n.d.). https://huggingface.co/decapoda-research/llama-7b-hf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, Y., Li, C., Wang, X., Zhang, B. (2023). Application and Research on a Large Model Training Method Based on Instruction Fine-Tuning in Domain-Specific Tasks. In: Chen, E., et al. Big Data. BigData 2023. Communications in Computer and Information Science, vol 2005. Springer, Singapore. https://doi.org/10.1007/978-981-99-8979-9_14
Download citation
DOI: https://doi.org/10.1007/978-981-99-8979-9_14
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8978-2
Online ISBN: 978-981-99-8979-9
eBook Packages: Computer ScienceComputer Science (R0)
