Novel Data Generation Frameworks for Mitigating Data Scarcity in Dialogue Systems = 대화 시스템에서의 데이터 부족 문제 완화를 위한 새로운 데이터 생성 프레임워크

다국어 초록 (Multilingual Abstract)

Dialogue systems, which include conversational agents, open-domain dialog sys- tems (ODD), and task-oriented dialog systems (TOD), play a crucial role in advancing human-computer interaction. Despite significant advancements driven by Large Lan- guage Models (LLMs), these systems still face critical challenges, especially in achiev- ing specific conversational goals and adapting to dynamic conversational contexts. A primary obstacle is the scarcity of domain-specific data, which is crucial for training task-oriented systems and informative dialogue systems but often limited or costly to obtain. This paper explores innovative frameworks designed to overcome these chal- lenges related to data scarcity. We introduce novel methodologies for automating the creation of dialogue datasets, with a specific focus on resolving database search ambi- guities and generating high-quality, information-seeking dialogues from standard text passages. Additionally, we propose a unique multi-passage-to-dialog framework that addresses the complexities of topic shifts in dialogues, a persistent issue even in ad- vanced LLMs. By integrating these novel frameworks, this research not only addresses the inherent limitations of current dialogue systems but also sets a new standard in the generation and management of conversational data, aiming to enhance the practicality and scalability of dialogue systems across various applications. keywords: Dialogue Systems, Data Scarcity, Dialogue Generation Framework, Human-Computer Interaction student number: 2019-26192

국문 초록 (Abstract)

대화 시스템은 대화형 에이전트, 개방형 대화 시스템(ODD), 과제 지향적 대화 시스템(TOD)을 포함해 인간-컴퓨터 상호작용 발전에 중요한 역할을 한다. 그러나 대규모 언어 모델(LLM)의 발전에도 불구하고, 대화 시스템은 여전히 특정 대화 목표를 달성하고 변화하는 대화 상황에 적응하는 데 어려움을 겪고 있다. 주된 문제 중 하나는 필수적이면서도 비용이 많이 드는 도메인 특화 데이터가 부족한 것이다. 이 논문은 데이터 부족 문제를 해결하기 위해 설계된 새로운 대화 데이터 생성 프레임워크를 탐구한다. 우리는 데이터베이스 검색의 모호성을 해결하고 표준 텍스트 구절에서 고품질의 정보 탐색 대화를 생성하는 새로운 자동화 방법론을 소개한다. 또한, 대화 중 주제 변화의 복잡성을 처리할 수 있는 독특한 다중 구절-대화 생성 프레임워크를 제안한다. 이러한 프레임워크를 통합함으로써, 본 연구는 기존 대화 시스템의 한계를 넘어 대화 데이터의 생성 및 관리에 새로운 기준을 제시하고, 다양한 응용 분야에서 대화 시스템의 실용성과 확장성을 향상시키고자 한다. 주요어: 대화 시스템, 데이터 부족의 한계점, 대화 데이터 생성 프레임워크, 인간-컴퓨터 상호작용
학번: 2019-26192

목차 (Table of Contents)

• Abstract i
• Contents ii
• List of Tables vi
• List of Figures ix
• 1 INTRODUCTION 1
• 1.1 Dialogue Systems . 2
• 1.2 Data Scarcity Challenges in Dialogue System . 4
• 1.3 Novel Frameworks for Mitigating Data Scarcity Challenges 6
• 1.4 Dissertation Structure and Roadmap 7
• 2 Background 9
• 2.1 Dialogue Systems . 9
• 2.1.1 ODD : Open-domain Dialogue System 10
• 2.1.2 TOD : Task-Oriented Dialogue System 11
• 2.1.3 ConvQA : Information-seeking Dialogue 12
• 2.1.4 Recent Trends and Challenges 13
• 2.2 Dialogue Data Generation 15
• 2.2.1 Dialogue Generation Frameworks 15
• 2.2.2 Machine-Generated vs. Human-Generated Data 16
• ii
• 2.2.3 Role of Language Models . 16
• 2.2.4 Challenges and Future Directions 17
• 3 Dialogue Generation for Disambiguating DB Searches 19
• 3.1 Related Works 22
• 3.1.1 Task-Oriented Dialogue System . 22
• 3.1.2 Schema-Guided Dialogue (SGD) dataset 22
• 3.1.3 Database Search results disambiguation task 24
• 3.2 Comparison-Based database search Ambiguity handling for dialogue . 24
• 3.2.1 Task proposal 24
• 3.2.2 Task formulation 25
• 3.3 Disambiguating Schema-guided Dialogue (DSD) Dataset . 25
• 3.3.1 Dataset Construction . 27
• 3.3.2 Statistics . 31
• 3.3.3 Human evaluation 31
• 3.3.4 Example dialogues per domain 31
• 3.4 Experiment . 34
• 3.4.1 Dialogue Generation (DG) . 34
• 3.4.2 Dialogue State Tracking (DST) . 37
• 3.4.3 Named Entity Prediction (NER) . 40
• 3.4.4 Domain adaptation ability of CBA 42
• 3.5 Conclusion . 44
• 4 Context-aware Passage2Dialog Generation 45
• 4.1 Related Works 48
• 4.1.1 Conversational Question-Answering 48
• 4.1.2 Dialog Inpainting 48
• 4.1.3 Reference-free Dialog Metrics 49
• 4.2 Dialogizer 49
• iii
• 4.2.1 Dialog Reconstruction 51
• 4.2.2 Question-Answer Matching 52
• 4.2.3 Topic-aware Dialog Generation . 52
• 4.2.4 Inference : Autoregressive Generation . 53
• 4.2.5 Re-ranking with contextual relevance . 54
• 4.3 Experiments . 54
• 4.3.1 Model implementation 55
• 4.3.2 Generated Datasets 55
• 4.3.3 Automatic Evaluation 60
• 4.3.4 Human and GPT-4 Evaluation 62
• 4.3.5 Application to Text Retrieval 62
• 4.4 Analysis 68
• 4.4.1 Ablation Study . 68
• 4.4.2 Question Types . 70
• 4.4.3 Case Study 72
• 4.5 Conclusion . 72
• 5 Multi-Passage2Dialog Generation for Mitigating Topic Shift 73
• 5.1 Related Works 76
• 5.1.1 Topic Shift Dialogue Systems 76
• 5.1.2 Passage to Dialogue (P2D) . 76
• 5.2 MP2D . 78
• 5.2.1 Find Path & Retrieve Passages 79
• 5.2.2 Generate Questions 80
• 5.3 Experiment . 82
• 5.3.1 Automatic Evaluation 82
• 5.3.2 Human & GPT-4 Evaluation 85
• 5.4 Benchmark . 86
• 5.4.1 TS-WikiDialog . 86
• iv
• 5.4.2 The Struggle of LLM in Topic Shift turns 87
• 5.5 Application . 87
• 5.5.1 Topic Segmentation 89
• 5.5.2 Topic Shift Detection . 90
• 5.5.3 Enhancing LLM in Topic Shift turns 93
• 5.6 Case Study . 93
• 5.7 Generated Topic Shift Dialogue Examples 94
• 5.8 Conclusion . 94
• 6 Summary and Contributions 97
• Abstract (In Korean) 121
• Acknowlegement 122
• v

참고문헌 (Reference)

1. Topic segmentation, M. Purver, Spoken language understanding: systems for extracting semantic information from speech, pp. 291–317, , 2011

2. Gpt-4 technical report, R. OpenAI, arXiv, pp. 2303–08774, 2023, , 2023

3. Elsevier oa cc-by corpus, D. Kershaw and, R. Koeling, arXiv preprint arXiv:2008.00774, , 2020

4. Dialogue-based user models, A. Kobsa, W. Wahlster and, Proceedings of the IEEE, vol. 74, no. 7, pp. 948–960, , 1986

5. End-to-end memory networks, S. Sukhbaatar, R. Fergus et al., J. Weston, Advances in neural information processing systems, vol. 28, , 2015

6. Dialogue systemA brief review, S. Singh, S. Arora, K. Batra and, arXiv preprint arXiv:1306.4134, , 2013

7. A survey of large language models, T. Tang, K. Zhou, B. Zhang, Y. Hou, X. Wang, Y. Min, W. X. Zhao, Z. Dong et al., J. Zhang, J. Li, arXiv preprint arXiv:2303.18223, 2023, , 2023

8. Large language models in medicine, K. Elangovan, L. Gutierrez, D. S. J. Ting, T. F. Tan and, D. S. W. Ting, A. J. Thirunavukarasu, vol. 29, no. 8, pp. 1930–1940, 2023, , 1930

9. Topic-segmentation of dialogue in, J. Arguello and, C. Rosé, Proceedings of the analyzing conversations in text and speech, pp. 42–49, , 2006

10. Topicoriented dialogue summarization, J. Zhang and, C. Zong, J. Zhu, H. Lin, F. Zhai, L. Xiang, Y. Zhou, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023, , 2023

11. Language models are few-shot learners, A. Neelakantan, P. Dhariwal, M. Subbiah, T. Brown, A. Askell et al., P. Shyam, B. Mann, N. Ryder, G. Sastry, J. D. Kaplan, Advances in neural information processing systems, vol. 33, pp. 1877–1901, , 2020

12. Question generation from concept maps, N. K. Person, A. M. Olney, A. C. Graesser and, Dialogue & Discourse, vol. 3, no. 2, pp. 75–99, , 2012

13. A repository of conversational datasets, P. Budzianowski, N. Mrkši´c, M. Henderson, I. Casanueva, G. Spithourakis, S. Coope, P.-H. Su, D. Gerz, G. Kumar, I. Vuli´c et al., arXiv preprint arXiv:1904.06472, , 2019

14. A survey on empathetic dialogue systems, F. Z. Xing and, E. Cambria, K. L. Nguyen, Y. Ma, Information Fusion, vol. 64, pp. 50–70, , 2020

15. Learning end-to-end goal-oriented dialog, A. Bordes, Y.-L. Boureau and, J. Weston, arXiv preprint arXiv:1605.07683, , 2016

16. Topic-aware multi-turn dialogue modeling, H. Zhao and, Z. Zhang, Y. Xu, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14176–14184, , 2021

17. Conversational question answeringA survey, W. E. Zhang, Y. Zhang, Q. Z. Sheng, M. Zaib, A. Mahmood and, Knowledge and Information Systems, vol. 64, no. 12, pp. 3151–3195, , 2022

18. Dynamic interpretation and dialogue theory, H. Bunt, The structure of multimodal dialogue, vol. 2, pp. 139–166, , 1999

19. Fact or fictionVerifying scientific claims, L. L. Wang, S. Lin, D. Wadden, A. Cohan and, M. van Zuylen, K. Lo, H. Hajishirzi, arXiv preprint arXiv:2004.14974, , 2020

20. Emergent abilities of large language models, D. Yogatama, C. Raffel, B. Zoph, R. Bommasani, D. Metzler et al., M. Bosma, D. Zhou, S. Borgeaud, Y. Tay, J. Wei, arXiv preprint arXiv:2206.07682, , 2022

21. End-to-end learning of task-oriented dialogs, B. Liu and, I. Lane, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 67–73, , 2018

22. LamdaLanguage models for dialog applications, J. Hall, Y. Du et al., R. Thoppilan, H.-T. Cheng, D. De Freitas, T. Bos, A. Kulshreshtha, A. Jin, L. Baker, N. Shazeer, arXiv preprint arXiv:2201.08239, , 2022

23. The curious case of neural text degeneration, Y. Choi, M. Forbes and, L. Du, A. Holtzman, J. Buys, arXiv preprint arXiv:1904.09751, , 2019

24. Recent advances in neural question generation, W. Lei, L. Pan, M.-Y. Kan, T.-S. Chua and, arXiv preprint arXiv:1905.08949, , 2019

25. BloomberggptA large language model for finance, V. Dabravolski, P. Kambadur, D. Rosenberg and, M. Dredze, G. Mann, S. Gehrmann, S. Wu, O. Irsoy, S. Lu, arXiv preprint arXiv:2303.17564, 2023, , 2023

26. CoqaA conversational question answering challenge, C. D. Manning, S. Reddy, D. Chen and, Transactions of the Association for Computational Linguistics, vol. 7, pp. 249–266, , 2019

27. A simple language model for task-oriented dialogue, R. Socher, B. McCann, C.-S. Wu, E. Hosseini-Asl, S. Yavuz and, Advances in Neural Information Processing Systems, vol. 33, pp. 20179–20191, , 2020

28. An empirical study of topic transition in dialogue, B. R. Cowan and, C. Saam, E. Gilmartin, V. Wade, B. Spillane, M. Soni, arXiv preprint arXiv:2111.14188, , 2021

29. Dialog inpaintingTurning documents into dialogs in, A. T. Chaganty, V. Y. Zhao, Q. M. Rashid, K. Guu, Z. Dai, M. Green and, A. Amini, International Conference on Machine Learning, pp. 4558–4586, PMLR, , 2022

30. Discourse segmentation of multi-party conversation, K. McKeown, H. Jing, E. Fosler-Lussier and, M. Galley, Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pp. 562–569, , 2003

31. The dialog state tracking challenge seriesA review, J. D. Williams, M. Henderson, A. Raux and, Dialogue & Discourse, vol. 7, no. 3, pp. 4–33, , 2016

32. news-pleaseA generic news crawler and extractor in, N. Meuschke, C. Breitinger and, B. Gipp, F. Hamborg, 15th International Symposium of Information Science (ISI 2017), pp. 218–223, , 2017

33. Deep reinforcement learning for dialogue generation, W. Monroe, J. Gao and, J. Li, D. Jurafsky, M. Galley, A. Ritter, arXiv preprint arXiv:1606.01541, , 2016

34. DynaevalUnifying turn and dialogue level evaluation, G. Lee and, T. Friedrichs, H. Li, L. F. D’Haro, Y. Zhang, Y. Chen, C. Zhang, arXiv preprint arXiv:2106.01112, , 2021

35. Language models are unsupervised multitask learners, D. Amodei, A. Radford, R. Child, J. Wu, I. Sutskever, et al., D. Luan, OpenAI blog, vol. 1, no. 8, p. 9, , 2019

36. Open-retrieval conversational question answering in, C. Qu, W. B. Croft and, M. Qiu, L. Yang, C. Chen, M. Iyyer, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 539–548, , 2020

37. CiderConsensus-based image description evaluation in, D. Parikh, C. Lawrence Zitnick and, R. Vedantam, Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575, , 2015

38. Is chatgpt a good nlg evaluator? a preliminary study, Y. Liang, J. Wang, Z. Li, H. Shi, F. Meng, J. Qu and, J. Zhou, J. Xu, arXiv preprint arXiv:2303.04048, 2023, , 2023

39. RougeA package for automatic evaluation of summaries, C.-Y. Lin, Text summarization branches out, pp. 74–81, , 2004

40. User modeling in dialog systemsPotentials and hazards, A. Kobsa, AI & society, vol. 4, pp. 214–231, , 1990

41. Introduction to the special issue on the web as corpus, G. Grefenstette, A. Kilgarriff and, Computational linguistics, vol. 29, no. 3, pp. 333–347, , 2003

42. TiageA benchmark for topic-shift aware dialog modeling, C. Xiong, A. Copestake, Z. Liu and, Z. Liu, H. Xie, arXiv preprint arXiv:2109.04562, , 2021

43. Multitasking information seeking and searching processes, H. C. Ozmutlu and, A. Spink, S. Ozmutlu, vol. 53, no. 8, pp. 639–652, , 2002

44. The future of ai in medicinea perspective from a chatbot, M. R. King, Annals of Biomedical Engineering, vol. 51, no. 2, pp. 291–295, 2023, , 2023

45. Towards automatic detection and tracking of topic change, F. Holz and, S. Teresniak, in International Conference on Intelligent Text Processing and Computational Linguistics, pp. 327–339, Springer, , 2010

46. Multi-domain task completion dialog challenge ii at dstc9, L. Luo, Z. Zhang, L. Liden, S. Shayandeh, J. Li, S. Shukla et al., Q. Zhu, K. Huang, B. Peng, R. Liang, Proceedings of the 35th AAAI Conference on Artificial Intelligence, Ninth Dialog System Technology Challenge Workshop, , 2021

47. Neural dialogue generation methods in open domainA survey, B. Sun and, K. Li, Natural Language Processing Research, vol. 1, no. 3-4, pp. 56–70, , 2021

48. OttersOne-turn topic transitions for open-domain dialogue, V. Rieser, I. Konstas and, D. M. Howcroft, K. Sevegnani, arXiv preprint arXiv:2105.13710, , 2021

49. Taskmaster-1Toward a realistic and diverse dialog dataset, A. Cedilnik and, A. Neelakantan, A. Dubey, D. Duckworth, K. Krishnamoorthi, B. Goodrich, S. Yavuz, C. Sankar, K.-Y. Kim, B. Byrne, arXiv preprint arXiv:1909.05358, , 2019

50. DailydialogA manually labelled multi-turn dialogue dataset, X. Shen, S. Niu, Y. Li, W. Li, H. Su, Z. Cao and, arXiv preprint arXiv:1710.03957, , 2017

51. Trec cast 2019The conversational assistance track overview, J. Callan, C. Xiong and, J. Dalton, arXiv preprint arXiv:2003.13624, , 2020

52. A survey on dialog managementRecent advances and challenges, H. Yu, Y. Dai, C. Tang, Y. Li and, Y. Jiang, J. Sun, arXiv preprint arXiv:2005.02233, , 2020

53. Building task-oriented dialogue systems for online shopping, M. Zhou, Z. Yan, J. Zhou and, P. Chen, N. Duan, Z. Li, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, , 2017

54. Chatgpt outperforms crowd-workers for text-annotation tasks, F. Gilardi, M. Kubli, M. Alizadeh and, arXiv preprint arXiv:2303.15056, 2023, , 2023

55. Information-seeking chatDialogues driven by topic-structure, M. Stede and, D. Schlangen, in Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04), , 2004

56. QrfaA data-driven model of information-seeking dialogues in, S. Vakulenko, C. Di Ciccio and, M. de Rijke, K. Revoredo, Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany,, Proceedings, Part I 41, pp. 541–557, Springer, , 2019

57. Analysis of conversational topic shiftsA multiple case study, Y. Joanette, L. J. Garcia and, Brain and language, vol. 58, no. 1, pp. 92–114, , 1997

58. Generating clarifying questions for information retrieval in, G. Lueck, P. Bennett and, N. Craswell, H. Zamani, S. Dumais, Proceedings of the web conference 2020, pp. 418–428, , 2020

59. Music search and iotHow people (really) use voice assistants, J. Kaye, F. Bentley, T. Ammari, J. Y. Tsai and, ACM Transactions on Computer-Human Interaction (TOCHI), vol. 26, no. 3, pp. 1–28, , 2019

60. Sentence-bertSentence embeddings using siamese bert-networks, I. Gurevych, N. Reimers and, arXiv preprint arXiv:1908.10084, , 2019

61. Task-oriented dialogue system as natural language generation, J. Guo, Y. Dai, W. Wang, Z. Zhang, W. Luo, B. Chen and, Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2698–2703, , 2022

62. A survey on dialogue systemsRecent advances and new frontiers, J. Tang, H. Chen, D. Yin and, X. Liu, Acm Sigkdd Explorations Newsletter, vol. 19, no. 2, pp. 25–35, , 2017

63. Challenges in building intelligent open-domain dialog systems, M. Huang, J. Gao, X. Zhu and, ACM Transactions on Information Systems (TOIS), vol. 38, no. 3, pp. 1–32, , 2020

64. Few-shot natural language generation for task-oriented dialog, C. Zhu, B. Peng, C. Li, J. Li, X. Li, J. Gao, M. Zeng and, arXiv preprint arXiv:2002.12328, , 2020

65. GptevalNlg evaluation using gpt-4 with better human alignment, C. Zhu, Y. Xu, S. Wang, R. Xu and, Y. Liu, D. Iter, arXiv preprint arXiv:2303.16634, 2023, , 2023

66. IntellibotA dialogue-based chatbot for the insurance industry, O. K. Hussain, M. Nuruzzaman and, Knowledge-Based Systems, vol. 196, p. 105810, , 2020

67. Neural enquirerLearning to query tables with natural language, H. Li and, Z. Lu, P. Yin, B. Kao, arXiv preprint arXiv:1512.00965, , 2015

68. Development of dialogue management system for banking services, S. Rustamov, E. Alasgarov, A. Bayramova and, Applied Sciences, vol. 11, no. 22, p. 10995, , 2021

69. FaithdialA faithful benchmark for information-seeking dialogue, S. Reddy, S. Milton, O. Zaiane, M. Yu, E. M. Ponti and, E. Kamalloo, N. Dziri, Transactions of the Association for Computational Linguistics, vol. 10, pp. 1473–1490, , 2022

70. Ir evaluation methods for retrieving highly relevant documents, J. Kekäläinen, K. Järvelin and, ACM SIGIR Forum, vol. 51, pp. 243–250, ACM New York, NY, USA, , 2017

71. Recent advances and challenges in task-oriented dialog systems, M. Huang and, Z. Zhang, R. Takanobu, Q. Zhu, X. Zhu, Science China Technological Sciences, vol. 63, no. 10, pp. 2011–2027, , 2020

72. Answer-focused and position-aware neural question generation in, J. Liu, Y. Ma and, Y. Lyu, X. Sun, W. He, S. Wang, Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3930–3939, , 2018

73. Bleua method for automatic evaluation of machine translation in, T. Ward and, S. Roukos, W.-J. Zhu, K. Papineni, Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, , 2002

74. Long-term control for dialogue generationMethods and evaluation, R. Ramakrishnan, R. McDonald, M. Schilman, K. Q. Weinberger and, H. B. Narangodage, arXiv preprint arXiv:2205.07352, , 2022

75. Ms marcoA human generated machine reading comprehension dataset, J. Gao, X. Song, R. Majumder and, T. Nguyen, S. Tiwary, M. Rosenberg, L. Deng, choice, vol. 2640, p. 660, , 2016

76. Multi-granularity prompts for topic shift detection in dialogue, X. Chu, J. Lin, Y. Fan, Q. Zhu, P. Li and, arXiv preprint arXiv:2305.14006, 2023, , 2023

77. Building a conversational agent overnight with dialogue self-play, A. Bapna, G. Tür, L. Heck, A. Rastogi, D. Hakkani-Tür, P. Shah, N. Nayak and, arXiv preprint arXiv:1801.04871, , 2018

78. Can large language models be an alternative to human evaluations?, C.-H. Chiang and, H.-y. Lee, arXiv preprint arXiv:2305.01937, 2023, , 1937

79. Survey on virtual assistantGoogle assistant siri cortana alexa in, A. S. Tulshan and, S. N. Dhage, Advances in Signal Processing and Intelligent Recognition Systems: 4th International Symposium SIRS 2018, Bangalore, India, September 19–22 Revised Selected Papers 4, pp. 190–201, Springer, , 2018

80. A network-based end-to-end trainable task-oriented dialogue system, M. Gasic, L. M. Rojas-Barahona, S. Ultes and, T.-H. Wen, D. Vandyke, N. Mrksic, S. Young, P.-H. Su, arXiv preprint arXiv:1604.04562, , 2016

81. Towards standard criteria for human evaluation of chatbotsA survey, H. Li, H. Liang and, arXiv preprint arXiv:2105.11197, , 2021

82. Alexa, siri, cortana, and more: an introduction to voice assistants, M. B. Hoy, Medical reference services quarterly, vol. 37, no. 1, pp. 81–88, , 2018

83. DlgnetA transformer-based model for dialogue response generation in, E. Mueller, O. Oluwatobi and, Proceedings of the 2nd workshop on natural language processing for conversational AI, pp. 54–62, , 2020

84. Taketopicshift aware knowledge selection for dialogue generation in, W. Wang, F. Meng, Z. Lin, J. Zhou, L. Wang and, J. Li, C. Yang, Proceedings of the 29th International Conference on Computational Linguistics, pp. 253–265, , 2022

85. Training language models to follow instructions with human feedback, X. Jiang, D. Almeida, C. L. Wainwright, A. Ray et al., S. Agarwal, K. Slama, P. Mishkin, L. Ouyang, C. Zhang, J. Wu, arXiv preprint arXiv:2203.02155, , 2022

86. Queens are powerful tooMitigating gender bias in dialogue generation, A. Williams, E. Dinan, A. Fan, J. Weston, D. Kiela and, J. Urbanek, arXiv preprint arXiv:1911.03842, , 2019

87. Evaluating the ripple effects of knowledge editing in language models, M. Geva, O. Yoran, E. Biran, R. Cohen, A. Globerson and, arXiv preprint arXiv:2307.12976, 2023, , 2023

88. Automatic evaluation of summaries using n-gram cooccurrence statistics, C.-Y. Lin and, E. Hovy, Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pp. 150–157, , 2003

89. UbarTowards fully end-to-end task-oriented dialog system with gpt-2 in, X. Quan, Y. Li and, Y. Yang, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 14230–14238, , 2021

90. A survey of available corpora for building data-driven dialogue systems, L. Charlin and, R. Lowe, J. Pineau, I. V. Serban, P. Henderson, arXiv preprint arXiv:1512.05742, , 2015

91. Database search results disambiguation for task-oriented dialog systems, C. Sankar, P. Crook, Z. Yu and, S. Kottur, S. Shayandeh, A. Geramifard, A. Beirami, K. Qian, arXiv preprint arXiv:2112.08351, , 2021

92. Dialogue response ranking training with large-scale human feedback data, B. Dolan, Y. Zhang, M. Galley, C. Brockett and, X. Gao, arXiv preprint arXiv:2009.06978, , 2020

93. How chat gpt can transform autodidactic experiences and open education?, M. Firat, 2023, , 2023

94. Knowledge graph refinementA survey of approaches and evaluation methods, H. Paulheim, Semantic web, vol. 8, no. 3, pp. 489–508, , 2017

95. Large language models as in-context ai generators for quality-diversity, A. Cully, B. Lim, M. Flageat and, arXiv preprint arXiv:2404.15794, 2024, , 2024

96. Towards an automatic turing testLearning to evaluate dialogue responses, R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard-Gontier, J. Pineau, Y. Bengio and, arXiv preprint arXiv:1708.07149, , 2017

97. Answer-based adversarial training for generating clarification questions, H. Daumé III, S. Rao and, arXiv preprint arXiv:1904.02281, , 2019

98. Are large language models good evaluators for abstractive summarization?, L. Cheng, L. Bing, C. Shen, Y. You and, arXiv preprint arXiv:2305.13091, 2023, , 2023

99. Bert with history answer embedding for conversational question answering, W. B. Croft, L. Yang, C. Qu, Y. Zhang and, M. Iyyer, M. Qiu, Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp. 1133–1136, , 2019

100. A full-text learning to rank dataset for medical information retrieval in, A. Sokolov and, V. Boteva, S. Riezler, D. Gholipour, Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, Proceedings 38, pp. 716–722, Springer, , 2016

101. CrosswozA large-scale chinese cross-domain task-oriented dialogue dataset, M. Huang, Q. Zhu, K. Huang, X. Zhu and, Z. Zhang, Transactions of the Association for Computational Linguistics, vol. 8, pp. 281–295, , 2020

102. Open-domain question answering goes conversational via question rewriting, R. Anantha, S. Vakulenko, S. Longpre, S. Pulman and, Z. Tu, S. Chappidi, arXiv preprint arXiv:2010.04898, , 2020

103. SlugnerdsA named entity recognition tool for open domain dialogue systems, K. K. Bowden, M. Walker, J. Wu, A. Misra and, S. Oraby, arXiv preprint arXiv:1805.03784, , 2018

104. Fusing task-oriented and open-domain dialogues in conversational agents in, J. Ni and, T. Young, V. Pandelea, F. Xing, E. Cambria, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 11622–11629, , 2022

105. Measuring conversational uptakeA case study on studentteacher interactions, D. Jurafsky and, T. Hashimoto, H. Hill, J. Cohen, Z. Mancenido, J. Liu, D. Demszky, arXiv preprint arXiv:2106.03873,, , 2021

106. Open-domain dialogue generationWhat we can do cannot do and should do next, K. Kann, J. Koh, A. Ebrahimi, S. Dudy and, A. Roncone, Proceedings of the 4th Workshop on NLP for Conversational AI, pp. 148–165, , 2022

107. Recent advances in deep learning based dialogue systemsA systematic survey, V. Pandelea, F. Xue and, T. Young, J. Ni, E. Cambria, Artificial intelligence review, vol. 56, no. 4, pp. 3055–3155, 2023, , 2023

108. Sequence-to-sequence data augmentation for dialogue language understanding, T. Liu, Y. Liu, Y. Hou, W. Che and, arXiv preprint arXiv:1807.01554, , 2018

109. do you follow me? A survey of recent approaches in dialogue state tracking, B. Favre, L. M. Rojas-Barahona and, L. Jacqmin, arXiv preprint arXiv:2207.14627,, , 2022

110. Conversations in the crowdCollecting data for task-oriented dialog learning, E. Kamar and, W. Lasecki, D. Bohus, Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 1, pp. 2–5, , 2013

111. Transferable multi-domain state generator for task-oriented dialogue systems, C. Xiong, R. Socher and, E. Hosseini-Asl, C.-S. Wu, P. Fung, A. Madotto, arXiv preprint arXiv:1905.08743, , 2019

112. Controllable open-ended question generation with a new question type ontology, L. Wang, S. Cao and, arXiv preprint arXiv:2107.00152, , 2021

113. Exploring voice assistant risks and potential with technology-based users. in, A. Hinderks, J. Thomaschewski, M. Rauschenberger and, A. M. Klein, WEBIST, pp. 147–154, , 2020

114. Tod-bertPre-trained natural language understanding for task-oriented dialogue, R. Socher and, S. Hoi, C.-S. Wu, C. Xiong, arXiv preprint arXiv:2004.06871, , 2020

115. UsrAn unsupervised and reference free evaluation metric for dialog generation, M. Eskenazi, S. Mehri and, arXiv preprint arXiv:2005.00456, , 2020

116. BertPre-training of deep bidirectional transformers for language understanding, K. Toutanova, M. W. Chang, K. Lee and, J. Devlin, arXiv preprint arXiv:1810.04805, , 2018

117. Retrieving and readingA comprehensive survey on open-domain question answering, J. Zheng, W. Lei, F. Zhu, S. Poria and, C. Wang, T.-S. Chua, arXiv preprint arXiv:2101.00774, , 2021

118. Asking clarifying questions in open-domain information-seeking conversations in, F. Crestani and, H. Zamani, M. Aliannejadi, W. B. Croft, Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pp. 475–484, , 2019

119. BondBert-assisted open-domain named entity recognition with distant supervision, R. Wang, S. Er, C. Zhang, C. Liang, T. Zhao and, Y. Yu, H. Jiang, in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1054–1064, , 2020

120. Few-shot dialogue generation without annotated dataA transfer learning approach, A. Eshghi and, I. Shalyminov, O. Lemon, S. Lee, arXiv preprint arXiv:1908.05854, , 2019

121. Keyword extraction from short texts with a text-to-text transfer transformer in, A. Mikołajczyk, A. Wawrzy´nski, P. P˛ezik, B. Nito´n and, M. Ogrodniczuk, Recent Challenges in Intelligent Information and Database Systems: 14th Asian Conference, ACIIDS 2022, Ho Chi Minh City, Vietnam, Proceedings, pp. 530–542, Springer, , 2022

122. Ordinal and attribute aware response generation in a multimodal dialogue system, M. Firdaus, H. Chauhan, A. Ekbal and, P. Bhattacharyya, Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 5437–5447, , 2019

123. Sgd-xA benchmark for robust generalization in schema-guided dialogue systems in, R. Gupta, H. Lee, Y. Wu, B. Zhang and, A. Rastogi, Y. Cao, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 10938– 10946, , 2022

124. Microsoft dialogue challengeBuilding end-to-end task-completion dialogue systems, J. Liu and, Y. Wang, X. Li, S. Sun, S. Panda, J. Gao, arXiv preprint arXiv:1807.11125, , 2018

125. DialogizerContext-aware conversational-qa dataset generation from textual sources, H. Bae, Y. Kim, Y. Hwang, H. Lee and, J. Bang, K. Jung, arXiv preprint arXiv:2311.07589, 2023, , 2023

126. Exploring the limits of transfer learning with a unified text-to-text transformer, A. Roberts, C. Raffel, Y. Zhou, M. Matena, S. Narang, P. J. Liu, W. Li and, N. Shazeer, K. Lee, vol. 21, no. 1, pp. 5485–5551, , 2020

127. Unsupervised dialogue topic segmentation with topic-aware contrastive learning in, H. Gao, T.-E. Lin, F. Huang and, Y. Li, Y. Wu, M. Yang, R. Wang, Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2481–2485, 2023, , 2023

128. A brief overview of chatgptThe history status quo and potential future development, S. He, S. Sun, Q.-L. Han and, Y. Tang, T. Wu, K. Liu, J. Liu, IEEE/CAA Journal of Automatica Sinica, vol. 10, no. 5, pp. 1122–1136, 2023, , 2023

129. DialogptLarge-scale generative pre-training for conversational response generation, B. Dolan, X. Gao, C. Brockett, Y. Zhang, M. Galley, J. Liu and, J. Gao, Y.-C. Chen, S. Sun, arXiv preprint arXiv:1911.00536, , 2019

130. SpecterDocument-level representation learning using citation-informed transformers, S. Feldman, A. Cohan, I. Beltagy, D. Downey and, D. S. Weld, arXiv preprint arXiv:2004.07180, , 2020

131. DragonA dialoguebased robot for assistive navigation with visual language grounding, A. Hasan, J. Lin, D. L. McPherson, W. A. Rogers and, K. Hong, Z. Mizrachi, S. Liu, P. Chang, R. Wang, K. Driggs-Campbell, IEEE Robotics and Automation Letters, 2024, , 2024

132. Two are better than oneAn ensemble of retrieval-and generation-based dialog systems, D. Zhao and, M. Zhang, X. Li, R. Yan, Y. Song, arXiv preprint arXiv:1610.07149, , 2016

133. Grounded conversation generation as guided traverses in commonsense knowledge graphs, C. Xiong and, Z. Liu, Z. Liu, H. Zhang, arXiv preprint arXiv:1911.02707, , 2019

134. Simulating user satisfaction for the evaluation of task-oriented dialogue systems in, S. Zhang, K. Balog, Z. Chen and, W. Sun, M. de Rijke, P. Ren, Z. Ren, Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2499–2506, , 2021

135. Learning to select the relevant history turns in conversational question answering in, A. Mahmood and, M. Zaib, Q. Z. Sheng, Y. Zhang, S. Sagar, W. E. Zhang, International Conference on Web Information Systems Engineering, pp. 334–348, Springer, 2023, , 2023

136. Two-level transformer and auxiliary coherence modeling for improved text segmentation, S. Somasundaran et al., Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7797–7804, , 2020

137. RqugeReference-free metric for evaluating question generation by answering the question, A. Fan, A. Mohammadshahi, T. Scialom, P. Yanki, M. Yazdani, M. Saeidi, J. Henderson and, arXiv preprint arXiv:2211.01482, , 2022

138. Approximating interactive human evaluation with self-play for open-domain dialog systems, N. Jones, R. Picard, J. H. Shen, N. Jaques, A. Lapedriza and, A. Ghandeharioun, C. Ferguson, Advances in Neural Information Processing Systems, vol. 32, , 2019

139. Chatgpt for good? on opportunities and challenges of large language models for education, E. Kasneci, E. Hüllermeier et al., K. Seßler, S. Küchemann, G. Groh, F. Fischer, D. Dementieva, S. Günnemann, M. Bannert, U. Gasser, Learning and individual differences, vol. 103, p. 102274, , 2023

140. Question generation for reading comprehension assessment by modeling how and what to ask, J. R. Dexter, B. Ghanem, S. M. von der Ohe and, L. L. Coleman, A. Fyshe, arXiv preprint arXiv:2204.02908, , 2022

141. Towards scalable multi-domain conversational agentsThe schema-guided dialogue dataset in, R. Gupta and, S. Sunkara, P. Khaitan, A. Rastogi, X. Zang, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8689–8696, , 2020

142. GradeAutomatic graphenhanced coherence metric for evaluating open-domain dialogue systems, J. Qin, L. Lin and, Z. Ye, L. Huang, X. Liang, arXiv preprint arXiv:2010.03994, , 2020

143. The ai doctor is inA survey of task-oriented dialogue systems for healthcare applications, M. Valizadeh and, N. Parde, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6638–6660, , 2022

144. MeteorAn automatic metric for mt evaluation with improved correlation with human judgments, A. Lavie, S. Banerjee and, in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, , 2005

145. Bad actor good advisorExploring the role of large language models in fake news detection in, D. Wang and, B. Hu, Y. Shi, P. Qi, Y. Li, J. Cao, Q. Sheng, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 22105–22113, 2024, , 2105

146. Semantically conditioned lstm-based natural language generation for spoken dialogue systems, S. Young, N. Mrksic, T.-H. Wen, P.-H. Su, M. Gasic, D. Vandyke and, arXiv preprint arXiv:1508.01745, , 2015

147. Addressing domain changes in task-oriented conversational agents through dialogue adaptation, B. Magnini, T. Labruna and, in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 149–158, 2023, , 2023

148. Automated interactive domain-specific conversational agents that understand human dialogs in, G. Gupta, P. Padalkar, K. Basu, A. Rajasekharan, J. Arias and, Y. Zeng, International Symposium on Practical Aspects of Declarative Languages, pp. 204–222, Springer, 2024, , 2024

149. Building a role specified open-domain dialogue system leveraging large-scale language models, D. Kwak, S.-W. Lee and, D. Ham, W. Park, S. Kim, S. Bae, S. Kang, arXiv preprint arXiv:2205.00176,, , 2022

150. Paraqaa question answering dataset with paraphrase responses for single-turn conversation in, B. Banerjee, J. Lehmann, K. Singh and, E. Kacupaj, European semantic web conference, pp. 598–613, Springer, , 2021

151. A sequential matching framework for multi-turn response selection in retrieval-based chatbots, Z. Li and, Y. Wu, C. Xu, M. Zhou, C. Xing, W. Wu, Computational Linguistics, vol. 45, no. 1, pp. 163–197, , 2019

152. History present and futureEnhancing dialogue generation with few-shot history-future prompt in, Y. Wang, Y. Wang, F. Mi, X. Jiang and, Q. Liu, P. Zhou, Y. Li, J. Liu, ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2023, , 2023

153. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, B.-H. Tseng, M. Gaši´c, S. Ultes, I. Casanueva, T.-H. Wen, P. Budzianowski, O. Ramadan and, arXiv preprint arXiv:1810.00278, , 2018

154. Investigating evaluation of open-domain dialogue systems with human generated multiple references, A. Pavel, T. Zhao, S. Mehri, P. Gupta, J. P. Bigham, M. Eskenazi and, arXiv preprint arXiv:1907.10568, , 2019

155. Multiwoz 2.2A dialogue dataset with additional annotation corrections and state tracking baselines, A. Rastogi, X. Zang, S. Sunkara, J. Chen, R. Gupta, J. Zhang and, arXiv preprint arXiv:2007.12720, , 2020

156. The study of the application of a web-based chatbot system on the teaching of foreign languages in, J. Jia, Society for Information Technology & Teacher Education International Conference, pp. 1201–1207 Association for the Advancement of Computing in Education (AACE), , 2004

157. The ubuntu dialogue corpusA large dataset for research in unstructured multi-turn dialogue systems, N. Pow, R. Lowe, J. Pineau, I. Serban and, arXiv preprint arXiv:1506.08909, , 2015

158. Automatic question generationa review of methodologies datasets evaluation metrics and applications, N. Mulla and, P. Gharpure, vol. 12, no. 1, pp. 1–32, 2023, , 2023

159. QrelscoreBetter evaluating generated questions with deeper understanding of context-aware relevance, L. Wu, X. Wang, S. Tang and, B. Liu, arXiv preprint arXiv:2204.13921, , 2022

160. Building an arabic flight booking dialogue system using a hybrid rule-based and data driven approach, A.-H. Al-Ajmi and, N. Al-Twairesh, IEEE Access, vol. 9, pp. 7043–7053, , 2021

161. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training, O. Agarwal, S. Shakeri and, R. Al-Rfou, H. Ge, arXiv preprint arXiv:2010.12688, , 2020

162. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders, T. Zhao, M. Eskenazi, R. Zhao and, arXiv preprint arXiv:1703.10960, , 2017

163. Learning new skills after deploymentImproving open-domain internet-driven dialogue with human feedback, J. Weston, M. Ung, K. Arora, Y.-L. Boureau and, J. Xu, M. Komeili, arXiv preprint arXiv:2208.03270, , 2022

164. TopicrefineJoint topic prediction and dialogue response generation for multi-turn end-to-end dialogue system, Z. Zhou, K.-F. Wong, G. P. C. Fung and, M. Cui, H. Wang, arXiv preprint arXiv:2109.05187, , 2021

165. Look before you hopConversational question answering over knowledge graphs using judicious context expansion in, P. Christmann, R. Saha Roy, J. Singh and, G. Weikum, A. Abujabal, Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 729–738, , 2019

166. Toward the implementation of a topic specific dialogue based natural language chatbot as an undergraduate advisor in, S. Ghose and, J. J. Barua, 2013 international conference on informatics, electronics and vision (ICIEV), pp. 1– 5, IEEE, , 2013

167. RisawozA large-scale multidomain wizard-of-oz dataset with rich semantic annotations for task-oriented dialogue modeling, D. Xiong, J. Quan, Z. Li and, S. Zhang, Q. Cao, arXiv preprint arXiv:2010.08738, , 2020

168. Complex sequential question answeringTowards learning to converse over linked question answer pairs with a knowledge graph in, K. Sankaranarayanan and, V. Pahuja, S. Chandar, M. Khapra, A. Saha, Proceedings of the AAAI conference on artificial intelligence, vol. 32, , 2018

169. How not to evaluate your dialogue systemAn empirical study of unsupervised evaluation metrics for dialogue response generation, C.-W. Liu, M. Noseworthy, R. Lowe, L. Charlin and, I. V. Serban, J. Pineau, arXiv preprint arXiv:1603.08023, , 2016

170. et GalaxyA generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection in, F. Huang, Y. Wu, M. Yang, D. Liu, Y. Dai, P. Jiang, L. Si al., W. He, Z. Cao, Y. Zheng, Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 10749–10757, 2022., , 2022

171. A systematic review on artificial intelligence dialogue systems for enhancing english as foreign language students interactional competence in the university, S. Wibowo, C. Zhai and, Computers and Education: Artificial Intelligence, vol. 4, p. 100134, 2023, , 2023