태스크 디자인 방식이 크라우드소싱 기반의 학습데이터 수집결과에 미치는 영향 연구 - 대화형 에이전트 개발을 위한 자연어 학습데이터 수집을 중심으로 - = Study on the Effect of Task Design Method on Quality of Collecting Training Data Based on Crowdsourcing - Focusing on Natural Language Training Data Collection for Conversational Agent Development -

국문 초록 (Abstract)

대화형 에이전트(챗봇, 스마트스피커)를 이용한 서비스 개발이 보편화되고 있다. 에이전트를 훈련시키기 위해서는 많은 양의 ‘대화’가 확보되어야 하지만, 초기 디자인에서는 이를 확보하기가 쉽지 않다. 이러한 난점을 해결하고자 다양한 발화데이터 수집하는 방안이 모색되고 있다. 본 연구에서는 아마존 미캐니컬터크를 활용해 발화데이터를 모으는 시도를 해보았다. 수집 결과가 참여자에게 태스크를 어떻게 제시하느냐에 따라 영향을 받기에 태스크 디자인 요소(태스크 양, 보너스 보상 방식, Social Proof 기반 설명 방식)를 비교한 실험을 진행하였다. 분석 결과, 태스크 양이 많아지면 수집된 데이터의 질이 떨어졌으며, 보너스 보상 방식을 제공할 때 수집의 효율성이 높아졌다. 소셜프루프(Social Proof) 기반 설명 방식은 수집 결과의 다양성은 높이지만 효율성은 떨어뜨리는 것으로 나타났다. 본 연구는 크라우드소싱 기반의 학습데이터 수집에 있어 태스크 디자인 요소에 대한 디자인 가이드를 제언한다는 점에서 연구 의의가 있다.

다국어 초록 (Multilingual Abstract)

Service development using a conversational agent(chatbot, smart speaker) is becoming popular. A large amount of ‘conversation’ must be secured to train an agent, but it is not easy to get it in the initial design. To solve the difficulty, various methods for collecting training data are being sought. In this study, we attempted to collect training data using MTurk. We compared the task design elements(task amount, bonus compensation method, explanation method based social proof) to be influenced by how the results were presented to participants. As a result of analysis, the quality of the collected data decreased when the amount of task increased, and the efficiency of collection increased when providing bonus compensation method. Providing explanation method based social proof show that the diversity of the collection results is higher but the efficiency is lowered. This study is meaningful in that it suggests a design guide for task design elements in collecting training data based on crowdsourcing.

참고문헌 (Reference)

1 "https://www.mturk.com/"

2 "https://www.ibm.com/watson/kr-ko/developercloud/conversation.html"

3 "https://spacy.io/"

4 Vashistha, A., "You Can Always Do Better!: The Impact of Social Proof on Participant Response Bias" ACM 552-, 2018

5 Tur, G., "What is left to be understood in ATIS?" IEEE 19-24, 2010

6 Bloodgood, M., "Using Mechanical Turk to build machine translation evaluation sets" Association for Computational Linguistics 208-211, 2010

7 Jiang, Y., "Understanding task design trade-offs in crowdsourced paraphrase collection" 2 : 103-109, 2017

8 Lane, I., "Tools for collecting speech corpora via Mechanical-Turk" Association for Computational Linguistics 184-187, 2010

9 Howe, J., "The rise of crowdsourcing" 14 (14): 1-4, 2006

10 Jurafsky, D., "Speech and language processing(Vol. 3)" Pearson 2014

11 Banko, M., "Scaling to very very large corpora for natural language disambiguation" Association for Computational Linguistics 26-33, 2001

12 Vashistha, A., "Respeak: A Voice-based, Crowd-powered Speech Transcription System" ACM 1855-1866, 2017

13 Allahbakhsh, M., "Quality control in crowdsourcing systems : Issues and directions" 17 (17): 76-81, 2013

14 Daniel, F., "Quality control in crowdsourcing : A survey of quality attributes, assessment techniques, and assurance actions" 51 (51): 7-, 2018

15 Myers, C., "Patterns for How Users Overcome Obstacles in Voice User Interfaces" ACM 6-, 2018

16 Chen, J. J., "Opportunities for crowdsourcing research on amazon mechanical turk" 5 (5): 2011

17 Choi, J. D., "It depends: Dependency parser comparison using a web-based evaluation tool" 1 : 387-396, 2015

18 Kim, J. K., "Intent detection using semantically enriched word embeddings" IEEE 414-419, 2016

19 Wooten, D. B., "Informational influence and the ambiguity of product experience : Order effects on the weighting of evidence" 7 (7): 79-99, 1998

20 Cialdini, R. B., "Influence: Science and practice(Vol. 4)" Pearson education 2009

21 Das, S., "Increasing security sensitivity with social proof: A large-scale experimental confirmation" ACM 739-749, 2014

22 Mason, W., "Financial incentives and the performance of crowds" ACM 77-85, 2009

23 Malu, M., "Encouraging personal storytelling by example" ACM 611-612, 2012

24 Bapat, R., "Effective Crowdsourced Generation of Training Data for Chatbots Natural Language Understanding" Springer 114-128, 2018

25 Wang, W. Y., "Crowdsourcing the acquisition of natural language corpora: Methods and observations" IEEE 73-78, 2012

26 Snow, R., "Cheap and fast but is it good?: evaluating non-expert annotations for natural language tasks" Association for Computational Linguistics 254-263, 2008

27 Mao, K., "A survey of the use of crowdsourcing in software engineering" 126 : 57-84, 2017