거대언어모델을 활용한 설문조사의 현재와 한계 = The Current State and Limitations of Large Language Models-Based Surveys

다국어 초록 (Multilingual Abstract)

Social scientists are exploring various ways to utilize the rapidly evolving large language models in their research. One way to use large language models for research is to employ them to conduct surveys. Because people’s responses vary by sociodemographic variables, the idea of this survey is to assign personas to a large language model with specific prompting and collect the responses. This article identifies the current status of this methodology and illustrates three fundamental limitations based on the conceptual framework of sociology. First, this approach may overestimate the influence of structure on an individual. Second, it is difficult to reflect the limited universal nature of social facts in particular societies. Third, the large language models we use are themselves social products that have already been internally censored to conform to social norms, and thereby may not necessarily represent actual public opinion. Therefore, instead of merely replicating existing survey methods using large-scale language models, I propose studying the societal changes these models will bring or creatively redesigning survey methods to leverage LLMs effectively.

참고문헌 (Reference)

1 "한국조사협회 윤리강령"

2 이경택 ; 이화정 ; 현경보, "유·무선전화 병행조사에 대한 연구 : 2011년 서울시장 보궐선거 여론조사 사례" 13 (13): 135-158, 2012

3 뒤르켐, 에밀, "사회학적 방법의 규칙들" 이른비 2021

4 Statham, Simon, "‘Wrap Our Arms around Them Here in Ireland’ : Social Media Campaigns in the Irish Abortion Referendum" 33 (33): 539-557, 2022

5 Jungherr, Andreas, "Why the Pirate Party Won the German Election of 2009 or the Trouble with Predictions: A Response to Tumasjan, A., Sprenger, T. O., Sander, P. G., & Welpe, I. M. “Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment" 30 (30): 229-234, 2012

6 Santurkar, Shibani, "Whose Opinions Do Language Models Reflect?" 202 : 29971-30004, 2023

7 Aher, Gati V., "Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies" 202 : 337-371, 2023

8 Sarstedt, Marko, "Using Large Language Models to Generate Silicon Samples in Consumer and Marketing Research : Challenges, Opportunities, and Guidelines" 41 (41): 1254-1270, 2024

9 Hilbert, Martin, "The World’s Technological Capacity to Store, Communicate, and Compute Information" 332 (332): 60-65, 2011

10 Bisbee, James, "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models" 1-16, 2024

11 Murthy, Dhiraj, "Sociology of Twitter/X : Trends, Challenges, and Future Research Directions" 50 : 2024

12 Rane, Halim, "Social Media, Social Movements and the Diffusion of Ideas in the Arab Uprisings" 18 (18): 97-111, 2012

13 Anthropic, "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet"

14 Perez, Ethan, "Red Teaming Language Models with Language Models"

15 Ganguli, Deep, "Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned"

16 Sun, Seungjong, "Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information"

17 Samvelyan, Mikayel, "Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts"

18 Tumasjan, Andranik, "Predicting Elections with Twitter: What 140 Characters Reveal about Political Sentiment" 4 (4): 178-185, 2010

19 Argyle, Lisa P., "Out of One, Many : Using Language Models to Simulate Human Samples" 31 (31): 337-351, 2023

20 Shafayat, Sheikh, "Multi-FAct: Assessing Multilingual LLMs’ Multi-Regional Knowledge Using FActScore"

21 Bai, Xuechunzi, "Measuring Implicit Bias in Explicitly Unbiased Large Language Models"

22 Horton, John J, "Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?" National Bureau of Economic Research 2023

23 Yoo, Kang Min, "HyperCLOVA X Technical Report"

24 Jakesch, Maurice, "Human Heuristics for AI-Generated Language are Flawed" 120 (120): e2208839120-, 2023

25 Dong, Xiangjue, "Disclosure and Mitigation of Gender Bias in LLMs"

26 Lazer, David M. J., "Computational Social Science : Obstacles and Opportunities" 369 (369): 1060-1062, 2020

27 Lazer, David, "Computational Social Science" 323 (323): 721-723, 2009

28 Bail, Christopher A, "Can Generative AI Improve Social Science?" 121 (121): e2314021121-, 2024

29 MacKenzie, Donald A, "An Engine, Not a Camera: How Financial Models Shape Markets" Mit Press 2008

30 Kim, Junsol, "AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction"

31 Grossmann, Igor, "AI and the Transformation of Social Science Research" 380 (380): 1108-1109, 2023

32 Wihbey, John, "AI and Epistemic Risk for Democracy: A Coming Crisis of Public Knowledge?" Northeastern University Ethics Institute 2024

33 Pellert, Max, "AI Psychometrics : Assessing the Psychological Profiles of Large Language Models through Psychometric Inventories" 19 (19): 808-826, 2023

34 Park, Peter S., "AI Deception : A Survey of Examples, Risks, and Potential Solutions" 5 (5): 100988-, 2024