청각과 시각의 교차감각 매핑에 의한 소리 시각화 연구 = A Sound Visualization Using Cross-Modal Mappings Between Auditory and Visual Sense

다국어 초록 (Multilingual Abstract)

Background Cross-modality of auditory and visual senses is the interaction between these two different sensory channels. Today media environments are evolving towards the reinforcement of multi-modality. In various fields using media, the interaction between hearing and sight is an element that strengthens communication effect and user’s immersion level. Therefore, it is needed to study the way to visualize sound for the coming digital media era, by analyzing the framework of interactions between two different senses: auditory and visual.
Methods Firstly, we discussed about mechanisms of synesthesia and cross-modal integration. Then the cross-modal patterns of auditory and visual senses were analyzed by the cross-case analysis methodology. And the properties of cross-modality of auditory and visual senses was derived. Also, data applied with properties of audio-visual cross-modal integration was used to illustrate matching framework in variety of ways.
Result Consistent association between pitch and brightness was found in all studies. Chroma had the biggest relevance to overtone and was also related to pitch, volume, and tempo. We could not define exact match of color sets for synesthesia and cross-modal integration. However, there was a tendency of showing similar color responses through shared emotional experiences from musical and tactile experiences.
Conclusion Until now, most of the researches for audio-visual synesthesia were conducted with limited conditions and too small number of samples to generalize the result. In this study, however, we were able to derive the pattern of synesthesia and cross-modal integration by combining studies performed under variety of conditions and methods. These studies were conducted in different cultural areas and times. Also, based on gathered data, we could study the way to visualize sound from media. Various frameworks for mapping auditory data with visual data were used.

참고문헌 (Reference)

1 황지영, "전자음향의 시각화에 관한 연구" 2011

2 이윤정, "영화에서의 사운드 역할과 분석 - “인셉션”을 중심으로 -" 한국디지털디자인학회 12 (12): 221-230, 2012

3 Ward, J., "소리가 보이는 사람들" 흐름출판 2016

4 Yin, R. K., "사례연구방법" 한경사 2016

5 McLuhan, M., "미디어의 이해" 민음사 2002

6 Manovich, L., "뉴미디어의 언어" 커뮤니케이션북스 2014

7 이재현, "뉴미디어 이론" 커뮤니케이션북스 2013

8 Ramachandran, V. S., "뇌는 어떻게 세상을 보는가" 바다출판사 2006

9 김용철, "공감각의 형성요인에 관한 연구 : 촉감, 음감, 색감의 관계성을 중심으로" 홍익대학교 대학원 2005

10 Maryon, E., "The Science of Tone-Color" CC Birchard & Co 1924

11 Cytowic, R. E., "Synesthesia: A Union of the Senses" MIT 2002

12 Martino, G., "Synesthesia : Strong and weak" 10 (10): 61-65, 2001

13 de Thornley Head, P., "Synaesthesia: pitch-colour isomorphism in RGB-space?" 42 (42): 164-174, 2006

14 Ward, J., "Sound-colour synaesthesia : To what extent does it use cross-modal mechanisms common to us all?" 42 (42): 264-280, 2006

15 Manuel K., "Processing – Generative Design Tutorials"

16 Samuel, L., "Processing Cubes"

17 Marks, L. E., "On colored-hearing synesthesia : cross-modal translations of sensory dimensions" 82 (82): 303-331, 1975

18 Palmer, S. E., "Music–color associations are mediated by emotion" 110 (110): 8836-8841, 2013

19 Herriott, R. E., "Multisite qualitative policy research : Optimizing description and generalizability" 12 (12): 14-19, 1983

20 Rogers, G. L., "Four cases of pitch-specific chromesthesia in trained musicians with absolute pitch" 15 (15): 198-207, 1987

21 Yin R. K., "Community crime prevention: Does it work" Sage 294-308, 1986

22 Cruzes, D. S., "Case studies synthesis: a thematic, cross-case, and narrative synthesis worked example" 20 (20): 1634-1665, 2015

23 Sadato, N., "Activation of the primary visual cortex by Braille reading in blind subjects" 380 (380): 526-528, 1996

24 Giannakis, K., "A comparative evaluation of auditory-visual mappings for sound visualisation" 11 (11): 297-307, 2006

25 Haack, P. A., "A case study of a chromesthetic" 29 (29): 85-90, 1981

26 양혜린, "3D-Touch 인터페이스에서 지각된 촉각-시각 모달리티의 일치성이 사용자 인지반응에 미치는 영향" 110-112, 2016