이미지의 인지적 특징 정량화를 통한 CNN-ViT 하이브리드 미학 평가 모델 = CNN-ViT Hybrid Aesthetic Evaluation Model Based on Quantification of Cognitive Features in Images

국문 초록 (Abstract)

본 논문에서는 이미지의 지역적 및 전역적 특징을 결합하여 이미지의 미학적 품질을 자동으로 평가할 수 있는 CNN-ViT 하이브리드 모델을 제안한다. 이 접근 방식에서는 CNN을 사용하여 색상 및 객체 배치와 같은 지역적 특징을 추출하고, ViT를 통해 전역적 특징을 반영하여 이미지의 미학적 가치를 분석한다. Color composition은 입력 이미지에서 주요 색상을 추출해 생성한 컬러팔레트를 CNN에 통과시켜 얻은 값이며, Rule of Third는 이미지 속 오브젝트가 삼등분할점에 얼마나 근접한지를 정량적으로 평가한 점수로 사용된다. 이러한 값들은 모델에 이미지의 주요 평가 요소인 색채와 공간 균형에 대한 정보를 제공한다. 모델은 이를 바탕으로 이미지의 점수와 색상, 공간의 균형 간에 연관성을 분석하며, 인간의 평가 분포와 유사한 점수를 추측하도록 설계되었다. 실험 결과, AADB 이미지 데이터베이스에서 스피어만순위상관계수(SRCC)에서는 0.716을 기록하여 순위 예측에서 더 일관된 결과를제공 했으며, 피어슨상관계수(LCC)에서도 0.72을 기록하여 기존 연구 모델보다 2~4% 정도 향상된 결과를 보였다.

다국어 초록 (Multilingual Abstract)

This paper proposes a CNN-ViT hybrid model that automatically evaluates the aesthetic quality of images bycombining local and global features. In this approach, CNN is used to extract local features such as color andobject placement, while ViT is employed to analyze the aesthetic value of the image by reflecting global features.
Color composition is derived by extracting the primary colors from the input image, creating a color palette, andthen passing it through the CNN. The Rule of Thirds is quantified by calculating how closely objects in the imageare positioned near the thirds intersection points. These values provide the model with critical information aboutthe color balance and spatial harmony of the image. The model then analyzes the relationship between thesefactors to predict scores that align closely with human judgment. Experimental results on the AADB image databaseshow that the proposed model achieved a Spearman's Rank Correlation Coefficient (SRCC) of 0.716, indicatingmore consistent rank predictions, and a Pearson Correlation Coefficient (LCC) of 0.72, which is 2~4% higher thanexisting models.

참고문헌 (Reference)

1 Schwarz, K., "Will people like your image? Learning the aesthetic space" IEEE 2048-2057, 2018

2 Simonyan, K., "Very deep convolutional networks for large-scale image recognition"

3 Li, K., "UniFormer : Unified transformer for efficient spatiotemporal representation learning"

4 Stepanova, E., "The impact of color palettes on the prices of paintings" 56 (56): 755-773, 2019

5 Karp, A, "The elements of color" 5 (5): 180-, 1972

6 Gao, F., "Style-adaptive photo aesthetic rating via convolutional neural networks and multi-task learning" 395 : 247-254, 2020

7 Lu, X., "RAPID : Rating pictorial aesthetics using deep learning" 457-466, 2014

8 Kong, S., "Photo aesthetics ranking network with attributes and content adaptation" Springer 662-679, 2016

9 Kong, S., "Photo aesthetics ranking network with attributes and content adaptation" Springer 662-679, 2016

10 Talebi, H., "NIMA : Neural image assessment" 27 (27): 3998-4011, 2018

11 Howard, A. G., "MobileNets : Efficient convolutional neural networks for mobile vision applications"

12 Wu, O., "Learning to predict the perceived visual quality of photos" IEEE 225-232, 2011

13 Krizhevsky, A., "ImageNet classification with deep convolutional neural networks" 60 : 84-90, 2017

14 Ke, Y., "Image aesthetics assessment using composite features from transformer and CNN" 29 (29): 2483-2494, 2023

15 Deng, Y., "Image aesthetic assessment : An experimental survey" 34 (34): 80-106, 2017

16 Szegedy, C., "Going deeper with convolutions" IEEE 1-9, 2015

17 Schultze, S., "Explaining Image Aesthetics Assessment : An Interactive Approach" 20-28, 2023

18 Tan, J. C., "Efficient palette-based decomposition and recoloring of images via RGBXY-space geometry" 37 (37): 1-10, 2018

19 Lu, P., "Deep conditional color harmony model for image aesthetic assessment" IEEE 2845-2850, 2018

20 Gulati, A., "Conformer : Convolutionaugmented transformer for speech recognition"

21 O’Donovan, P., "Color compatibility from large datasets" 30 (30): 1-12, 2011

22 Lokeshdhakar, "Color Thief"

23 Droste, M., "Bauhaus 1919~1933, Dansk produktion: Book Service I/S"

24 Lo, K. Y., "Assessment of photo aesthetics with efficiency" IEEE 2186-2189, 2012

25 Tong, J., "An interpretable approach for automatic aesthetic assessment of remote sensing images" 16 : 1077439-, 2022

26 Dosovitskiy, A., "An image is worth 16×16 words : Transformers for image recognition at scale"

27 Perona, F. R., "An application for aesthetic quality assessment in photography with interpretability features" 23 (23): 2021

28 Schloss, K. B., "Aesthetics of color combinations" SPIE 2010

29 Li, C., "Aesthetic visual quality assessment of paintings" 3 (3): 236-252, 2009

30 Nishiyama, M., "Aesthetic quality classification of photographs based on color harmony" IEEE 33-40, 2011

31 Murray, N., "AVA : A large-scale database for aesthetic visual analysis" IEEE 2408-2415, 2012

32 Anwar, A., "A survey on image aesthetic assessment"

33 Li, Xuewei, "A novel feature fusion method for computing image aesthetic quality" 8 : 63043-63054, 2020

34 Leder, H., "A model of aesthetic appreciation and aesthetic judgments" 95 (95): 489-508, 2004