http://chineseinput.net/에서 pinyin(병음)방식으로 중국어를 변환할 수 있습니다.
변환된 중국어를 복사하여 사용하시면 됩니다.
ProbAttnGuard: Reinforcing Transformers with Robust Stochastic Attention
김서린(Seorin Kim),김미현(Mihyeon Kim),김영빈(Youngbin Kim) 대한전자공학회 2023 대한전자공학회 학술대회 Vol.2023 No.6
Recent advances in Transformer-based models have shown remarkable performance in text classification tasks. However, these models can be vulnerable to out-of-distribution (OOD) and adversarial attack scenarios. In this paper, we propose a robust Transformer model that leverages stochastic attention to improve its resilience against such challenges. By sampling attention weights from probability distributions during training, our model achieves better test accuracy on both in-distribution and OOD datasets, as well as adversarial attack datasets, compared to deterministic vanilla models. We conduct experiments using various text classification tasks, including sentiment analysis and news article classification. Our results demonstrate that the proposed model exhibits improved performance and reduced performance gap between in-distribution and OOD or adversarial attack datasets. In future work, we aim to investigate the effect of contextual priors and develop more advanced models for enhanced robustness.