Paying Attention To Descriptions Generated By Image Captioning Models

Abstract

To bridge the gap between humans and machines in image understanding and describing, we need further insight into how people describe a perceived scene. In this paper, we study the agreement between bottom-up saliency-based visual attention and object referrals in scene description constructs. We investigate the properties of human-written descriptions and machine-generated ones. We then propose a saliency-boosted image captioning model in order to investigate benefits from low-level cues in language models. We learn that (1) humans mention more salient objects earlier than less salient ones in their descriptions, (2) the better a captioning model performs, the better attention agreement it has with human descriptions, (3) the proposed saliencyboosted model, compared to its baseline form, does not improve significantly on the MS COCO database, indicating explicit bottom-up boosting does not help when the task is well learnt and tuned on a data, (4) a better generalization is, however, observed for the saliency-boosted model on unseen data.

Publication Date

12-22-2017

Publication Title

Proceedings of the IEEE International Conference on Computer Vision

Volume

2017-October

Number of Pages

2506-2515

Document Type

Article; Proceedings Paper

Personal Identifier

scopus

DOI Link

https://doi.org/10.1109/ICCV.2017.272

Socpus ID

85041928364 (Scopus)

Source API URL

https://api.elsevier.com/content/abstract/scopus_id/85041928364

This document is currently not available here.

Share

COinS