Oct. 2023, 2023 IEEE 12th Global Conference on Consumer Electronic, Bronze Prize GCCE2023 Excellent Student Poster Award　　　　　　　　　　　　　　　

Oct. 2023, 2023 IEEE 12th Global Conference on Consumer Electronic, Silver Prize GCCE2023 Excellent Paper Award　　　　　　　　　　　　　　　

Jul. 2023, 2023 ICCE-TW Best Paper Award Honorable Metion　　　　　　　　　　　　　　　

Feb. 2023, 2022 IEEE Sapporo Section Encouragement Award 3件　　　　　　　　　　　　　　　

Feb. 2023, 2022 IEEE Sapporo Section Student Paper Contest, Encouraging Prize　　　　　　　　　　　　　　　

Jan. 2023, International Workshop on Advanced Image Technology (IWAIT2023) Best Paper Award　　　　　　　　　　　　　　　

Dec. 2022, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Dec. 2022, 令和4年度電気・情報関係学会北海道支部連合大会 3件　　　　　　　　　　　　　　　

Oct. 2022, 2022 IEEE 11th Global Conference on Consumer Electronic, Bronze Prize GCCE2022 Excellent Student Paper Award　　　　　　　　　　　　　　　

Oct. 2022, 2022 IEEE 11th Global Conference on Consumer Electronic, Silver Prize GCCE2022 Excellent Poster Award　　　　　　　　　　　　　　　

Oct. 2022, 2022 IEEE 11th Global Conference on Consumer Electronic, Silver Prize GCCE2022 Excellent Student Poster Award　　　　　　　　　　　　　　　

Sep. 2022, 土木学会 土木情報学システム開発賞　　　　　　　　　　　　　　　

Aug. 2022, MIRU 2022 学生奨励賞2件　　　　　　　　　　　　　　　

Mar. 2022, IEEE LifeTech 2022 WIE Excellent Poster Award　　　　　　　　　　　　　　　

Feb. 2022, 2021 IEEE Sapporo Section Student Paper Contest, Best Presentation Award　　　　　　　　　　　　　　　

Feb. 2022, 2021 IEEE Sapporo Section Encouragement Award 2件　　　　　　　　　　　　　　　

Jan. 2022, International Workshop on Advanced Image Technology (IWAIT2022) Best Paper Award　　　　　　　　　　　　　　　

Dec. 2021, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Dec. 2021, 令和3年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 3件　　　　　　　　　　　　　　　

Oct. 2021, The 1st Hokkaido Young Professionals Workshop Best Student Presentation Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Gold Prize GCCE2021 Excellent Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Gold Prize GCCE2021 Excellent Student Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Silver Prize GCCE2021 Excellent Student Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, GCCE2021 Outstanding Paper Award　　　　　　　　　　　　　　　

Jun. 2021, 映像情報メディア学会丹羽高柳賞論文賞　　　　　　　　　　　　　　　

Mar. 2021, IEEE LifeTech 2021 Excellent Student Paper Award for Oral Presentation, 2nd Prize　　　　　　　　　　　　　　　

Mar. 2021, 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, Excellent Poster (On-site) Award Winners: Bronze Prize　　　　　　　　　　　　　　　

Mar. 2021, ACM Multimedia Asia 2020, Best Paper Runner-up Award　　　　　　　　　　　　　　　

Feb. 2021, 2020 IEEE Sapporo Section Student Paper Awards, Best Paper Award　　　　　　　　　　　　　　　

Feb. 2021, 2020 IEEE Sapporo Section Student Paper Awards, Encouragement Paper Award　　　　　　　　　　　　　　　

Nov. 2020, 令和2年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 3件　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize IEEE GCCE2020 Excellent Student Paper Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize GCCE2020 Excellent Poster Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize IEEE GCCE2020 Excellent Demo! Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE2020 Excellent Paper Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Bronze Prize GCCE2020 Excellent Paper Award　　　　　　　　　　　　　　　

Jun. 2020, 映像情報メディア学会丹羽高柳賞論文賞　　　　　　　　　　　　　　　

May 2020, 2020 ICCE-TW Best Paper Award Honorable Metion　　　　　　　　　　　　　　　

Feb. 2020, The 2019 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　

Feb. 2020, The 2019 IEEE Sapporo Section Student Paper Contest Encouraging Prize 3件　　　　　　　　　　　　　　　

Dec. 2019, 令和元年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 2件　　　　　　　　　　　　　　　
小川 貴弘

Dec. 2019, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　
小川 貴弘

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE 2019 Excelent Paper Award　　　　　　　　　　　　　　　
小川 貴弘

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Outstanding Prize IEEE GCCE 2019 Excelent Demo! Award　　　　　　　　　　　　　　　
小川 貴弘

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE 2019 Excelent Poster Award　　　　　　　　　　　　　　　
小川 貴弘

Mar. 2019, 2019 IEEE 1st Global Conference on Life Sciences and Technologies, 2nd Prize IEEE Lifetech 2019 Excellent Paper Award　　　　　　　　　　　　　　　
小川 貴弘

Feb. 2019, The 2018 IEEE Sapporo Section Encouragement Award 2件　　　　　　　　　　　　　　　
小川 貴弘

Feb. 2019, The 2018 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　
小川 貴弘

Jan. 2019, The 2019 joint International Workshop on Advanced Image Technology & International Forum on Medical Imaging in Asia IWAIT Best Paper Award　　　　　　　　　　　　　　　
小川 貴弘

Dec. 2018, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　
小川 貴弘

Dec. 2018, 平成30年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　
小川 貴弘

Oct. 2018, 2018 IEEE 7th Global Conference on Consumer Electronics, 1st Prize IEEE GCCE 2018 Excellent Poster Award　　　　　　　　　　　　　　　
小川 貴弘

Oct. 2018, 2018 IEEE 7th Global Conference on Consumer Electronics, IEEE GCCE 2018 Outstanding Paper Award　　　　　　　　　　　　　　　
小川 貴弘

2018, The 2017 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2018, The 2017 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　
Takahiro Ogawa

2018, 平成29年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2017, 2017 IEEE 6th Global Conference on Consumer Electronics, IEEE GCCE 2017 Outstanding Poster Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2017, 精密工学会画像応用技術専門委員会・映像情報メディア学会メディア工学研究委員会合同サマーセミナー 優秀発表賞　　　　　　　　　　　　　　　
小川貴弘

2017, 電子情報通信学会 学術奨励賞　　　　　　　　　　　　　　　
小川貴弘

2017, The 2016 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2017, The 2016 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　
Takahiro Ogawa

2017, 平成28年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2017, International Workshop on Advanced Image Technology (IWAIT2017) Best Paper Award　　　　　　　　　　　　　　　
小川貴弘

2016, 平成27年度 SIP若手奨励賞　　　　　　　　　　　　　　　
小川貴弘

2016, The 2015 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2016, The 2015 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　
Takahiro Ogawa

2016, 平成27年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞 (2件)　　　　　　　　　　　　　　　
小川貴弘

2016, 2016 IEEE 5th Global Conference on Consumer Electronics 1st Prize IEEE GCCE 2016 Excellent Poster Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2016, 映像情報メディア学会, 論文査読功労賞　　　　　　　　　　　　　　　
小川貴弘

2015, 平成27年度 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　
小川貴弘

2015, The 2014 IEEE Sapporo Section Student Paper Contest Best Presentation Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2015, International Workshop on Advanced Image Technology (IWAIT2015) Best Paper Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2015, IEEE GCCE 2015 Excellent Poster Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2015, IEEE GCCE 2015 Outstanding Poster Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2014, IEEE GCCE 2014 Undergraduate Poster Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2013, 平成25年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2011, 平成23年度信号処理学生奨励賞 (2件)　　　　　　　　　　　　　　　
小川貴弘

2011, 平成23年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2011, 映像情報メディア学会 学生優秀発表賞　　　　　　　　　　　　　　　
小川貴弘

2010, 平成22年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2010, 2010 IEEE Sapporo Section Student Member Best Presentation Award　　　　　　　　　　　　　　　
Takahiro Ogawa

2009, 電子情報通信学会論文賞　　　　　　　　　　　　　　　
小川貴弘

2008, 平成20年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2008, 2008 IEEE Sapporo Section Student Member Encouraging Prize　　　　　　　　　　　　　　　
Takahiro Ogawa

2007, 平成19年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2007, IEEE International Conference on Consumer Electronics, IEEE Consumer Electronics Society Japan Chapter Young Scientist Paper Award　　　　　　　　　　　　　　　
小川貴弘

2006, 2006 IEEE Sapporo Section Student Paper Contest Award　　　　　　　　　　　　　　　
小川貴弘

2005, 精密工学会画像応用技術専門委員会・映像情報メディア学会メディア工学研究委員会合同サマーセミナー優秀発表賞　　　　　　　　　　　　　　　
小川貴弘

2005, 平成17年度電気情報関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　
小川貴弘

2005, 映像情報メディア学会 研究奨励賞　　　　　　　　　　　　　　　
小川貴弘

Graph Convolutional Network-based Sports Skill-level Recognition via Deep Metric Learning
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2024 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC), 15, 1, 4, IEEE, 02 Jul. 2024
International conference proceedings

Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions.
Yuya Moroto, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 11, 3440, 3440, Jun. 2024
Scientific journal

Expert–Novice Level Classification Using Graph Convolutional Network Introducing Confidence-Aware Node-Level Attention Mechanism
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
Sensors, 24, 10, 3033, 3033, MDPI AG, 10 May 2024
Scientific journal, In this study, we propose a classification method of expert–novice levels using a graph convolutional network (GCN) with a confidence-aware node-level attention mechanism. In classification using an attention mechanism, highlighted features may not be significant for accurate classification, thereby degrading classification performance. To address this issue, the proposed method introduces a confidence-aware node-level attention mechanism into a spatiotemporal attention GCN (STA-GCN) for the classification of expert–novice levels. Consequently, our method can contrast the attention value of each node on the basis of the confidence measure of the classification, which solves the problem of classification approaches using attention mechanisms and realizes accurate classification. Furthermore, because the expert–novice levels have ordinalities, using a classification model that considers ordinalities improves the classification performance. The proposed method involves a model that minimizes a loss function that considers the ordinalities of classes to be classified. By implementing the above approaches, the expert–novice level classification performance is improved.

Analysis of Continual Learning Techniques for Image Generative Models with Learned Class Information Management.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 10, 3087, 3087, May 2024
Scientific journal

A Novel Frame-Selection Metric for Video Inpainting to Enhance Urban Feature Extraction.
Yuhu Feng, Jiahuan Zhang, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 10, 3035, 3035, May 2024
Scientific journal

Algal Bed Region Segmentation Based on a ViT Adapter Using Aerial Images for Estimating CO2 Absorption Capacity.
Guang Li 0008, Ren Togo, Keisuke Maeda, Akinori Sako, Isao Yamauchi, Tetsuya Hayakawa, Shigeyuki Nakamae, Takahiro Ogawa 0001, Miki Haseyama
Remote. Sens., 16, 10, 1742, 1742, May 2024
Scientific journal

Confidence-Aware Spatial-Temporal Attention Graph Convolutional Network for Skeleton-Based Expert-Novice Level Classification
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 14 Apr. 2024
International conference proceedings

Parameter-efficient tuning of cross-modal retrieval for a specific database via trainable textual and visual prompts.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Int. J. Multim. Inf. Retr., 13, 1, 14, 14, Mar. 2024
Scientific journal

Flexibly manipulating popularity bias for tackling trade-offs in recommendation.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Inf. Process. Manag., 61, 2, 103606, 103606, Mar. 2024
Scientific journal

Text-Guided Image Editing Based on Post Score for Gaining Attention on Social Media.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 3, 921, 921, Feb. 2024
Scientific journal

AUTOMATIC RECOGNITION OF ALGAL BED AREAS BASED ON A LARGE-SCALE SEMANTIC SEGMENTATION MODEL FOR ESTIMATING CO₂ ABSORPTION BY BLUE CARBON
LI Guang, TOGO Ren, MAEDA Keisuke, SAKO Akinori, YAMAUCHI Isao, HAYAKAWA Tetsuya, NAKAMAE Shigeyuki, OGAWA Takahiro, HASEYAMA Miki
Japanese Journal of JSCE, 80, 17, n/a, Japan Society of Civil Engineers, 2024
Japanese, Measuring the CO₂ absorption of algal beds is one of the key issues for achieving carbon neutrality, but identifying the area of algal beds from UAV images requires a great deal of labor and experience. In this study, we propose a method for automatic recognition of algal beds using UAV images. The proposed method uses a model that enables semantic domain segmentation at the pixel level, and employs ViT-Adapter, one of the latest models. The advantage of this technique is that it effectively utilizes the knowledge of a trained large-scale model to recognize algal beds, and it can identify algal beds at the pixel level by adjusting the parameters of the model. In this study, we conducted learning using mask images of visually identified algal beds from aerial photographs, and further examined data expansion and other processing to adapt the learning to UAV images. The effectiveness of this method was verified through a demonstration using UAV images of the Erimo coast of Hokkaido.

Generative Dataset Distillation: Balancing Global Structure and Local Details.
Longzhen Li, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CVPR Workshops, 7664, 7671, 2024
International conference proceedings

Cross-domain Few-shot In-context Learning for Enhancing Traffic Sign Recognition.
Yaozong Gan, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2407.05814, 2024
Scientific journal

Zero-shot Composed Image Retrieval Considering Query-target Relationship Leveraging Masked Image-text Pairs.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2406.18836, 2024
Scientific journal

Prompt-based Personalized Federated Learning for Medical Visual Question Answering.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2402.09677, 2024
Scientific journal

Importance-Aware Adaptive Dataset Distillation.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2401.15863, 2024
Scientific journal

Multi-Object Editing in Personalized Text-To-Image Diffusion Model Via Segmentation Guidance.
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 8140, 8144, 2024
International conference proceedings

Caption Unification for Multi-View Lifelogging Images Based on In-Context Learning with Heterogeneous Semantic Contents.
Masaya Sato, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 8085, 8089, 2024
International conference proceedings

Enhancing Noisy Label Learning Via Unsupervised Contrastive Loss with Label Correction Based on Prior Knowledge.
Masaki Kashiwagi, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 6235, 6239, 2024
International conference proceedings

Prompt-Based Personalized Federated Learning for Medical Visual Question Answering.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1821, 1825, 2024
International conference proceedings

Importance-aware adaptive dataset distillation.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Neural Networks, 172, 106154, 106154, 2024
Scientific journal

Dataset Distillation Using Parameter Pruning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 107, 6, 936, 940, 2024
Scientific journal

Expert-novice level classification using engineers’ motion data in subway tunnel inspections - Introduction of explainable graph convolutional network -
SEINO Tatsuki, SAITO Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 5, 1, 101, 109, Japan Society of Civil Engineers, 2024
Japanese, Skill transfer to young engineers from senior engineers is a very important task in infrastructure equipment inspection. To support the skill transfer, an analysis method of the key factors of senior engineers skill is needed. However, conventional research has been limited to skill-level classification or analysis of the relationship between the skill level and biological data such as eye gaze and motion obtained from the engineers. This paper presents a method of classifying the skill level and visualization of its key factors to support the skill transfer. The proposed method employs a graph convolutional network introducing a novel attention mechanism for the classification and visualization.

Accident risk estimation with depth information in construction site videos
GOKA Ryota, MAEDA Keisuke, TOGO Ren, OGAWA Takahiro, HASEYAMA Miki
Proceedings of the Annual Conference of JSAI, JSAI2024, 2C6GS701, 2C6GS701, The Japanese Society for Artificial Intelligence, 2024
Japanese, In the construction industry, reducing accident risk and improving safety is one of the high-priority tasks. Recently, several methods have been proposed to estimate the contact accident risk with heavy machinery on construction sites for enhancing safety. Conventional studies based on deep learning estimate the risk by using relations within the image space of detected workers and machinery captured in construction site videos. However, these approaches focus on the distance between detected objects in the image, leading to the problem that accident risk is overestimated even when there is distance between objects in the real world. In this study, to consider 3D spatial information in videos, we propose a method for estimating the accident risk with visual features regarding depth information. Experimental results show that the proposed method performs better than existing methods in estimating the contact accident risk.

Reinforcing Pre-trained Models Using Counterfactual Images.
Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2406.13316, 2024
Scientific journal

Generative Dataset Distillation: Balancing Global Structure and Local Details.
Longzhen Li, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2404.17732, 2024
Scientific journal

Enhancing Generative Class Incremental Learning Performance with Model Forgetting Approach.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2403.18258, 2024
Scientific journal

Automatic Findings Generation for Distress Images Using In-Context Few-Shot Learning of Visual Language Model Based on Image Similarity and Text Diversity.
Yuto Watanabe, Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
J. Robotics Mechatronics, 36, 2, 353, 364, 2024
Scientific journal

Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender Systems.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 12, 23626, 23635, 2024
Scientific journal

Zero-Shot Traffic Sign Recognition Based on Midlevel Feature Matching
Yaozong Gan, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 23, 23, 9607, 9607, MDPI AG, 04 Dec. 2023
Scientific journal, Traffic sign recognition is a complex and challenging yet popular problem that can assist drivers on the road and reduce traffic accidents. Most existing methods for traffic sign recognition use convolutional neural networks (CNNs) and can achieve high recognition accuracy. However, these methods first require a large number of carefully crafted traffic sign datasets for the training process. Moreover, since traffic signs differ in each country and there is a variety of traffic signs, these methods need to be fine-tuned when recognizing new traffic sign categories. To address these issues, we propose a traffic sign matching method for zero-shot recognition. Our proposed method can perform traffic sign recognition without training data by directly matching the similarity of target and template traffic sign images. Our method uses the midlevel features of CNNs to obtain robust feature representations of traffic signs without additional training or fine-tuning. We discovered that midlevel features improve the accuracy of zero-shot traffic sign recognition. The proposed method achieves promising recognition results on the German Traffic Sign Recognition Benchmark open dataset and a real-world dataset taken from Sapporo City, Japan.

Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 22, 9287, 9287, Nov. 2023
Scientific journal

Skill Level Classification Using Motion Data via Spatial Temporal Graph Convolutional Network
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2023 IEEE 12th Global Conference on Consumer Electronics (GCCE), IEEE, 10 Oct. 2023
International conference proceedings

Visual Emotion Recognition Through Multimodal Cyclic-Label Dequantized Gaussian Process Latent Variable Model.
Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
Journal of Robotics and Mechatronics, 35, 5, 1321, 1330, Oct. 2023
Scientific journal

Self-supervised learning for gastritis detection with gastric X-ray images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
International Journal of Computer Assisted Radiology and Surgery, 18, 10, 1841, 1848, Oct. 2023
Scientific journal

Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding.
Yusuke Akamatsu, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 15, 6903, 6903, Aug. 2023
Scientific journal

Off-Screen Sound Separation Based on Audio-visual Pre-training Using Binaural Audio
Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama
SENSORS, 23, 9, 4540, 4540, MDPI, May 2023
English, Scientific journal, This study proposes a novel off-screen sound separation method based on audio-visual pre-training. In the field of audio-visual analysis, researchers have leveraged visual information for audio manipulation tasks, such as sound source separation. Although such audio manipulation tasks are based on correspondences between audio and video, these correspondences are not always established. Specifically, sounds coming from outside a screen have no audio-visual correspondences and thus interfere with conventional audio-visual learning. The proposed method separates such off-screen sounds based on their arrival directions using binaural audio, which provides us with three-dimensional sensation. Furthermore, we propose a new pre-training method that can consider the off-screen space and use the obtained representation to improve off-screen sound separation. Consequently, the proposed method can separate off-screen sounds irrespective of the direction from which they arrive. We conducted our evaluation using generated video data to circumvent the problem of difficulty in collecting ground truth for off-screen sounds. We confirmed the effectiveness of our methods through off-screen sound detection and separation tasks.

Multimodal Natural Language Explanation Generation for Visual Question Answering Based on Multiple Reference Data
He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama
ELECTRONICS, 12, 10, MDPI, May 2023
English, Scientific journal, As deep learning research continues to advance, interpretability is becoming as important as model performance. Conducting interpretability studies to understand the decision-making processes of deep learning models can improve performance and provide valuable insights for humans. The interpretability of visual question answering (VQA), a crucial task for human-computer interaction, has garnered the attention of researchers due to its wide range of applications. The generation of natural language explanations for VQA that humans can better understand has gradually supplanted heatmap representations as the mainstream focus in the field. Humans typically answer questions by first identifying the primary objects in an image and then referring to various information sources, both within and beyond the image, including prior knowledge. However, previous studies have only considered input images, resulting in insufficient information that can lead to incorrect answers and implausible explanations. To address this issue, we introduce multiple references in addition to the input image. Specifically, we propose a multimodal model that generates natural language explanations for VQA. We introduce outside knowledge using the input image and question and incorporate object information into the model through an object detection module. By increasing the information available during the model generation process, we significantly improve VQA accuracy and the reliability of the generated explanations. Moreover, we employ a simple and effective feature fusion joint vector to combine information from multiple modalities while maximizing information preservation. Qualitative and quantitative evaluation experiments demonstrate that the proposed method can generate more reliable explanations than state-of-the-art methods while maintaining answering accuracy.

Boosting automatic COVID-19 detection performance with self-supervised learning and batch knowledge ensembling.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Comput. Biol. Medicine, 158, 106877, 106877, May 2023
Scientific journal

Canonical Correlation Analysis Introducing Label Dequantization for Visual Emotion Recognition
SAITO Naoki, MAEDA Keisuke, OGAWA Takahiro, ASAMIZU Satoshi, HASEYAMA Miki
電子情報通信学会論文誌D 情報・システム, J106-D, 5, 337, 348, The Institute of Electronics, Information and Communication Engineers, 01 May 2023
Japanese, Supervised multi-view canonical correlation analysis via cyclic label dequantization (sMVCCA-CLD) for visual emotion recognition is presented in this paper. In the CCA approach, the dimension of latent common space is limited to the minimum dimension among those of all features. The dimension of label features i.e., the number of classes for label information, tends to be lower than those of the other features. Then the dimension of the latent common space constructed by CCA becomes lower. Therefore, there is a possibility of misssing important information that is necessary for the estimation from the latent common space due to the dimensionality constraint. To overcome this constraint, sMVCCA-CLD increases the dimension of the label features by the label dequantization process, and estimates the canonical correlation between multi-view features. In addition, sMVCCA-CLD performs the label dequantization considering that the emotions are represented by a cyclic model, e.g., Plutchik's and Mikel's wheels. Consequently, the construction of the latent common space for the accurate recognition of emotions becomes feasible.

COVID-19 detection based on self-supervised transfer learning using chest X-ray images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Int. J. Comput. Assist. Radiol. Surg., 18, 4, 715, 722, Apr. 2023
Scientific journal

Estimation of Degradation Degree in Road Infrastructure Based on Multi-Modal ABN Using Contrastive Learning.
Takaaki Higashi, Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 3, 1657, 1657, Feb. 2023
Scientific journal

Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 3, 1057, 1057, Feb. 2023
Scientific journal

Automatic generation of findings for distress images using visual language model—Introduction of few-shot learning based on similar image retrieval—
WATANABE Yuto, OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 223, 232, Japan Society of Civil Engineers, 2023
Japanese, In this study, we propose a novel method for automatic generation of findings using a visual language model to support the efficient creation of findings in inspection records for infrastructure facilities. It is essential for the creation of inspection records to write findings, which are sentences that include judgments and opinions of engineers in addition to what can be recognized from the distress image. However, there has been little discussion on the direct automatic generation of findings, and it is expected to realize generation methods to support the efficient creation of findings. With this background, in this paper, we introduce few-shot learning based on the similarity of distress images to the visual language model, which is an application of large language models attracted much attention in recent years and enables text output with a highly accurate understanding of both vision and language. By using past inspection records including images similar to the distress images, we can efficiently consider the relationship between the distress images and findings from a small number of pairs of them. In the last part of this paper, we confirm the effectiveness of the proposed method through experiments generating findings from the distress images included in the inspection records of bridges.

Distress estimation of road attachments based on attention-based multiple instance learning considering the diversity of background of images
WATANABE Koshi, OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 482, 489, Japan Society of Civil Engineers, 2023
Japanese, In this paper, we propose a distress estimation method for road attachments. Road attachments, such as signs and lighting, are equipped over a huge number and wide area, and therefore it is desired to achieve automatic inspection by using drones to reduce the burden on the inspectors. While captured images by drones include the diversity of the background including ground, sky, and road surfaces, the previous methods did not consider the diversity of the background of captured images of road attachments. This study proposes the distress estimation method via attention-based multiple instance learning to address this issue. We input patches of the images into the estimation model to distinguish between the background area and road attachment area and assign importance weight, or attention, to each patch. By performing this strategy, we realize the distress estimation method considering the diversity of the background area of images. In the experiment, we achieve a classification accuracy of about 70 % using images of actual road attachments confirming the effectiveness of this research approach.

Multi-task classification of distress types and deterioration levels for infrastructure maintenance
OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 807, 814, Japan Society of Civil Engineers, 2023
Japanese, This paper proposes a multi-task classification method that classifies both the distress type and deterioration level at the same time. Conventionally, classifying the deterioration level has been conducted for each distress type by using multiple models. In contrast, the proposed method enables classification of the deterioration level without assigning a distress type to the distress image in advance, by training a single model through loss minimization considering the distress type and deterioration level. In the last part of the paper, it is verified that the proposed method can achieve classification performance equivalent to models constructed for each distress type with a single model by using images of actual distress that have occurred on infrastructure.

Advanced AI research for enhancing the efficiency of infrastructure maintenance and management
MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 982, 989, Japan Society of Civil Engineers, 2023
Japanese, With the development and advancement of AI technology, research on the application to the field of infrastructure maintenance is actively progressing. Many of these studies focus on the development of learning theories that consider the characteristics of images obtained in infrastructure maintenance. The effectiveness of AI has been demonstrated in various tasks such as crack detection, classification of defect types, and estimation of degradation levels. On the other hand, to truly enhance the operational efficiency through AI, it is necessary to construct AI systems considering the practical business. Furthermore, to improve AI and continuously utilize AI, it is necessary to acquire images suitable for AI development in operations. Therefore, this paper introduces the learning theories that have been developed for images obtained in infrastructure maintenance, previous research on AI with essential function for the practical business and the authors’ idea on efficient image acquisition.

Automatic detection of dead trees using in-vehicle video based on semantic segmentation
OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 686, 693, Japan Society of Civil Engineers, 2023
Japanese, This paper proposes a method for automatic detection of dead trees using in-vehicle videos. The proposed method extracts vegetation regions from videos containing various objects based on semantic segmentation. Then, it detects dead trees from the extracted vegetation regions using color information. By presenting the dead tree regions detected by the proposed method to engineers, they can find dead trees efficiently. In the last part of this paper, the effectiveness of the proposed method is verified through experiments using actual in-vehicle videos.

Feature Integration via Back-Projection Ordering Multi-Modal Gaussian Process Latent Variable Model for Rating Prediction.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 3125, 3129, 2023
International conference proceedings

Multi-View Variational Recurrent Neural Network for Human Emotion Recognition Using Multi-Modal Biological Signals.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2925, 2929, 2023
International conference proceedings

Video-Music Retrieval with Fine-Grained Cross-Modal Alignment.
Yuki Era, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2005, 2009, 2023
International conference proceedings

Text-Guided Facial Image Manipulation for Wild Images via Manipulation Direction-Based Loss.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 361, 365, 2023
International conference proceedings

Text-to-image Diffusion Model Suppressing Catastrophic Forgetting via Elastic Weight Consolidation.
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 831, 832, 2023
International conference proceedings

Deterioration Level Estimation for Infrastructures Considering Noisy Labels via DivideMix.
Masaki Kashiwagi, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 829, 830, 2023
International conference proceedings

Novel Feature Extraction for Classification of Auditory-visual Stimuli from fNIRS Signals.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 759, 760, 2023
International conference proceedings

A Controllable Recoloring Method for Novel Views Using Segment Anything Model.
Haoyang Wang, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 587, 588, 2023
International conference proceedings

Caption Unification for Multiple Viewpoint Lifelogging Images and Its Verification.
Masaya Sato, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 415, 416, 2023
International conference proceedings

Improving Visual Counterfactual Explanation Models for Image Classification via CLIP.
Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 390, 391, 2023
International conference proceedings

DEGRADATION DEGREE ESTIMATION OF DISTRESS IMAGE IN ROAD INFRASTRUCTURE USING MULTI-DATASET CONTRASTIVE LEARNING
東孝明, 小川直輝, 前田圭介, 小川貴弘, 長谷山美紀
AI・データサイエンス論文集(Web), 4, 2, 2023

Few-Shot Personalized Saliency Prediction Using Tensor Regression for Preserving Structural Global Information.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2307.02799, 2023
Scientific journal

Personalized Content Recommender System via Non-verbal Interaction Using Face Mesh and Facial Expression.
Yuya Moroto, Rintaro Yanagi, Naoki Ogawa, Kyohei Kamikawa, Keigo Sakurai, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ACM Multimedia, 9399, 9401, 2023
International conference proceedings

Gromov-Wasserstein Autoencoders.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICLR, 2023
International conference proceedings

Proficiency-level Estimation Using Heterogeneous Features via Label Dequantized CCA.
Tatsuki Seino, Naoki Saito 0006, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
ICCE-Taiwan, 813, 814, 2023
International conference proceedings

Parameter-efficient Tuning of a Pre-trained Model via Prompt Learning in Cross-modal Retrieval.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 811, 812, 2023
International conference proceedings

Binaural Audio Generation with Data Augmentation from 360° Videos.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 795, 796, 2023
International conference proceedings

Prediction of Shoot Events by Considering Spatio-temporal Relations of Multimodal Features.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 793, 794, 2023
International conference proceedings

Shoot Event Prediction in Soccer Considering Expected Goals Based on Players' Positions.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 449, 450, 2023
International conference proceedings

Estimation of Amyloid-β Positivity Using QSM Images Considering Age Information.
Tsubasa Kunieda, Ren Togo, Noriko Nishioka, Yukie Shimizu, Shiro Watanabe, Kenji Hirata, Keisuke Maeda, Takahiro Ogawa 0001, Kohsuke Kudo, Miki Haseyama
ICCE-Taiwan, 165, 166, 2023
International conference proceedings

A Medical Domain Visual Question Generation Model via Large Language Model.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 163, 164, 2023
International conference proceedings

Defense Against Black-Box Adversarial Attacks Via Heterogeneous Fusion Features.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Binauralization Robust To Camera Rotation Using 360° Videos.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Learning Graph Laplacian from Intrinsic Patterns via Gaussian Process.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Estimation of Visual Contents from Human Brain Signals via VQA Based on Brain-Specific Attention.
Ryo Shichida, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Class-Aware Shared Gaussian Process Dynamic Model.
Ryosuke Sawata, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Improving Dropout in Graph Convolutional Networks for Recommendation via Contrastive Loss.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Multi-Label Classification in Anime Illustrations Based on Hierarchical Attribute Relationships.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 10, 4798, 4798, 2023
Scientific journal

Prediction of Shooting Events in Soccer Videos Using Complete Bipartite Graphs and Players' Spatial-Temporal Relations.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 9, 4506, 4506, 2023
Scientific journal

Material Compound-Property Retrieval Using Electron Microscope Images for Rubber Material Development.
Rintaro Yanagi, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 88258, 88264, 2023
Scientific journal

Hierarchical Multi-Label Attribute Classification With Graph Convolutional Networks on Anime Illustration.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 35447, 35456, 2023
Scientific journal

SpectralMAP: Approximating Data Manifold With Spectral Decomposition
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE ACCESS, 11, 31530, 31540, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, Dimensionality reduction is widely used to visualize complex high-dimensional data. This study presents a novel method for effective data visualization. Previous methods depend on local distance measurements for data manifold approximation. This leads to unreliable results when a data manifold locally oscillates because of some undesirable effects, such as noise effects. In this study, we overcome this limitation by introducing a dual approximation of a data manifold. We roughly approximate a data manifold with a neighborhood graph and prune it with a global filter. This dual scheme results in local oscillation robustness and yields effective visualization with explicit global preservation. We consider a global filter based on principal component analysis frameworks and derive it with the spectral information of the original high-dimensional data. Finally, we experiment with multiple datasets to verify our method, compare its performance to that of state-of-the-art methods, and confirm the effectiveness of our novelty and results.

Summarizing Data Structures with Gaussian Process and Robust Neighborhood Preservation
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2022, PT V, 13717, 157, 173, SPRINGER INTERNATIONAL PUBLISHING AG, 2023
English, International conference proceedings, Latent variable models summarize high-dimensional data while preserving its many complex properties. This paper proposes a locality-aware and low-rank approximated Gaussian process latent variable model (LolaGP) that can preserve the global relationship and local geometry in the derivation of the latent variables. We realize the global relationship by imitating the sample similarity non-linearly and the local geometry based on our newly constructed neighborhood graph. Formally, we derive LolaGP from GP-LVM and implement a locality-aware regularization to reflect its adjacency relationship. The neighborhood graph is constructed based on the latent variables, making the local preservation more resistant to noise disruption and the curse of dimensionality than the previous methods that directly construct it from the high-dimensional data. Furthermore, we introduce a new lower bound of a log-posterior distribution based on low-rank matrix approximation, which allows LolaGP to handle larger datasets than the conventional GP-LVM extensions. Our contribution is to preserve both the global and local structures in the derivation of the latent variables using the robust neighborhood graph and introduce the scalable lower bound of the log-posterior distribution. We conducted an experimental analysis using synthetic as well as images with and without highly noise disrupted datasets. From both qualitative and quantitative standpoint, our method produced successful results in all experimental settings.

Recallable Question Answering-Based Re-Ranking Considering Semantic Region for Cross-Modal Retrieval
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE OPEN JOURNAL OF SIGNAL PROCESSING, 4, 1, 11, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, Question answering (QA)-based re-ranking methods for cross-modal retrieval have been recently proposed to further narrow down similar candidate images. The conventional QA-based re-ranking methods provide questions to users by analyzing candidate images, and the initial retrieval results are re-ranked based on the user's feedback. Contrary to these developments, only focusing on performance improvement makes it difficult to efficiently elicit the user's retrieval intention. To realize more useful QA-based re-ranking, considering the user interaction for eliciting the user's retrieval intention is required. In this paper, we propose a QA-based re-ranking method with considering two important factors for eliciting the user's retrieval intention: query-image relevance and recallability. Considering the query-image relevance enables to only focus on the candidate images related to the provided query text, while, focusing on the recallability enables users to easily answer the provided question. With these procedures, our method can efficiently and effectively elicit the user's retrieval intention. Experimental results using Microsoft Common Objects in Context and computationally constructed dataset including similar candidate images show that our method can improve the performance of the cross-modal retrieval methods and the QA-based re-ranking methods.

Interpretable Visual Question Answering Referring to Outside Knowledge.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2303.04388, 2023
Scientific journal

Text-Guided Image Manipulation via Generative Adversarial Network With Referring Image Segmentation-Based Guidance.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 42534, 42545, 2023
Scientific journal

Cross-Modal Image Retrieval Considering Semantic Relationships With Many-to-Many Correspondence Loss.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 10675, 10686, 2023
Scientific journal

Similar interior coordination image retrieval with multi-view features
Ren Togo, Yuki Honma, Maiku Abe, Takahiro Ogawa, Miki Haseyama
International Journal of Multimedia Information Retrieval, 11, 4, 731, 740, Springer Science and Business Media LLC, 26 Aug. 2022
Scientific journal

Brain Decoding of Multiple Subjects for Estimating Visual Information Based on a Probabilistic Generative Model
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
SENSORS, 22, 16, 6148, 6148, MDPI, Aug. 2022
English, Scientific journal, Brain decoding is a process of decoding human cognitive contents from brain activities. However, improving the accuracy of brain decoding remains difficult due to the unique characteristics of the brain, such as the small sample size and high dimensionality of brain activities. Therefore, this paper proposes a method that effectively uses multi-subject brain activities to improve brain decoding accuracy. Specifically, we distinguish between the shared information common to multi-subject brain activities and the individual information based on each subject's brain activities, and both types of information are used to decode human visual cognition. Both types of information are extracted as features belonging to a latent space using a probabilistic generative model. In the experiment, an publicly available dataset and five subjects were used, and the estimation accuracy was validated on the basis of a confidence score ranging from 0 to 1, and a large value indicates superiority. The proposed method achieved a confidence score of 0.867 for the best subject and an average of 0.813 for the five subjects, which was the best compared to other methods. The experimental results show that the proposed method can accurately decode visual cognition compared with other existing methods in which the shared information is not distinguished from the individual information.

Defect Detection of Subway Tunnels Using Advanced U-Net Network
An Wang, Ren Togo, Takahiro Ogawa, Miki Haseyama
SENSORS, 22, 6, 2330, 2330, MDPI, Mar. 2022
English, Scientific journal, In this paper, we present a novel defect detection model based on an improved U-Net architecture. As a semantic segmentation task, the defect detection task has the problems of background-foreground imbalance, multi-scale targets, and feature similarity between the background and defects in the real-world data. Conventionally, general convolutional neural network (CNN)-based networks mainly focus on natural image tasks, which are insensitive to the problems in our task. The proposed method has a network design for multi-scale segmentation based on the U-Net architecture including an atrous spatial pyramid pooling (ASPP) module and an inception module, and can detect various types of defects compared to conventional simple CNN-based methods. Through the experiments using a real-world subway tunnel image dataset, the proposed method showed a better performance than that of general semantic segmentation including state-of-the-art methods. Additionally, we showed that our method can achieve excellent detection balance among multi-scale defects.

Microscopy and biomimetics: the NanoSuit® method and image retrieval platform
Takahiko Hariyama, Yasuharu Takaku, Hideya Kawasaki, Masatsugu Shimomura, Chiyo Senoh, Yumi Yamahama, Atsushi Hozumi, Satoru Ito, Naoto Matsuda, Satoshi Yamada, Toshiya Itoh, Miki Haseyama, Takahiro Ogawa, Naoki Mori, Shuhei So, Hidefumi Mitsuno, Masahiro Ohara, Shuhei Nomura, Masao Hirasaka
Microscopy, 71, 1, 1, 12, Oxford University Press (OUP), 29 Jan. 2022
Scientific journal, Abstract

This review aims to clarify a suitable method towards achieving next-generation sustainability. As represented by the term ‘Anthropocene’, the Earth, including humans, is entering a critical era; therefore, science has a great responsibility to solve it. Biomimetics, the emulation of the models, systems and elements of nature, especially biological science, is a powerful tool to approach sustainability problems. Microscopy has made great progress with the technology of observing biological and artificial materials and its techniques have been continuously improved, most recently through the NanoSuit® method. As one of the most important tools across many facets of research and development, microscopy has produced a large amount of accumulated digital data. However, it is difficult to extract useful data for making things as biomimetic ideas despite a large amount of biological data. Here, we would like to find a way to organically connect the indispensable microscopic data with the new biomimetics to solve complex human problems.

DETERIORATION LEVEL CLASSIFICATION OF DISTRESS IMAGE CONSIDERING CORRELATION AMONG HETEROGENEOUS FEATURES AND ATTENTION MAP CONFIDENCE
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀
AI・データサイエンス論文集(Web), 3, J2, 2022

WINTER ROAD SURFACE CONDITION CLASSIFICATION USING DEEP LEARNING WITH FOCAL LOSS BASED ON TEXT AND IMAGE INFORMATION
諸戸祐哉, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀
AI・データサイエンス論文集(Web), 3, J2, 2022

DISTRESS DETECTION BASED ON VISION TRANSFORMER USING EGOCENTRIC VIDEOS WHILE INSPECTING IN SUBWEY TUNNELS
櫻井慶悟, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀
AI・データサイエンス論文集(Web), 3, J2, 2022

CLASSIFICATION OF INSPECTION ACTION OF ENGINEERS FOR EFFICIENT VIDEO PRESENTATION TO ASSIST ASSESSMENT OF INFRASTRUCTURE FACILITY DISTRESS
上川恭平, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀
AI・データサイエンス論文集(Web), 3, J2, 2022

Popularity-Aware Graph Social Recommendation for Fully Non-Interaction Users.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Proceedings of the 4th ACM International Conference on Multimedia in Asia(MMAsia), 30, 5, ACM, 2022
International conference proceedings

Affective Embedding Framework with Semantic Representations from Tweets for Zero-Shot Visual Sentiment Prediction.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Proceedings of the 4th ACM International Conference on Multimedia in Asia(MMAsia), 6, 7, ACM, 2022
International conference proceedings

Visual Sentiment Prediction Using Cross-Way Few-Shot Learning Based on Knowledge Distillation.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3838, 3842, IEEE, 2022
International conference proceedings

Human-Centric Image Retrieval with Gaze-Based Image Captioning.
Yuhu Feng, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3828, 3832, IEEE, 2022
International conference proceedings

Few-Shot Personalized Saliency Prediction with Similarity of Gaze Tendency Using Object-Based Structural Information.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3823, 3827, IEEE, 2022
International conference proceedings

Gaussian Distributed Graph Constrained Multi-Modal Gaussian Process Latent Variable Model for Ordinal Labeled Data.
Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3798, 3802, IEEE, 2022
International conference proceedings

GCN-Based Multi-Modal Multi-Label Attribute Classification in Anime Illustration Using Domain-Specific Semantic Features.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 2021, 2025, IEEE, 2022
International conference proceedings

Trend Prediction of Students' Mock Examination Results Using Matrix Completion.
Yutaka Yamada, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 891, 892, IEEE, 2022
International conference proceedings

Shoot Event Prediction from Soccer Videos by Considering Players' Spatio-Temporal Relations.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 406, 407, IEEE, 2022
International conference proceedings

Refinement of Gaze-based Image Caption for Image Retrieval.
Yuhu Feng, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 272, 273, IEEE, 2022
International conference proceedings

Brain Decoding of Multiple Subjects for Estimating Visual Information Based on a Probabilistic Generative Model.
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 16, 6148, 6148, 2022
Scientific journal

Boosting Automatic COVID-19 Detection Performance with Self-Supervised Learning and Batch Knowledge Ensembling.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.09281, 2022
Scientific journal

COVID-19 Detection Based on Self-Supervised Transfer Learning Using Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.09276, 2022
Scientific journal

Union-set Multi-source Model Adaptation for Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.02785, 2022
Scientific journal

RGMIM: Region-Guided Masked Image Modeling for COVID-19 Detection.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2211.00313, 2022
Scientific journal

Rubber Material Retrieval System using Electron Microscope Images for Rubber Material Development.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MMAsia, 44, 3, ACM, 2022
International conference proceedings

Disentangled Image Attribute Editing in Latent Space via Mask-Based Retention Loss.
Shunya Ohaga, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MMAsia, 25, 7, ACM, 2022
International conference proceedings

Assessment of Image Manipulation Using Natural Language Description: Quantification of Manipulation Direction.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1046, 1050, IEEE, 2022
International conference proceedings

Improving Model Adaptation for Semantic Segmentation by Learning Model-Invariant Features with Multiple Source-Domain Models.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 421, 425, IEEE, 2022
International conference proceedings

A Multimodal Interpretable Visual Question Answering Model Introducing Image Caption Processor.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 777, 778, IEEE, 2022
International conference proceedings

Cross-modal Image Retrieval Considering Semantic Relationships with Object Information.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 775, 776, IEEE, 2022
International conference proceedings

Free-viewpoint Sports Video Generation Based on Dynamic NeRF Considering Time Series.
Masato Kawai, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 408, 409, IEEE, 2022
International conference proceedings

Content-based Image Retrieval Using Effective Synthesized Images from Different Camera Views via pixelNeRF.
Yuki Era, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 404, 405, IEEE, 2022
International conference proceedings

Analysis of Relationships between Visual Cognitive Contents and Response of Each Brain Region via Visual Question Answering.
Ryo Shichida, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 402, 403, IEEE, 2022
International conference proceedings

GCN-based Collaborative Filtering Considering Personality Bias.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 278, 279, IEEE, 2022
International conference proceedings

Prediction of Amyloid-β Positivity Using QSM Images Based on Bootstrap Your Own Latent.
Tsubasa Kunieda, Ren Togo, Noriko Nishioka, Yukie Shimizu, Shiro Watanabe, Kenji Hirata, Keisuke Maeda, Takahiro Ogawa 0001, Kohsuke Kudo, Miki Haseyama
GCCE, 137, 138, IEEE, 2022
International conference proceedings

Cross-platform Recommendation Considering Common Users' Preferences Based on Preference Propagation GraphNet.
Kazuki Yamamoto, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 135, 136, IEEE, 2022
International conference proceedings

Union-Set Multi-source Model Adaptation for Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ECCV (29), 579, 595, Springer, 2022
International conference proceedings

Trial Analysis of the Relationship between Taste and Biological Information Obtained While Eating Strawberries for Sensory Evaluation.
Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Shin-ichi Adachi, Fumiaki Yoshizawa, Miki Haseyama
Sensors, 22, 23, 9496, 9496, 2022
Scientific journal

Distress Detection in Subway Tunnel Images via Data Augmentation Based on Selective Image Cropping and Patching.
Keisuke Maeda, Saya Takada, Tomoki Haruyama, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 22, 8932, 8932, 2022
Scientific journal

Compressed gastric image generation based on soft-label dataset distillation for medical data sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Comput. Methods Programs Biomed., 227, 107189, 107189, 2022
Scientific journal

Dataset Complexity Assessment Based on Cumulative Maximum Scaled Area Under Laplacian Spectrum.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14743, 2022
Scientific journal

Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14635, 2022
Scientific journal

Dataset Distillation using Parameter Pruning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14609, 2022
Scientific journal

Dataset Distillation for Medical Dataset Sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14603, 2022
Scientific journal

Gromov-Wasserstein Autoencoders.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.07007, 2022
Scientific journal

TriBYOL: Triplet BYOL for Self-Supervised Representation Learning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2206.03012, 2022
Scientific journal

Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2206.03009, 2022
Scientific journal

Generating Captions of Imagined content from Human Brain Activities Applying An Image Captioning Model.
Saya Takada, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
4th IEEE Global Conference on Life Sciences and Technologies(LifeTech), 614, 615, IEEE, 2022
International conference proceedings

Knowledge-Guided Sequential Recommendation with Reinforcement Learning Using Empirical Distribution Function.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
4th IEEE Global Conference on Life Sciences and Technologies(LifeTech), 187, 188, IEEE, 2022
International conference proceedings

Transformer Based Multimodal Scene Recognition in Soccer Videos.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICME Workshops, 1, 6, IEEE, 2022
International conference proceedings

Scene Retrieval in Soccer Videos by Spatial-temporal Attention with Video Vision Transformer.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 453, 454, IEEE, 2022
International conference proceedings

Multi-scale Defect Detection from Subway Tunnel Images with Spatial Attention Mechanism.
An Wang, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 305, 306, IEEE, 2022
International conference proceedings

Action Classification Based on LSTM Using First and Third Person Videos of Engineers Inspecting Bridges.
Tsuyoshi Masuda, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 303, 304, IEEE, 2022
International conference proceedings

TriBYOL: Triplet BYOL for Self-Supervised Representation Learning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 3458, 3462, IEEE, 2022
International conference proceedings

Divergence-Guided Feature Alignment for Cross-Domain Object Detection.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 2240, 2244, IEEE, 2022
International conference proceedings

Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1371, 1375, IEEE, 2022
International conference proceedings

Regularization Meets Enhanced Multi-Stage Fusion Features: Making CNN More Robust against White-Box Adversarial Attacks.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 14, 5431, 5431, 2022
Scientific journal

Controllable Music Playlist Generation Based on Knowledge Graph and Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 10, 3722, 3722, 2022
Scientific journal

Learning intra-domain style-invariant representation for unsupervised domain adaptation of semantic segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Pattern Recognit., 132, 108911, 108911, 2022
Scientific journal

Dataset complexity assessment based on cumulative maximum scaled area under Laplacian spectrum.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Multimedia Tools and Applications, 81, 22, 32287, 32303, 2022
Scientific journal

User-centric multimodal feature extraction for personalized retrieval of tumblr posts.
Kazuma Ohtomo, Ryosuke Harakawa, Takahiro Ogawa 0001, Miki Haseyama, Masahiro Iwahashi
Multimedia Tools and Applications, 81, 2, 2979, 3003, 2022
Scientific journal

Chain centre loss: A psychology inspired loss function for image sentiment analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Neurocomputing, 495, 118, 128, 2022
Scientific journal

Generative Adversarial Network Including Referring Image Segmentation For Text-Guided Image Manipulation.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4818, 4822, IEEE, 2022
International conference proceedings

Human Emotion Recognition Using Multi-Modal Biological Signals Based On Time Lag-Considered Correlation Maximization.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4683, 4687, IEEE, 2022
International conference proceedings

Distributed Label Dequantized Gaussian Process Latent Variable Model for Multi-View Data Integration.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4643, 4647, IEEE, 2022
International conference proceedings

Variational Bayesian Graph Convolutional Network for Robust Collaborative Filtering.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 3908, 3912, IEEE, 2022
International conference proceedings

Time-Lag Aware Latent Variable Model for Prediction of Important Scenes Using Baseball Videos and Tweets.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 7, 2465, 2465, 2022
Scientific journal

Refining Graph Representation for Cross-Domain Recommendation Based on Edge Pruning in Latent Space.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
IEEE Access, 10, 12503, 12509, 2022
Scientific journal

Interactive Re-ranking via Object Entropy-Guided Question Answering for Cross-Modal Image Retrieval.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ACM Trans. Multim. Comput. Commun. Appl., 18, 3, 68, 17, 2022
Scientific journal

Deterioration Level Estimation Based on Convolutional Neural Network Using Confidence-Aware Attention Mechanism for Infrastructure Inspection.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 1, 382, 382, 2022
Scientific journal

Domain Adaptive Cross-Modal Image Retrieval via Modality and Domain Translations
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E104A, 6, 866, 875, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Jun. 2021
English, Scientific journal, Various cross-modal retrieval methods that can retrieve images related to a query sentence without text annotations have been proposed. Although a high level of retrieval performance is achieved by these methods, they have been developed for a single domain retrieval setting. When retrieval candidate images come from various domains, the retrieval performance of these methods might be decreased. To deal with this problem, we propose a new domain adaptive cross-modal retrieval method. By translating a modality and domains of a query and candidate images, our method can retrieve desired images accurately in a different domain retrieval setting. Experimental results for clipart and painting datasets showed that the proposed method has better retrieval performance than that of other conventional and state-of-the-art methods.

Database-adaptive Re-ranking for Enhancing Cross-modal Image Retrieval.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MM '21: ACM Multimedia Conference, 3816, 3825, ACM, 2021
International conference proceedings

Interpretable Representation Learning on Natural Image Datasets via Reconstruction in Visual-Semantic Embedding Space.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
2021 IEEE International Conference on Image Processing(ICIP), 2473, 2477, IEEE, 2021
International conference proceedings

Cross-Domain Recommendation Method Based On Multi-Layer Graph Analysis With Visual Information.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
ICIP, 2688, 2692, IEEE, 2021
International conference proceedings

Time-Lag Aware Multi-Modal Variational Autoencoder Using Baseball Videos And Tweets For Prediction Of Important Scenes.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2678, 2682, IEEE, 2021
International conference proceedings

Segmentation-Aware Text-Guided Image Manipulation.
Tomoki Haruyama, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2433, 2437, IEEE, 2021
International conference proceedings

Few-Shot Personalized Saliency Prediction using Person Similarity based on Collaborative Multi-Output Gaussian Process Regression.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1469, 1473, IEEE, 2021
International conference proceedings

Interest Level Estimation via Multi-Modal Gaussian Process Latent Variable Factorization.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1209, 1213, IEEE, 2021
International conference proceedings

Deep Metric Network Via Heterogeneous Semantics for Image Sentiment Analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1039, 1043, IEEE, 2021
International conference proceedings

Correlation-Aware Attention Branch Network Using Multi-Modal Data For Deterioration Level Estimation Of Infrastructures.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1014, 1018, IEEE, 2021
International conference proceedings

User Background Information-Aware Music Recommendation with Reinforcement Learning-Based Knowledge Graph Exploration.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Degradation Level Estimation of Road Structures via Attention Branch Network with Text Data.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Cross-view Self-supervised Learning via Momentum Statistics in Batch Normalization.
Guang Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Triplet Self-Supervised Learning for Gastritis Detection with Scarce Annotations.
Guang Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 787, 788, IEEE, 2021
International conference proceedings

Defense Against Image Captioning Attacks via A Robust and Stable Recurrent Neural Network.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 785, 786, IEEE, 2021
International conference proceedings

Text-Guided Image Manipulation for Desired Region Using Referring Image Segmentation.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 661, 662, IEEE, 2021
International conference proceedings

Multi-label Image Recognition Based on Multi-modal Graph Convolutional Networks Using Captioning Features.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 273, 274, IEEE, 2021

A Trial of Fine-grained Classification of Expert-novice Level Using Bio-signals While Inspecting Subway Tunnels.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 204, 205, IEEE, 2021
International conference proceedings

Listener Recommendation for Artist Based on Knowledge Graph and Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 202, 203, IEEE, 2021
International conference proceedings

Movie Rating Estimation Based on Weakly Supervised Multi-modal Latent Variable Model.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 195, 196, IEEE, 2021
International conference proceedings

Detection of Off-screen Sound Based on Loss Function of Self-supervised Audio-visual Spatialization.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 193, 194, IEEE, 2021
International conference proceedings

Visual Sentiment Prediction Using Few-shot Learning via Distribution Relations of Visual Features.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 191, 192, IEEE, 2021
International conference proceedings

Multi-class Similar Scene Retrieval in Soccer Videos: A Scene Confusion Reduction Method Based on Combination of Long and Short Frame Sequences.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 117, 118, IEEE, 2021
International conference proceedings

Sports Action Detection Based on Self-Supervised Feature Learning and Object Detection.
Tsuyoshi Masuda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 54, 55, IEEE, 2021
International conference proceedings

Analysis of Social Trends Related to COVID-19 Pandemic Utilizing Social Media Data.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
GCCE, 43, 44, IEEE, 2021
International conference proceedings

Estimating Imagined Images from Brain Activities via Visual Question Answering.
Saya Takada, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 35, 36, IEEE, 2021
International conference proceedings

Action Classification from Egocentric Videos Using Reinforcement Learning-based Pose Estimation.
Shunya Ohaga, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 9, 10, IEEE, 2021
International conference proceedings

Graph Analysis-based Recommendation via Entity Embeddings Using Wikipedia.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 5, 6, IEEE, 2021
International conference proceedings

Reliable Estimation of Deterioration Levels via Late Fusion Using Multi-View Distress Images for Practical Inspection.
Keisuke Maeda, Naoki Ogawa, Takahiro Ogawa 0001, Miki Haseyama
Journal of Imaging, 7, 12, 273, 273, 2021
Scientific journal

Feature Integration Through Semi-Supervised Multimodal Gaussian Process Latent Variable Model With Pseudo-Labels for Interest Level Estimation.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 9, 163843, 163850, 2021
Scientific journal

Preliminary study of AI-assisted diagnosis using FDG-PET/CT for axillary lymph node metastasis in patients with breast cancer
Zongyao Li, Kazuhiro Kitajima, Kenji Hirata, Ren Togo, Junki Takenaka, Yasuo Miyoshi, Kohsuke Kudo, Takahiro Ogawa, Miki Haseyama
EJNMMI Research, 11, 1, 2021
Scientific journal, Background: To improve the diagnostic accuracy of axillary lymph node (LN) metastasis in breast cancer patients using 2-[18F]FDG-PET/CT, we constructed an artificial intelligence (AI)-assisted diagnosis system that uses deep-learning technologies. Materials and methods: Two clinicians and the new AI system retrospectively analyzed and diagnosed 414 axillae of 407 patients with biopsy-proven breast cancer who had undergone 2-[18F]FDG-PET/CT before a mastectomy or breast-conserving surgery with a sentinel lymph node (LN) biopsy and/or axillary LN dissection. We designed and trained a deep 3D convolutional neural network (CNN) as the AI model. The diagnoses from the clinicians were blended with the diagnoses from the AI model to improve the diagnostic accuracy. Results: Although the AI model did not outperform the clinicians, the diagnostic accuracies of the clinicians were considerably improved by collaborating with the AI model: the two clinicians' sensitivities of 59.8% and 57.4% increased to 68.6% and 64.2%, respectively, whereas the clinicians' specificities of 99.0% and 99.5% remained unchanged. Conclusions: It is expected that AI using deep-learning technologies will be useful in diagnosing axillary LN metastasis using 2-[18F]FDG-PET/CT. Even if the diagnostic performance of AI is not better than that of clinicians, taking AI diagnoses into consideration may positively impact the overall diagnostic accuracy.

Personalized recommendation of Tumblr posts using graph convolutional networks with preference-aware multimodal features
Kazuma Ohtomo, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, Masahiro Iwahashi
ITE Transactions on Media Technology and Applications, 9, 1, 54, 61, 2021
Scientific journal, Tumblr is a popular micro-blogging service on which users can share posts comprising text and images. This paper presents a method for personalizing post recommendations for each user from a large number of posts. Specifically, we develop a supervised multi-variational auto encoder considering user preference (SMVAE-UP). SMVAE-UP can extract relationships between text and image features by considering class information representing a user’s preference for each post; thus, preference-aware multimodal features can be calculated. Furthermore, for each target user, a network that enables comparison between a user and posts in the same feature space is constructed using the preference-aware multimodal features and metadata on posts. By applying graph convolutional networks (GCNs) to the network constructed for each target user, an accurate recommendation matching each user’s preferred posts becomes feasible. Experimental results for real-world datasets including six users and 99, 844 posts show the effectiveness of our method.

User-selectable event summarization in unedited raw soccer video via multimodal bidirectional LSTM
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 9, 1, 42, 53, 2021
Scientific journal, A new method that generates user-selectable event summaries from unedited raw soccer videos is presented in this paper. Since there are more unedited raw soccer videos than broadcasted/distributed soccer videos and unedited videos have various viewers, it is necessary to analyze these videos for meeting the demands of various viewers. The proposed method introduces a multimodal CNN-BiLSTM architecture for analyzing unedited raw soccer videos. This architecture extracts candidate scenes for event summarization from unedited soccer videos and classifies these scenes into typical events. Finally, our method generates user-selectable event summaries by simultaneously considering the importance of candidate scenes and the event classification results. Experimental results using real unedited raw soccer videos show the effectiveness of our method.

Interior coordination image retrieval with object-detection-based and color features
Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings of SPIE - The International Society for Optical Engineering, 11766, 2021
International conference proceedings, This paper presents a new interior coordination image retrieval method using object-detection-based and color features. Interior coordination requires consideration of objects' positional information and the overall atmosphere of the room simultaneously. However, similar image retrieval methods considering the coordination characteristics have not been proposed. In the proposed method, we extract different types of features from interior coordination images and realize the similar interior coordination image retrieval based on our newly derived features.

Interest estimation method based on 2D pose features on security camera
Yuki Honma, Ren Togo, Maiku Abe, Takahiro Ogawa, Miki Haseyama
Proceedings of SPIE - The International Society for Optical Engineering, 11766, 2021
International conference proceedings, This paper proposes a customer interest estimation method using security camera to meet the demand of the retail industry. In the field of retail industry, it is considered that the understanding of customers' interests in the real store can be used for various marketing activities such as the product development and the layout of the store. Then, it is important to pay attention to customers' behavior in the real store. Their behavior is often recorded by the cameras installed in the store for security purposes. A method for estimating their interests from the videos of the security camera is presented in this paper. The novelty of our method is three-fold. Firstly, the experimental data of subjects in our group were taken by using the security camera already installed in the real store. Secondly, we used a pre-trained posture estimation model and treated the results as the features to be trained by a two-layer neural network model. Finally, a professional have annotated the subjects' interests. The effectiveness of our method was confirmed by comparing with benchmark supervised machine learning models.

Cross-domain recommendation based on multilayer graph analysis using subgraph representation
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
Proceedings of SPIE - The International Society for Optical Engineering, 11766, 2021
International conference proceedings, This paper presents cross-domain recommendation based on multilayer graph analysis using subgraph representation. The proposed method constructs two graphs in source and target domains utilizing user-item embedding and trains link relationships between the users' embedding features on each above graph via graph convolutional networks considering subgraph representation. Thus, the proposed method can obtain features with high representation ability, and this is the main contribution of this paper. Then the proposed method can estimate the user's embedding features in the target domain from those in the source domain and recommend items to users by using the estimated features. Experiments on real-world e-commerce datasets verify the effectiveness of the proposed method.

A note on detection of sports action based on temporal cycle consistency learning
Tsuyoshi Masuda, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings of SPIE - The International Society for Optical Engineering, 11766, 2021
International conference proceedings, This paper presents a method for action detection based on Temporal Cycle Consistency(TCC) Learning. The proposed method realizes the action detection of flexible length segments based on a frame-level action prediction technique. We enable calculation of similarities for spatio-temporal features based on TCC to detect target actions from input videos. Finally, our method determines temporal segments by smoothing the frame-level action detection result. Experimental results show the validity of the proposed method.

IR Questioner: QA-based Interactive Retrieval System.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
ICMR 2021 - Proceedings of the 2021 International Conference on Multimedia Retrieval, 611, 614, ACM, 2021
International conference proceedings, Image retrieval from a given text query (text-to-image retrieval) is one of the most essential systems, and it is effectively utilized for databases (DBs) on the Web. To make them more versatile and familiar, a retrieval system that is adaptive even for personal DBs such as images in smartphones and lifelogging devices should be considered. In this paper, we present a novel text-to-image retrieval system that is specialized for personal DBs. With the cross-modal scheme and the question-answering scheme, the developed system enables users to obtain the desired image effectively even from personal DBs. Our demo is available at https://sites.google.com/view/ir-questioner/.

Cross-Domain Semi-Supervised Deep Metric Learning for Image Sentiment Analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 4150, 4154, IEEE, 2021
International conference proceedings, This paper presents a novel method on image sentiment analysis called cross-domain semi-supervised deep metric learning (CDSS-DML). The proposed method has two contributions. Firstly, since previous researches on image sentiment analysis suffer from the limit of a small amount of well-labeled data, which occurs a decrease in accuracy of classification, CDSS-DML breaks through the limit by training with unlabeled data based on a teacher-student model. Secondly, the proposed method overcomes the difficulty of distribution shift between well-labeled and unlabeled data by jointing three losses. Especially, the proposed method constructs an effective latent space with the joint loss considering the inter-class and the intra-class correlations for image sentiments. From experimental results, the performance improvement with CDSS-DML is confirmed.

Feature Integration via Semi-Supervised Ordinally Multi-Modal Gaussian Process Latent Variable Model.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 4130, 4134, IEEE, 2021
International conference proceedings, This paper presents a method of feature integration via semi-supervised ordinally multi-modal Gaussian process latent variable model (Semi-OMGP). The proposed method transforms multimodal features into common latent variables suitable for users’ interest level estimation. For dealing with the multi-modal features, the proposed method newly derives Semi-OMGP. Semi-OMGP has two contributions. First, Semi-OMGP is suitable for integration between heterogeneous modalities with different distributions by assuming that the similarity matrices of these modalities as observations are generated from latent variables. Second, Semi-OMGP can efficiently use label information by introducing an operator considering the ordinal grade into the prior distribution of latent variables when obtained label information is partially given. Semi-OMGP can simultaneously realize the above contributions, and successful multi-modal feature integration becomes feasible. Experimental results show the effectiveness of the proposed method.

Multi-Modal Label Dequantized Gaussian Process Latent Variable Model for Ordinal Label Estimation.
Masanao Matsumoto, Keisuke Maeda, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 3985, 3989, IEEE, 2021
International conference proceedings, This paper presents multi-modal label dequantized Gaussian process latent variable model (mLDGP) for ordinal label estimation. mLDGP is constructed based on a probabilistic generative model via Gaussian process and realizes accurate calculation of common latent space from multi-view features including low-dimensional ordinal label features. Conventional methods have a problem that the dimension of the common latent space was limited to that of the label feature, and an enough expressive latent space cannot be obtained. mLDGP, which is constructed by introducing our novel label dequantization mechanism into the objective function of multi-modal Gaussian process latent variable model (GPLVM), can increase the dimension of label features. Then mLDGP can calculate the effective latent space. Furthermore, mLDGP can estimate projection transforming unknown features of test samples into the common latent space, which was a problem of the conventional GPLVMs. From experimental results obtained by applying our method to the product rating estimation on the online shopping website, it is confirmed that accuracy improvement using mLDGP becomes feasible compared to various methods.

Semantic-Aware Unpaired Image-to-Image Translation for Urban Scene Images.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 2150, 2154, IEEE, 2021
International conference proceedings, Unpaired image-to-image (I2I) translation methods have been developed for several years. Present methods do not take into consideration semantic information of the original image, which may perform well on simple datasets of uncomplicated scenes, however, fail in complex datasets of scenes involving abundant objects, such as urban scenes. To tackle this problem, in this paper, we reasonably modify the previous problem setting and present a novel semantic-aware method. Specifically, in training, we use additional semantic label maps of training images, while in the test, no labels are required. We originally adopt a semantic knowledge distillation strategy to acquire semantic information from the labels and construct a particular normalization layer to introduce semantic information. Being aware of the pixel-level semantic information, our method can realize better I2I translation than the previous methods. Experiments are conducted on benchmark datasets of urban scenes to validate the effectiveness of our method.

Classification of Expert-Novice Level Using Eye Tracking And Motion Data via Conditional Multimodal Variational Autoencoder.
Yusuke Akamatsu, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 1360, 1364, IEEE, 2021
International conference proceedings, Sensor data from wearable devices have been utilized to analyze dif- ferences between experts and novices. Previous studies attempted to classify the expert-novice level from sensor data based on super- vised learning methods. However, these approaches need to col- lect enough training data covering various novices' sensor patterns. In this paper, we propose a semi-supervised anomaly detection ap- proach that requires only sensor data of experts for training and iden- tifies those of novices as anomalies. Our proposed anomaly detec- tion model named conditional multimodal variational autoencoder (CMVAE) has the following two technical contributions: (i) con- sidering action information of persons and (ii) utilizing multimodal sensor data, i:e:; eye tracking data and motion data in this case. The proposed method is evaluated on sensor data measured when expert and novice soccer players were shooting, dribbling, and doing soc- cer ball juggling. Experimental results show that CMVAE can more accurately classify the expert-novice level than previous supervised learning methods and anomaly detection methods using other VAEs.

Estimation of Visual Features of Viewed Image From Individual and Shared Brain Information Based on FMRI Data Using Probabilistic Generative Model.
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 1335, 1339, IEEE, 2021
International conference proceedings, This paper presents a method for estimation of visual features based on brain responses measured when subjects view images. The proposed method estimates visual features of viewed images by using both individual and shared brain information from functional magnetic resonance imaging (fMRI) data when subjects view images. To extract an effective latent space shared by multiple subjects from high dimensional fMRI data, a probabilistic generative model that can provide a prior distribution to the space is introduced into the proposed method. Also, the extraction of a robust feature space with respect to noise for the individual information becomes feasible via the proposed probabilistic generative model. This is the first contribution of our method. Furthermore, the proposed method constructs a decoder transforming brain information into visual features based on collaborative use of both estimated spaces for individual and shared brain information. This is the second contribution of our method. Experimental results show that the proposed method improves the estimation accuracy of the visual features of viewed images.

Human-Centered Favorite Music Classification Using EEG-Based Individual Music Preference Via Deep Time-Series CCA.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2021-June, 1320, 1324, IEEE, 2021
International conference proceedings, A method to classify a user's like or dislike musical pieces based on the extraction of his or her music preference is proposed in this paper. New scheme of Canonical Correlation Analysis (CCA), called Deep Time-series CCA (DTCCA), which can consider the correlation between two sets of input features with considering the time-series relation lurked in each input data is exploited to realize the aforementioned classification. One of the most difference between DTCCA and existing other CCAs is enabling to consider the above time-series relation, and thus DTCCA make the individual electroencephalogram (EEG)-based favorite music classification more effective than the methods using one of other CCAs instead of DTCCA since EEG and audio signals are respectively time-series data. Experimental results show that DTCCA-based favorite music classification outperformed not only method using original features without CCA but also methods using other existing CCAs including even state-of-the-art CCA.

Deterioration level estimation via neural network maximizing category-based ordinally supervised multi-view canonical correlation.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Multim. Tools Appl., 80, 15, 23091, 23112, 2021
Scientific journal, A deterioration level estimation method via neural network maximizing category-based ordinally supervised multi-view canonical correlation is presented in this paper. This paper focuses on real world data such as industrial applications and has two contributions. First, a novel neural network handling multi-modal features transforms original features into features effectively representing deterioration levels in transmission towers, which are one of the infrastructures, with consideration of only correlation maximization. It can be realized by setting projection matrices maximizing correlations between multiple features into weights of hidden layers. That is, since the proposed network has only a few hidden layers, it can be trained from a small amount of training data. Second, since there exist diverse characteristics and an ordinal scale in deterioration levels, the proposed method newly derives category-based ordinally supervised multi-view canonical correlation analysis (Co-sMVCCA). Co-sMVCCA enables estimation of effective projection considering both within-class divergence and the ordinal scale between classes. Experimental results showed that the proposed method realizes accurate deterioration level estimation.

Disentangled Representation Learning in Real-World Image Datasets via Image Segmentation Prior.
Nao Nakagawa, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 110880, 110888, 2021
Scientific journal, We propose a novel method that can learn easy-to-interpret latent representations in real-world image datasets using a VAE-based model by splitting an image into several disjoint regions. Our method performs object-wise disentanglement by exploiting image segmentation and alpha compositing. With remarkable results obtained by unsupervised disentanglement methods for toy datasets, recent studies have tackled challenging disentanglement for real-world image datasets. However, these methods involve deviations from the standard VAE architecture, which has favorable disentanglement properties. Thus, for disentanglement in images of real-world image datasets with preservation of the VAE backbone, we designed an encoder and a decoder that embed an image into disjoint sets of latent variables corresponding to objects. The encoder includes a pre-trained image segmentation network, which allows our model to focus only on representation learning while adopting image segmentation as an inductive bias. Evaluations using real-world image datasets, CelebA and Stanford Cars, showed that our method achieves improved disentanglement and transferability.

Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 84971, 84981, 2021
Scientific journal, A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only 'tweets and corresponding multiple previous events' but also 'events and corresponding multiple following posted tweets', the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.

Self-Supervised Learning for Gastritis Detection with Gastric X-Ray Images.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
CoRR, abs/2104.02864, 2021
Scientific journal

Soft-Label Anonymous Gastric X-ray Image Distillation.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
CoRR, abs/2104.02857, 2021
Scientific journal

Human Emotion Estimation Using Multi-Modal Variational AutoEncoder with Time Changes.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
LifeTech 2021 - 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, 67, 68, IEEE, 2021
International conference proceedings, A human emotion estimation method via feature integration using multi-modal variational autoencoder (MVAE) with time changes is presented in this paper. To utilize multimodal information such as gaze and brain activity data including some noises, the proposed method newly introduces MVAE into the human emotion estimation. Furthermore, the proposed MVAE can consider the changes in bio-signals with time and reduce the effect of noises caused in bio-signals by using the probabilistic variation. Experimental results with that of some state-of-the-art methods indicate that the proposed method is effective.

Music Playlist Generation Based on Graph Exploration Using Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama
LifeTech 2021 - 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, 53, 54, IEEE, 2021
International conference proceedings, Spreading of music streaming platforms that use playlists to make recommendations, automatic playlist generation has been actively researched. Recently, it has been reported that playlists that have high diversity and smooth track transitions increase user satisfaction. Our previous method that used a two-dimensional space as a reinforcement learning environment has achieved these demands, but there remains the problem that the content of multi-dimensional acoustic features cannot be retained accurately. To solve this problem, in this paper, we present a new method of music playlist generation based on reinforcement learning using a graph structure constructed from multi-dimensional acoustic features directly. The new playlist generation provides greater diversity and smoother track transitions than the previous method. Experimental results are shown for verifying the effectiveness of the proposal method.

Question Answering from Brain Activity Data via Decoder Based on Neural Networks.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
LifeTech 2021 - 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, 51, 52, IEEE, 2021
International conference proceedings, We build a model that can estimate what subjects recognize from functional magnetic resonance imaging (fMRI) data via a visual question answering (VQA) model. The VQA model can generate an answer to a question about an image. We convert fMRI signals into image features via an fMRI decoder based on the relationship between the fMRI signals and the image features extracted from the gazed image. Then this allows the VQA model to answer a visual question from the fMRI signals measured while the subject is gazing at the image. Though brain decoding, which interprets what humans recognize, has become overwhelmingly popular in neuroscience, they often suffer from the small datasets of brain activity data. To overcome the small size of datasets of fMRI signals, we introduce an fMRI decoder based on neural networks that have a high expressive ability. Even when we do not have enough fMRI signals, the proposed method derives the answer to what a person is looking at from fMRI signals. Experimental results on several datasets show that our method allows us to answer a question about gazed images from fMRI signals.

Distress Image Retrieval for Infrastructure Maintenance via Self-Trained Deep Metric Learning Using Experts' Knowledge.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 65234, 65245, 2021
Scientific journal, Distress image retrieval for infrastructure maintenance via self-trained deep metric learning using experts' knowledge is proposed in this paper. Since engineers take multiple images of a single distress part for inspection of road structures, it is necessary to construct a similar distress image retrieval method considering the input of multiple images to support determination of the level of deterioration. Thus, the construction of an image retrieval method while selecting an effective input from multiple images is described in this paper. The proposed method performs deep metric learning by using a small number of effective images labeled by experts' knowledge with information about their effectiveness and a large number of unlabeled images via a self-training approach. Specifically, an end-to-end learning approach that performs retraining of the model by assigning pseudo-labels to these unlabeled images according to the output confidence of the model is achieved. Thus, the proposed method can select an effective image from multiple images that are input at the retrieval as a query image. This is the main contribution of this paper. As a result, the proposed method realizes highly accurate retrieval of similar distress images considering the actual situation of inspection in which multiple images of a distress part are input.

Text-Guided Style Transfer-Based Image Manipulation Using Multimodal Generative Models.
Ren Togo, Megumi Kotera, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 64860, 64870, 2021
Scientific journal, A new style transfer-based image manipulation framework combining generative networks and style transfer networks is presented in this paper. Unlike conventional style transfer tasks, we tackle a new task, text-guided image manipulation. We realize style transfer-based image manipulation that does not require any reference style images and generate a style image from the user's input sentence. In our method, since an initial reference input sentence for a content image can automatically be given by an image-to-text model, the user only needs to update the reference sentence. This scheme can help users when they do not have any images representing the desired style. Although this text-guided image manipulation is a new challenging task, quantitative and qualitative comparisons showed the superiority of our method.

Rubber Material Property Prediction Using Electron Microscope Images of Internal Structures Taken under Multiple Conditions.
Ren Togo, Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 21, 6, 2088, 2088, 2021
Scientific journal, A method for prediction of properties of rubber materials utilizing electron microscope images of internal structures taken under multiple conditions is presented in this paper. Electron microscope images of rubber materials are taken under several conditions, and effective conditions for the prediction of properties are different for each rubber material. Novel approaches for the selection and integration of reliable prediction results are used in the proposed method. The proposed method enables selection of reliable results based on prediction intervals that can be derived by the predictors that are each constructed from electron microscope images taken under each condition. By monitoring the relationship between prediction results and prediction intervals derived from the corresponding predictors, it can be determined whether the target prediction results are reliable. Furthermore, the proposed method integrates the selected reliable results based on Dempster–Shafer (DS) evidence theory, and this integration result is regarded as a final prediction result. The DS evidence theory enables integration of multiple prediction results, even if the results are obtained from different imaging conditions. This means that integration can even be realized if electron microscope images of each material are taken under different conditions and even if these conditions are different for target materials. This nonconventional approach is suitable for our application, i.e., property prediction. Experiments on rubber material data showed that the evaluation index mean absolute percent error (MAPE) was under 10% by the proposed method. The performance of the proposed method outperformed conventional comparative property estimation methods. Consequently, the proposed method can realize accurate prediction of the properties with consideration of the characteristic of electron microscope images described above.

Detection of Important Scenes in Baseball Videos via a Time-Lag-Aware Multimodal Variational Autoencoder.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 21, 6, 2045, 2045, 2021
Scientific journal, A new method for the detection of important scenes in baseball videos via a time-lag-aware multimodal variational autoencoder (Tl-MVAE) is presented in this paper. Tl-MVAE estimates latent features calculated from tweet, video, and audio features extracted from tweets and videos. Then, important scenes are detected by estimating the probability of the scene being important from estimated latent features. It should be noted that there exist time-lags between tweets posted by users and videos. To consider the time-lags between tweet features and other features calculated from corresponding multiple previous events, the feature transformation based on feature correlation considering such time-lags is newly introduced to the encoder in MVAE in the proposed method. This is the biggest contribution of the Tl-MVAE. Experimental results obtained from actual baseball videos and their corresponding tweets show the effectiveness of the proposed method.

Perceived Image Decoding From Brain Activity Using Shared Information of Multi-Subject fMRI Data.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 26593, 26606, 2021
Scientific journal, Decoding a person's cognitive contents from evoked brain activity is becoming important in the field of brain-computer interaction. Previous studies have decoded a perceived image from functional magnetic resonance imaging (fMRI) activity by constructing brain decoding models that were trained with a single subject's fMRI data. However, accurate decoding is still challenging since fMRI data acquired from only a single subject have several disadvantageous characteristics such as small sample size, noisy nature, and high dimensionality. In this article, we propose a method to decode categories of perceived images from fMRI activity using shared information of multi-subject fMRI data. Specifically, by aggregating fMRI data of multiple subjects that contain a large number of samples, we extract a low-dimensional latent representation shared by multi-subject fMRI data. Then the latent representation is nonlinearly transformed into visual features and semantic features of the perceived images to identify categories from various candidate categories. Our approach leverages rich information obtained from multi-subject fMRI data and improves the decoding performance. Experimental results obtained by using two public fMRI datasets showed that the proposed method can more accurately decode categories of perceived images from fMRI activity than previous approaches using a single subject's fMRI data.

Supervised Fractional-Order Embedding Multiview Canonical Correlation Analysis via Ordinal Label Dequantization for Image Interest Estimation.
Masanao Matsumoto, Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 21810, 21822, 2021
Scientific journal, Supervised fractional-order embedding multiview canonical correlation analysis via ordinal label dequantization (SFEMCCA-OLD) for image interest estimation is presented in this paper. SFEMCCA-OLD is a CCA method that realizes accurate integration of features including low-dimensional ordinal label features. In general, since information is lost due to a limitation of the number of classes, i.e., the dimension of ordinal label information is smaller than those of other features, derivation of highly accurate integration of features is difficult. In SFEMCCA-OLD, the dimension of the ordinal label information can be increased by estimation of the canonical correlation between multiview features. We call this approach ordinal label dequantization. In addition, by introducing a fractional-order technique, our method can calculate optimal projections for noisy data such as real data. Experimental results show that the accuracy of SFEMCCA-OLD for image interest estimation is better than that of recent CCA-based methods.

ESTIMATION OF DRILLING ENERGY BASED ON ONLINE LEARNING FROM TUNNEL CUTTING FACE IMAGE
山本健太郎, 藤後廉, 小川貴弘, 長谷山美紀
土木学会論文集 F3(土木情報学)(Web), 77, 1, 794, 795, 2021, [Peer-reviewed]
English, International conference proceedings

Land Use in Habitats Affects Metal Concentrations in Wild Lizards around a Former Lead Mining Site
Rio Doya, Shouta M.M. Nakayama, Hokuto Nakata, Haruya Toyomaki, John Yabe, Kaampwe Muzandu, Yared B. Yohannes, Andrew Kataba, Golden Zyambo, Takahiro Ogawa, Yoshitaka Uchida, Yoshinori Ikenaka, Mayumi Ishizuka
Environmental Science and Technology, 54, 22, 14474, 14481, 17 Nov. 2020
Scientific journal, We investigated the potential effects of different land use and other environmental factors on animals living in a contaminated environment. The study site in Kabwe, Zambia, is currently undergoing urban expansion, while lead contamination from former mining activities is still prevalent. We focused on a habitat generalist lizards (Trachylepis wahlbergii). The livers, lungs, blood, and stomach contents of 224 lizards were analyzed for their lead, zinc, cadmium, copper, nickel, and arsenic concentrations. Habitat types were categorized based on vegetation data obtained from satellite images. Multiple regression analysis revealed that land use categories of habitats and three other factors significantly affected lead concentrations in the lizards. Further investigation suggested that the lead concentrations in lizards living in bare fields were higher than expected based on the distance from the contaminant source, while those in lizards living in green fields were lower than expected. In addition, the lead concentration of lungs was higher than that of the liver in 19% of the lizards, implying direct exposure to lead via dust inhalation besides digestive exposure. Since vegetation reduces the production of dust from surface soil, it is plausible that dust from the mine is one of the contamination sources and that vegetation can reduce exposure to this.

Important Scene Detection Based on Anomaly Detection using Long Short-Term Memory for Baseball Highlight Generation
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020, 28 Sep. 2020
International conference proceedings, This paper presents an important scene detection method based on anomaly detection using a Long Short-Term Memory (LSTM) for baseball highlight generation. In order to deal with multi-view time series features calculated from tweets and videos, we adopt an anomaly detection method using LSTM. LSTM which can maintain a long-term memory is effective for training such features. Introduction of LSTM into important scene detection of baseball videos is the biggest contribution of this paper. Experimental results show high detection performance by our method.

An Estimation Method of Candidate Region for Superimposing Information Based on Gaze Tracking Data in Soccer Videos
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2020 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020, 28 Sep. 2020
International conference proceedings, A novel method estimating candidate regions for superimposing information in soccer videos based on gaze tracking data is presented in this paper. The proposed method generates a likelihood map based on visual attention regions based on the gaze tracking data and detection results of objects such as players and soccer goals in soccer videos. Candidate regions for superimposing information are estimated by using the likelihood map. Experimental results show that the proposed method realizes effective candidate region estimation.

Estimation of Person-Specific Visual Attention via Selection of Similar Persons
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-Taiwan 2020, 28 Sep. 2020
International conference proceedings, This paper presents a method for estimation of person-specific visual attention based on estimated similar persons' visual attention. For improving the estimation performance of person-specific visual attention, the proposed method uses the dataset including the large number of images and corresponding gaze data of many persons not including the target person and trains an estimation model based on deep learning. By using the estimated visual attention of similar persons for the target image, the proposed method estimates the visual attention of the target person with the small amount of gaze data. Experimental results show that the proposed method is effective for estimation of person-specific visual attention.

Chronic atrophic gastritis detection with a convolutional neural network considering stomach regions
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
World Journal of Gastroenterology, 26, 25, 3650, 3659, 07 Jul. 2020
Scientific journal, BACKGROUND The risk of gastric cancer increases in patients with Helicobacter pylori-associated chronic atrophic gastritis (CAG). X-ray examination can evaluate the condition of the stomach, and it can be used for gastric cancer mass screening. However, skilled doctors for interpretation of X-ray examination are decreasing due to the diverse of inspections. AIM To evaluate the effectiveness of stomach regions that are automatically estimated by a deep learning-based model for CAG detection. METHODS We used 815 gastric X-ray images (GXIs) obtained from 815 subjects. The ground truth of this study was the diagnostic results in X-ray and endoscopic examinations. For a part of GXIs for training, the stomach regions are manually annotated. A model for automatic estimation of the stomach regions is trained with the GXIs. For the rest of them, the stomach regions are automatically estimated. Finally, a model for automatic CAG detection is trained with all GXIs for training. RESULTS In the case that the stomach regions were manually annotated for only 10 GXIs and 30 GXIs, the harmonic mean of sensitivity and specificity of CAG detection were 0.955 ± 0.002 and 0.963 ± 0.004, respectively. CONCLUSION By estimating stomach regions automatically, our method contributes to the reduction of the workload of manual annotation and the accurate detection of the CAG.

胃X線画像を用いたAIによるH.pylori感染識別と今後の展望　　　　　　　　　　　　　　　
藤後 廉, 小川 貴弘, 間部 克裕, 加藤 元嗣, 長谷山 美紀
日本消化器がん検診学会雑誌, 58, 2, 127, 127, (一社)日本消化器がん検診学会, Mar. 2020
Japanese

Image retrieval based on supervised local regression and global alignment with relevance feedback for insect identification
Keisuke Maeda, Susumu Genma, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 8, 3, 140, 150, 2020
Scientific journal, A method for image retrieval based on supervised local regression and global alignment (sLRGA) with relevance feedback for insect identification is presented in this paper. Based on the novel sLRGA, which is an extended version of LRGA, the proposed method estimates ranking scores for image retrieval in such a way that the neighborhood structure of a feature space of the database can be optimally preserved with consideration of class information. This is the main contribution of this paper. By measuring the relevance between all of the images and the query image in the database, sLRGA realizes accurate image retrieval. Furthermore, when positive/negative labels to retrieved images are given by users, the proposed method can improve image retrieval performance considering the query relevance information via use of both relevance feedback and sLRGA. This is the second contribution of this paper. Experimental results show the effectiveness of the proposed method.

A method for player importance prediction from player network using gaze position estimated by LSTM
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 8, 3, 151, 160, 2020
Scientific journal, A novel method for player importance prediction from a player network using gaze positions estimated by Long Short-Term Memory (LSTM) in soccer videos is presented in this paper. By newly using an estimation model of gaze positions trained by gaze tracking data of experienced persons, it is expected that the importance of each player can be predicted. First, we generate a player network by utilizing the estimated gaze positions and first-arrival regions representing players' connections, e.g., passes between players. The gaze positions are estimated by LSTM that is newly trained from the gaze tracking data of experienced persons. Second, the proposed method predicts the importance of each player by applying the Hypertext Induced Topic Selection (HITS) algorithm to the constructed network. Consequently, prediction of the importance of each player based on soccer tactic knowledge of experienced persons can be realized without constantly obtaining gaze tracking data.

Interpretable convolutional neural network including attribute estimation for image classification
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 8, 2, 111, 124, 2020
Scientific journal, An interpretable convolutional neural network (CNN) including attribute estimation for image classification is presented in this paper. Although CNNs perform highly accurate image classification, the reason for the classification results obtained by the neural networks is not clear. In order to provide interpretation of CNNs, the proposed method estimates attributes, which explain elements of objects, in an intermediate layer of the network. This enables improvement of the interpretability of CNNs, and it is the main contribution of this paper. Furthermore, the proposed method uses the estimated attributes for image classification in order to enhance its accuracy. Consequently, the proposed method not only provides interpretation of CNNs but also realizes improvement in the performance of image classification.

Multimodal important scene detection in far-view soccer videos based on single deep neural architecture
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 8, 2, 89, 99, 2020
Scientific journal, The details of the matches of soccer can be estimated from visual and audio sequences, and they correspond to the occurrence of important scenes. Therefore, the use of these sequences is suitable for important scene detection. In this paper, a new multimodal method for important scene detection from visual and audio sequences in far-view soccer videos based on a single deep neural architecture is presented. A unique point of our method is that multiple classifiers can be realized by a single deep neural architecture that includes a Convolutional Neural Network-based feature extractor and a Support Vector Machine-based classifier. This approach provides a solution to the problem of not being able to simultaneously optimize different multiple deep neural architectures from a small amount of training data. Then we monitor confidence measures output from this architecture for the multimodal data and enable their integration to obtain the final classification result.

Image Retrieval with Data Augmentation of Sentence Labels Based on Paraphrasing
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - TAIWAN (ICCE-TAIWAN), IEEE, 2020
English, International conference proceedings, Text-based image retrieval is a fundamental study in the field of information retrieval. Recent text-based image retrieval methods employ deep neural networks (here-inafter referred to as deep neural TBIR) to retrieve a desired image from a sentence query and achieve the state-of-the-art performance in TBIR. To improve the retrieval performance of the deep neural TBIR method further, it is essential to prepare diverse sentence labels in training data. However, it takes a lot of effort to prepare diverse sentence labels in training data. To address this problem, we propose a novel deep neural TBIR method with data augmentation of the sentence labels in training data. Experimental results show the effectiveness of the proposed method.

Interactive re-ranking for cross-modal retrieval based on object-wise question answering.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings of the 2nd ACM International Conference on Multimedia in Asia, MMAsia 2020, 37, 7, ACM, 2020
International conference proceedings, Cross-modal retrieval methods retrieve desired images from a query text by learning relationships between texts and images. This retrieval approach is one of the most effective ways in the easiness of query preparation. Recent cross-modal retrieval is convenient and accurate when users input a query text that can uniquely identify the desired image. Meanwhile, users frequently input ambiguous query texts, and these ambiguous queries make it difficult to obtain the desired images. To alleviate these difficulties, in this paper, we propose a novel interactive cross-modal retrieval method based on question answering (QA) with users. The proposed method analyses candidate images and asks users about information that can narrow retrieval candidates effectively. By only answering the questions generated by the proposed method, users can reach their desired images even from an ambiguous query text. Experimental results show the effectiveness of the proposed method.

Similar scene retrieval in soccer videos with weak annotations by multimodal use of bidirectional LSTM.
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Proceedings of the 2nd ACM International Conference on Multimedia in Asia, MMAsia 2020, 27, 8, ACM, 2020
International conference proceedings, This paper presents a novel method to retrieve similar scenes in soccer videos with weak annotations via multimodal use of bidirectional long short-term memory (BiLSTM). The significant increase in the number of different types of soccer videos with the development of technology brings valid assets for effective coaching, but it also increases the work of players and training staff. We tackle this problem with a nontraditional combination of pre-trained models for feature extraction and BiLSTMs for feature transformation. By using the pre-trained models, no training data is required for feature extraction. Then effective feature transformation for similarity calculation is performed by applying BiLSTM trained with weak annotations. This transformation allows for highly accurate capture of soccer video context from less annotation work. In this paper, we achieve an accurate retrieval of similar scenes by multimodal use of this BiLSTM-based transformer trainable with less human effort. The effectiveness of our method was verified by comparative experiments with state-of-the-art using actual soccer video dataset.

胃X線画像を用いたAIによるH.pylori感染識別と今後の展望
藤後廉, 小川貴弘, 間部克裕, 加藤元嗣, 長谷山美紀
日本消化器がん検診学会雑誌(Web), 58, 2, 2020

SIMILAR INSPECTION DATA RETRIEVAL FOR ROAD STRUCTURE INSPECTION BASED ON CANONICAL CORRELATION BETWEEN EYE TRACKING DATA AND INSPECTION RECORDS
前田圭介, 斉藤僚汰, 高橋翔, 小川貴弘, 長谷山美紀
土木学会論文集 F3(土木情報学)(Web), 76, 1, 2020

Quantitative Analysis of Engineer's Skill Using Wearable Sensor Data while Inspecting Highway Bridge.
Genki Suzuki, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies, 111, 112, IEEE, 2020
International conference proceedings, This paper presents quantitative analysis of engineer's skill using wearable sensor data while inspecting a highway bridge. This paper analyzes differences of behavior between experienced and inexperienced engineers by using bio-signals obtained from wearable sensor data during inspections. Specifically, by analyzing each engineer's movement distance and the number of steps during inspections, the differences of behavior patterns between experienced and inexperienced engineers can be revealed.

Estimating Viewed Images with Natural Language Question Answering from fMRI Data.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies, 99, 100, IEEE, 2020
International conference proceedings, We propose a model for free-form visual question answering (VQA) from human brain activity. The task of VQA is leading to an answer given an image and a question about the image. Given brain activity data measured by functional magnetic resonance imaging (fMRI) and a natural language question in terms of the viewed image, the proposed method can provide an accurate natural language answer with the VQA algorithm. Visual questions selectively approach various areas of an image such as objects and backgrounds. As a result, a more detailed understanding of the image and complex reasoning are typically needed than general image captioning models. In this paper, we propose a method of answering a given question about a viewed image from fMRI data based on the VQA algorithm. We estimate the relation between fMRI data and visual features extracted from viewed images. Based on the relationship, we convert fMRI data into visual features. Finally, the proposed method can answer to a visual question from fMRI data measured while subjects are viewing images. Experimental results show that the proposed method enables accurate answering for questions about viewed images.

Distress Level Classification of Road Infrastructures via CNN Generating Attention Map.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies, 97, 98, IEEE, 2020
International conference proceedings, This paper proposes distress level classification of road infrastructures via convolutional neural networks (CNNs) generating an attention map. In the proposed method, performance of distress level classification is improved by utilizing an attention map to emphasize locations contributing to classification results. Furthermore, the attention map is also utilized to explain the reasons for the classification results. Thus, the proposed method realizes both performance improvement of distress level classification and provision of the interpretability for the classification results. Finally, experimental results verify the effectiveness of the proposed method.

Interest Estimation for Images Using Eye Gaze-based Visual and Text Features via DLPCCA.
Masanao Matsumoto, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
LifeTech 2020 - 2020 IEEE 2nd Global Conference on Life Sciences and Technologies, 3, 4, IEEE, 2020
International conference proceedings, This paper presents a new method to realize estimation performance improvement of user-specific interests for images. The proposed method computes projections which transform visual and text features to eye gaze-based features that reflect user's interests by utilizing discriminative locality preserving canonical correlation analysis (DLPCCA). DLPCCA can calculate projections suitable for interest estimation by considering the class information and locality of data structure. Experimental results are shown for verifying the effectiveness of our method.

Mvgan Maximizing Time-Lag Aware Canonical Correlation for Baseball Highlight Generation.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2020, 1, 6, IEEE, 2020
International conference proceedings, This paper presents multi-view unsupervised generative adversarial network maximizing time-lag aware canonical correlation (Mv-GAN) for baseball highlight generation. MvGAN has the following two contributions. First, MvGAN utilizes textual, visual and audio features calculated from tweets and videos as multi-view features. MvGAN which adopts these multi-view features is the effective work for highlight generation of baseball videos. Second, since there is a temporal difference between posted tweets and the corresponding events, MvGAN introduces a novel feature embedding scheme considering a time-lag between textual features and other features. Specifically, the proposed method newly derives the time-lag aware canonical correlation maximization of these multi-view features. This is the biggest contribution of this paper. Furthermore, since MvGAN is an unsupervised method for highlight generation, a large amount of training data with annotation is not needed. Thus, the proposed method has high applicability to the real world.

Generation of Viewed Image Captions From Human Brain Activity Via Unsupervised Text Latent Space.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 2521, 2525, IEEE, 2020
International conference proceedings, Generation of human cognitive contents based on the analysis of functional magnetic resonance imaging (fMRI) data has been actively researched. Cognitive contents such as viewed images can be estimated by analyzing the relationship between fMRI data and semantic information of viewed images. In this paper, we propose a new method generating captions for viewed images from human brain activity via a novel robust regression scheme. Unlike conventional generation methods using image feature representations, the proposed method makes use of more semantic text feature representations, which are more suitable for the caption generation. We construct a text latent space with unlabeled images not used for the training, and the fMRI data are regressed to the text latent space. Besides, we newly make use of unlabeled images not used for the training phase to improve caption generation performance. Finally, the proposed method can generate captions from the fMRI data measured while subjects are viewing images. Experimental results show that the proposed method enables accurate caption generation for viewed images.

Multimodal Image-to-Image Translation for Generation of Gastritis Images.
Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 2466, 2470, IEEE, 2020
International conference proceedings, We present a new multimodal image-to-image translation model for the generation of gastritis images using X-ray and blood inspection results. In clinical situations, clinicians estimate the prognosis of the target disease by considering multiple inspection results. Similarly, we take a multimodal approach in the task of gastric cancer risk prediction. Visual characteristics of the gastric X-ray image and blood index values are highly related in the evaluation of gastric cancer risk. If we can generate a prediction image from blood index values, it contributes to the clinicians' sophisticated and integrated diagnosis. Hence, we learn a model that can map non-gastritis images to gastritis images based on the blood index values. Although this is a challenging multimodal task in medical image analysis, experimental results showed the effectiveness of our model.

Image Retrieval With Lingual And Visual Paraphrasing Via Generative Models.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2020-October, 2431, 2435, IEEE, 2020
English, International conference proceedings, A new approach that improves text-based image retrieval (hereinafter referred to as TBIR) performance is proposed in this paper. TBIR methods aim to retrieve a desired image related to a query text. Especially, recent TBIR methods allow us to retrieve images considering word relationships by using a sentence as a query. In these TBIR methods, it is necessary to uniquely identify a desired image from similar images using a single query sentence. However, the diverse expressive styles for a query sentence make it difficult to uniquely identify a desired image. In this paper, we propose a novel TBIR method with paraphrasing on multiple representation spaces. Specifically, by paraphrasing a query sentence on lingual and visual representation spaces, the proposed method can retrieve a desired image from various perspectives and then it can uniquely identify a desired image from similar images. Comprehensive experimental results show the effectiveness of the proposed method.

Variational Autoencoder Based Unsupervised Domain Adaptation For Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 2426, 2430, IEEE, 2020
International conference proceedings, Unsupervised domain adaptation, which transfers supervised knowledge from a labeled domain to an unlabeled domain, remains a tough problem in the field of computer vision, especially for semantic segmentation. Some methods inspired by adversarial learning and semi-supervised learning have been developed for unsupervised domain adaptation in semantic segmentation and achieved outstanding performances. In this paper, we propose a novel method for this task. Like adversarial learning-based methods using a discriminator to align the feature distributions from different domains, we employ a variational autoencoder to get to the same destination but in a non-adversarial manner. Since the two approaches are compatible, we also integrate an adversarial loss into our method. By further introducing pseudo labels, our method can achieve state-of-the-art performances on two benchmark adaptation scenarios, GTA5-toCITYSCAPES and SYNTHIA-to-CITYSCAPES.

Important Scene Detection Of Baseball Videos Via Time-Lag Aware Deep Multiset Canonical Correlation Maximization.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 1236, 1240, IEEE, 2020
International conference proceedings, This paper presents a new important scene detection method of baseball videos based on correlation maximization between heterogeneous modalities via time-lag aware deep multiset canonical correlation analysis (Tl-dMCCA). The technical contributions of this paper are twofold. First, textual, visual and audio features calculated from tweets and videos are adopted as multi-view time series features. Since Tl-dMCCA which utilizes these features includes the unsupervised embedding scheme via deep networks, the proposed method can flexibly express the relationship between heterogeneous features. Second, since there is the time-lag between posted tweets and the corresponding multiple previous events, Tl-dMCCA considers the time-lag relationships between them. Specifically, we newly introduce the representation of such time-lags into the derivation of their covariance matrices. By considering time-lags via Tl-dMCCA, the proposed method correctly detects important scenes.

Soft-Label Anonymous Gastric X-Ray Image Distillation.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 305, 309, IEEE, 2020
International conference proceedings, This paper presents a soft-label anonymous gastric X-ray image distillation method based on a gradient descent approach. The sharing of medical data is demanded to construct high-accuracy computeraided diagnosis (CAD) systems. However, the large size of the medical dataset and privacy protection are remaining problems in medical data sharing, which hindered the research of CAD systems. The idea of our distillation method is to extract the valid information of the medical dataset and generate a tiny distilled dataset that has a different data distribution. Different from model distillation, our method aims to find the optimal distilled images, distilled labels and the optimized learning rate. Experimental results show that the proposed method can not only effectively compress the medical dataset but also anonymize medical images to protect the patient's private information. The proposed approach can improve the efficiency and security of medical data sharing.

Estimation Of Visual Contents Based On Question Answering From Human Brain Activity.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 61, 65, IEEE, 2020
International conference proceedings, We propose an estimation method for free-form Visual Question Answering (VQA) from human brain activity, brain decoding VQA. The task of VQA in the field of computer vision is generating an answer given an image and a question about its contents. The proposed method can realize answering arbitrary visual questions about images from brain activity measured by functional Magnetic Resonance Imaging (fMRI) while viewing the same images. We enable estimating various information from brain activity via a unique VQA model, which can realize a more detailed understanding of images and complex reasoning. In addition, we newly make use of un-labeled images not used in the training phase to improve the performance of the transformation, since fMRI datasets are generally small. The proposed method can answer a visual question from a little amount of fMRI data measured while subjects are viewing images.

Feature Integration Via Geometrical Supervised Multi-View Multi-Label Canonical Correlation For Incomplete Label Assignment.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2020-October, 46, 50, IEEE, 2020
International conference proceedings, This paper presents feature integration via geometrical supervised multi-view multi-label canonical correlation analysis (GSM2CCA) for incomplete label assignment. The problem of incomplete labels is frequently encountered in the multi-label classification problem where the training labels are obtained via crowd-sourcing. In such a situation, consideration of only the label correlation, which is the basic approach, is not suitable for improvement of representation ability of features. For dealing with the incomplete label assignment, GSM2CCA constructs effective feature embedding space providing the discriminant ability by introducing both the multi-label correlation and feature similarity of the original feature space into its objective function. Since novel integrated features with high discriminant ability can be calculated by our GSM2CCA, performance improvement of multi-label classification with the incomplete label assignment is realized. The main contribution of this paper is the realization of the effective feature integration via the adoption of the combination use of label similarity and locality preserving projection of heterogeneous features for solving the problem of the incomplete label assignment. The effectiveness of GSM2CCA by applying GSM2CCA-based feature integration to heterogeneous features calculated from various convolutional neural network models is verified via experimental results.

Unsupervised Domain Adaptation for Semantic Segmentation with Symmetric Adaptation Consistency.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2020-May, 2263, 2267, IEEE, 2020
International conference proceedings, Unsupervised domain adaptation, which leverages label information from other domains to solve tasks on a domain without any labels, can alleviate the problem of the scarcity of labels and expensive labeling costs faced by supervised semantic segmentation. In this paper, we utilize adversarial learning and semi-supervised learning simultaneously to solve the task of unsupervised domain adaptation in semantic segmentation. We propose a new approach that trains two segmentation models with the adversarial learning symmetrically and further introduces the consistency between the outputs of the two models into the semi-supervised learning to improve the accuracy of pseudo labels which significantly affect the final adaptation performance. We achieve state-of-the-art semantic segmentation performance on the GTA5-to-Cityscapes scenario, a widely used benchmark setting in unsupervised domain adaptation.

Multi-View Bayesian Generative Model for Multi-Subject FMRI Data on Brain Decoding of Viewed Image Categories.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2020-May, 1215, 1219, IEEE, 2020
International conference proceedings, Brain decoding studies have demonstrated that viewed image categories can be estimated from human functional magnetic resonance imaging (fMRI) activity. However, there are still limitations with the estimation performance because of the characteristics of fMRI data and the employment of only one modality extracted from viewed images. In this paper, we propose a multi-view Bayesian generative model for multi-subject fMRI data to estimate viewed image categories from fMRI activity. The proposed method derives effective representations of fMRI activity by utilizing multi-subject fMRI data. In addition, we associate fMRI activity with multiple modalities, i.e., visual features and semantic features extracted from viewed images. Experimental results show that the proposed method outperforms existing state-of-the-art methods of brain decoding.

Interest Level Estimation Based on Feature Integration Considering Distribution of Partially Paired User's Behavior, Videos and Posters.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 944, 945, IEEE, 2020
International conference proceedings, This paper presents a method for interest level estimation based on feature integration considering distribution of partially paired user's behavior, videos and posters. The proposed method collaboratively uses videos, their corresponding poster images, and user's behavior to estimate the interest levels. For dealing with the multi-view data, the proposed method newly derives semi-supervised divergence-aware multi-set canonical correlation analysis (SDMCCA). SDMCCA has two contributions. First, since the number of viewable videos is limited, consideration of unviewed videos is necessary, and SDMCCA adopts the semi-supervised approach. Second, information which data have is different due to the difference between similarities to other data. SDMCCA can simultaneously realize the above points and realize successful interest level estimation. Experimental results show the effectiveness of the proposed method.

Music Playlist Generation Based on Reinforcement Learning Using Acoustic Feature Map.
Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 942, 943, IEEE, 2020
International conference proceedings, With the spread of streaming services that make playlist-based recommendation, automatic music playlist generation has been actively researched. Traditionally, most of music playlist generation methods have focused on the sounds and the genres of songs that have been listened by a user. However, users are bored and unsatisfied with the music playlist that fits users' preferences overly. In this paper, we propose a new music playlist generation method based on reinforcement learning using an acoustic feature map. Our reinforcement learning-based music playlist generation can overlook the whole songs since an agent explores all of the features on the map. This new playlist generation can achieve high diversity and smooth track transitions. Experimental results show the effectiveness of the proposed method.

Estimation of Images Matched with Audio-Induced Brain Activity via Modified DGCCA.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 940, 941, IEEE, 2020
International conference proceedings, This paper presents a method that can match images with audio-induced brain activity via modified deep generalized canonical correlation analysis (modified DGCCA). Modified DGCCA that contains the Pearson correlation coefficient in its loss function is newly derived for embedding not only images and audio but also audio-induced brain activity calculated from functional near-infrared spectroscopy (fNIRS) into the same feature space. By inputting audio and its corresponding brain activity into modified DGCCA, the estimation of their best matched images becomes feasible. The experiment result shows the effectiveness of the proposed method. The main contribution of this paper is the introduction of a new method for matching images with audio-induced brain activity.

Estimation of User-Specific Visual Attention Considering Individual Tendency toward Gazed Objects.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 745, 746, IEEE, 2020
International conference proceedings, An estimation method of user-specific visual attention considering individual tendency toward gazed objects is presented in this paper. For realization the user-specific visual attention for images, it can be effective to use the past gaze tendency of a target user and gaze data obtained from other users. However, the collaborative use of these information is difficult since there may be no gaze data obtained from other users. Then the proposed method focuses on the saliency map and enables the collaborative use of those information for estimation of user-specific visual attention. It is confirmed that the estimation accuracy of the proposed method improved by approximately 20% compared to the state-of-the-art saliency estimation method by performing the experiment.

Estimation of Viewed Images Using Individual and Shared Brain Responses.
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 716, 717, IEEE, 2020
International conference proceedings, This paper presents a method to estimate viewed images by using brain responses measured when persons view images. The proposed method predicts visual features of viewed images from not only individual brain responses but also shared brain responses of multiple subjects. Specifically, the proposed method simultaneously estimates the following two projection matrices: a matrix that converts individual brain responses into visual features and a matrix that converts shared brain responses into visual features. The collaborative use of two types of brain responses is the biggest contribution of this paper. Experimental results show that the proposed method improves the estimation accuracy in comparison with methods considering only individual or shared brain responses.

Cross-domain Recommendation via Multi-layer Graph Analysis Using User-item Embedding.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 714, 715, IEEE, 2020
International conference proceedings, This paper presents cross-domain recommendation via multi-layer graph analysis using user-item embedding. The proposed method constructs two graphs in source and target domains utilizing user-item embedding, respectively. By training relationship between the user's embedding features in these two graphs, the proposed method realizes the cross-domain recommendation via the multi-layer graphs. This is the main contribution of this paper. Then the proposed method can estimate the user's embedding in the target domain from that in the source domain. Finally, our method can recommend items to users via the estimated user's embedding. Experiments on real-world datasets verify the effectiveness of the proposed method.

Question Answering for Estimation of Seen Image Contents from Multi-subject fMRI Responses.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 712, 713, IEEE, 2020
International conference proceedings, Estimating seen image contents based on the analysis of functional magnetic resonance imaging (fMRI) data has been actively researched. So far, it has been necessary to train a model with individual fMRI data and construct models for each task. In this paper, we propose an estimation method via Visual Question Answering (VQA) from fMRI data from multi-subjects. The task of VQA in the field of computer vision is generating answers when given an image and questions about its contents. The proposed method enables generation accurate answers via the VQA model when given fMRI data and questions about the seen images. Besides, we newly introduce training with multi-subject fMRI data since fMRI datasets are generally small due to the burden for subjects. Then we can realize the estimation of the contents of seen images from fMRI data of a subject whose data are not used in the training phase.

Face Synthesis via User Manipulation of Disentangled Latent Representation.
Nao Nakagawa, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 692, 693, IEEE, 2020
International conference proceedings, In this paper, we propose a novel face synthesis method whose process can be conditioned not only on labels but also on latent variables as 'unknown' labels corresponding to unrevealed factors. We extend Variational Autoencoder (VAE) without any additional networks to introduce conditional generation, disentangled representation, and adversarial learning into one autoencoder. Since previous conditional generative models require the annotation of labels to condition them on, disentanglement, i.e., the unsupervised discovery of generative factors enables users to generate face images more flexibly and more efficiently. Moreover, although generative adversarial networks (GANs) have problems of mode collapse and instability of the learning process, adversarial learning on VAE in an introspective way achieves both the variation of results and the stability of generation. Evaluations on the CelebFaces Attributes Dataset (CelebA) show that our method can generate face images following users' conditioning both on the known and the 'unknown' labels.

Complexity Evaluation of Medical Image Data for Classification Problem Based on Spectral Clustering.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 667, 669, IEEE, 2020
International conference proceedings, Deep convolutional neural networks (DCNNs) have been popular with medical image classification problems in recent years. However, training a DCNN model on the sizeable medical dataset requires repeated manipulation to achieve the desired results and hence is time-consuming. Since there is an inevitable link between DCNN training results and the complexity of the medical dataset, it is essential to accurately evaluate the medical dataset's complexity before training the DCNN models. In this paper, we propose an efficient method to assess the medical dataset's complexity based on spectral clustering. The experimental results show that the medical dataset complexity calculated with our approach is not time-consuming and has a high correlation with DCNN test accuracy.

Important Scene Prediction of Baseball Videos Using Twitter and Video Analysis Based on LSTM.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE 9th Global Conference on Consumer Electronics, GCCE 2020, 636, 637, IEEE, 2020
International conference proceedings, This paper presents a novel important scene prediction method using baseball videos and tweets on Twitter based on a Long Short-Term Memory (LSTM). For considering baseball videos and tweets, the proposed method utilizes textual, visual and audio features. Introducing these multi-modal features into the important scene prediction of baseball videos is the first work. In order to deal with multi-modal time-series features constructed from textual, visual and audio features, the proposed method adopts LSTM which is effective for training such multimodal time-series features by maintaining a long-term memory. The effectiveness of the proposed method is confirmed by experimental results.

Brain Decoding of Viewed Image Categories via Semi-Supervised Multi-View Bayesian Generative Model.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Trans. Signal Process., 68, 5769, 5781, 2020
Scientific journal, Brain decoding has shown that viewed image categories can be estimated from evoked functional magnetic resonance imaging (fMRI) activity. Recent studies attempted to estimate viewed image categories that were not used for training previously. Nevertheless, the estimation performance is limited since it is difficult to collect a large amount of fMRI data for training. This paper presents a method to accurately estimate viewed image categories not used for training via a semi-supervised multi-view Bayesian generative model. Our model focuses on the relationship between fMRI activity and multiple modalities, i.e., visual features extracted from viewed images and semantic features obtained from viewed image categories. Furthermore, in order to accurately estimate image categories not used for training, our semi-supervised framework incorporates visual and semantic features obtained from additional image categories in addition to image categories of training data. The estimation performance of the proposed model outperforms existing state-of-the-art models in the brain decoding field and achieves more than 95% identification accuracy. The results also have shown that the incorporation of additional image category information is remarkably effective when the number of training samples is small. Our semi-supervised framework is significant for the brain decoding field where brain activity patterns are insufficient but visual stimuli are sufficient.

Few-Shot Personalized Saliency Prediction Based on Adaptive Image Selection Considering Object and Visual Attention.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 20, 8, 2170, 2170, 2020
Scientific journal, A few-shot personalized saliency prediction based on adaptive image selection considering object and visual attention is presented in this paper. Since general methods predicting personalized saliency maps (PSMs) need a large number of training images, the establishment of a theory using a small number of training images is needed. To tackle this problem, although finding persons who have visual attention similar to that of a target person is effective, all persons have to commonly gaze at many images. Thus, it becomes difficult and unrealistic when considering their burden. On the other hand, this paper introduces a novel adaptive image selection (AIS) scheme that focuses on the relationship between human visual attention and objects in images. AIS focuses on both a diversity of objects in images and a variance of PSMs for the objects. Specifically, AIS selects images so that selected images have various kinds of objects to maintain their diversity. Moreover, AIS guarantees the high variance of PSMs for persons since it represents the regions that many persons commonly gaze at or do not gaze at. The proposed method enables selecting similar users from a small number of images by selecting images that have high diversities and variances. This is the technical contribution of this paper. Experimental results show the effectiveness of our personalized saliency prediction including the new image selection scheme.

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 20, 7, 2146, 2146, 2020
Scientific journal, The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

Chronic gastritis classification using gastric X-ray images with a semi-supervised learning method based on tri-training.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Medical Biol. Eng. Comput., 58, 6, 1239, 1250, 2020
Scientific journal, High-quality annotations for medical images are always costly and scarce. Many applications of deep learning in the field of medical image analysis face the problem of insufficient annotated data. In this paper, we present a semi-supervised learning method for chronic gastritis classification using gastric X-ray images. The proposed semi-supervised learning method based on tri-training can leverage unannotated data to boost the performance that is achieved with a small amount of annotated data. We utilize a novel learning method named Between-Class learning (BC learning) that can considerably enhance the performance of our semi-supervised learning method. As a result, our method can effectively learn from unannotated data and achieve high diagnostic accuracy for chronic gastritis. [Figure not available: see fulltext.].

Multi-Task Convolutional Neural Network Leading to High Performance and Interpretability via Attribute Estimation.
Keisuke Maeda, Kazaha Horii, Takahiro Ogawa, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1609, 1612, 2020
Scientific journal, A multi-task convolutional neural network leading to high performance and interpretability via attribute estimation is presented in this letter. Our method can provide interpretation of the classification results of CNNs by outputting attributes that explain elements of objects as a judgement reason of CNNs in the middle layer. Furthermore, the proposed network uses the estimated attributes for the following prediction of classes. Consequently, construction of a novel multi-task CNN with improvements in both of the interpretability and classification performance is realized.

Inpainting via Sparse Representation Based on a Phaseless Quality Metric.
Takahiro Ogawa, Keisuke Maeda, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1541, 1551, 2020
Scientific journal, An inpainting method via sparse representation based on a new phaseless quality metric is presented in this paper. Since power spectra, phaseless features, of local regions within images enable more successful representation of their texture characteristics compared to their pixel values, a new quality metric based on these phaseless features is newly derived for image representation. Specifically, the proposed method enables spare representation of target signals, i.e., target patches, including missing intensities by monitoring errors converged by phase retrieval as the novel phaseless quality metric. This is the main contribution of our study. In this approach, the phase retrieval algorithm used in our method has the following two important roles: (1) derivation of the new quality metric that can be derived even for images including missing intensities and (2) conversion of phaseless features, i.e., power spectra, to pixel values, i.e., intensities. Therefore, the above novel approach solves the existing problem of not being able to use better features or better quality metrics for inpainting. Results of experiments showed that the proposed method using sparse representation based on the new phaseless quality metric outperforms previously reported methods that directly use pixel values for inpainting.

Heterogeneous-Graph-Based Video Search Reranking Using Topic Relevance.
Soh Yoshida, Mitsuji Muneyasu, Takahiro Ogawa, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1529, 1540, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, 2020
English, Scientific journal, In this paper, we address the problem of analyzing topics, included in a social video group, to improve the retrieval performance of videos. Unlike previous methods that focused on an individual visual aspect of videos, the proposed method aims to leverage the "mutual reinforcement" of heterogeneous modalities such as tags and users associated with video on the Internet. To represent multiple types of relationships between each heterogeneous modality, the proposed method constructs three subgraphs: user-tag, video-video, and video-tag graphs. We combine the three types of graphs to obtain a heterogeneous graph. Then the extraction of latent features, i.e., topics, becomes feasible by applying graph-based soft clustering to the heterogeneous graph. By estimating the membership of each grouped cluster for each video, the proposed method defines a new video similarity measure. Since the understanding of video content is enhanced by exploiting latent features obtained from different types of data that complement each other, the performance of visual reranking is improved by the proposed method. Results of experiments on a video dataset that consists of YouTube-8M videos show the effectiveness of the proposed method, which achieves a 24.3% improvement in terms of the mean normalized discounted cumulative gain in a search ranking task compared with the baseline method.

Deep convolutional neural network-based anomaly detection for organ classification in gastric X-ray examination.
Ren Togo, Haruna Watanabe, Takahiro Ogawa, Miki Haseyama
Comput. Biol. Medicine, 123, 103903, 103903, 2020
Scientific journal, Aim: The aim of this study was to determine whether our deep convolutional neural network-based anomaly detection model can distinguish differences in esophagus images and stomach images obtained from gastric X-ray examinations. Methods: A total of 6012 subjects were analyzed as our study subjects. Since the number of esophagus X-ray images is much smaller than the number of gastric X-ray images taken in X-ray examinations, we took an anomaly detection approach to realize the task of organ classification. We constructed a deep autoencoding gaussian mixture model (DAGMM) with a convolutional autoencoder architecture. The trained model can produce an anomaly score for a given test X-ray image. For comparison, the original DAGMM, AnoGAN, and a One-Class Support Vector Machine (OCSVM) that were trained with features obtained by a pre-trained Inception-v3 network were used. Results: Sensitivity, specificity, and the calculated harmonic mean of the proposed method were 0.956, 0.980, and 0.968, respectively. Those of the original DAGMM were 0.932, 0.883, and 0.907, respectively. Those of AnoGAN were 0.835, 0.833, and 0.834, respectively, and those of OCSVM were 0.932, 0.935, and 0.934, respectively. Experimental results showed the effectiveness of the proposed method for an organ classification task. Conclusion: Our deep convolutional neural network-based anomaly detection model has shown the potential for clinical use in organ classification.

Human-Centric Emotion Estimation Based on Correlation Maximization Considering Changes With Time in Visual Attention and Brain Activity.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 203358, 203368, 2020
Scientific journal, A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.

Estimation of Interest Levels From Behavior Features via Tensor Completion Including Adaptive Similar User Selection.
Keisuke Maeda, Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 126109, 126118, 2020
Scientific journal, A method for estimating interest levels from behavior features via tensor completion including adaptive similar user selection is presented in this paper. The proposed method focuses on a tensor that is suitable for data containing multiple contexts and constructs a third-order tensor in which three modes are 'products', 'users' and 'user behaviors and interest levels' for these products. By complementing this tensor, unknown interest level estimation of a product for a target user becomes feasible. For further improving the estimation performance, the proposed method adaptively selects similar users for the target user by focusing on converged estimation errors between estimated interest levels and known interest levels in the tensor completion. Furthermore, the proposed method can adaptively estimate the unknown interest from the similar users. This is the main contribution of this paper. Therefore, the influence of users having different interests is reduced, and accurate interest level estimation can be realized. In order to verify the effectiveness of the proposed method, we show experimental results obtained by estimating interest levels of users holding books.

Supervised Fractional-Order Embedding Geometrical Multi-View CCA (SFGMCCA) for Multiple Feature Integration.
Keisuke Maeda, Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 114340, 114353, 2020
Scientific journal, Techniques for integrating different types of multiple features effectively have been actively studied in recent years. Multiset canonical correlation analysis (MCCA), which maximizes the sum of pairwise correlations of inter-view (i.e., between different features), is one of the powerful methods for integrating different types of multiple features, and various MCCA-based methods have been proposed. This work focuses on a supervised MCCA variant in order to construct a novel effective feature integration framework. In this paper, we newly propose supervised fractional-order embedding geometrical multi-view CCA (SFGMCCA). This method constructs not only the correlation structure but also two types of geometrical structures of intra-view (i.e., within each feature) and inter-view simultaneously, thereby realizing more precise feature integration. This method also supports the integration of small sample and high-dimensional data by using the fractional-order technique. We conducted experiments using four types of image datasets, i.e., MNIST, COIL-20, ETH-80 and CIFAR-10. Furthermore, we also performed an fMRI dataset containing brain signals to verify the robustness. As a result, it was confirmed that accuracy improvements using SFGMCCA were statistically significant at the significance level of 0.05 compared to those using conventional representative MCCA-based methods.

Enhancing Cross-Modal Retrieval Based on Modality-Specific and Embedding Spaces.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 96777, 96786, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, A new approach that drastically improves cross-modal retrieval performance in vision and language (hereinafter referred to as & x201C;vision and language retrieval & x201D;) is proposed in this paper. Vision and language retrieval takes data of one modality as a query to retrieve relevant data of another modality, and it enables flexible retrieval across different modalities. Most of the existing methods learn optimal embeddings of visual and lingual information to a single common representation space. However, we argue that the forced embedding optimization results in loss of key information for sentences and images. In this paper, we propose an effective utilization of representation spaces in a simple but robust vision and language retrieval method. The proposed method makes use of multiple individual representation spaces through text-to-image and image-to-text models. Experimental results showed that the proposed approach enhances the performance of existing methods that embed visual and lingual information to a single common representation space.

Context-Aware Network Analysis of Music Streaming Services for Popularity Estimation of Artists.
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 48673, 48685, 2020, [Peer-reviewed]
Scientific journal, A novel trial for estimating popularity of artists in music streaming services (MSS) is presented in this paper. The main contribution of this paper is to improve extensibility for using multi-modal features to accurately analyze latent relationships between artists. In the proposed method, a novel framework to construct a network is derived by collaboratively using social metadata and multi-modal features via canonical correlation analysis. Different from conventional methods that do not use multi-modal features, the proposed method can construct a network that can capture social metadata and multi-modal features, i.e., a context-aware network. For effectively analyzing the context-aware network, a novel framework to realize popularity estimation of artists is developed based on network analysis. The proposed method enables effective utilization of the network structure by extracting node features via a node embedding algorithm. By constructing an estimator that can distinguish differences between the node features, the proposed method can archive accurate popularity estimation of artists. Experimental results using multiple real-world datasets that contain artists in various genres in Spotify, one of the largest MSS, are presented. Quantitative and qualitative evaluations show that our method is effective for both classifying and regressing the popularity.

Retrieval of similar scenes based on multimodal distance metric learning in soccer videos
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019, 10, 15, ACM, 15 Oct. 2019, [Peer-reviewed]
International conference proceedings, © 2019 Association for Computing Machinery. This paper presents a new method for retrieval of similar scenes based on multimodal distance metric learning in far-view soccer videos that broadly capture soccer fields and are not edited. We extract visual features and audio features from soccer video clips, and we extract text features from text data corresponding to these soccer video clips. In addition, distance metric learning based on Laplacian Regularized Metric Learning is performed to calculate the distances for each kind of features. Finally, by determining the final rank by integrating these distances, we realize successful multimodal retrieval of similar scenes from query scenes of soccer video clips. Experimental results show the effectiveness of our retrieval method.

Semantic Shot Classification in Baseball Videos Based on Similarities of Visual Features
K. Hirasawa, K. Maeda, T. Ogawa, M. Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 663, 664, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents a method of semantic shot classification in baseball videos based on similarities of visual features. Since it is difficult to prepare a large amount of training data with annotation, accurate event detection methods constructed from a small amount of training data are needed. In broadcast baseball video, since view angles of cameras are different for each event, shot change and event change have a close relationship. When visual features from shots are similar, events corresponding to shots are also similar, and a simple distance-based approach only focusing on training data is effective. Therefore, semantic shot classification based on visual features from a small amount of training data can be realized.

Region-based Distress Classification of Road Infrastructures via CNN Without Region Annotation
N. Ogawa, K. Maeda, T. Ogawa, M. Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 764, 765, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents region-based distress classification of road infrastructures via convolutional neural networks (CNN) without region annotation. Although CNNs are often used for classification tasks recently, CNNs trained from images which contain unnecessary regions cannot perform precise classification. Distress images of road infrastructures contain various unnecessary objects other than the target distress. Although target regions should be provided in order to achieve high performance, it is a time-confusing task for engineers. This paper focuses on removing unnecessary objects in the images without region annotation via an object detection method. Especially, by using a pre-trained object detection model with distress images of road infrastructures, distress regions in the images are detected automatically. Our proposed CNN trained from the obtained distress regions realizes precise distress classification.

Aesthetic style transfer through text-to-image synthesis and image-to-image translation
Megumi Kotera, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 492, 493, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents a style transfer method combining generative adversarial networks and style transfer networks. In the previous style transfer methods, transformation from one image to another has been proposed. On the other hand, our method enables style transfer from a text to an image. This will be helpful when there are no images that represent the desired style. Experimental results show the effectiveness of our method.

Voice-input multimedia information retrieval system based on text-to-image GAN
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 943, 944, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings, In this paper, we develop an integrated multimedia information retrieval system. By utilizing text-to-image Generative Adversarial Network and image-to-text model, the developed system enables users to retrieve an objective content utilizing voice as an input with high accuracy. Experimental results show the effectiveness of the developed system.

Detection of distress region from subway tunnel images via U-net-based deep semantic segmentation
An Wang, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 766, 767, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings, The maintenance and management of old infrastructures has become an urgent issue nowadays, as an important part of transportation infrastructures, the subway tunnels are facing the same problem as other infrastructures. In this paper, we present a distress detection method in subway tunnels using U-net. As one of the semantic segmentation neural networks, U-net shows promising performance in remote sensing image segmentation and medical image segmentation. We apply this network to a subway tunnel dataset and compare it with other three semantic segmentation methods. From our experiment, it is shown that the proposed approach achieves promising results in our task.

Estimation of Emotion Labels via Tensor-Based Spatiotemporal Visual Attention Analysis
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 4105, 4109, IEEE, Sep. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents emotion label estimation via tensor-based spatiotemporal visual attention analysis. It has been reported in the fields of psychology and neuroscience that human emotions are related to two elements, their visual attention change and objects included in a target image. Therefore, the proposed method focuses on the spatiotemporal change of visual attention of human gazing at objects in the target image and constructs two neural networks which enable the emotion label estimation considering both of the above two elements. Specifically, the proposed method newly constructs a fourth-order tensor, gaze and image tensor (GIT) whose modes correspond to the width, the height and the color channel of the target image and the time axis of visual attention which is used for representing the time change. Then the first network, which consists of general tensor discriminant analysis (GTDA) and extreme learning machine (ELM), estimates the emotion label from the fourth-order GIT with concerning their visual attention change. Furthermore, the second network, which consists of pre-trained convolutoinal neural network-based feature extraction, GTDA and ELM, enables the estimation from the second-order GIT including visual features obtained from objects focused at each time. Finally, the proposed method estimates emotion labels based on decision fusion of the outputs from the two networks. Experimental results show the effectiveness of the proposed method.

Scene Retrieval for Video Summarization Based on Text-to-Image gan
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 1825, 1829, IEEE, Sep. 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. We present a new scene retrieval method based on text-to-image Generative Adversarial Network (GAN) and its application to query-based video summarization. Text-to-image GAN is a deep learning method that can generate images from their corresponding sentences. In this paper, we reveal a characteristic that deep learning-based visual features extracted from images generated by text-to-image GAN include semantic information sufficiently. By utilizing the generated images as queries, the proposed method achieves higher scene retrieval performance than those of the stateof-the-art methods. In addition, we introduce a novel architecture that can consider order relationship of the input sentences to our method for realizing a target video summarization. Specifically, the proposed method generates multiple images thorough text-to-image GAN from multiple sentences summarizing target videos. Their summarized video can be obtained by performing the retrieval of corresponding scenes from the target videos according to the generated images with considering the order relationship. Experimental results show the effectiveness of the proposed method in the retrieval and summarization performance.

Gastritis Detection from Gastric X-Ray Images Via Fine-Tuning of Patch-Based Deep Convolutional Neural Network
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 1371, 1375, IEEE, Sep. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for gastritis detection from gastric X-ray images via fine-tuning techniques using a deep convolutional neural network (DCNN). DCNNs can learn parameters to capture high-dimensional features which express semantic contents of images by training on a large number of labeled images. However, lack of gastric X-ray images for training often occurs. To realize accurate detection with a small number of gastric X-ray images, the proposed method adopts fine-tuning techniques and newly introduces simple annotation of stomach regions to gastric X-ray images used for training. The proposed method fine-tunes a pre-trained DCNN with patches and three kinds of patch-level class labels considering not only the image-level ground truth ('gastritis'/'non-gastritis') but also the regions of a stomach since the outside of the stomach is not related to the image-level ground truth. In the test phase, by estimating the patch-level class labels with the fine-tuned DCNN, the proposed method enables the image-level class label estimation which excludes the effect of the unnecessary regions. Experimental results show the effectiveness of the proposed method.

Neural Network Maximizing Ordinally Supervised Multi-View Canonical Correlation for Deterioration Level Estimation
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 919, 923, IEEE, Sep. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a neural network maximizing ordinally supervised multi-view canonical correlation for deterioration level estimation. The contributions of this paper are twofold. First, in order to calculate features representing deterioration levels on transmission towers, which is one of the infrastructures, a novel neural network handling multi-modal features is constructed from a small amount of training data. Specifically, in our method, effective transformation to features with high discriminant ability without using many hidden layers is realized by setting projection matrices maximizing correlation between multiple features into hidden layer's weights. Second, since there exists ordinal scale in deterioration levels, the proposed method newly derives ordinally supervised multi-view canonical correlation analysis (OsMVCCA). OsMVCCA enables estimation of the effective projection considering not only label information but also their ordinal scales. Experimental results show that the proposed method realizes accurate deterioration level estimation.

Bilingual Lexicon Learning Using Tagged Images via Graph Trilateral Filter-based Feature Refinement
Yui Matsumoto, Shota Hamano, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2019 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-TW 2019, May 2019
International conference proceedings, A novel method to realize bilingual lexicon learning (BLL) using tagged images is presented in this paper. Different from existing methods that require parallel corpora, the proposed method enables extraction of semantically similar words by utilizing not such corpora but tagged images on image sharing services. The main contribution of this paper is derivation of a novel framework to refine visual features of tagged images based on graph trilateral filter-based smoothing. This enables reduction of the influence of noisy tags that are irrelevant to contents of images. As a result, accurate BLL becomes feasible by nearest neighbor search using the refined visual features.

Convolutional Sparse Coding-based Anomalous Event Detection in Surveillance Videos
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama
2019 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-TW 2019, May 2019
International conference proceedings, This paper presents a Convolutional Sparse Coding (CSC)-based anomalous event detection method in surveillance videos. The proposed method derives new features from reconstruction errors and sparse coefficient maps obtained by CSC, and the anomalous events are detected by a multi-layer network whose inputs are the above new features. Since such events, i.e., anomalous objects, have different characteristics in the sparse coefficient maps and their corresponding reconstruction errors, successful detection can be expected. Experimental results show high detection performance of own method.

User-Specific Visual Attention Estimation Based on Visual Similarity and Spatial Information in Images
Y. Moroto, K. Maeda, T. Ogawa, M. Haseyama
IEEE International Conference on Consumer Electronics – Taiwan (ICCE-TW), 479, 480, May 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for user-specific visual attention estimation based on visual similarities and spatial information in images. In order to estimate the user-specific visual attention, the proposed method calculates two kinds of saliency maps. One is constructed as a visual similarity-based saliency map, and the other is constructed by considering spatial information of objects in images. The proposed method performs a fusion of these two maps for considering visual similarities and spatial information. This is the biggest contribution of this paper. Therefore, improvement of the estimation performance of the user-specific visual attention is realized.

Estimating Viewed Image Categories from Human Brain Activity via Semi-supervised Fuzzy Discriminative Canonical Correlation Analysis
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2019-May, 1105, 1109, IEEE, May 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method to estimate viewed image categories from human brain activity via newly derived semi-supervised fuzzy discriminative canonical correlation analysis (Semi-FDCCA). The proposed method can estimate image categories from functional magnetic resonance imaging (fMRI) activity measured while subjects view images by making fMRI activity and visual features obtained from images comparable through Semi-FDCCA. To realize Semi-FDCCA, we first derive a new supervised CCA called FDCCA that can consider fuzzy class information based on image category similarities obtained from WordNet ontology. Second, we adopt SemiCCA that can utilize additional unpaired visual features in addition to pairs of fMRI activity and visual features in order to prevent overfitting to the limited pairs. Furthermore, Semi-FDCCA can be derived by combining FDCCA with SemiCCA. Experimental results show that Semi-FDCCA enables accurate estimation of viewed image categories.

Detection of gastritis by a deep convolutional neural network from double-contrast upper gastrointestinal barium X-ray radiography
Ren Togo, Nobutake Yamamichi, Katsuhiro Mabe, Yu Takahashi, Chihiro Takeuchi, Mototsugu Kato, Naoya Sakamoto, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
Journal of Gastroenterology, 54, 4, 321, 329, Apr. 2019, [Peer-reviewed]
Scientific journal, © 2018, Japanese Society of Gastroenterology. Background: Deep learning has become a new trend of image recognition tasks in the field of medicine. We developed an automated gastritis detection system using double-contrast upper gastrointestinal barium X-ray radiography. Methods: A total of 6520 gastric X-ray images obtained from 815 subjects were analyzed. We designed a deep convolutional neural network (DCNN)-based gastritis detection scheme and evaluated the effectiveness of our method. The detection performance of our method was compared with that of ABC (D) stratification. Results: Sensitivity, specificity, and harmonic mean of sensitivity and specificity of our method were 0.962, 0.983, and 0.972, respectively, and those of ABC (D) stratification were 0.925, 0.998, and 0.960, respectively. Although there were 18 false negative cases in ABC (D) stratification, 14 of those 18 cases were correctly classified into the positive group by our method. Conclusions: Deep learning techniques may be effective for evaluation of gastritis/non-gastritis. Collaborative use of DCNN-based gastritis detection systems and ABC (D) stratification will provide more reliable gastric cancer risk information.

The friction properties of firebrat scales
Yuji Hirai, Naoto Okuda, Naoki Saito, Takahiro Ogawa, Ryuichiro Machida, Shûhei Nomura, Masahiro Ôhara, Miki Haseyama, Masatsugu Shimomura
Biomimetics, 4, 1, 01 Mar. 2019
Scientific journal, Friction is an important subject for sustainability due to problems that are associated with energy loss. In recent years, micro- and nanostructured surfaces have attracted much attention to reduce friction; however, suitable structures are still under consideration. Many functional surfaces are present in nature, such as the friction reduction surfaces of snake skins. In this study, we focused on firebrats, Thermobia domestica, which temporary live in narrow spaces, such as piled papers, so their body surface (integument) is frequently in contact with surrounding substrates. We speculate that, in addition to optical, cleaning effects, protection against desiccation and enemies, their body surface may be also adapted to reduce friction. To investigate the functional effects of the firebrat scales, firebrat surfaces were observed using a field-emission scanning electron microscope (FE-SEM) and a colloidal probe atomic force microscope (AFM). Results of surface observations by FE-SEM revealed that adult firebrats are entirely covered with scales, whose surfaces have microgroove structures. Scale groove wavelengths around the firebrat's head are almost uniform within a scale but they vary between scales. At the level of single scales, AFM friction force measurements revealed that the firebrat scale reduces friction by decreasing the contact area between scales and a colloidal probe. The heterogeneity of the scales' groove wavelengths suggests that it is difficult to fix the whole body on critical rough surfaces and may result in a "fail-safe" mechanism.

Extraction of regions related to cardiac sarcoidosis in polar map images
Ren Togo, Takahiro Ogawa, Osamu Manabe, Kenji Hirata, Tohru Shiga, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 237, 238, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for extracting important regions for deep learning models in the identification of cardiac sarcoidosis using polar map images. Although deep learning-based detection methods have widely studied, they are still often called black boxes. Since high reliability for provided results from computer-aided diagnosis systems is important toward clinical applications, this problem should be solved. In this paper, we try to visualize important regions for deep learning-based models for improvement of understanding to clinicians. We monitor the variance of confidence of a model constructed with a deep learning-based feature and define it as a contribution value toward the estimated label. We visualize important regions for models based on the contribution value.

Estimation of emotions evoked by images based on multiple gaze-based CNN features
Taiga Matsui, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 194, 195, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for estimating emotions evoked by watching images based on multiple visual features considering relationship with gaze information. The proposed method obtains multiple visual features from multiple middle layers of a Convolutional Neural Network. Then the proposed method newly derives their gaze-based visual features maximizing correlation with gaze information by using Discriminative Locality Preserving Canonical Correlation Analysis. The final estimation result is calculated by integrating multiple estimation results obtained from these gaze-based visual features. Consequently, successful emotion estimation becomes feasible by using such multiple estimation results which correspond to different semantic levels of target images.

Estimation of users' interest levels using tensor completion with SemiCCA
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 239, 240, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a new method for estimation of users' interest levels using tensor completion with SemiCCA. The proposed method extracts new features maximizing correlation between features calculated from partially paired users' behavior and contents with semi-supervised canonical correlation analysis (SemiCCA). By this approach, we can successfully use the contents that users have not viewed for the interest level estimation. Moreover, our method utilizes the tensor completion to estimate unknown interest levels. Consequently, in the proposed method, accurate estimation of interest levels using SemiCCA and the tensor completion is realized. Experimental results are shown to verify the effectiveness of the proposed method by using actual data.

Classification of subcellular protein patterns in human cells with transfer learning
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 273, 274, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. In this paper, we present a deep learning method for classifying subcellular protein patterns in human cells. Our method is mainly based on transfer learning and utilizes a newly proposed loss function named focal loss to deal with the problem of severe class imbalance existing in the task. The performance of our method is evaluated by a MacroF1 score of total 28 classes, and the final MacroF1 score of our method is 0.706.

Estimation of visual attention via canonical correlation between visual and gaze-based features
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 229, 230, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for estimating visual attention via canonical correlation between visual and gaze-based features. The proposed method estimates user-specific visual attention by comparing a test image with training images including their corresponding individual eye gaze data in a common space. Specifically, canonical correlation analysis can derive projections which enable comparison between visual and gaze-based features in the common space. Therefore, given the new test image, our method projects its visual features to the common space and can estimate visual attention. Experimental results show the effectiveness of the proposed method.

Chronic gastritis detection from gastric X-ray images via deep autoencoding Gaussian mixture models
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 231, 232, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a detection method of chronic gastritis from gastric X-ray images. The conventional method cannot detect chronic gastritis accurately since the number of non-gastritis images is overwhelmingly larger than the number of gastritis images. To deal with this problem, the proposed method performs the detection of chronic gastritis by using Deep Autoencoding Gaussian Mixture Models (DAGMM) which is an anomaly detection approach. DAGMM enables construction of chronic gastritis detection model using only non-gastritis images. In addition, DAGMM is superior to conventional anomaly detection methods since the models of dimensionality reduction and density estimation can be learned simultaneously. Therefore, the proposed method realizes accurate detection of chronic gastritis by utilizing DAGMM.

Fine-tuning of pre-trained DCNN for gastritis detection from gastric X-ray images
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 196, 197, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a detection method of gastritis from gastric X-ray images using fine-tuning techniques. With the development of deep convolutional neural networks (DCNNs), DCNN-based methods have achieved more accurate performance than conventional machine learning methods using hand-crafted features in the field of medical image analysis. However, lack of training images often occurs in clinical situations even though DCNNs require a large amount of training images to avoid overfitting. Therefore, the proposed method aims to consider the clinical situations that a limited amount of the training images are available. By fine-tuning a DCNN pre-trained with a large amount of annotated natural images, we avoid overfitting and realize accurate detection of the gastritis with a small amount of the training images.

Bone metastatic tumor detection based on AnoGAN using CT images
Haruna Watanabe, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 235, 236, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. In this paper, we propose a method to detect bone metastatic tumors using computed tomography (CT) images. Bone metastatic tumors spread from primary cancer to other organs, and they can cause severe pain. Therefore, it is important to detect metastatic tumors earlier in addition to primary cancer. However, since metastatic tumors are very small, and they emerge from unpredictable regions in the body, collecting metastatic tumor images is difficult compared to primary cancer. In such a case, it can be considered that the idea of anomaly detection is suitable. The proposed method based on a generative adversarial network model trains with only non-metastatic bone tumor images and detects bone metastatic tumor in an unsupervised manner. Then the anomaly score is defined for each test CT image. Experimental results show the anomaly scores between non-metastatic bone tumor images and metastatic bone tumor images are clearly different. The anomaly detection approach may be effective for the detection of bone metastatic tumors in CT images.

Semi-supervised discriminative CCA for estimating viewed image categories from fMRI data
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 233, 234, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method that estimates viewed image categories from functional magnetic resonance imaging (fMRI) data via semi-supervised discriminative canonical correlation analysis (Semi-DCCA). We newly derive Semi-DCCA that enables direct comparison of fMRI data and visual features extracted from viewed images while taking into account the class information and additional visual features to avoid overfitting. The proposed method enables estimation of image categories from fMRI data measured when subjects view images by comparing fMRI data with visual features through Semi-DCCA. Experimental results show that Semi-DCCA can improve estimation performance of the viewed image categories.

Video classification based on user preferences with soft-bag multiple instance learning
Akira Toyoda, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 198, 199, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method to classify videos based on user preferences with soft-bag multiple instance learning (MIL). Our method classifies videos that a user has watched into two classes (preferred and not-preferred) with two-modal features extracted from the videos and brain signals measured while the user is watching the videos. Our method splits videos and brain signals into fixed-length segments and computes features used for classification from only a fixed-number of segments selected based on the idea of soft-bag MIL. By using the features computed from the selected segments, our method makes it possible to classify videos in the case that some videos that a user prefers contain some scenes the user does not prefer, and vice versa. Our main contribution allows methods classifying videos based on user preferences to treat such a case unlike conventional methods.

Estimation of Drilling Energy from Tunnel Cutting Face Image Based on Online Learning.
Kentaro Yamamoto, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 768, 769, IEEE, 2019
International conference proceedings, To realize the effective tunnel construction, it is important to grasp the characteristics of ground conditions. This paper presents an estimation method of drilling energy based on online learning from tunnel cutting face images. The proposed method realizes the estimation from a small amount of data by learning the image taken immediately before the target image using online learning since consecutive tunnel cutting faces are related to each other. The experimental results show the effectiveness of the proposed method.

Multimodal Retrieval of Similar Soccer Videos Based on Optimal Combination of Multiple Distance Measures.
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 665, 666, IEEE, 2019
International conference proceedings, This paper presents a new multimodal method for retrieval of similar soccer videos based on optimal combination of multiple distance measures. Our method first extracts three types of Convolutional Neural Network-based features focusing the players' actions, the audience's cheers and prompt reports. Then, by applying the optimal distance measure to each feature, we calculate the similarities between a query video and videos in a database. Finally, we realize accurate retrieval of similar soccer videos by integrating these similarities. Experiments on actual soccer videos demonstrate encouraging results.

Interest Estimation for Images Based on Eye Gaze-based Visual and Text Features.
Masanao Matsumoto, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 481, 482, IEEE, 2019
International conference proceedings, This paper presents an estimation method of user-specific interests for images. The proposed method computes a projection which maximizes the correlation between 'eye gaze data which are collected while watching images' and 'visual and text features' by utilizing Canonical Correlation Analysis (CCA). Since eye gaze data reflect user's interests, new visual and text features calculated by using obtained projections can be also expected to reflect user's interests. Then accurate estimation of user-specific interests for images via Support Vector Machine (SVM) becomes feasible from these features. Experimental results show the effectiveness of our method.

Estimation of User-Specific Visual Attention Based on Gaze Information of Similar Users.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 477, 478, IEEE, 2019, [Peer-reviewed]
International conference proceedings, This paper proposes an estimation of user-specific visual attention based on gaze information of similar users. The proposed method estimates the user-specific visual attention by using the eye gaze data of other similar users. Then the similar users are selected based on the past eye gaze data of the target user. Although introducing the eye gaze data of the similar users into the estimation of user-specific visual attention is a simple approach, it can break the limitation of estimation performance. This approach is the main contribution of this paper. Experimental results show the effectiveness.

Performance Prediction Method of Examinees Based on Matrix Completion.
Yutaka Yamada, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 229, 230, IEEE, 2019, [Peer-reviewed]
International conference proceedings, This paper presents a novel performance prediction method of examinees based on matrix completion. The proposed method newly introduces the matrix completion for predicting the performance of the entrance examination from the results of a mock exams. This approach using the matrix completion provides a solution to the problem that there is unknown information on the explanatory variable side. Specifically, we adopt singular value decomposition for solving the problem. Therefore, accurate prediction can be expected by the proposed method. This work is the first trial to realize predicting performance of examinees and this is the main contribution of this paper. Experimental results are shown for verifying the effectiveness of the proposed method.

Estimating Viewed Image Categories from fMRI Activity via Multi-view Bayesian Generative Model.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 127, 128, IEEE, 2019, [Peer-reviewed]
International conference proceedings, Researches for estimating what people view from their brain activity have attracted wide attention. Many existing methods focus on only relationship between brain activity and visual features extracted from viewed images. In this paper, we propose a multi-view Bayesian generative model (MVBGM), which adopts a new view, i.e., category features obtained from viewed images. MVBGM based on automatic feature selection under the Bayesian approach can also avoid overfitting caused by high dimensional features. Experimental results show that MVBGM can estimate viewed image categories from brain activity more accurately than existing methods.

The Extraction of Individual Music Preference Based on Deep Time-series CCA.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 15, 16, IEEE, 2019, [Peer-reviewed]
International conference proceedings, This paper presents a novel method to extract individual music preference. The novel CCA, named Deep Time-series Canonical Correlation Analysis (DTCCA), is proposed to realize the aforementioned extraction. In contrast with standard CCA, our novel DTCCA can deal with not only the correlation between the input features but also the time-series relation in each input those simultaneously. A DTCCA-based latent space of projected features effectively reflects relationship between his/her EEG features and corresponding audio features rather than standard CCA's one since the audio and EEG signals have time-series relation respectively.

Effectiveness Evaluation of Deep Features for Image Reconstruction from fMRI Signals.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 479, 480, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, Reconstruction of human cognitive contents based on analyzing of functional Magnetic Resonance Imaging (fMRI) signals has been actively researched. Cognitive contents such as seen images can be reconstructed by estimating the relation between fMRI signals and deep neural network (DNN) features extracted from seen images. In order to reconstruct seen images with high accuracy, translation fMRI signals into meaningful features is an important task. In this paper, we validate the reconstruction accuracy of seen images by using visual features with some DNN feature extraction models. Recent works for image reconstruction used VGG19 to extract visual features. However, newer models such as Inception-v3 and ResNet50 have been proposed and these models perform general object recognition with higher accuracy. Thus it is expected the accuracy of image reconstruction is improved when using features extracted by these newer models. Experimental results for images of five categories show the effectiveness of the use of visual features from newer DNN models.

Scene Retrieval Using Text-to-image GAN-based Visual Similarities and Image-to-text Model-based Textual Similarities.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 13, 14, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, Scene retrieval from a video database is a fundamental study in computer vision. Traditionally, content based retrieval methods can retrieve objective scenes with high accuracy by utilizing visual features. However, users cannot utilize content based retrieval methods when they cannot prepare query contents. To solve this problem, in this paper, we propose a novel content based scene retrieval method focusing on text-to-image Generative Adversarial Network and image-to-text model. By utilizing the proposed method, we can retrieve objective scenes in visual feature space with high accuracy even though it only utilizes a sentence as an input. Experimental results show the effectiveness of the proposed method.

Multi-classifier Decision: Integration of Multiple Brain Activity-based Classifications
Takahiro Ogawa, Kento Sugata, Ren Togo, Miki Haseyama
ITE Transactions on Media Technology and Applications, 7, 1, 36, 44, 2019, [Peer-reviewed]
English, Scientific journal, A novel method that integrates brain activity-based classifications obtained from multiple users is presented in this paper. The proposed method performs decision-level fusion (DLF) of the classifications using a kernelized version of extended supervised learning from multiple experts (KESLME), which is newly derived in this paper. In this approach, feature-level fusion of multiuser electroencephalogram (EEG) features is performed by multiset supervised locality preserving canonical correlation analysis (MSLPCCA). In the proposed method, the multiple classification results are obtained by classifiers separately constructed for the multiuser EEG features. Then DLF of these classification results becomes feasible based on KESLME, which can provide the final decision with consideration of the relationship between the MSLPCCA-based integrated EEG features and each classifier’s performance. In this way, a new multi-classifier decision technique, which depends only on users’ brain activities, is realized, and the performance in an image classification task becomes comparable to that of Inception-v3, one of the state-of-the-art deep convolutional neural networks.

Text-to-Image GAN-Based Scene Retrieval and Re-Ranking Considering Word Importance.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 169920, 169930, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a novel scene retrieval and re-ranking method based on a text-to-image Generative Adversarial Network (GAN). The proposed method generates an image from an input query sentence based on the text-to-image GAN and then retrieves a scene that is the most similar to the generated image. By utilizing the image generated from the input query sentence as a query, we can control semantic information of the query image at the text level. Furthermore, we introduce a novel interactive re-ranking scheme to our retrieval method. Specifically, users can consider the importance of each word within the first input query sentence. Then the proposed method re-generates the query image that reflects the word importance provided by users. By updating the generated query image based on the word importance, it becomes feasible for users to revise retrieval results through this re-ranking process. In experiments, we showed that our retrieval method including the re-ranking scheme outperforms recently proposed retrieval methods.

Estimating Regions of Deterioration in Electron Microscope Images of Rubber Materials via a Transfer Learning-Based Anomaly Detection Model.
Ren Togo, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 162395, 162404, 2019, [Peer-reviewed]
Scientific journal, A method for estimating regions of deterioration in electron microscope images of rubber materials is presented in this paper. Deterioration of rubber materials is caused by molecular cleavage, external force, and heat. An understanding of these characteristics is essential in the field of material science for the development of durable rubber materials. Rubber material deterioration can be observed by using on electron microscope but it requires much effort and specialized knowledge to find regions of deterioration. In this paper, we propose an automated deterioration region estimation method based on deep learning and anomaly detection techniques to support such material development. Our anomaly detection model, called Transfer Learning-based Deep Autoencoding Gaussian Mixture Model (TL-DAGMM), uses only normal regions for training since obtaining training data for regions of deterioration is difficult. TL-DAGMM makes use of extracted high representation features from a pre-trained deep learning model and can automatically learn the characteristics of normal rubber material regions. Regions of deterioration are estimated at the pixel level by calculated anomaly scores. Experiments on real rubber material electron microscope images demonstrated the effectiveness of our model.

Team Tactics Estimation in Soccer Videos Based on a Deep Extreme Learning Machine and Characteristics of the Tactics.
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 153238, 153248, 2019, [Peer-reviewed]
Scientific journal, A novel method for estimating team tactics in soccer videos based on a Deep Extreme Learning Machine (DELM) and unique characteristics of tactics is presented in this paper. The proposed method estimates the tactics of each team from players' formations and enables successful training from a limited amount of training data. Specifically, the estimation of tactics consists of two stages. First, by utilizing two DELMs corresponding to the two teams, the proposed method estimates the provisional tactics of each team. Second, the proposed method updates the team tactics based on unique characteristics of soccer tactics, the relationship between tactics of the two teams and information on ball possession. Consequently, since the proposed method estimates the team tactics that satisfy these characteristics, accurate estimation results can be obtained. In an experiment, the proposed method is applied to actual soccer videos to verify its effectiveness.

Query is GAN: Scene Retrieval With Attentional Text-to-Image Generative Adversarial Network.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 153183, 153193, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, Scene retrieval from input descriptions has been one of the most important applications with the increasing number of videos on the Web. However, this is still a challenging task since semantic gaps between features of texts and videos exist. In this paper, we try to solve this problem by utilizing a text-to-image Generative Adversarial Network (GAN), which has become one of the most attractive research topics in recent years. The text-to-image GAN is a deep learning model that can generate images from their corresponding descriptions. We propose a new retrieval framework, Query is GAN, based on the text-to-image GAN that drastically improves scene retrieval performance by simple procedures. Our novel idea makes use of images generated by the text-to-image GAN as queries for the scene retrieval task. In addition, unlike many studies on text-to-image GANs that mainly focused on the generation of high-quality images, we reveal that the generated images have reasonable visual features suitable for the queries even though they are not visually pleasant. We show the effectiveness of the proposed framework through experimental evaluation in which scene retrieval is performed from real video datasets.

Interest Level Estimation Based on Tensor Completion via Feature Integration for Partially Paired User's Behavior and Videos.
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 148576, 148585, 2019, [Peer-reviewed]
Scientific journal, A novel method for interest level estimation based on tensor completion via feature integration for partially paired users' behavior and videos is presented in this paper. The proposed method defines a novel canonical correlation analysis (CCA) framework that is suitable for interest level estimation, which is a hybrid version of semi-supervised CCA (SemiCCA) and supervised locality preserving CCA (SLPCCA) called semi-supervised locality preserving CCA (S2LPCCA). For partially paired users' behavior and videos in actual shops and on the Internet, new integrated features that maximize the correlation between partially paired samples by the principal component analysis (PCA)-mixed CCA framework are calculated. Then videos that users have not watched can be used for the estimation of users' interest levels. Furthermore, local structures of partially paired samples in the same class are preserved for accurate estimation of interest levels. Tensor completion, which can be applied to three contexts, videos, users and 'canonical features and interest levels,' is used for estimation of interest levels. Consequently, the proposed method realizes accurate estimation of users' interest levels based on S2LPCCA and the tensor completion from partially paired training features of users' behavior and videos. Experimental results obtained by applying the proposed method to actual data show the effectiveness of the proposed method.

Scene Retrieval from Multiple Resolution Generated Images Based on Text-to-image GAN
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), 2019-May, 1, 5, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, Text-to-image Generative Adversarial Network (GAN) is a deep learning model that generates an image from an input sentence. It is expressly attracting attentions because of its applicability of the generated images. However, many existing studies have still focused on generation of high-quality images, and there are few studies focusing on application of the generated images since text-to-image GANs still cannot produce visually pleasing images in the complicated tasks. In this paper, we apply a text-to-image GAN as a generator of query images for a scene retrieval task to show availability of the visually non-pleasant images. The proposed method utilizes a low-resolution generated image that focuses on a sentence and a high-resolution generated image that focuses on each word of the sentence to retrieve a desired scene. With this mechanism, the proposed method realizes a high-accuracy scene retrieval from a sentence input. Experimental results show the effectiveness of our method.

Semi-Supervised Learning Based on Tri-Training for Gastritis Classification using Gastric X-ray Images.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE International Symposium on Circuits and Systems, ISCAS 2019, Sapporo, Japan, May 26-29, 2019, 2019-May, 1, 5, IEEE, 2019, [Peer-reviewed]
International conference proceedings, This paper presents a method of semi-supervised learning based on tri-training for gastritis classification using gastric X-ray images. The proposed method is constructed based on the tri-training architecture, and the strategies of label smoothing regularization and random erasing augmentation are utilized in the method to enhance the performance. Although the task of gastritis classification is challenging, we report that the proposed semi-supervised learning method using only a small number of labeled data achieves 0.888 harmonic mean of sensitivity and specificity on test data composed of 615 patients.

Synthetic Image Generation for Gastritis Detection Based on Auxiliary Classifier Generative Adversarial Network.
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE International Symposium on Circuits and Systems, ISCAS 2019, Sapporo, Japan, May 26-29, 2019, 2019-May, 1, 5, IEEE, 2019, [Peer-reviewed]
International conference proceedings, With the development of convolutional neural networks (CNNs), CNN-based methods for medical image analysis have achieved more accurate performance than conventional machine learning methods using hand-crafted features. Although these methods utilize a large number of training images and realize high performance, lack of the training images often occurs in medical image analysis due to several reasons. This paper presents a novel image generation method to construct a dataset for gastritis detection from gastric X-ray images. The proposed method effectively utilizes two kinds of training images (gastritis and non-gastritis images) to generate images of each domain by introducing label conditioning into a generative model. Experimental results using real-world gastric X-ray images show the effectiveness of the proposed method.

Multi-feature Fusion Based on Supervised Multi-view Multi-label Canonical Correlation Projection.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, 2019-May, 3936, 3940, IEEE, 2019, [Peer-reviewed]
International conference proceedings, This paper presents multi-feature fusion based on supervised multi-view multi-label canonical correlation projection (sM2CP). The proposed method applies sM2CP-based feature fusion to multiple features obtained from various convolutional neural networks (CNNs) whose characteristics are different. Since new fused features with high representation ability can be obtained, performance improvement of multi-label classification is realized. Specifically, in order to tackle the multi-label problem, sM2CP introduces a label similarity information of label vectors into the objective function of supervised multi-view canonical correlation analysis. Thus, sM2CP can deal with complex label information such as multi-label annotation. The main contribution of this paper is the realization of feature fusion of multiple CNN features for the multi-label problem by introducing multi-label similarity information into the canonical correlation analysis-based feature fusion approach. Experimental results show the effectiveness of sM2CP, which enables effective fusion of multiple CNN features.

Novel Audio Feature Projection Using KDLPCCA-Based Correlation with EEG Features for Favorite Music Classification.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
IEEE Trans. Affect. Comput., 10, 3, 430, 444, 2019, [Peer-reviewed]
Scientific journal, A novel audio feature projection using Kernel Discriminative Locality Preserving Canonical Correlation Analysis (KDLPCCA)-based correlation with electroencephalogram (EEG) features for favorite music classification is presented in this paper. The projected audio features reflect individual music preference adaptively since they are calculated by considering correlations with the user’s EEG signals during listening to musical pieces that the user likes/dislikes via a novel CCA proposed in this paper. The novel CCA, called KDLPCCA, can consider not only a non-linear correlation but also local properties and discriminative information of each class sample, namely, music likes/dislikes. Specifically, local properties reflect intrinsic data structures of the original audio features, and discriminative information enhances the power of the final classification. Hence, the projected audio features have an optimal correlation with individual music preference reflected in the user’s EEG signals, adaptively. If the KDLPCCA-based projection that can transform original audio features into novel audio features is calculated once, our method can extract projected audio features from a new musical piece without newly observing individual EEG signals. Our method therefore has a high level of practicability. Consequently, effective classification of user’s favorite musical pieces via a Support Vector Machine (SVM) classifier using the new projected audio features becomes feasible. Experimental results show that our method for favorite music classification using projected audio features via the novel CCA outperforms methods using original audio features, EEG features and even audio features projected by other state-of-the-art CCAs.

Young Professionals Activities at GCCE 2018 [Society News].
Takahiro Ogawa, Shingo Yamaguchi
IEEE Consumer Electron. Mag., 8, 3, 6, 7, 2019, [Peer-reviewed]
Scientific journal

Cardiac sarcoidosis classification with deep convolutional neural network-based features using polar maps.
Ren Togo, Kenji Hirata, Osamu Manabe, Hiroshi Ohira, Ichizo Tsujino, Keiichi Magota, Takahiro Ogawa, Miki Haseyama, Tohru Shiga
Comput. Biol. Medicine, 104, 81, 86, 2019, [Peer-reviewed]
Scientific journal, Aims: The aim of this study was to determine whether deep convolutional neural network (DCNN)-based features can represent the difference between cardiac sarcoidosis (CS) and non-CS using polar maps. Methods: A total of 85 patients (33 CS patients and 52 non-CS patients) were analyzed as our study subjects. One radiologist reviewed PET/CT images and defined the left ventricle region for the construction of polar maps. We extracted high-level features from the polar maps through the Inception-v3 network and evaluated their effectiveness by applying them to a CS classification task. Then we introduced the ReliefF algorithm in our method. The standardized uptake value (SUV)-based classification method and the coefficient of variance (CoV)-based classification method were used as comparative methods. Results: Sensitivity, specificity and the harmonic mean of sensitivity and specificity of our method with the ReliefF algorithm were 0.839, 0.870 and 0.854, respectively. Those of the SUVmax-based classification method were 0.468, 0.710 and 0.564, respectively, and those of the CoV-based classification method were 0.655, 0.750 and 0.699, respectively. Conclusion: The DCNN-based high-level features may be more effective than low-level features used in conventional quantitative analysis methods for CS classification.

Convolutional sparse coding-based deep random vector functional link network for distress classification of road structures.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Comput. Aided Civ. Infrastructure Eng., 34, 8, 654, 676, 2019, [Peer-reviewed]
Scientific journal, © 2019 Computer-Aided Civil and Infrastructure Engineering This paper presents a convolutional sparse coding (CSC)-based deep random vector functional link network (CSDRN) for distress classification of road structures. The main contribution of this paper is the introduction of CSC into a feature extraction scheme in the distress classification. CSC can extract visual features representing characteristics of target images because it can successfully estimate optimal convolutional dictionary filters and sparse features as visual features by training from a small number of distress images. The optimal dictionaries trained from distress images have basic components of visual characteristics such as edge and line information of distress images. Furthermore, sparse feature maps estimated on the basis of the dictionaries represent both strength of the basic components and location information of regions having their components, and these maps can represent distress images. That is, sparse feature maps can extract key components from distress images that have diverse visual characteristics. Therefore, CSC-based feature extraction is effective for training from a limited number of distress images that have diverse visual characteristics. The construction of a novel neural network, CSDRN, by the use of a combination of CSC-based feature extraction and the DRN classifier, which can also be trained from a small dataset, is shown in this paper. Accurate distress classification is realized via the CSDRN.

Consensus Clustering of Tweet Networks via Semantic and Sentiment Similarity Estimation.
Ryosuke Harakawa, Shoji Takimura, Takahiro Ogawa, Miki Haseyama, Masahiro Iwahashi
IEEE Access, 7, 116207, 116217, 2019, [Peer-reviewed]
Scientific journal, Although Twitter has become an important source of information, the number of accessible tweets is too large for users to easily find their desired information. To overcome this difficulty, a method for tweet clustering is proposed in this paper. Inspired by the reports that network representation is useful for multimedia content analysis including clustering, a network-based approach is employed. Specifically, a consensus clustering method for tweet networks that represent relationships among the tweets' semantics and sentiment are newly derived. The proposed method integrates multiple clustering results obtained by applying successful clustering methods to the tweet networks. By integrating complementary clustering results obtained based on semantic and sentiment features, the accurate clustering of tweets becomes feasible. The contribution of this work can be found in the utilization of the features, which differs from existing network-based consensus clustering methods that target only the network structure. Experimental results for a real-world Twitter dataset, which includes 65 553 tweets of 25 datasets, verify the effectiveness of the proposed method.

Music Video Recommendation Based on Link Prediction Considering Local and Global Structures of a Network.
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 104155, 104167, 2019, [Peer-reviewed]
Scientific journal, A novel method for music video recommendation is presented in this paper. The contributions of this paper are two-fold. (i) The proposed method constructs a network, which not only represents relationships between music videos and users but also captures multi-modal features of music videos. This enables collaborative use of multi-modal features such as audio, visual, and textual features, and multiple social metadata that can represent relationships between music videos and users on video hosting services. (ii) A novel scheme for link prediction considering local and global structures of the network (LP-LGSN) is newly derived by fusing multiple link prediction scores based on both local and global structures. By using the LP-LGSN to predict the degrees to which users desire music videos, the proposed method can recommend users' desired music videos. The experimental results for a real-world dataset constructed from YouTube-8M show the effectiveness of the proposed method.

Synthetic Gastritis Image Generation via Loss Function-Based Conditional PGGAN.
Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 87448, 87457, 2019, [Peer-reviewed]
Scientific journal, © 2013 IEEE. In this paper, a novel synthetic gastritis image generation method based on a generative adversarial network (GAN) model is presented. Sharing medical image data is a crucial issue for realizing diagnostic supporting systems. However, it is still difficult for researchers to obtain medical image data since the data include individual information. Recently proposed GAN models can learn the distribution of training images without seeing real image data, and individual information can be completely anonymized by generated images. If generated images can be used as training images in medical image classification, promoting medical image analysis will become feasible. In this paper, we targeted gastritis, which is a risk factor for gastric cancer and can be diagnosed by gastric X-ray images. Instead of collecting a large amount of gastric X-ray image data, an image generation approach was adopted in our method. We newly propose loss function-based conditional progressive growing generative adversarial network (LC-PGGAN), a gastritis image generation method that can be used for a gastritis classification problem. The LC-PGGAN gradually learns the characteristics of gastritis in gastric X-ray images by adding new layers during the training step. Moreover, the LC-PGGAN employs loss function-based conditional adversarial learning so that generated images can be used as the gastritis classification task. We show that images generated by the LC-PGGAN are effective for gastritis classification using gastric X-ray images and have clinical characteristics of the target symptom.

Distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Advanced Engineering Informatics, 37, 79, 87, Elsevier Ltd, 01 Aug. 2018, [Peer-reviewed]
English, Scientific journal, This paper presents distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine (CMWELM). For distress classification, it is necessary to extract semantic features that can effectively distinguish multiple kinds of distress from a small amount of class-imbalanced data. In recent machine learning techniques such as general deep learning methods, since effective feature transformation from visual features to semantic features can be realized by using multiple hidden layers, a large amount of training data are required. However, since the amount of training data of civil structures becomes small, it becomes difficult to perform successful transformation by using these multiple hidden layers. On the other hand, CMWELM consists of two hidden layers. The first hidden layer performs feature transformation, which can directly extract the semantic features from visual features, and the second hidden layer performs classification with solving the class-imbalanced problem. Specifically, in the first hidden layer, the feature transformation is realized by using projections obtained by maximizing the canonical correlation between visual and text features as weight parameters of the hidden layer without designing multiple hidden layers. Furthermore, the second hidden layer enables successful training of our classifier by using weighting factors concerning the class-imbalanced problem. Consequently, CMWELM realizes accurate distress classification from a small amount of class-imbalanced data.

Strategy to develop convolutional neural network-based classifier for diagnosis of whole-body FDG PET images　　　　　　　　　　　　　　　
Keisuke Kawauchi, Kenji Hirata, Seiya Ichikawa, Osamu Manabe, Kentaro Kobayashi, Shiro Watanabe, Miki Haseyama, Takahiro Ogawa, Ren Togo, Tohru Shiga, Chietsugu Katoh
Society of Nuclear Medicine and Molecular Imaging Annual Meeting (SNMMI), Jun. 2018, [Peer-reviewed]
English, International conference proceedings

Use of deep convolutional neural network-based features for detection of cardiac sarcoidosis from polar map　　　　　　　　　　　　　　　
Ren Togo, Kenji Hirata, Osamu Manabe, Hiroshi Ohira, Ichizo Tsujino, Takahiro Ogawa, Miki Haseyama, Tohru Shiga
Society of Nuclear Medicine and Molecular Imaging Annual Meeting (SNMMI), Jun. 2018, [Peer-reviewed]
English, International conference proceedings

Graph-Based Video Search Reranking with Local and Global Consistency Analysis
Soh Yoshida, Takahiro Ogawa, Miki Haseyama, Mitsuji Muneyasu
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E101D, 5, 1430, 1440, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, May 2018, [Peer-reviewed]
English, Scientific journal, Video reranking is an effective way for improving the retrieval performance of text-based video search engines. This paper proposes a graph-based Web video search reranking method with local and global consistency analysis. Generally, the graph-based reranking approach constructs a graph whose nodes and edges respectively correspond to videos and their pairwise similarities. A lot of reranking methods are built based on a scheme which regularizes the smoothness of pairwise relevance scores between adjacent nodes with regard to a user's query. However, since the overall consistency is measured by aggregating only the local consistency over each pair, errors in score estimation increase when noisy samples are included within query-relevant videos' neighbors. To deal with the noisy samples, the proposed method leverages the global consistency of the graph structure, which is different from the conventional methods. Specifically, in order to detect this consistency, the propose method introduces a spectral clustering algorithm which can detect video groups, in which videos have strong semantic correlation, on the graph. Furthermore, a new regularization term, which smooths ranking scores within the same group, is introduced to the reranking framework. Since the score regularization is performed by both local and global aspects simultaneously, the accurate score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

Anonymous Image Data Generation from Gastric X-ray Images for Improving Gastritis Recognition Performance
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Consumer Electronics – Taiwan (ICCE-TW), May 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents an anonymous gastritis image generation method for improving gastritis recognition performance. We realize the generation of realistic gastritis images by considering label information. Experimental results showed that anonymous images generated by our method had a potential for a gastritis recognition task. Concretely, the recognition performance of a classifier constructed with the anonymous images outperformed the performance of the conventional image generation method-based classifier.

Sentiment-aware personalized tweet recommendation through multimodal FFM
Ryosuke Harakawa, Daichi Takehara, Takahiro Ogawa, Miki Haseyama
Multimedia Tools and Applications, 77, 14, 1, 19, Springer New York LLC, 29 Mar. 2018, [Peer-reviewed]
English, Scientific journal, For realizing quick and accurate access to desired information and effective advertisements or election campaigns, personalized tweet recommendation is highly demanded. Since multimedia contents including tweets are tools for users to convey their sentiment, users’ interest in tweets is strongly influenced by sentiment factors. Therefore, successful personalized tweet recommendation can be realized if sentiment in tweets can be estimated. However, sentiment factors were not taken into account in previous works and the performance of previous methods may be limited. To overcome the limitation, a method for sentiment-aware personalized tweet recommendation through multimodal Field-aware Factorization Machines (FFM) is newly proposed in this paper. Successful personalized tweet recommendation becomes feasible through the following three contributions: (i) sentiment factors are newly introduced into personalized tweet recommendation, (ii) users’ interest is modeled by deriving multimodal FFM that enables collaborative use of multiple factors in a tweet, i.e., publisher, topic and sentiment factors, and (iii) the effectiveness of using sentiment factors as well as publisher and topic factors is clarified from results of experiments using real-world datasets related to worldwide hot topics, “#trump”, “#hillaryclinton” and “#ladygaga”. In addition to showing the effectiveness of the proposed method, the applicability of the proposed method to other tasks such as advertisement and social analysis is discussed as a conclusion and future work of this paper.

Image retrieval based on LRGA algorithm and relevance feedback for insect identification
Susumu Gerund, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 3978, 3982, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents an image retrieval method based on local regression and global alignment (LRGA) algorithm and relevance feedback for insect identification. Based on LRGA algorithm, the proposed method enables estimation of ranking scores for image retrieval in such a way that the neighborhood structure of the database can be optimally preserved. This is the biggest contribution of this paper. Then our method measures relevance between the query image and all the images in the database and realizes retrieval of images based on the measured relevance. Furthermore, if positively labeled images obtained by a user are available, they are used as the query relevance information for the relevance feedback to improve the retrieval results. Experimental results show the effectiveness of our method.

Automatic estimation of deterioration level on transmission towers via deep extreme learning machine based on local receptive field
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 2379, 2383, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents an automatic estimation method of deterioration levels on transmission towers via Deep Extreme Learning Machine based on Local Receptive Field (DELM-LRF). Although Convolutional Neural Network (CNN) requires a large number of training images, it is difficult to prepare a sufficient number of training images of transmission towers. Thus, we generate a novel estimation method which enables training from a small number of training images. Specifically, we automatically extract image features based on Local Receptive Field (LRF) which combines convolution and pooling without using hand-craft features and estimate deterioration levels via Deep Extreme Learning Machine (DELM), which is a part of efficient deep learning methods. The derivation of DELM-LRF is the biggest contribution of this paper, and it can be trained from less training images compared to CNN. Experimental results show the effectiveness of DELM-LRF for the estimation of deterioration levels on transmission towers. Consequently, the proposed method makes it possible to approach challenging tasks with high expertise having difficulty in preparing enough images.

Detection of gastric cancer risk from X-ray images via patch-based convolutional neural network
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 2055, 2059, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel detection method of gastric cancer risk from X-ray images using the patch-based Convolutional Neural Network (CNN). Our method enables the training of the patch-based CNN which can accurately detect gastric cancer risk even though there is only the image-level ground truth. Furthermore, the proposed method can extract a feature vector that can represent the whole of symptoms associated with the presence or absence of the risk. Specifically, the proposed method selects the patches related to their true risk via the CNN, and it is the most innovative contribution of our method. Moreover, we extract the feature vector by applying the Bag-of-Feature representation to the output values from the CNN's intermediate layer obtained from the selected patches. Finally, the detection of gastric cancer risk is performed by inputting the extracted feature vector into Support Vector Machine. Experimental results confirm that the proposed method outperforms a previously reported method that combines the detection results obtained from X-ray images taken from multiple angles even though the proposed method only uses an X-ray image taken from a single angle, and we can achieve a higher performance than that of doctors.

Tag refinement based on multilingual tag hierarchies extracted from image folksonomy
Shota Hamano, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 1327, 1331, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for tag refinement using multilingual sources of tagged images in an image folksonomy. The proposed method enables accurate tag refinement by effectively leveraging multilingual sources of tags and considering the hierarchical structure of tags in the following way. First, synonymous tags across different languages are detected based on similarities between tagged images. In this stage, the proposed method utilizes visual similarities to effectively detect synonymous tags since the visual features extracted from images should be similar if they are assigned tags with the same meaning in different languages. Then hierarchical structure of the tags are extracted based on the similarity between the detected synonymous tags. The hierarchical structure provides hypernymous and hyponymous tags of the target tags, which are important for considering the relevance between tags and images. Consulting the hierarchical structure enables removal of irrelevant tags from the images and assignment of relevant tags to the images. The proposed method effectively utilizes tags in various languages in an image folksonomy. Experimental results show the effectiveness of introducing multilingual sources of tagged images for accuracy improvement in tag refinement.

MvLFDA-based video preference estimation using complementary properties of features
Akira Toyoda, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 635, 639, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents a new method to estimate users' video preferences using complementary properties of features via Multiview Local Fisher Discriminant Analysis (MvLFDA). The proposed method first extracts multiple visual features from video frames and electroencephalogram (EEG) features from users' EEG signals recorded during watching video. Then we calculate EEG-based visual features by applying Locality Preserving Canonical Correlation Analysis (LPCCA) to each visual feature and EEG features. The EEG-based visual features reflect users' preferences since the correlation between visual features and EEG features which reflect users' preferences is maximized. Next, MvLFDA, which is newly derived in this paper, integrates multiple EEG-based visual features. Since MvLFDA explores complementary properties of different features, it can be expected that the features obtained by integrating multiple EEG-based visual features are more effective for users' preference estimation than each EEG-based visual feature. The biggest contribution of this paper is the new derivation of MvLFDA. Then successful estimation of users' video preferences becomes feasible using features obtained by MvLFDA.

Automatic martian dust storm detection via decision level fusion basedondeep extreme learning machine
Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2017-, 435, 439, IEEE Computer Society, 20 Feb. 2018, [Peer-reviewed]
English, International conference proceedings, This paper presents an automatic Martian dust storm detection via decision level fusion (DLF) based on deep extreme learning machine (DELM). Since Martian images are taken in multi-wavelength bands, DLF techniques which output a final classification result by integrating multiple classification results are necessary. Furthermore, since the number of Martian images taken by satellites is different for each region, the number of the classification results to be integrated is different. Thus, we present a new DLF framework based on confidence values of the classification results. Specifically, we generate multiple extreme learning machines with kernel classifiers to obtain their classification results. Moreover, we monitor the classification results as confidence values and select the same number of the classification results with high confidence for each region. Finally, these selected results can be integrated by using a DLF based on DELM, which is a multilayered ELM. This integration framework is the biggest contribution of our method. Experimental results show the effectiveness of the DLF based on DELM.

Preliminary study of automatic gastric cancer risk classification from photofluorography
Ren Togo, Kenta Ishihara, Katsuhiro Mabe, Harufumi Oizumi, Takahiro Ogawa, Mototsugu Kato, Naoya Sakamoto, Shigemi Nakajima, Masahiro Asaka, Miki Haseyama
World Journal of Gastrointestinal Oncology, 10, 2, 62, 70, Baishideng Publishing Group Co, 15 Feb. 2018, [Peer-reviewed]
English, Scientific journal, AIM To perform automatic gastric cancer risk classification using photofluorography for realizing effective mass screening as a preliminary study. METHODS We used data for 2100 subjects including X-ray images, pepsinogen ? and ? levels, PG?/PG? ratio, Helicobacter pylori (H. pylori ) antibody, H. pylori eradication history and interview sheets. We performed two-stage classification with our system. In the first stage, H. pylori infection status classification was performed, and H. pylori -infected subjects were automatically detected. In the second stage, we performed atrophic level classification to validate the effectiveness of our system. RESULTS Sensitivity, specificity and Youden index (YI) of H. pylori infection status classification were 0.884, 0.895 and 0.779, respectively, in the first stage. In the second stage, sensitivity, specificity and YI of atrophic level classification for H. pylori -infected subjects were 0.777, 0.824 and 0.601, respectively. CONCLUSION Although further improvements of the system are needed, experimental results indicated the effectiveness of machine learning techniques for estimation of gastric cancer risk.

A Novel Framework for Estimating Viewer Interest by Unsupervised Multimodal Anomaly Detection
Yuma Sasaka, Takahiro Ogawa, Miki Haseyama
IEEE Access, 6, 8340, 8350, Institute of Electrical and Electronics Engineers Inc., 09 Feb. 2018, [Peer-reviewed]
English, Scientific journal, A reliable method to estimate viewer interest is highly sought after for human-centered video information retrieval. A method that estimates viewer interest while users are watching Web videos is presented in this paper. The method uses a framework for anomaly detection based on collaborative use of facial expression and biological signals such as electroencephalogram (EEG) signals. To the best of our knowledge, there have been no studies that have taken into account two actual mechanisms of the behavior of users while they are watching Web videos. First, whereas most Web videos garner very little attention, a small number attract millions of views. Therefore, a framework for anomaly detection is newly applied to facial expression and EEG in order to model the imbalanced distribution of popularity. Second, since the number of Web videos that are labeled by users as interesting/not interesting is generally too small to estimate viewer interest by a supervised approach, the proposed method utilizes parametric techniques for anomaly detection, which estimates viewer interest in an unsupervised way. Unlike some related studies for estimating viewer interest, our method takes into account actual mechanisms of the behavior of users while they are watching Web videos by utilizing parametric techniques for anomaly detection. Then viewer interest can be estimated on the basis of an anomaly score calculated from our proposed method. Consequently, successful estimation of viewer interest based on a framework for anomaly detection, via collaborative use of facial expression and biological signals, becomes feasible.

Accurate estimation of personalized video preference using multiple users' viewing behavior
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
IEICE Transactions on Information and Systems, E101D, 2, 481, 490, Institute of Electronics, Information and Communication, Engineers, IEICE, 01 Feb. 2018, [Peer-reviewed]
English, Scientific journal, A method for accurate estimation of personalized video preference using multiple users' viewing behavior is presented in this paper. The proposed method uses three kinds of features: a video, user's viewing behavior and evaluation scores for the video given by a target user. First, the proposed method applies Supervised Multiview Spectral Embedding (SMSE) to obtain lower-dimensional video features suitable for the following correlation analysis. Next, supervised Multi-View Canonical Correlation Analysis (sMVCCA) is applied to integrate the three kinds of features. Then we can get optimal projections to obtain new visual features, "canonical video features" reflecting the target user's individual preference for a video based on sMVCCA. Furthermore, in our method, we use not only the target user's viewing behavior but also other users' viewing behavior for obtaining the optimal canonical video features of the target user. This unique approach is the biggest contribution of this paper. Finally, by integrating these canonical video features, Support Vector Ordinal Regression with Implicit Constraints (SVORIM) is trained in our method. Consequently, the target user's preference for a video can be estimated by using the trained SVORIM. Experimental results show the effectiveness of our method.

Tourism Category Classification on Image Sharing Services Through Estimation of Existence of Reliable Results.
Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, ICMR 2018, Yokohama, Japan, June 11-14, 2018, 493, 496, ACM, 2018, [Peer-reviewed]
International conference proceedings, A new tourism category classification method through estimation of existence of reliable classification results is presented in this paper. The proposed method obtains two kinds of classification results by applying a convolutional neural network to tourism images and applying a Fuzzy K-nearest neighbor algorithm to geotags attached to the tourism images. Then the proposed method estimates existence of reliable classification results in the above two results. If the reliable result is included, the result is selected as the final classification result. If any reliable result is not included, the final result is obtained by another approach based on a multiple annotator logistic regression model. Consequently, the proposed method enables accurate classification based on the new estimation scheme.

A Human-Centered Neural Network Model with Discriminative Locality Preserving Canonical Correlation Analysis for Image Classification.
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Image Processing, ICIP 2018, Athens, Greece, October 7-10, 2018, 2366, 2370, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a human-centered neural network model with discriminative locality preserving canonical correlation analysis (DLPCCA) for image classification. Although construction of multiple hidden layers adopted in recent deep learning methods is effective for extracting semantic features, a large amount of training images is required. In order to extract effective features for image classification successfully from a small amount of training images, the proposed method transforms visual features by using biological information obtained from image viewers as auxiliary information. The proposed method consists of two hidden layers. By constructing the first hidden layer, which can maximize canonical correlation between visual features and features based on biological information, the effective feature transformation can be realized. Specifically, the proposed method uses DLPCCA, which considers label information and preserves local structures. The second hidden layer constructed based on Extreme Learning Machine (ELM) enables classification. Consequently, since the first hidden layer performs the effective feature transformation, the proposed neural network model realizes accurate image classification from a quite small amount of training images.

Anonymous Gastritis Image Generation via Adversarial Learning from Gastric X-Ray Images.
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Image Processing, ICIP 2018, Athens, Greece, October 7-10, 2018, 2082, 2086, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents an anonymous gastritis image generation method based on a generative adversarial network approach. Since clinical individual data include highly confidential information, they must be handled carefully. Although data sharing is demanded to construct large-scale medical image datasets for deep learning-based recognition tasks, managing and annotating these data have been conducted manually. The proposed method enables the generation of anonymous images by an adversarial learning approach. Experimental results show that generated images by our method contribute to a gastritis recognition task. This will be helpful for constructing large-scale medical image datasets effectively.

Tracking topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery.
Array,Takahiro Ogawa, Miki Haseyama
Multimedia Tools Appl., 77, 16, 20297, 20324, 2018, [Peer-reviewed]
Scientific journal, A method to track topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery is presented in this paper. The proposed method enables users to understand the evolution of topics over time for discovering Web videos in which they are interested. A framework that enables extraction and tracking of the hierarchical structure, which contains Web video groups with various degrees of semantic broadness, is newly derived as follows: Based on network analysis using multimodal features, i.e., features of video contents and metadata, our method extracts the hierarchical structure and salient keywords that represent contents of each Web video group. Moreover, salient keyword matching, which is newly developed by considering salient keyword distribution, semantic broadness of each Web video group and initial topic relevance, is applied to each hierarchical structure obtained in different time stamps. Unlike methods in previous works, by considering the semantic broadness as well as the salient keyword distribution, our method can overcome the problem of the desired semantic broadness of topics being different depending on each user. Also, the initial topic relevance enables correction of the gap from an initial topic at the start of tracking. Consequently, it becomes feasible to track the evolution of topics over time for finding Web videos in which the users are interested. Experimental results for real-world datasets containing YouTube videos verify the effectiveness of the proposed method.

Estimation of Deterioration Levels of Transmission Towers via Deep Learning Maximizing Canonical Correlation Between Heterogeneous Features.
Array,Sho Takahashi, Array,Array
J. Sel. Topics Signal Processing, 12, 4, 633, 644, 2018, [Peer-reviewed]
Scientific journal, This paper presents estimation of deterioration levels of transmission towers via deep learning maximizing the canonical correlation between heterogeneous features. In the proposed method, we newly construct a correlation-maximizing deep extreme learning machine (CMDELM) based on a local receptive field (LRF). For accurate deterioration level estimation, it is necessary to obtain semantic information that effectively represents deterioration levels. However, since the amount of training data for transmission towers is small, it is difficult to perform feature transformation by using many hidden layers such as general deep learning methods. In CMDELM-LRF, one hidden layer, which maximizes the canonical correlation between visual features and text features obtained from inspection text data, is newly inserted. Specifically, by using projections obtained by maximizing the canonical correlation as weight parameters of the hidden layer, feature transformation for extracting semantic information is realized without designing many hidden layers. This is the main contribution of this paper. Consequently, CMDELM-LRF realizes accurate deterioration level estimation from a small amount of training data.

Estimating the Quality of Fractal Compressed Images Using Lacunarity.
Megumi Takezawa, Hirofumi Sanada, Takahiro Ogawa, Miki Haseyama
IEICE Transactions, 101-A, 6, 900, 903, 2018, [Peer-reviewed]
Scientific journal, In this paper, we propose a highly accurate method for estimating the quality of images compressed using fractal image compression. Using an iterated function system, fractal image compression compresses images by exploiting their self-similarity, thereby achieving high levels of performance; however, we cannot always use fractal image compression as a standard compression technique because some compressed images are of low quality. Generally, sufficient time is required for encoding and decoding an image before it can be determined whether the compressed image is of low quality or not. Therefore, in our previous study, we proposed a method to estimate the quality of images compressed using fractal image compression. Our previous method estimated the quality using image features of a given image without actually encoding and decoding the image, thereby providing an estimate rather quickly; however, estimation accuracy was not entirely sufficient. Therefore, in this paper, we extend our previously proposed method for improving estimation accuracy. Our improved method adopts a new image feature, namely lacunarity. Results of simulation showed that the proposed method achieves higher levels of accuracy than those of our previous method.

Selection of Significant Brain Regions Based on MvGTDA and TS-DLF for Emotion Estimation.
Array,Array,Miki Haseyama
IEEE Access, 6, 32481, 32492, 2018, [Peer-reviewed]
Scientific journal, In this paper, we propose a novel method for estimating human emotion using functional brain images. The final goal of our study is contribution to affective brain computer interfaces (aBCIs), which use neuropsychological signals. In the proposed method, we newly derive multiview general tensor discriminant analysis (MvGTDA) in order to reveal significant brain regions and accurately estimate human emotion evoked by visual stimuli. This is because it is important to find activation of multiple brain regions for estimating emotional states. Since we regard a 'Brodmann area' as a 'view' and introduce $L-{1}$-norm regularization for these views, MvGTDA can eliminate non-crucial Brodmann areas and select significant ones. Moreover, in general studies on functional brain images based on machine learning methodologies, there is an overfitting problem caused by a small sample size. Therefore, revealing significant Brodmann areas based on MvGTDA has another important role, i.e., solving the overfitting problem. By inputting estimation results respectively obtained from the significant areas and the MvGTDA-based feature, tensor-based supervised decision-level fusion (TS-DLF) integrates them and outputs the final estimation result of the user's emotion. In experiments, we showed the effectiveness of our method by using actual functional brain images and we revealed the significant brain regions in emotional states.

Simultaneous Realization of Multiple Music Video Applications Based on Heterogeneous Network Analysis Via Latent Link Estimation.
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Multimedia and Expo, ICME 2018, San Diego, CA, USA, July 23-27, 2018, 2018-July, 1, 6, IEEE Computer Society, 2018, [Peer-reviewed]
International conference proceedings, To help users seek desired music videos and create attractive music videos, many methods that realize applications such as music video recommendation, captioning and generation have been proposed. In this paper, a novel method that realizes these applications simultaneously on the basis of heterogeneous network analysis via latent link estimation is proposed. To the best of our knowledge, this work is the first attempt to realize music video recommendation, captioning and generation simultaneously. The proposed method enables latent link estimation with consideration of multimodal information and multiple social metadata obtained from music videos via Laplacian multiset canonical correlation analysis. Thus, it becomes feasible to construct a heterogeneous network that enables direct comparison of audio, visual and textual information of music videos and user information on the same feature space. Furthermore, link prediction on the obtained heterogeneous network enables association with (i) user information and their desired audio information; (ii) audio information and textual information that describes contents of musical pieces; and (iii) audio information and visual information that represents contents of musical pieces visually. As a result, support for (i) music video recommendation; (ii) captioning; and (iii) generation becomes feasible, respectively. Experimental results for a real-world dataset constructed by using YouTube-8M show the effectiveness of the proposed method.

Interest Level Estimation of Items via Matrix Completion Based on Adaptive User Matrix Construction.
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Multimedia and Expo, ICME 2018, San Diego, CA, USA, July 23-27, 2018, 2018-July, 1, 6, IEEE Computer Society, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a novel method for interest level estimation of items via matrix completion based on adaptive user matrix construction. The proposed method introduces a new criterion for adaptively constructing a user matrix that consists of user behavior features and interest levels, which are evaluated by target users and similar users. In the estimation, the matrix completion via rank minimization using the truncated nuclear norm is applied to the constructed matrix. The proposed method enables both of the interest level estimation of the target users and the selection of the similar users suitable for the estimation by monitoring errors caused in the matrix completion algorithm. The caused errors indicate the minimum differences between the estimated interest levels and true ones, and they can be regarded as the criterion for both of the optimal estimation and the adaptive selection. Furthermore, the proposed method uses weight matrices for decreasing an influence of missing data on the estimation. Consequently, accurate estimation of the interest levels becomes feasible by using the adaptively constructed matrix. Experimental results obtained by applying the proposed method to users' behavior and interest data show the effectiveness of the proposed method.

Sfemcca: Supervised Fractional-Order Embedding Multiview Canonical Correlation Analysis for Video Preference Estimation.
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, 2018-April, 3086, 3090, IEEE, 2018, [Peer-reviewed]
International conference proceedings, In this paper, we present supervised fractional-order embedding multiview canonical correlation analysis (SFEMCCA). SFEMCCA is a CCA method realizing the following three points: (1) learning noisy data with small number of samples and large number of dimensions, (2) multiview learning that can integrate three or more kinds of features, and (3) supervised learning using labels corresponding to the samples. In real data, it is necessary to deal with high dimensional noisy data with limited number of samples, and there are many cases where three or more kinds of multimodal and supervised data are treated in order to calculate more accurate projections. Therefore, SFEMCCA, which takes the above advantages (1)-(3) into account, is effective for data obtained from real environments. From experimental results, it was confirmed that accuracy improvements using SFEMCCA were statistically significant compared to the several conventional methods of supervised multiview CCA.

Semi-Supervised Multiple Feature Fusion for Video Preference Estimation.
Akira Toyoda, Takahiro Ogawa, Miki Haseyama
2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, 2018-April, 891, 895, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a new method to estimate user preferences for videos based on multiple feature fusion via semi-supervised Multiview Local Fisher Discriminant Analysis (sMvLFDA). The proposed method first extracts multiple visual features from videos and functional near-infrared spectroscopy (fNIRS) features from fNIRS signals recorded during watching videos. Next, we apply Locality Preserving Canonical Correlation Analysis (LPCCA) to each visual feature and fNIRS features and project each visual feature to the new feature spaces (fNIRS-based visual feature spaces). Consequently, since the correlation between each visual feature and fNIRS features which reflect user preferences is maximized, we can transform visual features into features which also reflect user preferences. In addition, we newly introduce sMvLFDA and fuse multiple fNIRS-based visual features via sMvLFDA. sMvLFDA fuses features while using labeled samples and unlabeled samples simultaneously to reduce overfitting to the labeled samples. Furthermore, sMvLFDA adequately uses complementary properties in multiple features. Therefore, it can be expected that the fused features are more effective for estimation of user preferences than each fNIRS-based visual feature. The main contribution of this paper is the new derivation of sMvLFDA. Consequently, by using the fused features, it becomes feasible to estimate user preferences for videos successfully.

Binary Sparse Representation Based on Arbitrary Quality Metrics and Its Applications.
Takahiro Ogawa, Sho Takahashi, Naofumi Wada, Akira Tanaka, Miki Haseyama
IEICE Transactions, 101-A, 11, 1776, 1785, 2018, [Peer-reviewed]
Scientific journal, Binary sparse representation based on arbitrary quality metrics and its applications are presented in this paper. The novelties of the proposed method are twofold. First, the proposed method newly derives sparse representation for which representation coefficients are binary values, and this enables selection of arbitrary image quality metrics. This new sparse representation can generate quality metric-independent subspaces with simplification of the calculation procedures. Second, visual saliency is used in the proposed method for pooling the quality values obtained for all of the parts within target images. This approach enables visually pleasant approximation of the target images more successfully. By introducing the above two novel approaches, successful image approximation considering human perception becomes feasible. Since the proposed method can provide lower-dimensional subspaces that are obtained by better image quality metrics, realization of several image reconstruction tasks can be expected. Experimental results showed high performance of the proposed method in terms of two image reconstruction tasks, image inpainting and super-resolution.

Favorite Video Estimation Based on Multiview Feature Integration via KMvLFDA.
Array,Array,Miki Haseyama
IEEE Access, 6, 63833, 63842, 2018, [Peer-reviewed]
Scientific journal, This paper presents a novel method for favorite video estimation based on multiview feature integration via kernel multiview local fisher discriminant analysis (KMvLFDA). The proposed method first extracts electroencephalogram (EEG) features from users' EEG signals recorded while watching videos and multiple visual features from videos. Then, multiple EEG-based visual features are obtained by applying locality preserving canonical correlation analysis to EEG features and each visual feature. Next, KMvLFDA, which is newly derived in this paper, explores the complementary properties of different features and integrates the multiple EEG-based visual features. In addition, by using KMvLFDA, between-class scatter is maximized and within-class scatter is minimized in the integrated feature space. Consequently, it can be expected that the new features that are obtained by the above integration are more effective than each of the EEG-based visual features for the estimation of users' favorite videos. The main contribution of this paper is the new derivation of KMvLFDA. Successful estimation of users' favorite videos becomes feasible by using the new features obtained via KMvLFDA.

Favorite Video Classification Based on Multimodal Bidirectional LSTM.
Array,Array, Keisuke Maeda, Miki Haseyama
IEEE Access, 6, 61401, 61409, 2018, [Peer-reviewed]
Scientific journal, Video classification based on the user's preference (information of what a user likes: WUL) is important for realizing human-centered video retrieval. A better understanding of the rationale of WUL would greatly contribute to the support for successful video retrieval. However, a few studies have shown the relationship between information of what a user watches and WUL. A new method that classifies videos on the basis of WUL using video features and electroencephalogram (EEG) signals collaboratively with a multimodal bidirectional Long Short-Term Memory (Bi-LSTM) network is presented in this paper. To the best of our knowledge, there has been no study on WUL-based video classification using video features and EEG signals collaboratively with LSTM. First, we newly apply transfer learning to the WUL-based video classification since the number of labels (liked or not liked) attached to videos by users is small, and it is difficult to classify videos based on WUL. Furthermore, we conduct a user study for showing that the representation of psychophysiological signals calculated from Bi-LSTM is effective for the WUL-based video classification. Experimental results showed that our deep neural network feature representations can distinguish WUL for each subject.

Estimation of Important Scenes in Soccer Videos Based on Collaborative Use of Audio-Visual CNN Features.
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 710, 711, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a novel method for estimating important scenes in soccer videos based on collaborative use of audio-visual Convolutional Neural Network (CNN) features. In soccer games, since game situations influence not only players' movements but also audiences' cheers, analyses of their audio and visual sequences are useful for the estimation of important scenes. In our method, such scenes are estimated from audio and visual CNN features via support vector machine (SVM) in each feature. Furthermore, by applying weighted majority voting based on confidences defined from the SVM-based estimation results, accurate estimation of important scenes becomes feasible. Experimental results show the effectiveness of our method.

Twitter Followee Recommendation Based on Multimodal FFM Considering Social Relations.
Shoji Takimura, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 204, 205, IEEE, 2018, [Peer-reviewed]
International conference proceedings, A method for Twitter followee recommendation based on multimodal field-aware factorization machines considering social relations (MFFM-SR) is presented. MFFM-SR enables collaborative use of textual and visual features and social relations unlike conventional methods. Specifically, for distinguishing users' interest, visual features are extracted from images in their tweets and icons as well as textual features and social relations. Furthermore, to construct a model that accurately represents users' interest, MFFM-SR that enables calculation of high-level features via estimation of latent relationships among the obtained features and social relations is derived. By using the constructed model, successful followee recommendation becomes feasible.

Estimation of Viewed Image Categories via CCA Using Human Brain Activity.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 202, 203, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a method to estimate viewed image categories via canonical correlation analysis (CCA) using human brain activity measured by functional magnetic resonance imaging (fMRI). The proposed method enables estimation of image categories that a subject viewed by using only the subject's brain activity. Specifically, the proposed method calculates the projection matrices that enable direct comparison between human brain activity and images that subjects viewed through CCA. After projecting the human brain activity and the viewed images on the same latent space, k-Nearest Neighbor (k-NN) is performed to estimate the viewed image categories from only human brain activity. Through the projection matrices, the proposed method can increase training data for k-NN even if a large number of pairs of human brain activity and images cannot be prepared. Experimental results for ten subjects show the effectiveness of the proposed method.

Missing Image Data Estimation via Iterative Convolutional Sparse Coding.
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 200, 201, IEEE, 2018, [Peer-reviewed]
International conference proceedings, A novel method for missing image data estimation is presented in this paper. The proposed method realizes accurate estimation of missing image data by iterating dictionary learning and Convolutional Sparse Coding (CSC). Specifically, our method iterates estimation of missing image data via CSC by using a dictionary that is constructed from a target image, and reconstruction of the dictionary by using the obtained estimation results. As the main contribution of our paper, the proposed method enables the missing image data estimation by using more suitable dictionaries obtained by this iterative scheme. Experimental results show high missing image data estimation performance by the proposed method.

Image Retrieval from Vague Description Based on AttnGAN
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2018 IEEE 7TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE 2018), 198, 199, IEEE, 2018, [Peer-reviewed]
English, International conference proceedings, Image retrieval plays an important role in the information society. Many studies have been conducted to improve accuracy of the image retrieval. However, there exists a major limitation in their input methods. For example, if users only have a vague description that does not include detailed information such as its name and do not have an appropriate input image, it is difficult to retrieve their desired images. To solve this problem, we propose a novel image retrieval method that enables retrieval of a desired image from a vague description. In the proposed method, we generate a query image from a vague description through an Attentional Generative Adversarial Network. By using the generated query image, the proposed method enables users to retrieve images even if they do not have a clear retrieval description as an input. Experimental results show the effectiveness of our method.

Detection of Deformed PPG Waveforms Based on a Ratio of Their Average Accelerations.
Ken Kawakami, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 196, 197, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a new method to detect deformed Photoplethysmogram (PPG) waveforms for sufficient accuracy of signal processing. The PPG waveforms have been applied in many health indicators, such as blood pressure, blood viscosity and blood vessel elasticity. Usually, the measurements using a sensitive signals require user awareness so that all the PPG waveforms are kept accurate. Namely, accuracy of the calculated indicators becomes lower when the PPG waveform is deformed due to motion artifacts. In particular, detection methods of deformed PPG waveforms are important for incorporating the health indicators into general fitness trackers to find the correct waveform or to remove deformed PPG waveforms from the measurement. Therefore, we propose a new method which detects a badly formed PPG waveform by monitoring a ratio of average accelerations. Experimental results show the effectiveness of the method for detecting a deformed PPG waveform.

Blood Circulation Based on PPG Signals for Thermal Comfort Evaluation.
Ken Kawakami, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 194, 195, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a new index for monitoring transition of blood circulation from Photoplethysmogram (PPG) signals for thermal comfort evaluation of users. Heat dissipation reaction through the dilation of blood vessels is person's intrinsic ability to control the thermal comfort. When body temperature is higher than normal temperature, blood circulation changes according to the dilation of blood vessels in distal end of the extremities. Blood circulation is often evaluated by an index of peripheral resistance corresponding to changes in blood flow velocity such as systolic/diastolic ratio S/D of flow velocities, resistance index and the pulsatility index. Unfortunately, such an index cannot be utilized in daily life for healthcare with using fitness trackers since the blood flow velocity is measured by either an ultrasonic Doppler blood flowmeter (UDF) or a Laser-Doppler flowmeter. Therefore, we propose a new index which is easily acquirable from PPG signals. First, a couple of variables correlating to the blood flow velocity is calculated from a rate of volumetric strain of Photoplethysmogram signals. Then the new index can be obtained as difference of these variables. Experimental results show the effectiveness of this index by confirming high correlation with S/D of UDF.

Team Tactics Estimation in Soccer Videos via Deep Extreme Learning Machine Based on Players Formation.
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 116, 117, IEEE, 2018, [Peer-reviewed]
International conference proceedings, A method of team tactics estimation in soccer videos is presented in this paper. Our method enables estimation of basic tactics in each team on the basis of the Deep-Extreme Learning Machine (DELM) by using features of players formation. In the soccer games, team tactics relate to each other team. Therefore, the proposed method obtains final estimation results by utilizing two DELMs of each team and their relationship. Since the proposed method takes into consideration the relevance of the estimated tactics in each team, we realize accurate tactics estibimation. Experimental results using actual soccer videos showed the effectiveness of our method.

User-centric Visual Attention Estimation Based on Relationship between Image and Eye Gaze Data.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, Nara, Japan, October 9-12, 2018, 73, 74, IEEE, 2018, [Peer-reviewed]
International conference proceedings, This paper presents a method for estimating user-centric visual attention based on the relationship between image and eye gaze data. The proposed method focuses on relationship between visual features calculated from images and saliency values calculated from eye gaze data. Specifically, our method calculates the saliency map of each training image by using individual eye gaze data obtained from only these images. Furthermore, from the pairs of visual features and the gaze-based saliency, the estimation of user-centric saliency from a new test image becomes feasible. Our contribution is the construction of a simple but successful estimation model which can train the relationship from limited amount of individual eye gaze data. Experimental results show the effectiveness of the proposed method.

A Language-Independent Ontology Construction Method Using Tagged Images in Folksonomy
Shota Hamano, Takahiro Ogawa, Miki Haseyama
IEEE Access, 6, 2930, 2942, Institute of Electrical and Electronics Engineers Inc., 21 Dec. 2017, [Peer-reviewed]
English, Scientific journal, This paper presents a language-independent ontology (LION) construction method that uses tagged images in an image folksonomy. Existing multilingual frameworks that construct an ontology deal with concepts translated on the basis of parallel corpora, which are not always available
however, the proposed method enables LION construction without parallel corpora by using visual features extracted from tagged images as the alternative. In the proposed method, visual similarities in tagged images are leveraged to aggregate synonymous concepts across languages. The aggregated concepts take on intrinsic semantics of themselves, while they also hold distinct characteristics in different languages. Then relationships between concepts are extracted on the basis of visual and textual features. The proposed method constructs a LION whose nodes and edges correspond to the aggregated concepts and relationships between them, respectively. The LION enables successful image retrieval across languages since each of the aggregated concepts can be referred to in different languages. Consequently, the proposed method removes the language barriers by providing an easy way to access a broader range of tagged images for users in the folksonomy, regardless of the language they use.

Video preference estimation using fNIRS signals
Akira Toyoda, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a new method for video preference estimation using functional near-infrared spectroscopy signals (fNIRS signals). The proposed method first computes fNIRS features from fNIRS signals recorded while users are watching videos and multiple visual features from these videos. Next, by applying Locality Preserving Canonical Correlation Analysis to fNIRS features and each visual feature, we can obtain multiple new visual features. In addition, Multiview Local Fisher Discriminant Analysis fuses multiple new visual features and optimizes within and between class scatter in the fused feature space while using complementary properties in these features. Consequently, we can realize video preference estimation by using the fused features.

Personalized tweet recommendation based on field-aware factorization machines with adaptive field organization
Shoji Takimura, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, A novel method for personalized tweet recommendation based on Field-aware Factorization Machines (FFMs) with adaptive field organization is presented in this paper. The proposed method realizes accurate recommendation of tweets in which users are interested by the following two contributions. First, sentiment factors such as opinions, thoughts and feelings included in tweets are newly introduced into FFMs in addition to their publisher and topic factors. Second, the proposed method newly enables adaptive organization of fields via canonical correlation analysis for multiple features extracted from each tweet. Experimental results for real-world datasets confirm the performance improvement of personalized tweet recommendation through the two contributions.

Construction of network using heterogeneous social metadata for music video recommendation
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, A novel method to construct a network based on heterogeneous features obtained from music videos and social metadata for music video recommendation is presented in this paper. The proposed method enables construction of the network that can accurately associate users with music videos corresponding to their preference by the collaborative use of audio and textual features obtained from music videos and social metadata 'related videos', 'tags', and 'keywords' through sub-sampled canonical correlation analysis. By performing link prediction on the obtained network, our method enables users to obtain desired music videos that are not linked to each other in the network but corresponding to users' preference, that is, music video recommendation becomes feasible. Experimental results for real-world datasets show the effectiveness of our method.

Interest level estimation based on matrix completion via rank minimization
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for interest level estimation based on matrix completion via rank minimization. The proposed method estimates interest levels of target objects from human behavior features which are extracted during selecting these objects. Specifically, by adopting matrix completion via rank minimization, unknown interest levels can be estimated. Furthermore, the proposed method can also estimate unknown interest levels with some missing behavior features which are not correctly extracted by sensors. Experimental results show the effectiveness of the proposed method.

Aesthetic quality assessment of images via Supervised Locality Preserving CCA
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, Aesthetic quality assessment plays an important role in how people organize large image collections. Many studies on aesthetic quality assessment are based on design of hand-crafted features without considering whether attributes conveyed by images can actually affect image aesthetics. This paper presents an aesthetic quality assessment method which uses new visual features. The proposed method utilizes Supervised Locality Preserving Canonical Correlation Analysis (SLPCCA) to derive the new features which maximize correlation between attributes and visual features. Finally, by applying ridge regression to the SLPCCA-based features, successful aesthetic quality assessment is realized.

Preference estimation for video recommendation using DCNN features and viewing behavior
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a personalized preference estimation method for video recommendation. Our method not only uses deep convolutional neural network (DCNN)-based video features but also transforms them based on user's viewing behavior in order to improve accuracy of preference estimation for a video. Specifically, we adopt supervised multi-view canonical correlation analysis (sMVCCA) in order to calculate 'canonical video features', which have a maximal correlation between the following three kinds of features: a video, user's viewing behavior and user's evaluation scores for the video. By using the canonical video features, our method can estimate the user's personalized preference for a video more accurately than using only the DCNN-based video features. Experimental results show the effectiveness of our method.

Image classification for trend prediction based on integration of fNIRS and visual features
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2017 IEEE 6th Global Conference on Consumer Electronics, GCCE 2017, 2017-, 1, 2, Institute of Electrical and Electronics Engineers Inc., 19 Dec. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method of image classification for trend prediction based on integration of visual and fNIRS features. It is expected that classification of images in the same object category in terms of generation enables trend prediction. However, since images in the same object category have similar visual features, a limit of accuracy exists for image classification by using only visual features. To overcome this problem, we utilize fNIRS features which represent brain activity in addition to visual features. Specifically, we apply Discriminative Locality Preserving Canonical Correlation Analysis (DLPCCA) to fNIRS and visual features for utilizing them collaboratively. The main contribution of this paper is the improvement of classification performance of images in the same object category for trend prediction by using the visual features projected to the DLPCCA-based space.

Wiener-Based Inpainting Quality Prediction
Takahiro Ogawa, Akira Tanaka, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 10, 2614, 2626, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Oct. 2017, [Peer-reviewed]
English, Scientific journal, A Wiener-based inpainting quality prediction method is presented in this paper. The proposed method is the first method that can predict inpainting quality both before and after the intensities have become missing even if their inpainting methods are unknown. Thus, when the target image does not include any missing areas, the proposed method estimates the importance of intensities for all pixels, and then we can know which areas should not be removed. Interestingly, since this measure can be also derived in the same manner for its corrupted image already including missing areas, the expected difficulty in reconstruction of these missing pixels is predicted, i.e., we can know which missing areas can be successfully reconstructed. The proposed method focuses on expected errors derived from the Wiener filter, which enables least-squares reconstruction, to predict the inpainting quality. The greatest advantage of the proposed method is that the same inpainting quality prediction scheme can be used in the above two different situations, and their results have common trends. Experimental results show that the inpainting quality predicted by the proposed method can be successfully used as a universal quality measure.

Extracting hierarchical structure of content groups from different social media platforms using multiple social metadata
Daichi Takehara, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
MULTIMEDIA TOOLS AND APPLICATIONS, 76, 19, 20249, 20272, SPRINGER, Oct. 2017, [Peer-reviewed]
English, Scientific journal, A novel scheme for retrieving users' desired contents, i.e., contents with topics in which users are interested, from multiple social media platforms is presented in this paper. In existing retrieval schemes, users first select a particular platform and then input a query into the search engine. If users do not specify suitable platforms for their information needs and do not input suitable queries corresponding to the desired contents, it becomes difficult for users to retrieve the desired contents. The proposed scheme extracts the hierarchical structure of content groups (sets of contents with similar topics) from different social media platforms, and it thus becomes feasible to retrieve desired contents even if users do not specify suitable platforms and do not input suitable queries. This paper has two contributions: (1) A new feature extraction method, Locality Preserving Canonical Correlation Analysis with multiple social metadata (LPCCA-MSM) that can detect content groups without the boundaries of different social media platforms is presented in this paper. LPCCA-MSM uses multiple social metadata as auxiliary information unlike conventional methods that only use content-based information such as textual or visual features. (2) The proposed novel retrieval scheme can realize hierarchical content structuralization from different social media platforms. The extracted hierarchical structure shows various abstraction levels of content groups and their hierarchical relationships, which can help users select topics related to the input query. To the best of our knowledge, an intensive study on such an application has not been conducted; therefore, this paper has strong novelty. To verify the effectiveness of the above contributions, extensive experiments for real-world datasets containing YouTube videos and Wikipedia articles were conducted.

Visualizing Web Images Using Fisher Discriminant Locality Preserving Canonical Correlation Analysis
Kohei Tateno, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 9, 2005, 2016, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Sep. 2017, [Peer-reviewed]
English, Scientific journal, A novel dimensionality reduction method, Fisher Discriminant Locality Preserving Canonical Correlation Analysis (FDLPCCA), for visualizing Web images is presented in this paper. FDLP-CCA can integrate two modalities and discriminate target items in terms of their semantics by considering unique characteristics of the two modalities. In this paper, we focus onWeb images with text uploaded on Social Networking Services for these two modalities. Specifically, text features have high discriminate power in terms of semantics. On the other hand, visual features of images give their perceptual relationships. In order to consider both of the above unique characteristics of these two modalities, FDLPCCA estimates the correlation between the text and visual features with consideration of the cluster structure based on the text features and the local structures based on the visual features. Thus, FDLP-CCA can integrate the different modalities and provide separated manifolds to organize enhanced compactness within each natural cluster.

Biomimetics Image Retrieval Platform
Miki Haseyama, Takahiro Ogawa, Sho Takahashi, Shuhei Nomura, Masatsugu Shimomura
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 8, 1563, 1573, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2017, [Peer-reviewed]
English, Scientific journal, Biomimetics is a new research field that creates innovation through the collaboration of different existing research fields. However, the collaboration, i.e., the exchange of deep knowledge between different research fields, is difficult for several reasons such as differences in technical terms used in different fields. In order to overcome this problem, we have developed a new retrieval platform, "Biomimetics image retrieval platform," using a visualization-based image retrieval technique. A biological database contains a large volume of image data, and by taking advantage of these image data, we are able to overcome limitations of text-only information retrieval. By realizing such a retrieval platform that does not depend on technical terms, individual biological databases of various species can be integrated. This will allow not only the use of data for the study of various species by researchers in different biological fields but also access for a wide range of researchers in fields ranging from materials science, mechanical engineering and manufacturing. Therefore, our platform provides a new path bridging different fields and will contribute to the development of biomimetics since it can overcome the limitation of the traditional retrieval platform.

Deterioration Level Estimation on Transmission Towers via Extreme Learning Machine based on Combination Use of Local Receptive Field and Principal Component Analysis　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), 457, 458, Jul. 2017, [Peer-reviewed]
English, International conference proceedings

Effectiveness Evaluation of Imaging Direction for Estimation of Gastritis Regions on Gastric X-ray Images　　　　　　　　　　　　　　　
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
International Technical Conference on Circuits, Systems, Computers, and Communications (ITC-CSCC), 459, 460, May 2017, [Peer-reviewed]
English, International conference proceedings

Helicobacter Pylori infection detection from gastric X-ray images based on feature fusion and decision fusion
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
COMPUTERS IN BIOLOGY AND MEDICINE, 84, 69, 78, PERGAMON-ELSEVIER SCIENCE LTD, May 2017, [Peer-reviewed]
English, Scientific journal, In this paper, a fully automatic method for detection of Helicobacter pylori (H. pylori) infection is presented with the aim of constructing a computer-aided diagnosis (CAD) system. In order to realize a CAD system with good performance for detection of H. pylori infection, we focus on the following characteristic of stomach X-ray examination. The accuracy of X-ray examination differs depending on the symptom of H. pylori infection that is focused on and the position from which X-ray images are taken. Therefore, doctors have to comprehensively assess the symptoms and positions. In order to introduce the idea of doctors' assessment into the CAD system, we newly propose a method for detection of H. pylori infection based on the combined use of feature fusion and decision fusion. As a feature fusion scheme, we adopt Multiple Kernel Learning (MKL). Since MKL can combine several features with determination of their weights, it can represent the differences in symptoms. By constructing an MKL classifier for each position, we can obtain several detection results. Furthermore, we introduce confidence-based decision fusion, which can consider the relationship between the classifier's performance and the detection results. Consequently, accurate detection of H. pylori infection becomes possible by the proposed method. Experimental results obtained by applying the proposed method to real X-ray images show that our method has good performance, close to the results of detection by specialists, and indicate that the realization of a CAD system for determining the risk of H. pylori infection is possible.

Tracking hierarchical structure of web video groups based on salient keyword matching including semantic broadness estimation
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2016 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2016 - Proceedings, 1238, 1242, Institute of Electrical and Electronics Engineers Inc., 19 Apr. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method to track the hierarchical structure of Web video groups on the basis of salient keyword matching including semantic broadness estimation. To the best of our knowledge, this paper is the first work to perform extraction and tracking of the hierarchical structure simultaneously. Specifically, the proposed method first extracts the hierarchical structure of Web video groups and salient keywords of them on the basis of an improved scheme of our previously reported method. Moreover, to calculate similarities between Web video groups obtained in different time stamps, salient keyword matching is newly developed by considering both co-occurrences of the salient keywords and semantic broadness of each Web video group. Consequently, tracking of the hierarchical structure over time becomes feasible to easily understand popularity trends of many Web videos for realizing effective retrieval.

Distress Classification of Class Imbalanced Data for Maintenance Inspection of Road Structures in Express Way　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Conference on Civil and Building Engineering Informatics in conjunction with Conference on Computer Applications in Civil and Hydraulic Engineering (ICCBEI & CCACHE), 182, 185, Apr. 2017, [Peer-reviewed]
English, International conference proceedings

Human-Centered Video Feature Selection via mRMR-SCMMCCA for Preference Extraction
Takahiro Ogawa, Yoshiaki Yamaguchi, Satoshi Asamizu, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 2, 409, 412, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Feb. 2017, [Peer-reviewed]
English, Scientific journal, This paper presents human-centered video feature selection via mRMR-SCMMCCA (minimum Redundancy and Maximum Relevance-Specific Correlation Maximization Multiset Canonical Correlation Analysis) algorithm for preference extraction. The proposed method derives SCMMCCA, which simultaneously maximizes two kinds of correlations, correlation between video features and users' viewing behavior features and correlation between video features and their corresponding rating scores. By monitoring the derived correlations, the selection of the optimal video features that represent users' individual preference becomes feasible.

Human-Centered Video Feature Selection via mRMR-SCMMCCA for Preference Extraction
OGAWA Takahiro, YAMAGUCHI Yoshiaki, ASAMIZU Satoshi, HASEYAMA Miki
IEICE Transactions on Information and Systems, E100.D, 2, 409, 412, The Institute of Electronics, Information and Communication Engineers, 2017
English, This paper presents human-centered video feature selection via mRMR-SCMMCCA (minimum Redundancy and Maximum Relevance-Specific Correlation Maximization Multiset Canonical Correlation Analysis) algorithm for preference extraction. The proposed method derives SCMMCCA, which simultaneously maximizes two kinds of correlations, correlation between video features and users' viewing behavior features and correlation between video features and their corresponding rating scores. By monitoring the derived correlations, the selection of the optimal video features that represent users' individual preference becomes feasible.

Extracting Hierarchical Structure of Web Video Groups Based on Sentiment-Aware Signed Network Analysis
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE ACCESS, 5, 16963, 16973, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2017, [Peer-reviewed]
English, Scientific journal, Sentiment in multimedia contents has an influence on their topics, since multimedia contents are tools for social media users to convey their sentiment. Performance of applications such as retrieval and recommendation will be improved if sentiment in multimedia contents can be estimated; however, there have been few works in which such applications were realized by utilizing sentiment analysis. In this paper, a novel method for extracting the hierarchical structure of Web video groups based on sentiment-aware signed network analysis is presented to realize Web video retrieval. First, the proposed method estimates latent links between Web videos by using multimodalfeatures of contents and sentiment features obtained from texts attached to Web videos. Thus, our method enables construction of a signed network that reflects not only similarities but also positive and negative relations between topics of Web videos. Moreover, an algorithm to optimize a modularity-based measure, which can adaptively adjust the balance between positive and negative edges, was newly developed. This algorithm detects Web video groups with similar topics at multiple abstraction levels; thus, successful extraction of the hierarchical structure becomes feasible. By providing the hierarchical structure, users can obtain an overview of many Web videos and it becomes feasible to successfully retrieve the desired Web videos. Results of experiments using a new benchmark dataset, YouTube-8M, validate the contributions of this paper, i.e., 1) the first attempt to utilize sentiment analysis for Web video grouping and 2) a novel algorithm for analyzing a weighted signed network derived from sentiment and multimodal features.

PERSONALIZED VIDEO PREFERENCE ESTIMATION BASED ON EARLY FUSION USING MULTIPLE USERS'VIEWING BEHAVIOR
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 3006, 3010, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for personalized video preference estimation based on early fusion using multiple users' viewing behavior. The proposed method adopts supervised Multi-View Canonical Correlation Analysis (sMVCCA) to estimate correlation between different types of features. Specifically, we estimate optimal projections maximizing the correlation between three features of video, target user's viewing behavior and evaluation scores for video. Then novel video features (canonical video features), which reflect the target user's individual preference, are obtained by the estimated projections. Furthermore, our method computes sMVCCA-based canonical video features by using multiple users' viewing behavior and a target user's evaluation scores. This non-conventional approach using the multiple users' viewing behavior for the preference estimation of the target user is the biggest contribution of our method, and it enables early fusion of the canonical video features. Consequently, successful video recommendation that reflects the users' individual preference can be expected via the evaluation score prediction from the integrated canonical video features. Experimental results show the effectiveness of our method.

EXEMPLAR-BASED IMAGE COMPLETION VIA NEW QUALITY MEASURE BASED ON PHASELESS TEXTURE FEATURES
Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 1827, 1831, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents an exemplar-based image completion via a new quality measure based on phaseless texture features. The proposed method derives a new quality measure obtained by monitoring errors caused in power spectra, i.e., errors of phaseless texture features, converged through phase retrieval. Even if a target patch includes missing pixels, this measure enables selection of the best matched patch including the most similar texture features for realizing the exemplar-based image completion. Furthermore, since the phaseless texture features are robust to various changes such as spatial gaps and luminance changes, the new quality measure successfully provides the best matched patch from few training examples. Then, by solving an optimization problem that retrieves the phase of the target patch from the phaseless texture features of the best matched patch, its missing areas can be reconstructed. Consequently, accurate image completion using the new quality measure becomes feasible. Subjective and quantitative experimental results are shown to verify the effectiveness of our method using the new quality measure.

EMOTION ESTIMATION VIA TENSOR-BASED SUPERVISED DECISION-LEVEL FUSION FROM MULTIPLE BRODMANN AREAS
Kento Sugata, Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 999, 1003, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method that estimates human emotion based on tensor-based supervised decision-level fusion (TS-DLF) from multiple Brodmann areas (BAs). From multiple brain data corresponding to these BAs captured by functional magnetic resonance imaging (fMRI), our method performs general tensor discriminant analysis (GTDA) to obtain features which can reflect the user's emotion. Furthermore, since the dimension of the obtained features becomes lower, this can avoid overfitting in the following training procedure of estimators. Next, by separately using the transformed BA data obtained after GTDA, we obtain multiple estimation results of the user's emotion based on logistic tensor regression (LTR). Then our method realizes the decision of the final result based on TS-DLF from the multiple estimation results. This approach, i.e., the integration of the multiple BAs' results for the whole-brain data, is the biggest contribution of this paper. TS-DLF successfully integrates the multiple estimation results with considering the performance of the LTR-based estimator constructed for each BA. Experimental results show that our method outperforms state-of-the-art approaches, and the effectiveness of our method can be confirmed.

Adaptive Subspace-Based Inverse Projections via Division Into Multiple Sub-Problems for Missing Image Data Restoration
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON IMAGE PROCESSING, 25, 12, 5971, 5986, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Dec. 2016, [Peer-reviewed]
English, Scientific journal, This paper presents adaptive subspace-based inverse projections via division into multiple sub-problems (ASIP-DIMSs) for missing image data restoration. In the proposed method, a target problem for estimating missing image data is divided into multiple sub-problems, and each sub-problem is iteratively solved with the constraints of other known image data. By projection into a subspace model of image patches, the solution of each sub-problem is calculated, where we call this procedure "subspace-based inverse projection" for simplicity. The proposed method can use higher dimensional subspaces for finding unique solutions in each sub-problem, and successful restoration becomes feasible, since a high level of image representation performance can be preserved. This is the biggest contribution of this paper. Furthermore, the proposed method generates several subspaces from known training examples and enables derivation of a new criterion in the above framework to adaptively select the optimal subspace for each target patch. In this way, the proposed method realizes missing image data restoration using ASIP-DIMS. Since our method can estimate any kind of missing image data, its potential in two image restoration tasks, image inpainting and super-resolution, based on several methods for multivariate analysis is also shown in this paper.

A Web video retrieval method using hierarchical structure of Web video groups
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
MULTIMEDIA TOOLS AND APPLICATIONS, 75, 24, 17059, 17079, SPRINGER, Dec. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a Web video retrieval method that uses hierarchical structure of Web video groups. Existing retrieval systems require users to input suitable queries that identify the desired contents in order to accurately retrieve Web videos; however, the proposed method enables retrieval of the desired Web videos even if users cannot input the suitable queries. Specifically, we first select representative Web videos from a target video dataset by using link relationships between Web videos obtained via metadata "related videos" and heterogeneous video features. Furthermore, by using the representative Web videos, we construct a network whose nodes and edges respectively correspond to Web videos and links between these Web videos. Then Web video groups, i.e., Web video sets with similar topics are hierarchically extracted based on strongly connected components, edge betweenness and modularity. By exhibiting the obtained hierarchical structure of Web video groups, users can easily grasp the overview of many Web videos. Consequently, even if users cannot write suitable queries that identify the desired contents, it becomes feasible to accurately retrieve the desired Web videos by selecting Web video groups according to the hierarchical structure. Experimental results on actual Web videos verify the effectiveness of our method.

Classifying Insects from SEM Images Based on Optimal Classifier Selection and D-S Evidence Theory
Takahiro Ogawa, Akihiro Takahashi, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E99A, 11, 1971, 1980, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, an insect classification method using scanning electron microphotographs is presented. Images taken by a scanning electron microscope (SEM) have a unique problem for classification in that visual features differ from each other by magnifications. Therefore, direct use of conventional methods results in inaccurate classification results. In order to successfully classify these images, the proposed method generates an optimal training dataset for constructing a classifier for each magnification. Then our method classifies images using the classifiers constructed by the optimal training dataset. In addition, several images are generally taken by an SEM with different magnifications from the same insect. Therefore, more accurate classification can be expected by integrating the results from the same insect based on Dempster-Shafer evidence theory. In this way, accurate insect classification can be realized by our method. At the end of this paper, we show experimental results to confirm the effectiveness of the proposed method.

Estimation of salient regions related to chronic gastritis using gastric X-ray images
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
COMPUTERS IN BIOLOGY AND MEDICINE, 77, 9, 15, PERGAMON-ELSEVIER SCIENCE LTD, Oct. 2016, [Peer-reviewed]
English, Scientific journal, Since technical knowledge and a high degree of experience are necessary for diagnosis of chronic gastritis, computer-aided diagnosis (CAD) systems that analyze gastric X-ray images are desirable in the field of medicine. Therefore, a new method that estimates salient regions related to chronic gastritis/non-gastritis for supporting diagnosis is presented in this paper. In order to estimate salient regions related to chronic gastritis/non-gastritis, the proposed method monitors the distance between a target image feature and Support Vector Machine (SVM)-based hyperplane for its classification. Furthermore, our method realizes removal of the influence of regions outside the stomach by using positional relationships between the stomach and other organs. Consequently, since the proposed method successfully estimates salient regions of gastric X-ray images for which chronic gastritis and non-gastritis are unknown, visual support for inexperienced clinicians becomes feasible. (C) 2016 Elsevier Ltd. All rights reserved.

Category Classification of Tourism Images in Image Sharing Services
SAITO Naoki, OGAWA Takahiro, ASAMIZU Satoshi, HASEYAMA Miki
電子情報通信学会論文誌D 情報・システム, J99-D, 9, 848, 860, The Institute of Electronics, Information and Communication Engineers, 01 Sep. 2016, [Peer-reviewed]
Japanese, In this paper, we propose a category classification method of tourism images in image sharing services. First, the proposed method performs classification of tourism images by applying a fuzzy K-nearest neighbor algorithm to the location feature vectors and newly estimates their confidence measures. Second, if the confidence measures of the classification results are lower, the proposed method performs decision level fusion of related visual and textual tag feature vectors to obtain final classification results. Consequently, the proposed method enables successful integration of multiple classification results obtained from the several kinds of features based on the above two stage classification scheme.

NMF-Based Spectral Reflectance Estimation From Image Pairs Including Near-Infrared Components
Takahiro Ogawa, Yuta Igarashi, Miki Haseyama
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 26, 5, 855, 867, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, May 2016, [Peer-reviewed]
English, Scientific journal, In this paper, a novel spectral reflectance estimation method from image pairs including near-infrared (NIR) components based on nonnegative matrix factorization (NMF) is presented. The proposed method enables estimation of spectral reflectance from only two kinds of input images: 1) an image including both visible light components and NIR components and 2) an image including only NIR components. These two images can be easily obtained using a general digital camera without an infrared-cut filter and one with a visible light-cut filter, respectively. Since RGB values of these images are obtained according to spectral sensitivity of the image sensor, the spectrum power distribution of the light source and the spectral reflectance, we have to solve the inverse problem for estimating the spectral reflectance. Therefore, our method approximates spectral reflectance by a linear combination of several bases obtained by applying NMF to a known spectral reflectance data set. Then estimation of the optimal solution to the above problem becomes feasible based on this approximation. In the proposed method, NMF is used for obtaining the bases used in this approximation from a characteristic that the spectral reflectance is a nonnegative component. Furthermore, the proposed method realizes simple approximation of the spectrum power distribution of the light source with direct and scattered light components. Therefore, estimation of spectral reflectance becomes feasible using the spectrum power distribution of the light source in our method. In the last part of this paper, we show some simulation results to verify the performance of the proposed method. The effectiveness of the proposed method is also shown using the method for several applications that are closely related to spectral reflectance estimation. Although our method is based on a simple scheme, it is the first method that realizes the estimation of the spectral reflectance and the spectrum power distribution of the light source from the above two kinds of images taken by general digital cameras and provides breakthroughs to several fundamental applications.

A Most Resource-Consuming Disease Estimation Method from Electronic Claim Data Based on Labeled LDA
Yasutaka Hatakeyama, Takahiro Ogawa, Hironori Ikeda, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E99D, 3, 763, 768, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Mar. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a method to estimate the most resource-consuming disease from electronic claim data based on Labeled Latent Dirichlet Allocation (Labeled LDA). The proposed method models each electronic claim from its medical procedures as a mixture of resource-consuming diseases. Thus, the most resource-consuming disease can be automatically estimated by applying Labeled LDA to the electronic claim data. Although our method is composed of a simple scheme, this is the first trial for realizing estimation of the most resource-consuming disease.

Distress Classification of Road Structures via Multiple Classifier-based Bayesian Network　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Workshop on Advanced Image Technology (IWAIT), 1, 4, 2016, [Peer-reviewed]
English, International conference proceedings

Decision Level Fusion-based Team Tactics Estimation in Soccer Videos
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, A decision-level fusion (DLF)-based team tactics estimation method in soccer videos is newly presented. In our method, tactics estimation based on audio-visual and formation features is newly adopted since the tactics of the soccer game are closely related to the audio-visual sequences and player positions. Therefore, by using these features, we classify the tactics via Support. Vector Machine (SVM). Furthermore, by applying DIA' to the SVM-based classification results, the two modalities are integrated to obtain more accurate tactics estimation results. Some results of experiments verify the superiority of our method.

Estimating Human Emotion Evoked by Visual Stimuli Using fMRI Data
Kento Sugata, Takahiro Ogawa, Miki Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method that estimates human emotion evoked by visual stimuli using functional magnetic resonance imaging (fMRI) data. First, in our method, preprocessing and masking procedures are applied to the fMRI data. These procedures provide the multiple brain data corresponding to Brodmann areas (BA). In most cases, the dimensionality of fMRI data and the BA data is larger than the number of observations, and this results in overfilling. Thus, in order to reduce the dimensionality, we apply general tensor discriminant analysis (GTDA), which can take into account the information related to the users' emotion. Then multiple estimation results of the users' emotion are obtained from support vector machine by separately using the multiple BA data obtained after the dimensionality reduction via GTDA. Furthermore, our method obtains the final estimation result from effective supervised decision-level fusion of the above estimation results.

Retrieval of Similar Inspection Records Based on Metric Learning Using Fixperienced Inspectors' Evaluation
Ryota Saito, Sho Takahashi, Takahiro Ogawa, Miki Hasayama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a retrieval method of similar inspection records in road structures based on metric learning using experienced inspectors' evaluation. Inspection records of road structures include images and text-based information such as category of distress, damaged parts and degree of damage. The proposed method calculates distances from query inspection records, and rank lists of retrieval results are obtained for each feature. In this approach, the distance quantification are updated on the basis of experienced inspectors' evaluation. Finally, the proposed method obtains retrieval results by integrating the multiple rank lists. The experimental results show the effectiveness of the proposed method.

A Touism Category Classification Method Based on Estimation of Reliable Decision
Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose a tourism category classification method based on estimation of reliable decision. The proposed method performs tourism category classification using location, visual, and textual tag features obtained from tourism images in image sharing services. As the biggest contribution of this paper, the proposed method performs successful classification based on two classification results obtained from a fuzzy K-nearest neighbor algorithm using the location features and a decision level fusion approach using the visual and textual tag features. The proposed method enables estimation of reliable decision from above two classifiers.

Novel Video Feature-based Favorite Video Estimation Using Users' Viewing Behavior and Evaluation
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents novel video feature-based favorite video estimation method. In the proposed method, we use three features, videos, users' viewing behavior and users' evaluation scores for these videos. In order to calculate the novel video features, Multiset Canonical Correlations Analysis (MCCA) is applied to these features to integrate the different types of features. Specifically, MCCA maximizes the sum of three kinds of correlations between three pairs of these features. Then the novel video features that represent the users' individual preference can be obtained by using the projection maximizing the three correlations. Finally, Supported Vector Ordinal Regression (SVOR) is trained by using the novel video features to estimate favorite videos. Experimental results show the effectiveness of our method.

Different Language Association Using Mutual Information Between Tags and Visual Features
Shota Hamano, Takahiro Ogawa, Mild Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for associating tags in one language with the tags representing the same meaning in another language. Since recent image search and sharing services highly rely on annotations like tags with images for obtaining the desired images, the proposed method utilizes the visual features extracted from images with tags. In the proposed method, mutual information between tags and visual features are calculated. Tag similarity is then calculated based on the mutual information. Mutual information takes into consideration the relevance between tags and visual features. Therefore, the similarity based on the mutual information represents tag-to-tag relationships more effectively than direct use of the visual features. Experimental results show the effectiveness of the proposed method in associating English tags with Japanese tags representing the same meanings.

Image Retrieval for Identification of Insects Based on Saliency Map and Distance Metric Learning
Susumu Genma, Takahiro Ogawa, Miki Haseyama
2016 IEEE 5TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS, 1, 2, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents an image retrieval method for insect identification based on saliency map and distance metric learning. First, the proposed method extracts regions of insects from target images by using saliency map and calculates visual features from the extracted insect regions. Next, in order to realize accurate retrieval of insects based on the calculated features, distance metric learning is newly adopted. Consequently, through users' evaluation in the retrieval, optimal distance can be obtained for the calculated visual features to obtain successful retrieval results, and the identification of insects becomes feasible. Experimental results show the effectiveness of our method.

Distress Classification of Road Structures via Decision Level Fusion
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 2016, DSP, 589, 593, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, A distress classification method of road structures via decision level fusion is presented in this paper. In order to classify various kinds of distresses accurately, the proposed method integrates multiple classification results with considering their performance, and this is the biggest contribution of this paper. By introducing this approach, it becomes feasible to adaptively integrate the multiple classification results based on the accuracy of each classifier for a target sample. Consequently, realization of the accurate distress classification can be expected. Experimental results show that our method outperforms existing methods.

Multimodal Interest Level Estimation via Variational Bayesian Mixture of Robust CCA
Yuma Sasaka, Takhiro Ogawa, Miki Haseyama
MM'16: PROCEEDINGS OF THE 2016 ACM MULTIMEDIA CONFERENCE, 387, 391, ASSOC COMPUTING MACHINERY, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method which estimates interest level while watching videos, based on collaborative use of facial expression and biological signals such as electroencephalogram (EEG) and electrocardiogram (ECG). To the best of our knowledge, no studies have been carried out on the collaborative use of facial expression and biological signals for estimating interest level. Since training data, which is used for estimating interest level, is generally small and imbalanced, Variational Bayesian Mixture of Robust Canonical Correlation Analysis (VBMRCCA) is newly applied to facial expression and biological signals, which are obtained from users while they are watching the videos. Unlike some related works, VBMRCCA is used to obtain the posterior distributions which represent the latent correlation between facial expression and biological signals in our method. Then, the users' interest level can be estimated by comparing the posterior distributions of the positive class data with those of the negative. Consequently, successful interest level estimation, via collaborative use of facial expression and biological signals, becomes feasible.

An Accurate Mortality Prediction Method Based on Decision-level Fusion of Existing ICU Scoring Systems
Yasutaka Hatakeyama, Takahiro Ogawa, Hirokazu Tanaka, Miki Haseyama
PROCEEDINGS OF 2016 INTERNATIONAL SYMPOSIUM ON INFORMATION THEORY AND ITS APPLICATIONS (ISITA 2016), 126, 130, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose a mortality prediction method based on decision-level fusion (DLF) of existing intensive unit care (ICU) scoring systems. First, the proposed method obtains severity scores from the existing ICU scoring systems. Furthermore, we construct classifiers that categorize patients into survivors or non-survivors. Next, patient feature vectors are extracted based on the mortality rates that are estimated from the obtained severity scores by using a non-linear least squares method to obtain other types of classification results. In order to obtain the final severity score for each patient, we integrate the obtained multiple classification results based on DLF that can estimate the final severity scores. Finally, we performed the proposed method to actual ICU patient data and verified the effectiveness of the proposed method. Thus, the proposed method can realize accurate mortality prediction without any additional work by using the existing ICU scoring systems.

GRAPH-BASED WEB VIDEO SEARCH RERANKING THROUGH CONSISTENCY ANALYSIS USING SPECTRAL CLUSTERING
Soh Yoshida, Takahiro Ogawa, Miki Haseyaina
2016 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA & EXPO (ICME), 2016-August, 1, 6, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper proposes a graph-based Web video search reranking method through consistency analysis using spectral clustering. Graph-based reranking is effective for refining text-based video search results. Generally, this approach constructs a graph where the vertices are videos and the edges reflect their pairwise similarities. A lot of reranking methods are built based on a scheme which regularizes the smoothness of pairwise ranking scores between adjacent nodes. However, since the overall consistency is measured by aggregating the individual consistency over each pair, errors in score estimation increase when noisy samples are included within their neighbors. To deal with the noisy samples, different from the conventional methods, the proposed method models the global consistency of the graph structure. Specifically, in order to detect this consistency, the propose method introduces a spectral clustering algorithm which can detect video groups, whose videos have strong semantic correlation, on the graph. Furthermore, a new regularization term, which smooths ranking scores within the same group, is introduced to the reranking framework. Since score regularization is performed by both local and global aspect simultaneously, the accurate score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

HIERARCHICAL CONTENT GROUP DETECTION FROM DIFFERENT SOCIAL MEDIA PLATFORMS USING WEB LINK STRUCTURE
Daichi Takehara, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2016-August, 479, 483, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for hierarchical content group detection from different social media platforms, which can reveal hierarchical structure of content groups. In this paper, content groups are defined as sets of contents with similar topics. Based on the revealed hierarchical structure, our method enables users to efficiently find the desired contents from large amount of contents placed in diversified social media platforms. The main contributions of this paper are twofold. First, effective latent features for comparing the contents placed in different social media platforms can be extracted by the combination use of the correlation between features obtained from different social media platform and the Web link structure. Second, the hierarchical structure of the content groups, which captures their various abstraction levels, can be revealed by hierarchically detecting their content groups. Experimental results on the real-world dataset containing YouTube videos and Wikipedia articles show the effectiveness of our method.

NOVEL FAVORITE MUSIC CLASSIFICATION USING EEG-BASED OPTIMAL AUDIO FEATURES SELECTED VIA KDLPCCA
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING PROCEEDINGS, 2016, ICASSP, 759, 763, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method of favorite music classification using EEG-based optimal audio features. To select audio features related to user's music preference, our method utilizes a relationship between EEG features obtained from the user's EEG signals during listening to music and their corresponding audio features since EEG signals of human reflect his/her music preference. Specifically, cross-loadings, whose components denote the degree of the relationship, are calculated based on Kernel Discriminative Locality Preserving Canonical Correlation Analysis (KDLPCCA) which is newly derived in the proposed method. In contrast with standard CCA, KDLPCCA can consider (1) non-linear correlation, (2) class information and (3) local structures of input EEG and audio features, simultaneously. Therefore, KDLPCCA-based cross-loadings can reflect best correlation between the user's EEG and corresponding audio signals. Then an optimal set of audio features related to his/her music preference can be obtained by employing the cross-loadings as novel criteria for feature selection. Consequently, our method realizes favorite music classification successfully by using the EEG-based optimal audio features.

Improvement of video coding efficiency based on sparse contractive mapping approach
Zaixing He, Takahiro Ogawa, Sho Takahashi, Miki Haseyama, Xinyue Zhao
NEUROCOMPUTING, 173, P3, 1898, 1907, ELSEVIER SCIENCE BV, Jan. 2016, [Peer-reviewed]
English, Scientific journal, This paper presents a new method for improving video coding efficiency based on a sparse contractive mapping approach. The proposed method introduces a new sparse contractive mapping approach to replace the traditional intra prediction in the video coding standards such as H.264/AVC. Specifically, the intra- and its following inter-frame are respectively approximated by the sparse representation, satisfying contractive mapping. Then these two frames can be reconstructed from an arbitraryinitial image by utilizing a few representation coefficients. With this advantage, the proposed method reduces the total amount of bits by removing MBs in the target I frame, whose approximation performance is higher than the others in the encoder. Furthermore, by transmitting the representation coefficients of the removed MBs, these MBs can be accurately reconstructed in the decoder. Since the reconstruction performance is better than that of the conventional approach, the proposed method can remove more MBs from the target video sequences, and reduction of total amount of bits can be feasible. Therefore, the proposed method realizes the improvement of the video coding efficiency. Some experimental results are shown to verify the superior performance of the proposed method to that of H.264/AVC. The results also demonstrate that the bit-saving performance of the proposed method is comparable to that of H.2651 HEVC. (C) 2015 Elsevier B.V. All rights reserved.

Bregman pooling: Feature-space local pooling for image classification
Alameen Najjar, Takahiro Ogawa, Miki Haseyama
International Journal of Multimedia Information Retrieval, 4, 4, 247, 259, Springer London, 01 Dec. 2015, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a novel feature-space local pooling method for the commonly adopted architecture of image classification. While existingmethods partition the feature space based on visual appearance to obtain pooling bins, learning more accurate space partitioning that takes semantics into account boosts performance even for a smaller number of bins. To this end, we propose partitioning the feature space over clusters of visual prototypes common to semantically similar images (i.e., images belonging to the same category). The clusters are obtained by Bregman co-clustering applied offline on a subset of training data. Therefore, being aware of the semantic context of the input image, our features have higher discriminative power than do those pooled from appearance-based partitioning. Testing on four datasets (Caltech-101, Caltech-256, 15 Scenes, and 17 Flowers) belonging to three different classification tasks showed that the proposed method outperforms methods in previous works on local pooling in he feature space for less feature dimensionality. Moreover, when implemented within a spatial pyramid, our method achieves comparable results on three of the datasets used.

Perceptually Optimized Missing Texture Reconstruction via Neighboring Embedding
Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E98A, 8, 1709, 1717, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2015, [Peer-reviewed]
English, Scientific journal, Perceptually optimized missing texture reconstruction via neighboring embedding (NE) is presented in this paper. The proposed method adopts the structural similarity (SSIM) index as a measure for representing texture reconstruction performance of missing areas. This provides a solution to the problem of previously reported methods not being able to perform perceptually optimized reconstruction. Furthermore, in the proposed method, a new scheme for selection of the known nearest neighbor patches for reconstruction of target patches including missing areas is introduced. Specifically, by monitoring the SSIM index observed by the proposed NE-based reconstruction algorithm, selection of known patches optimal for the reconstruction becomes feasible even if target patches include missing pixels. The above novel approaches enable successful reconstruction of missing areas. Experimental results show improvement of the proposed method over previously reported methods.

NOVEL IMAGE CLASSIFICATION BASED ON INTEGRATION OF EEG AND VISUAL FEATURES VIA MSLPCCA
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 957, 961, IEEE, 2015
English, International conference proceedings, This paper presents a novel image classification method based on integration of EEG and visual features. In the proposed method, we obtain classification results by separately using EEG and visual features. Furthermore, we merge the above classification results based on a kernelized version of Supervised learning from multiple experts and obtain the final classification result. In order to generate feature vectors used for the final image classification, we apply Multiset supervised locality preserving canonical correlation analysis (MSLPCCA), which is newly derived in the proposed method, to EEG and visual features. Our method realizes successful multimodal classification of images by the object categories that they contain based on MSLPCCA-based feature integration.

MISSING INTENSITY RESTORATION VIA ADAPTIVE SELECTION OF PERCEPTUALLY OPTIMIZED SUBSPACES
Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 1628, 1632, IEEE, 2015
English, International conference proceedings, A missing intensity restoration method via adaptive selection of perceptually optimized subspaces is presented in this paper. In order to realize adaptive and perceptually optimized restoration, the proposed method generates several subspaces of known textures optimized in terms of the structural similarity (SSIM) index. Furthermore, the SSIM-based missing intensity restoration is performed by a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Furthermore, the selection of the optimal subspace is realized by monitoring the SSIM index converged in the POCS algorithm, and the adaptive restoration becomes feasible. Experimental results show that our method outperforms existing methods.

Automatic Martian Dust Storm Detection from Multiple Wavelength Data Based on Decision Level Fusion
Maeda Keisuke, Ogawa Takahiro, Haseyama Miki
IMT, 10, 3, 473, 477, Information and Media Technologies Editorial Board, 2015
English, This paper presents automatic Martian dust storm detection from multiple wavelength data based on decision level fusion. In our proposed method, visual features are first extracted from multiple wavelength data, and optimal features are selected for Martian dust storm detection based on the minimal-Redundancy-Maximal-Relevance algorithm. Second, the selected visual features are used to train the Support Vector Machine classifiers that are constructed on each data. Furthermore, as a main contribution of this paper, the proposed method integrates the multiple detection results obtained from heterogeneous data based on decision level fusion, while considering each classifiers detection performance to obtain accurate final detection results. Consequently, the proposed method realizes successful Martian dust storm detection.

Image Classification Based on Integration of EEG and Visual Features Via LFDA-MSLPCCA
Kento Sugata, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 513, 514, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel image classification based on the integration of EEG and visual features. In the proposed method, we first obtain classification results by separately using EEG and visual features. Then we merge the above classification results based on kernelized version of Supervised Learning from Multiple Experts (KSLME) via Multiset Supervised Locality Preserving Canonical Correlation Analysis (MSLPCCA) to obtain final classification results. It should be noted that when the number of samples is fewer than the dimension of a sample data used in MSLPCCA, we have to reduce the dimension. Therefore, we propose MSLPCCA based on Local Fisher Discriminant Analysis (LFDA) which can take class information into account. Then the integration of all of the classifications results becomes feasible by MSLPCCA based on LFDA.

Web Image Visualization Considering Image Content Based on Visual and Tag Features
Kouhei Tateno, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 254, 255, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a Web image visualization considering image content based on visual and tag features. In this paper, we focus on tagged images on social media websites. Since these tags represent the image content according to the subjectivity of the user, using these tags is efficient for the image visualization. Thus, by using visual and tag features, the proposed method can take account of the semantic contents. Specifically, the proposed method applies Locality Preserving Canonical Correlation Analysis (LPCCA) to these two features to obtain the dimensionality reduction results, i.e., the visualization result.

Video Genre Estimation from Relationship between Motion and Facial Features using SLPCCA
Yuma Sasaka, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 250, 251, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose an efficient video genre estimation method based on the relationship between facial features and motion features. In the proposed method, we utilize supervised locality preserving canonical correlation analysis (SLPCCA), which is derived in the proposed method, to maximize the correlation between facial features and motion features. Moreover, by using SLPCCA, we can consider not only the correlation but also class information. Finally, by applying Support Vector Machine (SVM) to the SLPCCA-based feature vectors, we realize a successful video genre estimation. Experimental results show the effectiveness of our method.

Missing Texture Reconstruction via Power Spectrum-based Sparse Representation
Yuma Tanaka, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 221, 222, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for missing texture reconstruction via power spectrum-based sparse representation. We reconstruct missing areas based on minimizing the mean square error between power spectra (P-MSE). In our method, missing areas are reconstructed by embedding some known patches. Mathematically, we obtain the optimal linear combination of measurement patches by P-MSE minimization. The optimization can be solved as a combinatorial problem based on sparse representaion. In this way, the optimal approximation which minimizes the P-MSE is obtained and we embed it in the missing area. Experimental results show effectiveness of our method for reconstructing texture images.

LDA-based Music Recommendation with CF-based Similar User Selection
Shohei Kinoshita, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 215, 216, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a Latent Dirichlet Allocation (LDA)-based music recommendation method with collaborative filtering (CF)-based similar user selection. By applying LDA to music, we can estimate latent topics of music. However, we have to effectively reduce the size of the target dataset applied to LDA in order to recommend music from a large dataset. Hence, we use CF techniques, which recommend items using evaluation information of users who have similar tastes to a target user. Therefore, the proposed method limits the size of the dataset by using information of similar users and enables the recommendation of music considering latent topics of music. By using the idea of CF, our method can use LDA for music recommendation. Experimental results show the effectiveness of our method.

Helicobacter Pylori Infection Detection From Gastric X-ray Images Using KLFDA-based Decision Fusion
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2015 IEEE 4TH GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 204, 205, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents the performance improvement of Helicobacter pylori (H.pylori) infection detection using Kernel Local Fisher Discriminant Analysis (KLFDA)-based decision fusion. As the biggest contribution of this paper, the proposed method extracts more discriminative features based on KLFDA for the decision fusion. Since the decision fusion employed in this paper can consider not only the detection results but also the visual features, by calculating more discriminative features via KLFDA, more accurate decision fusion becomes feasible. Furthermore, experimental results show the effectiveness of the proposed method.

Automatic martian dust storm detection from multiple wavelength data based on decision level fusion
Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IPSJ Transactions on Computer Vision and Applications, 7, 79, 83, Information Processing Society of Japan, 2015, [Peer-reviewed]
English, Scientific journal, This paper presents automatic Martian dust storm detection from multiple wavelength data based on decision level fusion. In our proposed method, visual features are first extracted from multiple wavelength data, and optimal features are selected for Martian dust storm detection based on the minimal-Redundancy-Maximal-Relevance algorithm. Second, the selected visual features are used to train the Support Vector Machine classifiers that are constructed on each data. Furthermore, as a main contribution of this paper, the proposed method integrates the multiple detection results obtained from heterogeneous data based on decision level fusion, while considering each classifier's detection performance to obtain accurate final detection results. Consequently, the proposed method realizes successful Martian dust storm detection.

Investigation of flow and sink initiation process in mold shapes in injection molding
Tetsushi Kaburagi, Masashi Kurose, Takahiro Ogawa, Hiroki Kuroiwa, Tomoyuki Iwasawa
International Journal of Automation Technology, 9, 1, 10, 18, Fuji Technology Press, 2015, [Peer-reviewed]
English, Scientific journal, Injection molding has faults, or sinks, caused by the shrinkage ofmaterials. Sinks should be inhibited since they greatly affect the dimensions of moldings. In this study, a mold that allows visual observation is employed, and the sink initiation process is analyzed and predicted based on the results of that analysis. This mold has two sections, one flat and one curved. The difference between the deformations in the flat and curved sections is investigated. Methods of inhibiting sinks are considered from the results of the analysis and injection molding experiments. Packing pressure is found to have a great effect on sinks.

Heterogeneous Graph-based Video Search Reranking using Web Knowledge via Social Media Network
Soh Yoshida, Takahiro Ogawa, Miki Haseyama
MM'15: PROCEEDINGS OF THE 2015 ACM MULTIMEDIA CONFERENCE, 871, 874, ASSOC COMPUTING MACHINERY, 2015, [Peer-reviewed]
English, International conference proceedings, Graph-based reranking is effective for refining text-based video search results by making use of the social network structure. Unlike previous works which only focus on an individual video graph, the proposed method leverages the mutual reinforcement of heterogeneous graphs, such as videos and their associated tags obtained by social influence mining. Specifically, propagation of information relevancy across different modalities is performed by exchanging information of inter- and intra-relations among heterogeneous graphs. The proposed method then formulates the video search reranking as an optimization problem from the perspective of Bayesian framework. Furthermore, in order to model the consistency over the modified video graph topology, a local learning regularization with a social community detection scheme is introduced to the framework. Since videos within the same social community have strong semantic correlation, the consistency score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

HELICOBACTER PYLORI INFECTION DETECTION FROM MULTIPLE X-RAY IMAGES BASED ON COMBINATION USE OF SUPPORT VECTOR MACHINE AND MULTIPLE KERNEL LEARNING
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2015-December, 4728, 4732, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a detection method of Helicobacter pylori (H. pylori) infection from multiple gastric X-ray images based on combination use of Support Vector Machine (SVM) and Multiple Kernel Learning (MKL). The proposed method firstly computes some types of visual features from multiple gastric X-ray images taken in several specific directions in order to represent the characteristics of X-ray images with H. pylori infection. Second, based on the minimal-Redundancy-Maximal-Relevance algorithm, we select the effective features for H. pylori infection detection from each type of visual feature and all visual features. The selected features are used to train the SVM classifier and the MKL classifier for each direction of gastric X-ray images. Finally, the proposed method integrates multiple detection results based on a late fusion scheme considering the detection performance of each classifier. Experimental results obtained by applying the proposed method to real X-ray images prove its effectiveness.

AUTOMATIC DETECTION OF MARTIAN DUST STORMS FROM HETEROGENEOUS DATA BASED ON DECISION LEVEL FUSION
Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2015-December, 2246, 2250, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents automatic detection of Martian dust storms from heterogeneous data (raw data, reflectance data and background subtraction data of the reflectance data) based on decision level fusion. Specifically, the proposed method first extracts image features from these data and selects optimal features for dust storm detection based on the minimal-Redundancy-Maximal-Relevance algorithm. Second, the selected image features are used to train the Support Vector Machine classifier that is constructed on each data. Furthermore, as a main contribution of this paper, the proposed method combines the multiple detection results obtained from the heterogeneous data based on decision level fusion with considering each classifier's detection performance to obtain accurate final detection results. Consequently, the proposed method realizes automatic and accurate detection of Martian dust storms.

EXTRACTION OF HIERARCHICAL STRUCTURE OF WEB COMMUNITIES INCLUDING SALIENT KEYWORD ESTIMATION FOR WEB VIDEO RETRIEVAL
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2015-December, 1021, 1025, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, In this paper, we present a method for extraction of hierarchical structure of Web communities including salient keyword estimation for Web video retrieval. The following two contributions of the proposed method enable retrieval of the desired Web videos even if users cannot input suitable queries that identify the desired contents. First, our method realizes the extraction of hierarchical structure of Web communities, i.e., Web video sets with similar topics by using heterogeneous features of Web videos and link relationships between Web videos obtained via metadata "related videos". Second, we can estimate salient keywords to identify the contents of each obtained Web community at a glance based on text attached to Web videos such as title, the heterogeneous features of Web videos and the link relationships between Web videos. Experimental results on actual Web videos verify that our method can realize accurate retrieval of the desired Web videos via the hierarchical structure of Web communities with their salient keywords.

Human-centered Favorite Music Estimation: EEG-based Extraction of Audio Features Reflecting Individual Preference
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 2015-September, 818, 822, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a human-centered method for favorite music estimation using EEG-based audio features. In order to estimate user's favorite musical pieces, our method utilizes his/her EEG signals for calculating new audio features suitable for representing the user's music preference. Specifically, projection, which transforms original audio features into the features reflecting the preference, is calculated by applying kernel Canonical Correlation Analysis (CCA) to the audio features and the EEG features which are extracted from the user's EEG signals during listening to favorite musical pieces. By using the obtained projection, the new EEG-based audio features can be derived since this projection provides the best correlation between the user's EEG signals and their corresponding audio signals. Thus, successful estimation of user's favorite musical pieces via a Support Vector Machine (SVM) classifier using the new audio features becomes feasible. Since our method does not need acquisition of EEG signals for obtaining new audio features from new musical pieces after calculating the projection, this indicates the high practicability of our method. Experimental results show that our method outperforms methods using original audio features or EEG features.

Algorithm for Sparse Representation Minimizing Mean Square Error of Power Spectrograms
Yuma Tanaka, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 2015-September, 618, 622, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, Sparse representation is an idea to approximate a target signal by a linear combination of a small number of sample signals, and it is utilized in various research fields. In this paper, we evaluate the approximation error of signals by the mean square error of power spectrograms (P-MSE). Specifically, we propose a P-MSE minimization algorithm for sparse representation. Our method minimizes the P-MSE by an iterative approach. Specifically, in each iteration, we find the optimal sample signal and optimize the corresponding coefficients by a gradient-based method. In this approach, our method can utilize the result of the previous iteration for fast and stable convergence in the optimization of the coefficients. Based on this algorithm, the sparse representation which minimizes the P-MSE becomes feasible. Experimental results show the effectiveness of our method in terms of the P-MSE minimization.

MISSING INTENSITY RESTORATION VIA ADAPTIVE SELECTION OF PERCEPTUALLY OPTIMIZED SUBSPACES
Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 2015-August, 1628, 1632, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, A missing intensity restoration method via adaptive selection of perceptually optimized subspaces is presented in this paper. In order to realize adaptive and perceptually optimized restoration, the proposed method generates several subspaces of known textures optimized in terms of the structural similarity (SSIM) index. Furthermore, the SSIM-based missing intensity restoration is performed by a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Furthermore, the selection of the optimal subspace is realized by monitoring the SSIM index converged in the POCS algorithm, and the adaptive restoration becomes feasible. Experimental results show that our method outperforms existing methods.

NOVEL IMAGE CLASSIFICATION BASED ON INTEGRATION OF EEG AND VISUAL FEATURES VIA MSLPCCA
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 2015-August, 957, 961, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel image classification method based on integration of EEG and visual features. In the proposed method, we obtain classification results by separately using EEG and visual features. Furthermore, we merge the above classification results based on a kernelized version of Supervised learning from multiple experts and obtain the final classification result. In order to generate feature vectors used for the final image classification, we apply Multiset supervised locality preserving canonical correlation analysis (MSLPCCA), which is newly derived in the proposed method, to EEG and visual features. Our method realizes successful multimodal classification of images by the object categories that they contain based on MSLPCCA-based feature integration.

Random combination for information extraction in compressed sensing and sparse representation-based pattern recognition
Zaixing He, Xinyue Zhao, Shuyou Zhang, Takahiro Ogawa, Miki Haseyama
NEUROCOMPUTING, 145, 160, 173, ELSEVIER SCIENCE BV, Dec. 2014, [Peer-reviewed]
English, Scientific journal, In compressed sensing and sparse representation-based pattern recognition, random projection with a dense random transform matrix is widely used for information extraction. However, the complicated structure makes dense random matrices computationally expensive and difficult in hardware implementation. This paper considers the simplification of the random projection method. First, we propose a simple random method, random combination, for information extraction to address the issues of dense random methods. The theoretical analysis and the experimental results show that it can provide comparable performance to those of dense random methods. Second, we analyze another simple random method, random choosing, and give its applicable occasions. The comparative analysis and the experimental results show that it works well in dense cases but worse in sparse cases. Third, we propose a practical method for measuring the effectiveness of the feature transform matrix in sparse representation-based pattern recognition. A matrix satisfying the Representation Residual Restricted Isometry Property can provide good recognition results. (C) 2014 Elsevier B.V. All rights reserved.

Adaptive missing texture reconstruction method based on kernel cross-modal factor analysis with a new evaluation criterion
Takahiro Ogawa, Mild Haseyama
SIGNAL PROCESSING, 103, 69, 83, ELSEVIER SCIENCE BV, Oct. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents an adaptive missing texture reconstruction method based on kernel cross-modal factor analysis (KCFA) with a new evaluation criterion. The proposed method estimates the latent relationship between two areas, which correspond to a missing area and its neighboring area, respectively, from known parts within the target image and realizes reconstruction of the missing textures. In order to obtain this relationship, KCFA is applied to each cluster containing similar known textures, and the optimal cluster is used for reconstructing each target missing area. Specifically, a new criterion obtained by monitoring errors caused in the latent space enables selection of the optimal cluster. Then each missing texture is adaptively estimated by the optimal cluster's latent relationship, which enables accurate reconstruction of similar textures. In our method, the above criterion is also used for estimating patch priority, which determines the reconstruction order of missing areas within the target image. Since patches, whose textures are accurately modeled by our KCFA-based method, can be selected by using the new criterion, it becomes feasible to perform successful reconstruction of the missing areas. Experimental results show improvements of our KCFA-based reconstruction method over previously reported methods. (C) 2013 Elsevier B.V. All rights reserved.

Player Tracking in Far-View Soccer Videos Based on Composite Energy Function
Kazuya Iwai, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E97D, 7, 1885, 1892, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Jul. 2014, [Peer-reviewed]
English, Scientific journal, In this paper, an accurate player tracking method in far-view soccer videos based on a composite energy function is presented. In far-view soccer videos, player tracking methods that perform processing based only on visual features cannot accurately track players since each player region becomes small, and video coding causes color bleeding between player regions and the soccer field. In order to solve this problem, the proposed method performs player tracking on the basis of the following three elements. First, we utilize visual features based on uniform colors and player shapes. Second, since soccer players play in such a way as to maintain a formation, which is a positional pattern of players, we use this characteristic for player tracking. Third, since the movement direction of each player tends to change smoothly in successive frames of soccer videos, we also focus on this characteristic. Then we adopt three energies: a potential energy based on visual features, an elastic energy based on formations and a movement direction-based energy. Finally, we define a composite energy function that consists of the above three energies and track players by minimizing this energy function. Consequently, the proposed method achieves accurate player tracking in far-view soccer videos.

A new method for error degree estimation in numerical weather prediction via MKDA-based ordinal regression
Takahiro Ogawa, Shintaro Takahashi, Sho Takahashi, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2014, 1, 115, 115, SPRINGER INTERNATIONAL PUBLISHING AG, Jul. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents a new method for estimating error degrees in numerical weather prediction via multiple kernel discriminant analysis (MKDA)-based ordinal regression. The proposed method tries to estimate how large prediction errors will occur in each area from known observed data. Therefore, ordinal regression based on KDA is used for estimating the prediction error degrees. Furthermore, the following points are introduced into the proposed approach. Since several meteorological elements are related to each other based on atmospheric movements, the proposed method merges such heterogeneous features in the target and neighboring areas based on a multiple kernel algorithm. This approach is based on the characteristics of actual meteorological data. Then, MKDA-based ordinal regression for estimating the prediction error degree of a target meteorological element in each area becomes feasible. Since the amount of training data obtained from known observed data becomes very large in the training stage of MKDA, the proposed method performs simple sampling of those training data to reduce the number of samples. We effectively use the remaining training data for determining the parameters of MKDA to realize successful estimation of the prediction error degree.

A Cross-Modal Approach for Extracting Semantic Relationships Between Concepts Using Tagged Images
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON MULTIMEDIA, 16, 4, 1059, 1074, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Jun. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents a cross-modal approach for extracting semantic relationships between concepts using tagged images. In the proposed method, we first project both text and visual features of the tagged images to a latent space using canonical correlation analysis (CCA). Then, under the probabilistic interpretation of CCA, we calculate a representative distribution of the latent variables for each concept. Based on the representative distributions of the concepts, we derive two types of measures: the semantic relatedness between the concepts and the abstraction level of each concept. Because these measures are derived from a cross-modal scheme that enables the collaborative use of both text and visual features, the semantic relationships can successfully reflect semantic and visual contexts. Experiments conducted on tagged images collected from Flickr show that our measures are more coherent to human cognition than the conventional measures that use either text or visual features, or the WordNet-based measures. In particular, a new measure of semantic relatedness, which satisfies the triangle inequality, obtains the best results among different distance measures in our framework. The applicability of our measures to multimedia-related tasks such as concept clustering, image annotation and tag recommendation is also shown in the experiments.

Effective Multiple Feature Fusion Using Topic Model for Social Image Visualization
Kouhei Tateno, Takahiro Ogawa, Miki Haseyama
2014 IEEE 3RD GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 182, 183, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents a multiple feature fusion method using topic model for social image visualization. Images in social media are represented from several aspects such as their visual information and tags. The proposed method extracts low-level features from social images and their tags and calculates their integrated high-level features. Specifically, the proposed method applies multilayer multimodal probabilistic Latent Semantic Analysis (mm-pLSA) to the low-level visual and tag features to obtain the high-level features. Then, by applying dimensionality reduction techniques to the obtained features, successful visualization becomes feasible.

Bayesian Network-based Distress Estimation Using Image Features in Road Structure Assessment
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2014 IEEE 3RD GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 169, 170, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents a Bayesian network-based method for estimating a distress of road structures from inspection data. The distress is represented by a damage of road structures and its degree. In the previous work, the distress was estimated by utilizing Bayesian network based on categories of road structures, details of road structures and damaged parts. However, inspection data include not only the above items but also images of the distress. Therefore, by introducing the use of the images to the previous work, improvement of the distress estimation accuracy can be expected. The proposed method calculates Bayesian network from inspection items and their corresponding images to perform the distress estimation. Experimental results show the effectiveness of the proposed method.

Popular Music Estimation Based on Topic Model Using Time Information and Audio Features
Shohei Kinoshita, Takahiro Ogawa, Miki Haseyama
2014 IEEE 3RD GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 102, 103, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents popular music estimation based on a topic model using time information and audio features. The proposed method calculates latent topic distribution using Latent Dirichlet Allocation to obtain more accurate music features. In this approach, we also use release date information of each music as time information for concerning the relationship between music trends and each age. Then, by using the obtained latent topic distribution features, the estimation of the popular music becomes feasible based on a Support Vector Machine classifier. Experimental results show the effectiveness of our method.

Missing Audio Segment Reconstruction Based on Sparse Representation with Power Spectrogram
Yuma Tanaka, Takahiro Ogawa, Miki Haseyama
2014 IEEE 3RD GLOBAL CONFERENCE ON CONSUMER ELECTRONICS (GCCE), 86, 87, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for reconstructing missing audio segments based on sparse representation with power spectrogram. In the proposed method, an error of power spectrograms is utilized as a quality measure representing reconstruction performance. Then the proposed method estimates missing segments based on sparse representation optimized with respect to the error of power spectrograms. This error minimization problem can be solved with a greedy algorithm by limitting the solution to only sparse one. By using our method, perceptually optimized reconstruction becomes feasible since missing segments are estimated by using the quality measure which represens auditory properties. Experimental results obtained by applying the proposed method to actual music signals from RWC Music Database show its effectiveness.

HELICOBACTER PYLORI INFECTION DETECTION FROM MULTIPLE X-RAY IMAGES BASED ON DECISION LEVEL FUSION
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2769, 2773, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents an automatic detection method of Helicobacter pylori (H. pylori) infection from multiple gastric X-ray images. As the biggest contribution of this paper, we combine multiple detection results based on a decision level fusion. In order to obtain multiple detection results, the proposed method first focuses on characteristics of gastric X-ray images with H. pylori infection and computes several visual features from multiple X-ray images taken in several specific directions. Second, we select effective features for H. pylori infection detection from all features based on the minimal-Redundancy-Maximal-Relevance algorithm, and the selected features are used to train the Support Vector Machine (SVM) classifiers that are constructed for each direction of gastric radiography. Therefore, the detection of H. pylori infection becomes feasible, and we can obtain multiple detection results from the SVM classifiers. Furthermore, we combine multiple detection results based on the decision level fusion scheme considering the detection performance of each SVM classifier. Experimental results obtained by applying the proposed method to real X-ray images prove the effectiveness of the proposed method.

2D SEMI-SUPERVISED CCA-BASED INPAINTING INCLUDING NEW PRIORITY ESTIMATION
Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1837, 1841, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents an inpainting method based on 2D semi-supervised canonical correlation analysis (2D semi-CCA) including new priority estimation. The proposed method estimates relationship, i.e., the optimal correlation, between missing area and its neighboring area from known parts within the target image by using 2D CCA. In this approach, we newly introduce a semi-supervised scheme into the 2D CCA for deriving the 2D semi-CCA which corresponds to a hybrid version of 2D CCA and 2D principle component analysis (2D PCA). This enables successful relationship estimation even if sufficient number of training pairs cannot be provided. Then, by using the obtained relationship, accurate estimation of the missing intensities can be realized. Furthermore, in the proposed method, errors caused in the new variate space obtained by the 2D semi-CCA are effectively used for deriving patch priority determining inpainting order of missing areas. Experimental results show our inpainting method can outperform previously reported methods.

NOVEL IMAGE CLASSIFICATION BASED ON DECISION-LEVEL FUSION OF EEG AND VISUAL FEATURES
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2014 Vol.8, 5874, 5878, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel image classification based on decision-level fusion of EEG and visual features. In the proposed method, we extract the EEG features from EEG signals recorded while users stare at images, and the visual features are computed from these images. Then the classification of images is performed based on Support Vector Machine (SVM) by separately using the EEG and visual features. Furthermore, we merge the above classification results based on Supervised Learning from Multiple Experts to obtain the final classification result. This method focuses on the classification accuracy calculated from each classification result. Therefore, although classification accuracy based on EEG and visual features are different from each other, our method realizes effective integration of these classification results. In addition, we newly derive a kernelized version of the method in order to realize more accurate integration of the classification results. Consequently, our method realizes successful multimodal classification of images by the object categories that they contain.

MISSING INTENSITY RESTORATION VIA PERCEPTUALLY OPTIMIZED SUBSPACE PROJECTION BASED ON ENTROPY COMPONENT ANALYSIS
Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2014 Vol.1, 175, 179, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, A missing intensity restoration method via perceptually optimized subspace projection based on entropy component analysis (ECA) is presented in this paper. The proposed method calculates the optimal subspace of known patches within a target image based on structural similarity (SSIM) index, and the optimal bases are determined based on ECA. Then missing intensity estimation whose results maximize the SSIM index is realized by using a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Experimental results show that our restoration method outperforms previously reported methods.

Image inpainting based on sparse representations with a perceptual metric
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2013, 179, 179, SPRINGER INTERNATIONAL PUBLISHING AG, Dec. 2013, [Peer-reviewed]
English, Scientific journal, This paper presents an image inpainting method based on sparse representations optimized with respect to a perceptual metric. In the proposed method, the structural similarity (SSIM) index is utilized as a criterion to optimize the representation performance of image data. Specifically, the proposed method enables the formulation of two important procedures in the sparse representation problem, 'estimation of sparse representation coefficients' and 'update of the dictionary', based on the SSIM index. Then, using the generated dictionary, approximation of target patches including missing areas via the SSIM-based sparse representation becomes feasible. Consequently, image inpainting for which procedures are totally derived from the SSIM index is realized. Experimental results show that the proposed method enables successful inpainting of missing areas.

Vocal segment estimation in music pieces based on collaborative use of EEG and audio features
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2013 Vol.2, 1197, 1201, IEEE, 18 Oct. 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel estimation method of segments including vocals in music pieces based on collaborative use of features extracted from electroencephalogram (EEG) signals recorded while users are listening to music pieces and features extracted from these audio signals. From extracted EEG features and audio features, we estimate segments including vocals based on Support Vector Machine (SVM) by separately utilizing these two features. Furthermore, the final classification results are obtained by integrating these estimation results based on supervised learning from multiple experts. Therefore, our method realizes multimodal estimation of segments including vocals in music pieces. Experimental results show the improvement of our method over the methods utilizing only EEG or audio features. © 2013 IEEE.

Missing Texture Reconstruction Method Based on Error Reduction Algorithm Using Fourier Transform Magnitude Estimation Scheme
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON IMAGE PROCESSING, 22, 3, 1252, 1257, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Mar. 2013, [Peer-reviewed]
English, Scientific journal, A missing texture reconstruction method based on an error reduction (ER) algorithm, including a novel estimation scheme of Fourier transform magnitudes is presented in this brief. In our method, Fourier transform magnitude is estimated for a target patch including missing areas, and the missing intensities are estimated by retrieving its phase based on the ER algorithm. Specifically, by monitoring errors converged in the ER algorithm, known patches whose Fourier transform magnitudes are similar to that of the target patch are selected from the target image. In the second approach, the Fourier transform magnitude of the target patch is estimated from those of the selected known patches and their corresponding errors. Consequently, by using the ER algorithm, we can estimate both the Fourier transform magnitudes and phases to reconstruct the missing areas.

Super-resolution for simultaneous realization of resolution enhancement and motion blur removal based on adaptive prior settings
Takahiro Ogawa, Daisuke Izumi, Akane Yoshizaki, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2013, 1, 17, SPRINGER INTERNATIONAL PUBLISHING AG, Feb. 2013, [Peer-reviewed]
English, Scientific journal, A super-resolution method for simultaneously realizing resolution enhancement and motion blur removal based on adaptive prior settings are presented in this article. In order to obtain high-resolution (HR) video sequences from motion-blurred low-resolution video sequences, both of the resolution enhancement and the motion blur removal have to be performed. However, if one is performed after the other, errors in the first process may cause performance deterioration of the subsequent process. Therefore, in the proposed method, a new problem, which simultaneously performs the resolution enhancement and the motion blur removal, is derived. Specifically, a maximum a posterior estimation problem which estimates original HR frames with motion blur kernels is introduced into our method. Furthermore, in order to obtain the posterior probability based on Bayes' rule, a prior probability of the original HR frame, whose distribution can adaptively be set for each area, is newly defined. By adaptively setting the distribution of the prior probability, preservation of the sharpness in edge regions and suppression of the ringing artifacts in smooth regions are realized. Consequently, based on these novel approaches, the proposed method can perform successful reconstruction of the HR frames. Experimental results show impressive improvements of the proposed method over previously reported methods.

AN EXTRACTION METHOD OF HIERARCHICAL WEB COMMUNITIES FOR WEB VIDEO RETRIEVAL
Ryosuke Harakawa, Yasutaka Hatakeyama, Takahiro Ogawa, Miki Haseyama
2013 20TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP 2013), 4397, 4401, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents an extraction method of hierarchical Web communities for Web video retrieval. In the proposed method, Web communities containing Web videos whose topics are similar to each other are extracted by using hyperlinks between Web pages including Web videos and their video features. Furthermore, we focus on graph structure of hyperlinks between Web pages including Web videos which belong to the Web communities. Then, by using strongly connected components and betweenness centrality of the graph, hierarchical structure of the Web communities can be estimated. Consequently, users can easily find Web videos including related topics in each hierarchy, and desired Web videos can be effectively retrieved.

Insect classification using Scanning Electron Microphotographs considering magnifications
Akihiro Takahashi, Takahiro Ogawa, Miki Haseyama
2013 IEEE International Conference on Image Processing, ICIP 2013 - Proceedings, 3269, 3273, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a method of insect classification using images taken by Scanning Electron Microscope (SEM) considering magnifications. Generally, when images of the same insects are taken by SEM with different magnifications, visual features of these images are different from each other. Thus, the proposed method adopts a new scheme which groups images of different magnifications in such a way that the classification performance becomes the highest. Then a classifier is constructed for each group, and the insect classification becomes feasible based on a target image magnification. In addition, by integrating the classification results of several images obtained from the same sample, i.e., the same insect, performance improvement of the insect classification considering magnifications can be realized. Experimental results show the effectiveness of the proposed method. © 2013 IEEE.

SPECTRAL REFLECTANCE ESTIMATION FROM VISIBLE LIGHT COMPONENTS AND NEAR-INFRARED COMPONENTS
Yuta Igarashi, Takahiro Ogawa, Miki Haseyama
2013 20TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP 2013), 2388, 2392, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for estimating a spectral reflectance from two kinds of input images: an image including both visible light components and near-infrared (NIR) components, and an image including only NIR components. From these input images, we estimate the spectral reflectance based on the Non-negative Matrix Factorization algorithm using spectral sensitivities of a digital camera. The estimated spectral reflectance enables several important applications. In this paper, the e ff ectiveness of the proposed method is verified by using the estimated spectral reflectance in the two image processing applications.

KCFA-BASED MISSING AREA RESTORATION INCLUDING NEW PRIORITY ESTIMATION
Takahiro Ogawa, Miki Haseyama
2013 20TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP 2013), 704, 708, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, A kernel cross-modal factor analysis (KCFA) based missing area restoration method including a new priority estimation scheme is presented in this paper. The proposed method estimates latent relationship between missing areas and their neighboring areas by deriving projection matrices minimizing their errors in the latent space based on KCFA. This latent relationship represented by the derived projection matrices is optimal for accurately restoring missing areas within the target image. Furthermore, the proposed method adopts a new priority estimation scheme which determines the restoration order of missing areas. Specifically, this priority is estimated based on the criterion representing the restoration performance derived from KCFA, and it enables adaptive selection of missing areas successfully restored by our method. Consequently, it becomes feasible to accurately perform the restoration of missing areas by using the proposed KCFA-based method. Experimental results show subjective and quantitative improvements of the proposed method over previously reported restoration methods.

Trial Realization of Human-Centered Multimedia Navigation for Video Retrieval
Miki Haseyama, Takahiro Ogawa
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION, 29, 2, 96, 109, TAYLOR & FRANCIS INC, Jan. 2013, [Peer-reviewed]
English, Scientific journal, A trial realization of human-centered navigation for video retrieval is presented in this article. This system consists of the following functions: (a) multimodal analysis for collaborative use of multimedia data, (b) preference extraction for the system to adapt to users' individual demands, and (c) adaptive visualization for users to be guided to their desired contents. By using these functions, users can find their desired video contents more quickly and accurately than with the conventional retrieval schemes since our system can provide new pathways to the desired contents. Experimental results verify the effectiveness of the proposed system.

A Review of Video Retrieval Based on Image and Video Semantic Understanding
Miki Haseyama, Takahiro Ogawa, Nobuyuki Yagi
ITE Transactions on Media Technology and Applications, 1, 1, 2, 9, 2013, [Peer-reviewed]
Research trends in new video retrieval based on image and video semantic understanding are presented in this paper. First, recent studies related to image and video semantic analysis are introduced to understand leading-edge multimedia retrieval technologies. Several works related to visualization interfaces for multimedia retrieval are also presented. Finally, trends in state-of-the-art studies and the future outlook are described.

A Method for Improving SVM-based Image Classification Performance Based on a Target Object Detection Scheme
Soh Yoshida, Hiroshi Okada, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 1, 3, 237, 243, 2013, [Peer-reviewed]
Scientific journal, This paper presents a new method to improve performance of SVM-based classification, which contains a target object detection scheme. The proposed method tries to detect target objects from training images and improve the performance of the image classification by calculating the hyperplane from the detection results. Specifically, the proposed method calculates a Support Vector Machine (SVM) hyperplane, and detects rectangular areas surrounding the target objects based on the distances between their feature vectors and the separating hyperplane in the feature space. Then modification of feature vectors becomes feasible by removing features that exist only in background areas. Furthermore, a new hyperplane is calculated by using the modified feature vectors. Since the removed features are not part of the target object, they are not relevant to the learning process. Therefore, their removal can improve the performance of the image classification. Experimental results obtained by applying the proposed methods to several existing SVM-based classification method show its effectiveness.

A Compressed Sensing-Based Low-Density Parity-Check Real-Number Code
Zaixing He, Takahiro Ogawa, Miki Haseyama, Xinyue Zhao, Shuyou Zhang
Radioengineering, 22, 3, 851, 860, 2013, [Peer-reviewed]
Scientific journal, In this paper, we propose a novel low-density parity-check real-number code, based on compressed sensing. A real-valued message is encoded by a coding matrix (with more rows than columns) and transmitted over an erroneous channel, where sparse errors (impulsive noise) corrupt the codeword. In the decoding procedure, we apply a structured sparse (low-density) parity-check matrix, the Permuted Block Diagonal matrix, to the corrupted output, and the errors can be corrected by solving a compressed sensing problem. A compressed sensing algorithm, Cross Low-dimensional Pursuit, is used to decode the code by solving this compressed sensing problem. The proposed code has high error correction performance and decoding efficiency. The comparative experimental results demonstrate both advantages of our code. We also apply our code to cryptography.

Novel Evaluation Criterion for Visualization of Image Search Results Based on Feature Integration Theory
Katsuki Kobayashi, Takahiro Ogawa, Miki Haseyama
ITE Transactions on Media Technology and Applications, 1, 4, 333, 342, 2013, [Peer-reviewed]
Scientific journal, This paper presents a new evaluation criterion for visualization of image search results based on the feature integration theory. This criterion is derived by combining two elements, visual saliency on visualization and grouping degree of similar images. Visual saliency, which is calculated from the feature integration theory, on visualization of image search results enables representation of users' attention, which is closely related to the effectiveness of finding images. Furthermore, since users perceive similar images that are close to each other as one group, grouping degree of similar images enables evaluation of the effectiveness when users find images similar to a desired image. Therefore, by combining visual saliency on visualization and grouping degree of similar images, we can derive the novel criterion and evaluate the effectiveness of visualization of image search results.

Performance of Spatial and Temporal Error Concealment Method for 3D DWT Video Coding in Packet Loss Channel
Hirokazu Tanaka, Sunmi Kim, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E95A, 11, 2015, 2022, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2012, [Peer-reviewed]
English, A new spatial and temporal error concealment method for three-dimensional discrete wavelet transform (3D DWT) video coding is analyzed. 3D DWT video coding employing dispersive grouping (DG) and two-step error concealment is an efficient method in a packet loss channel [20], [21]. In the two-step error concealment method, the interpolations are only spatially applied however, higher efficiency of the interpolation can be expected by utilizing spatial and temporal similarities. In this paper, we propose an enhanced spatial and temporal error concealment method in order to achieve higher error concealment (EC) performance in packet loss networks. In the temporal error concealment method, structural similarity (SSIM) index is employed for inter group of pictures (GOP) EC and minimum mean square error (MMSE) is used for intra GOP EC. Experimental results show that the proposed method can obtain remarkable performance compared with the conventional methods.

Optical Fast Circuit Switching Networks Employing Dynamic Waveband Tunnel
Takahiro Ogawa, Hiroshi Hasegawa, Ken-ichi Sato
IEICE TRANSACTIONS ON COMMUNICATIONS, E95B, 10, 3139, 3148, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Oct. 2012, [Peer-reviewed]
English, Scientific journal, We propose a novel dynamic hierarchical optical path network architecture that achieves efficient optical fast circuit switching. In order to complete wavelength path setup/teardown efficiently, the proposed network adaptively manages waveband paths and bundles of optical paths, which provide virtual mesh connectivity between node pairs for wavelength paths. Numerical experiments show that operational and facility costs are significantly reduced by employing the adaptive virtual waveband connections.

A Novel Framework for Extracting Visual Feature-Based Keyword Relationships from an Image Database
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E95A, 5, 927, 937, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, May 2012, [Peer-reviewed]
English, Scientific journal, In this paper, a novel framework for extracting visual feature-based keyword relationships from an image database is proposed. From the characteristic that a set of relevant keywords tends to have common visual features, the keyword relationships in a target image database are extracted by using the following two steps. First, the relationship between each keyword and its corresponding visual features is modeled by using a classifier. This step enables detection of visual features related to each keyword. In the second step, the keyword relationships are extracted from the obtained results. Specifically, in order to measure the relevance between two keywords, the proposed method removes visual features related to one keyword from training images and monitors the performance of the classifier obtained for the other keyword. This measurement is the biggest difference from other conventional methods that focus on only keyword co-occurrences or visual similarities. Results of experiments conducted using an image database showed the effectiveness of the proposed method.

A CROSS-MODAL APPROACH FOR EXTRACTING SEMANTIC RELATIONSHIPS OF CONCEPTS FROM AN IMAGE DATABASE
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
2012 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2012 Vol.4, 2373, 2376, IEEE, 2012, [Peer-reviewed]
English, International conference proceedings, This paper presents a cross-modal approach for extracting semantic relationships of concepts from an image database. First, canonical correlation analysis (CCA) is used to capture the cross-modal correlations between visual features and tag features in the database. Then, in order to measure inter-concept relationships and estimate semantic levels, the proposed method focuses on the distributions of images under the probabilistic interpretation of CCA. Results of experiments conducted by using an image database showed the improvement of the proposed method over existing methods.

PERCEPTUALLY OPTIMIZED SUBSPACE ESTIMATION FOR MISSING TEXTURE RECONSTRUCTION
Takahiro Ogawa, Miki Haseyama
2012 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2012 Vol.2, 1141, 1144, IEEE, 2012, [Peer-reviewed]
English, International conference proceedings, This paper presents a perceptually optimized subspace estimation method for missing texture reconstruction. The proposed method calculates the optimal subspace of known patches within a target image based on structural similarity (SSIM) index instead of calculating mean square error (MSE)-based eigenspace. Furthermore, from the obtained subspace, missing texture reconstruction whose results maximize the SSIM index is performed. In this approach, the non-convex maximization problem is reformulated as a quasi convex problem, and the reconstruction of the missing textures becomes feasible. Experimental results show that our method overcomes previously reported MSE-based reconstruction methods.

Support Vector Data Dscription-Based Method for Finding New Benthic Species and Estimating Their Taxonomy Position from Microscopic Images.
長谷川尭史, 小川貴弘, 渡邉日出海, 長谷山美紀
映像情報メディア学会誌(Web), 66, 7, 2012, [Peer-reviewed]

Image Inpainting by Block-Based Linear Regression with Optimal Block Selection
Akira Tanaka, Takahiro Ogawa, Miki Haseyama
2012 ASIA-PACIFIC SIGNAL AND INFORMATION PROCESSING ASSOCIATION ANNUAL SUMMIT AND CONFERENCE (APSIPA ASC), 1, 4, IEEE, 2012, [Peer-reviewed]
English, International conference proceedings, Estimation of missing entries in a multivariate data is one of classical problems in the field of statistical science. One of most popular approaches for this problem is linear regression based on the EM algorithm. When we consider to apply this approach to block-based image inpainting problems, we have additional information, that is, a target lost pixel could be included in multiple blocks, which implies that we have multiple candidates of estimates for the pixel. In such cases, we have to choose a good estimate among the multiple candidates. In this paper, we propose a novel image inpainting method incorporating optimal block selection in terms of the expected squared errors among multiple candidates of the estimate for the target pixel. Results of numerical examples are also shown to verify the efficacy of the proposed method.

Missing Image Data Reconstruction Based on Adaptive Inverse Projection via Sparse Representation
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON MULTIMEDIA, 13, 5, 974, 992, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Oct. 2011, [Peer-reviewed]
English, Scientific journal, In this paper, a missing image data reconstruction method based on an adaptive inverse projection via sparse representation is proposed. The proposed method utilizes sparse representation for obtaining low-dimensional subspaces that approximate target textures including missing areas. Then, by using the obtained low-dimensional subspaces, inverse projection for reconstructing missing areas can be derived to solve the problem of not being able to directly estimate missing intensities. Furthermore, in this approach, the proposed method monitors errors caused by the derived inverse projection, and the low-dimensional subspaces optimal for target textures are adaptively selected. Therefore, we can apply adaptive inverse projection via sparse representation to target missing textures, i.e., their adaptive reconstruction becomes feasible. The proposed method also introduces some schemes for color processing into the calculation of subspaces on the basis of sparse representation and attempts to avoid spurious color caused in the reconstruction results. Consequently, successful reconstruction of missing areas by the proposed method can be expected. Experimental results show impressive improvement of our reconstruction method over previously reported reconstruction methods.

Cross Low-Dimension Pursuit for Sparse Signal Recovery from Incomplete Measurements Based on Permuted Block Diagonal Matrix
Zaixing He, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E94A, 9, 1793, 1803, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Sep. 2011, [Peer-reviewed]
English, Scientific journal, In this paper, a novel algorithm, Cross Low-dimension Pursuit, based on a new structured sparse matrix, Permuted Block Diagonal (PBD) matrix, is proposed in order to recover sparse signals from incomplete linear measurements. The main idea of the proposed method is using the PBD matrix to convert a high-dimension sparse recovery problem into two (or more) groups of highly low-dimension problems and crossly recover the entries of the original signal from them in an iterative way. By sampling a sufficiently sparse signal with a PBD matrix, the proposed algorithm can recover it efficiently. It has the following advantages over conventional algorithms: (1) low complexity, i.e., the algorithm has linear complexity, which is much lower than that of existing algorithms including greedy algorithms such as Orthogonal Matching Pursuit and (2) high recovery ability, i.e., the proposed algorithm can recover much less sparse signals than even l(1)-norm minimization algorithms. Moreover, we demonstrate both theoretically and empirically that the proposed algorithm can reliably recover a sparse signal from highly incomplete measurements.

Missing Intensity Interpolation Using a Kernel PCA-Based POCS Algorithm and its Applications
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON IMAGE PROCESSING, 20, 2, 417, 432, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Feb. 2011, [Peer-reviewed]
English, Scientific journal, A missing intensity interpolation method using a kernel principal component analysis (PCA)-based projection onto convex sets (POCS) algorithm and its applications are presented in this paper. In order to interpolate missing intensities within a target image, the proposed method reconstructs local textures containing the missing pixels by using the POCS algorithm. In this reconstruction process, a nonlinear eigenspace is constructed from each kind of texture, and the optimal subspace for the target local texture is introduced into the constraint of the POCS algorithm. In the proposed method, the optimal subspace can be selected by monitoring errors converged in the reconstruction process. This approach provides a solution to the problem in conventional methods of not being able to effectively perform adaptive reconstruction of the target textures due to missing intensities, and successful interpolation of the missing intensities by the proposed method can be realized. Furthermore, since our method can restore any images including arbitrary-shaped missing areas, its potential in two image reconstruction tasks, image enlargement and missing area restoration, is also shown in this paper.

ADAPTIVE KPCA-BASED MISSING TEXTURE RECONSTRUCTION APPROACH INCLUDING CLASSIFICATION SCHEME VIA DIFFERENCE SUBSPACES
Takahiro Ogawa, Miki Haseyama
2011 18TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1133, 1136, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive kernel principal component analysis (KPCA) based missing texture reconstruction approach including a classification scheme via difference subspaces. The proposed method utilizes a KPCA-based nonlinear eigenspace, which is obtained from each kind of known texture within a target image, as a constraint for reconstructing missing textures with a constraint of known neighboring areas. Then since these two constraints are convex, we can estimate missing textures based on a projection onto convex sets (POCS) algorithm. Furthermore, in this approach, the proposed method derives a new criterion for selecting the optimal eigenspace by monitoring errors caused in the projection via a difference subspace of each kind of known texture. This provides a solution to conventional problems of not being able to perform accurate texture classification, and the adaptive reconstruction of missing textures can be realized by the proposed method. Experimental results show subjective and quantitative improvement of the proposed method over previously reported reconstruction methods.

LINEAR TIME DECODING OF REAL-FIELD CODES OVER HIGH ERROR RATE CHANNELS
Zaixing He, Takahiro Ogawa, Miki Haseyama
2011 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 3172, 3175, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel algorithm for decoding real-field codes over erroneous channels, where the encoded message is corrupted by sparse errors, i.e., impulsive noise. The main problem of decoding such a corrupted encoded message is to reconstruct the error vector; recently, a common way to reconstruct it is to find the sparsest solution to an underdetermined system that is constructed using a parity-check matrix. Unlike the conventional approaches reconstructing the high-dimensional error vector directly, the proposed method crossly recovers the elements of error vector from two (or several) groups of low-dimensional equations. Compared with the traditional algorithms, the proposed method can decode an encoded message with a much higher corruption rate. Furthermore, the complexity of our method is linear, which is much lower than those of the traditional methods. The experimental results verified the high error correction ability and speed of the proposed method.

ADAPTIVE RECONSTRUCTION METHOD OF MISSING TEXTURES BASED ON PERCEPTUALLY OPTIMIZED ALGORITHM
Takahiro Ogawa, Miki Haseyama
2011 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 2011 Vol.2, 1157, 1160, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive reconstruction method of missing textures based on structural similarity (SSIM) index. The proposed method firstly performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Furthermore, from the selected known textures, the missing texture reconstruction maximizing the SSIM index is performed. In this approach, the non-convex maximization problem is reformulated as a quasi convex problem, and the adaptive reconstruction of the missing textures becomes feasible. Experimental results show impressive improvement of the proposed method over previously reported reconstruction methods.

Adaptive example-based super-resolution using kernel PCA with a novel classification approach
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, 1, 29, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, An adaptive example-based super-resolution (SR) using kernel principal component analysis (PCA) with a novel classification approach is presented in this paper. In order to enable estimation of missing high-frequency components for each kind of texture in target low-resolution (LR) images, the proposed method performs clustering of high-resolution (HR) patches clipped from training HR images in advance. Based on two nonlinear eigenspaces, respectively, generated from HR patches and their corresponding low-frequency components in each cluster, an inverse map, which can estimate missing high-frequency components from only the known low-frequency components, is derived. Furthermore, by monitoring errors caused in the above estimation process, the proposed method enables adaptive selection of the optimal cluster for each target local patch, and this corresponds to the novel classification approach in our method. Then, by combining the above two approaches, the proposed method can adaptively estimate the missing high-frequency components, and successful reconstruction of the HR image is realized.

Music recommendation according to human motion based on kernel CCA-based relationship
Hiroyuki Ohkushi, Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, 121, 121, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, In this article, a method for recommendation of music pieces according to human motions based on their kernel canonical correlation analysis (CCA)-based relationship is proposed. In order to perform the recommendation between different types of multimedia data, i.e., recommendation of music pieces from human motions, the proposed method tries to estimate their relationship. Specifically, the correlation based on kernel CCA is calculated as the relationship in our method. Since human motions and music pieces have various time lengths, it is necessary to calculate the correlation between time series having different lengths. Therefore, new kernel functions for human motions and music pieces, which can provide similarities between data that have different time lengths, are introduced into the calculation of the kernel CCA-based correlation. This approach effectively provides a solution to the conventional problem of not being able to calculate the correlation from multimedia data that have various time lengths. Therefore, the proposed method can perform accurate recommendation of best matched music pieces according to a target human motion from the obtained correlation. Experimental results are shown to verify the performance of the proposed method.

Adaptive Single Image Superresolution Approach Using Support Vector Data Description
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, An adaptive single image superresolution (SR) method using a support vector data description (SVDD) is presented. The proposed method represents the prior on high-resolution (HR) images by hyperspheres of the SVDD obtained from training examples and reconstructs HR images from low-resolution (LR) observations based on the following schemes. First, in order to perform accurate reconstruction of HR images containing various kinds of objects, training HR examples are previously clustered based on the distance from a center of a hypersphere obtained for each cluster. Furthermore, missing high-frequency components of the target image are estimated in order that the reconstructed HR image minimizes the above distances. In this approach, the minimized distance obtained for each cluster is utilized as a criterion to select the optimal hypersphere for estimating the high-frequency components. This approach provides a solution to the problem of conventional methods not being able to perform adaptive estimation of the high-frequency components. In addition, local patches in the target low-resolution (LR) image are utilized as the training HR examples from the characteristic of self-similarities between different resolution levels in general images, and our method can perform the SR without utilizing any other HR images.

固有空間BPLP法の補間精度に関する解析
田中章, 小川貴弘, 長谷山美紀, 宮腰政明
電子情報通信学会論文誌 A, J94-A, 2, 2011, [Peer-reviewed]

A Study on Adaptive Spatial-Temporal Error Concealment method for Wavelet based Video Coding in Wireless Networks
KIM Sunmi, TANAKA Hirokazu, OGAWA Takahiro, HASEYAMA Miki
ITE Technical Report, 35, 165, 170, The Institute of Image Information and Television Engineers, 2011
English, In this paper, we present an adaptive spatial-temporal error concealment method for the wavelet based video coding in wireless networks. A three-dimensional discrete wavelet transform (3-D DWT) is performed 2-D spatial DWT coding and temporal DWT coding on a group of pictures (GOP). The transmission of compressed video suffers from errors such as packet losses that not only corrupt frame but also propagate successive frames. The proposed adaptive spatial-temporal error concealment method consists of spatial EC and temporal EC to overcome error propagation problems. The proposed method is concealing erroneous coefficients of the spatiotemporal low-frequency subband by their duplication information, and uses the proposed adaptive spatial-temporal concealment method to recover errors for entire video sequence. The performance of proposed method was evaluated with wireless packet transmission networks. Experimental results show the proposed method can serve robust and stable performance in error-prone environments.

Error-Resilient 3-D Wavelet Video Coding with Duplicated Lowest Sub-Band Coefficients and Two-Step Error Concealment Method
Sunmi Kim, Hirokazu Tanaka, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E93A, 11, 2173, 2183, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2010, [Peer-reviewed]
English, Scientific journal, In this paper we propose a two step error concealment algorithm based on an error resilient three dimensional discrete wavelet transform (3 D DWT) video coding scheme. The proposed scheme consists of an error resilient encoder duplicating the lowest sub band bit streams for dispersive grouped frames and an error concealment decoder. The error concealment method of this decoder is decomposed of two steps the first step is replacement of erroneous coefficients in the lowest sub band by the duplicated coefficients and the second step is interpolation of the missing wavelet coefficients by minimum mean square error (MMSE) estimation. The proposed scheme can achieve robust transmission over unreliable channels. Experimental results provide performance comparisons in terms of peak signal to noise ratio (PSNR) and demonstrate increased performances compared to state of the art error concealment schemes.

ADAPTIVE RECONSTRUCTION METHOD OF MISSING TEXTURES BASED ON INVERSE PROJECTION VIA SPARSE REPRESENTATION
Takahiro Ogawa, Miki Haseyama
2010 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME 2010), 352, 357, IEEE, 2010, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive reconstruction method of missing textures based on an inverse projection via sparse representation. The proposed method approximates original and corrupted textures in lower-dimensional subspaces by using the sparse representation technique. Then, this approach effectively solves problems of not being able to directly estimate an inverse projection for reconstructing missing textures. Furthermore, even if target textures contain missing areas, the proposed method enables adaptive generation of the subspaces by monitoring errors caused in their known neighboring textures by the estimated inverse projection. Consequently, since the optimal inverse projection is adaptively estimated for each texture, successful reconstruction of the missing areas can be expected. Experimental results show impressive improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

THE SIMPLEST MEASUREMENT MATRIX FOR COMPRESSED SENSING OF NATURAL IMAGES
Zaixing He, Takahiro Ogawa, Miki Haseyama
2010 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, 4301, 4304, IEEE, 2010, [Peer-reviewed]
English, International conference proceedings, There exist two main problems in currently existing measurement matrices for compressed sensing of natural images, the difficulty of hardware implementation and low sensing efficiency. In this paper, we present a novel simple and efficient measurement matrix, Binary Permuted Block Diagonal (BPBD) matrix. The BPBD matrix is binary and highly sparse (all but one or several "1"s in each column are "0"s). Therefore, it can simplify the compressed sensing procedure dramatically. The proposed measurement matrix has the following advantages, which cannot be entirely satisfied by existing measurement matrices. (1) It has easy hardware implementation because of the binary elements; (2) It has high sensing efficiency because of the highly sparse structure; (3) It is incoherent with different popular sparsity basis' like wavelet basis and gradient basis; (4) It provides fast and nearly optimal reconstructions. Moreover, the simulation results demonstrate the advantages of the proposed measurement matrix.

Missing Texture Reconstruction Method Based on Perceptually Optimized Algorithm
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2010, HINDAWI PUBLISHING CORPORATION, 2010, [Peer-reviewed]
English, Scientific journal, This paper presents a simple and effective missing texture reconstruction method based on a perceptually optimized algorithm. The proposed method utilizes the structural similarity (SSIM) index as a new visual quality measure for reconstructing missing areas. Furthermore, in order to adaptively reconstruct target images containing several kinds of textures, the following two novel approaches are introduced into the SSIM-based reconstruction algorithm. First, the proposed method performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Secondly, missing texture reconstruction that maximizes the SSIM index in the known neighboring areas is performed. In this approach, the nonconvex maximization problem is reformulated as a quasi convex problem, and adaptive reconstruction of the missing textures based on the perceptually optimized algorithm becomes feasible. Experimental results show impressive improvements of the proposed method over previously reported reconstruction methods.

Erratum: Error-Resilient 3-D Wavelet Video Coding with Duplicated Lowest Sub-Band Coefficients and Two-Step Error Concealment Method [IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E93.A (2010) , No. 11 pp.2173-2183]
KIM Sunmi, TANAKA Hirokazu, OGAWA Takahiro, HASEYAMA Miki
IEICE Trans. Fundamentals, 93, 12, 2763_e1, 2763_e1, The Institute of Electronics, Information and Communication Engineers, 2010
English

A Novel Video Retrieval Method Based on Web Community Extraction Using Features of Video Materials
Yasutaka Hatakeyama, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1961, 1969, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, A novel video retrieval method based on Web community extraction using audio and visual features and textual features of video materials is proposed in this paper. In this proposed method, canonical correlation analysis is applied to these three features calculated from video materials and their Web pages, and transformation of each feature into the same variate space is possible. The transformed variates are based on the relationships between visual, audio and textual features of video materials, and the similarity between video materials in the same feature space for each feature can be calculated. Next, the proposed method introduces the obtained similarities of video materials into the link relationship between their Web pages. Furthermore, by performing link analysis of the obtained weighted link relationship, this approach extracts Web communities including similar topics and provides the degree of attribution of video materials in each Web community for each feature. Therefore, by calculating similarities of the degrees of attribution between the Web communities extracted from the three kinds of features, the desired ones are automatically selected. Consequently, by monitoring the degrees of attribution of the obtained Web communities, the proposed method can perform effective video retrieval. Some experimental results obtained by applying the proposed method to video materials obtained from actual Web pages are shown to verify the effectiveness of the proposed method.

Adaptive Missing Texture Reconstruction Method Based on Kernel Canonical Correlation Analysis with a New Clustering Scheme
Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1950, 1960, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, a method for adaptive reconstruction of missing textures based on kernel canonical correlation analysis (CCA) with a new clustering scheme is presented. The proposed method estimates the correlation between two areas, which respectively correspond to a missing area and its neighboring area, from known parts within the target image and realizes reconstruction of the missing texture. In order to obtain this correlation, the kernel CCA is applied to each cluster containing the same kind of textures, and the optimal result is selected for the target missing area. Specifically, a new approach monitoring errors caused in the above kernel CCA-based reconstruction process enables selection of the optimal result. This approach provides a solution to the problem in traditional methods of not being able to perform adaptive reconstruction of the target textures due to missing intensities. Consequently, all of the missing textures are successfully estimated by the optimal cluster's correlation, which provides accurate reconstruction of the same kinds of textures. In addition, the proposed method can obtain the correlation more accurately than our previous works, and more successful reconstruction performance can be expected. Experimental results show impressive improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

An ER Algorithm-Based Method for Removal of Adherent Water Drops from Images Obtained by a Rear View Camera Mounted on a Vehicle in Rainy Conditions
Tomoki Hiramatsu, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1939, 1949, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, an ER (Error-Reduction) algorithm-based method for removal of adherent water drops from images obtained by a rear view camera mounted on a vehicle in rainy conditions is proposed. Since Fourier-domain and object-domain constraints are needed for any ER algorithm-based method, the proposed method introduces the following two novel constraints for the removal of adherent water drops. The first one is the Fourier-domain constraint that utilizes the Fourier transform magnitude of the previous frame in the obtained images as that of the target frame. Noting that images obtained by the rear view camera have the unique characteristics of objects moving like ripples because the rear view camera is generally composed of a fish-eye lens for a wide view angle, the proposed method assumes that the Fourier transform magnitudes of the target frame and the previous frame are the same in the polar coordinate system. The second constraint is the object-domain constraint that utilizes intensities in an area of the target frame to which water drops have adhered. Specifically, the proposed method models a deterioration process of intensities that are corrupted by the water drop adhering to the rear view camera lens. By utilizing these novel constraints, the proposed ER algorithm can remove adherent water drops from images obtained by the rear view camera. Experimental results that verify the performance of the proposed method are represented.

Kalman Filter-Based Error Concealment for Video Transmission
Shigeki Takahashi, Takahiro Ogawa, Hirokazu Tanaka, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 3, 779, 787, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Mar. 2009, [Peer-reviewed]
English, Scientific journal, A novel error concealment method using a Kalman filter is presented ill this paper, In order to successfully utilize the Kalman filter, its state transition and observation models that are suitable for the video error concealment are newly defined as follows. The state transition model represents the video decoding process by a notion-compensated prediction. Furthermore, the new observation model that represents all image blurring process is defined. and calculation of the Kalman gain becomes possible. The problem of the traditional methods is solved by using the Kalman filter in the proposed method, and accurate reconstruction of corrupted video frames, is achieved. Consequently. an effective error concealment method using the Kalman filter is realized. Experimental results showed that the proposed method has better performance than that of traditional methods.

A Kalman Filter-Based Method for Restoration of Images Obtained by an In-Vehicle Camera in Foggy Conditions
Tomoki Hiramatsu, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 2, 577, 584, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Feb. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, a Kalman filter-based method for restoration of video images acquired by an in-vehicle camera in foggy conditions is proposed. In order to realize Kalman filter-based restoration, the proposed method clips local blocks from the target frame by using a sliding window and regards the intensities in each block as elements of the state variable of the Kalman filter. Furthermore, the proposed method designs the following two models for restoration of foggy images. The first one is an observation model, which represents a fog deterioration model. The proposed method automatically determines all parameters of the fog deterioration model from only the foggy images to design the observation model. The second one is a non-linear state transition model, which represents the target frame in the original video image from its previous frame based on motion vectors. By utilizing the observation and state transition models, the correlation between successive frames can be effectively utilized for restoration, and accurate restoration of images obtained in foggy conditions can be achieved. Experimental results show that the proposed method has better performance than that of the traditional method based on the fog deterioration model.

A WEB COMMUNITY-BASED VIDEO RETRIEVAL METHOD USING CANONICAL CORRELATION ANALYSIS
Yasutaka Hatakeyama, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2009 16TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1-6, 805, +, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, This paper presents a Web community-based video retrieval method using canonical correlation analysis (CCA). In the proposed method, two novel approaches are introduced into the retrieval scheme of video materials on the Web. First, the CCA is applied to three kinds of video features, visual and audio features of video materials and textual features obtained from Web pages containing those video materials. This approach provides a solution of problems of traditional methods of not being able to calculate similarities between different kinds of video features. Furthermore, from the obtained similarities and link relationships of Web pages, a new adjacency matrix is defined, and link analysis can be applied to this matrix. Then, the Web communities of the video materials whose topics are similar to each other can be automatically extracted based on their features. Therefore, by ranking the video materials in the obtained Web community, accurate video retrieval can be realized.

ADAPTIVE RECONSTRUCTION METHOD OF MISSING TEXTURES BASED ON KERNEL CANONICAL CORRELATION ANALYSIS
Takahiro Ogawa, Miki Haseyama
2009 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOLS 1- 8, PROCEEDINGS, 1165, 1168, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive reconstruction method of missing textures based on kernel canonical correlation analysis (CCA). The proposed method calculates the correlation between two areas, which respectively correspond to a missing area and its neighbor area, from known parts within the target image and realizes the estimation of the missing textures. In order to obtain this correlation, the kernel CCA is applied to each set containing the same kind of textures, and the optimal result is selected for the target missing area. Specifically, a new approach monitoring errors caused in the above estimation process enables the selection of the optimal result. This approach provides a solution to the problem in traditional methods of not being able to perform adaptive reconstruction of the target textures due to the missing intensities. Experimental results show subjective and quantitative improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

AN EFFECTIVE FLOW ESTIMATION METHOD WITH PARTICLE FILTER BASED ON HELMHOLTZ DECOMPOSITION THEOREM
Norihiro Kakukou, Takahiro Ogawa, Miki Haseyaam
2009 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOLS 1- 8, PROCEEDINGS, 949, 952, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel flow estimation method with a particle filter based on a Helmholtz decomposition theorem. The proposed method extends a model of the Helmholtz decomposition theorem and enables the decomposition of flows into rotational, divergent, and translational components. From the extended model, the proposed method defines a state transition model and an observation model of the particle filter. Furthermore, the proposed method derives an observation density of the particle filter from an energy function based on the Helmholtz decomposition theorem. By utilizing these novel approaches, the proposed method provides a solution to the problem in the traditional ones of not being able to realize an effective flow estimation with the particle filter based on rotation, divergence, and translation, which are important geometric features. Consequently, the proposed method can accurately estimate the flows.

エッジの連続性を考慮したIFSによる画像の拡大
覚幸典弘, 小川貴弘, 長谷山美紀
電子情報通信学会論文誌 D, J92-D, 3, 2009, [Peer-reviewed]

Semantic Image Retrieval Based on POCS Algorithm Using Kernel PCA And Its Performance Verification
Takahiro Ogawa, Miki Haseyama
ISCE: 2009 IEEE 13TH INTERNATIONAL SYMPOSIUM ON CONSUMER ELECTRONICS, VOLS 1 AND 2, 342, 343, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, This paper presents a projection onto convex sets (POCS)-based semantic image retrieval method and its performance verification. The main contributions of the proposed method are twofold: introduction of nonlinear eigenspace of visual and semantic features into the constraint of the POCS-based semantic image retrieval algorithm and adaptive selection of the annotated images utilized for this algorithm. Then, by combining these two approaches., the semantic features of the query image are successfully estimated, and accurate image retrieval can be expected. Finally, relationship between the performance of the proposed method and the kinds of the kernel functions utilized for the kernel PICA is shown in this paper.

POCS-based annotation method using kernel PCA for semantic image retrieval
Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E91A, 8, 1915, 1923, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2008, [Peer-reviewed]
English, Scientific journal, A projection onto convex sets (POCS)-based annotation method for semantic image retrieval is presented in this paper. Utilizing database images previously annotated by keywords, the proposed method estimates unknown semantic features of a query image from its known visual features based on a POCS algorithm, which includes two novel approaches. First, the proposed method semantically assigns database images to some clusters and introduces a nonlinear eigenspace of visual and semantic features in each cluster into the constraint of the POCS algorithm. This approach accurately provides semantic features for each cluster by using its visual features in the least squares sense. Furthermore, the proposed method monitors the error converged by the POCS algorithm in order to select the optimal cluster including the query image. By introducing the above two approaches into the POCS algorithm, the unknown semantic features of the query image are successfully estimated from its known visual features. Consequently, similar images can be easily retrieved from the database based on the obtained semantic features. Experimental results verify the effectiveness of the proposed method for semantic image retrieval.

A KALMAN FILTER-BASED APPROACH FOR ADAPTIVE RESTORATION OF IN-VEHICLE CAMERA FOGGY IMAGES
Tomoki Hiramatsu, Takahiro Ogawa, Miki Haseyama
2008 15TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1-5, 3160, 3163, IEEE, 2008, [Peer-reviewed]
English, International conference proceedings, In this paper, a Kalman filter-based approach for adaptive restoration of video images acquired by an in-vehicle camera in foggy conditions is proposed. In order to realize Kalman filter-based restoration, the proposed method regards the intensities in each frame as elements of the state variable of the Kalman filter and designs the following two models for restoration of foggy images. The first one is an observation model, which represents a fog deterioration model. The second one is a non-linear state transition model, which represents the target frame in the original video image from its previous frame based on motion vectors. By utilizing the observation and state transition models, the correlation between successive frames can be effectively utilized for restoration. Further, the proposed method introduces a new estimation scheme of the parameter, which determines the deterioration characteristic in foggy conditions, into the Kalman filter algorithm. Consequently, since automatic determination of the fog deterioration model, which specifies the observation model, from only the foggy images is realized, the accurate restoration can be achieved. Experimental results show that the proposed method has better performance than that of the traditional method based on the fog deterioration model.

AN EFFECTIVE DETECTION METHOD OF ROTATIONAL AND DIVERGENT STRUCTURES IN STILL IMAGES BASED ON HELMHOLTZ DECOMPOSITION
Norihiro Kakukou, Takahiro Ogawa, Miki Haseyama
2008 15TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1-5, 2336, 2339, IEEE, 2008, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel detection method of rotational and divergent structures in still images based on Helmholtz decomposition. These structures are mathematical features in vector analysis. Traditionally, some detection methods of these structures in image sequences have been proposed. By using the Helmholtz decomposition, which can decompose flows into rotational and divergent components, the traditional methods can detect these structures in image sequences. However, the rotational and divergent structures in still images cannot be detected with the traditional methods. Therefore, the proposed method introduces a new criterion into the traditional schemes in order to realize the detection of the rotational and divergent structures in still images. This criterion is derived from two properties based on relation between still images and the flows, which are composed of the rotational and divergent components. Consequently, the detection of the rotational and divergent structures in still images can be achieved.

KERNEL PCA-BASED SEMANTIC FEATURE ESTIMATION APPROACH FOR SIMILAR IMAGE RETRIEVAL
Takahiro Ogawa, Miki Haseyama
2008 15TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1-5, 965, 968, IEEE, 2008, [Peer-reviewed]
English, International conference proceedings, A kernel PCA-based semantic feature estimation approach for similar image retrieval is presented in this paper. Utilizing database images previously annotated by keywords, the proposed method estimates unknown semantic features of a query image. First, our method performs semantic clustering of the database images and derives a new map from a nonlinear eigenspace of visual and semantic features in each cluster. This map accurately provides the semantic features for the images belonging to each cluster by using their visual features. Further, in order to select the optimal cluster including the query image, the proposed method monitors errors of the visual features caused by the semantic feature estimation process. Then, even if any semantics of the query image are unknown, its semantic features are successfully estimated by the optimal cluster. Experimental results verify the effectiveness of the proposed method for semantic image retrieval.

KERNEL PCA-BASED SEMANTIC FEATURE ESTIMATION APPROACH FOR SIMILAR IMAGE RETRIEVAL
Takahiro Ogawa, Miki Haseyama
2008 IEEE International Conference on Image Processing, Proceedings, 969, 972, IEEE, 2008, [Peer-reviewed]
English, International conference proceedings, A kernel PCA-based semantic feature estimation approach for similar image retrieval is presented in this paper. Utilizing database images previously annotated by keywords, tire proposed method estimates unknown semantic features of a query image. First, our method performs semantic clustering of the database images and derives a new map from a nonlinear eigenspace of visual and semantic features in each c aster. This map accurately provides the semantic features for the images belonging to each cluster by using their visual features. Further, in order to select the optional cluster including the query image, the proposed method monitors errors of the visual features caused by the Semantic feature estimation process. Then, even if any semantics of the query image arc unknown, its semantic features are successfully estimated by tire optimal cluster. Experimental results verify the effectiveness of the proposed method for semantic image retrieval.

Image recognition method using intensity gradient vectors
Masao Hiramoto, Takahiro Ogawa, Miki Haseyama
Systems and Computers in Japan, 38, 13, 15, 27, 30 Nov. 2007, [Peer-reviewed]
English, Scientific journal, This paper proposes a method for general image recognition based on the progress of increasing the pixels of image sensors and improving image quality. This method can also adapt to images which have undergone geometric transformations such as the rotation and movement of images. The proposed method uses a voting system that utilizes vectors. This method uses vectors which represent intensity gradients and vectors which show position to express images and also defines voting vectors and similarity for the recognition. In addition, the proposed method has characteristics of concentrating voting locations at an origin point if there are identical images such that the voting results do not influence geometric transformations. When we performed experiments on natural images including original images which may have undergone image processing such as Gaussian or median filtering and JPEG compression, we understood that distinct differences appeared in the similarities and that recognition was possible even if an artificial process was added to the images. Even further, when we examined recognition of images using the greatest number of voting points as an application of this method, we were able to show that the recognition capability was high and that a partial image contained in one image could also be recognized. © 2007 Wiley Periodicals, Inc.

POCS-Based texture reconstruction method using clustering scheme by kernel PCA
Takahiro Ogawa, Milki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E90A, 8, 1519, 1527, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2007, [Peer-reviewed]
English, Scientific journal, A new framework for reconstruction of missing textures in digital images is introduced in this paper. The framework is based on a projection onto convex sets (POCS) algorithm including a novel constraint. In the proposed method, a nonlinear eigenspace of each cluster obtained by classification of known textures within the target image is applied to the constraint. The main advantage of this approach is that the eigenspace can approximate the textures classified into the same cluster in the least-squares sense. Furthermore, by monitoring the errors converged by the POCS algorithm, a selection of the optimal cluster to reconstruct the target texture including missing intensities can be achieved. This POCS-based approach provides a solution to the problem in traditional methods of not being able to perform the selection of the optimal cluster due to the missing intensities within the target texture. Consequently, all of the missing textures are successfully reconstructed by the selected cluster's eigenspaces which correctly approximate the same kinds of textures. Experimental results show subjective and quantitative improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

POCS-based iterative reconstruction algorithm of missing textures
Takahiro Ogawa, Miki Haseyama
2007 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1-7, 3, 1229, 1232, IEEE, 2007, [Peer-reviewed]
English, International conference proceedings, In this paper, a new framework for texture reconstruction of missing areas, which exist all over the target image, is presented. The framework is based on a projection onto convex sets (POCS) algorithm including a novel constraint. In the proposed method, a nonlinear eigenspace of each cluster obtained by texture classification is applied to the constraint. Furthermore, by monitoring the errors converged by the POCS algorithm, selection of the optimal cluster for the target texture including missing intensities is realized in order to reconstruct it adaptively. Then, iterating the POCS-based procedures, our method renews the nonlinear eigenspaces and the reconstruction image, and outputs the reliable result. This approach provides a solution to the problem in traditional methods of not being able to perform adaptive reconstruction of the target textures due to the missing intensities. Experimental results show subjective and quantitative improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

Adaptive reconstruction method of missing texture based on projection onto convex sets
Takahiro Ogawa, Miki Haseyama
2007 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL I, PTS 1-3, PROCEEDINGS, 1, 697, 700, IEEE, 2007, [Peer-reviewed]
English, International conference proceedings, This paper presents a missing texture reconstruction method based on projection onto convex sets (POCS). The proposed method classifies textures within the target image into some clusters in a high-dimensional texture feature space. Further, for the target missing texture, our method performs a novel approach, that monitors the errors caused by the POCS algorithm in the feature space, and adaptively selects the optimal cluster including similar textures. Then, the missing texture is restored from these similar textures by a new POCS-based nonlinear subspace projection scheme. Consequently, since the proposed method realizes the nonconventional adaptive technique using the optimal nonlinear subspace, the accurate restoration result can be obtained. Experimental results show that our method achieves higher performance than the traditional method.

Restoration of missing intensity of still images by using optical flow
Takahiro Ogawa, Miki Haseyama, Hideo Kitajima
Systems and Computers in Japan, 37, 3, 49, 57, Mar. 2006, [Peer-reviewed]
English, Scientific journal, This paper proposes an accurate method for the restoration of missing intensities of still images by using the optical flow. It is important in restoration to reconstruct missing edges correctly. Therefore, this paper modifies the optical flow conventionally used for motion analysis in video images and applies it to the restoration of missing intensities. Further, the proposed method introduces a new index expressing the correlation of intensities between two pixels into the scheme for calculation of the optical flow in order to obtain a flow which gives more accurate estimated values. The optical flow calculated by this index provides the pixel from the neighborhood whose intensity is most similar to that of the target pixel, so that the estimated intensity is not affected by pixels whose intensities are quite different. Consequently, even when multiple edges pass through the missing area or the direction of the edge changes significantly inside the area, the proposed method can reconstruct the edges correctly. Some experimental results are presented in order to verify the high performance of the proposed method. © 2006 Wiley Periodicals, Inc.

An accurate ifs-based image enlargement method using line process
Norihiro Kakukou, Takahiro Ogawa, Miki Haseyama, Hideo Kitajima
2006 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, ICIP 2006, PROCEEDINGS, 2701, +, IEEE, 2006, [Peer-reviewed]
English, International conference proceedings, This paper proposes an effective image enlargement method based on an Iterated Function System(IFS), which is traditionally used for image coding. The IFS can reconstruct an image of a different size from the coding target image's. Based on this property, some methods enlarging images have been proposed by using the IFS. However, the images enlarged by the traditional methods suffer from block noise and edge discontinuity in the boundary between neighboring range blocks, which are units of the process in the IFS. The reasons for the problems of the traditional methods are that they use non-overlapping range blocks and do not consider edge continuity in the boundary between the neighboring range blocks. Therefore, the proposed method allows selection of overlapping range blocks in order to avoid the block noise. Further, the proposed method introduces a line process, which is used for edge detection, into the enlargement procedure. The edges obtained by using the line process can retain the edge continuity. Therefore, the images enlarged by the proposed method retain the edge continuity. Consequently, an accurate image enlargement can be achieved.

Restoration method of missing areas in still images using GMRF model
T Ogawa, M Haseyama, H Kitajima
2005 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), VOLS 1-6, CONFERENCE PROCEEDINGS, 4931, 4934, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, This paper proposes a GMRF-model based restoration method of missing areas in still images. The GMRF model used in the proposed method is realized by a new assumption that reasonably holds for an image source. This model can express important image features such as edges because of the use of the new assumption. Therefore, the proposed method restores the missing areas using the modified GMRF model and can correctly reconstruct the missing edges. Consequently, the proposed method achieves more accurate restoration than those of the traditional methods on both objective and subjective measures. Extensive experimental results demonstrate the improvement of the proposed method over the previous methods.

Reconstruction method of missing texture using error reduction algorithm
T Ogawa, M Haseyama, H Kitajima
2005 International Conference on Image Processing (ICIP), Vols 1-5, 2, 1389, 1392, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel reconstruction method of missing textures using an error reduction algorithm which is one of phase retrieval methods. The proposed method estimates the Fourier transform magnitude of the missing area from another area whose texture is similar in the obtained image. In order to realize this, a novel approach that monitors the errors caused by the error reduction algorithm is introduced into the selection scheme of the similar texture. Further, the proposed method estimates the phase of the target area by using the error reduction algorithm modified for the texture reconstruction and can restore the missing area accurately. Some experimental results show that the proposed method achieves more accurate restoration than that of the traditional methods.

Influence of Music Listening on the Cerebral Activity by Analyzing EEG.
Takahiro Ogawa, Satomi Ota, Shin-Ichi Ito, Yasue Mitsukura, Minoru Fukumi, Norio Akamatsu
Knowledge-Based Intelligent Information and Engineering Systems, 657, 663, Springer, 2005
International conference proceedings

Restoration method of missing areas in still images using GMRF model
Takahiro Ogawa, Miki Haseyama, Hideo Kitajima
Proceedings - IEEE International Symposium on Circuits and Systems, 4931, 4934, 2005, [Peer-reviewed]
English, International conference proceedings, This paper proposes a GMRF-model based restoration method of missing areas in still images. The GMRF model used in the proposed method is realized by a new assumption that reasonably holds for an image source. This model can express important image features such as edges because of the use of the new assumption. Therefore, the proposed method restores the missing areas using the modified GMRF model and can correctly reconstruct the missing edges. Consequently, the proposed method achieves more accurate restoration than those of the traditional methods on both objective and subjective measures. Extensive experimental results demonstrate the improvement of the proposed method over the previous methods. © 2005 IEEE.

A novel image recognition method based on feature-extraction vector scheme
M Hiramoto, T Ogawa, M Haseyama
ICIP: 2004 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1- 5, 2, 3049, 3052, IEEE, 2004, [Peer-reviewed]
English, International conference proceedings, This paper introduces a method for recognizing images using a new approach to expressing images as vectors. Using this expression method, an image is constructed from 2 types of vectors - vectors indicating positions and vectors denoting intensity gradients for those positions. When investigating the amount of difference between two images, similarities are evaluated by calculating voting densities in the image space, using the vectors making up the sample image in relation to the vectors expressing the reference image. The expression proposed is invariant to image rotation and by changing the resolution hierarchically, recognition using this expression is also adaptable to perspective and detail. Using this method, we carried out experimentation recognizing representative images from various fields and the results show that the method is effective in discriminating between them.

A Note on Classification of Expert-novice Level Using Motion and Gaze Data via Spatial Temporal Attention GCN
清野竜生, 斉藤直輝, 小川貴弘, 浅水仁, 長谷山美紀, 人工知能学会全国大会論文集(Web), 38th, 2024

Estimation of contact accident risk based on recurrent neural network introducing spatial-temporal attention
五箇亮太, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

Zero-shot high-risk situation detection based on object detection and pose estimation using fixed camera at construction site
大羽賀駿也, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

Prediction of event locations from urgent call using large language models
吉田将規, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

サッカー映像における時空間的関係を考慮したシュート予測の高精度化に関する検討 : 競技者のチーム情報に基づく完全二部グラフの導入—A Note on Accurate Shoot Prediction Considering Players' Spatio-temporal Relations in Soccer Videos : Introduction of Complete Bipartite Graph Based on Players' Team Information—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
五箇 亮太, 諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 23, 28, Feb. 2023
東京 : 映像情報メディア学会, Japanese

クラス情報を導入したグラフ表現による教師有り潜在変数モデルの高精度化に関する検討—A Note on Improvement of Supervised Latent Variable Model with Graph-Encoded Class Information—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
渡部 航史, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 29, 33, Feb. 2023
東京 : 映像情報メディア学会, Japanese

属性情報の階層関係を考慮したアニメイラストのマルチラベル分類に関する検討—A Note on Multi-label Image Classification in Animation Illustration Considering Hierarchical Relationships of Attributes—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
蘭 子文, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 5, 9, Feb. 2023
東京 : 映像情報メディア学会, Japanese

ユーザの視線情報を考慮したコンテンツベースの画像再検索に関する検討—A note on gaze-dependent image re-ranking for content-based image retrieval—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
馮 鈺虎, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 89, 93, Feb. 2023
東京 : 映像情報メディア学会, Japanese

A Note on Classification of Proficiency-level Using Motion Data though Spatial Temporal Attention Graph Convolutional Network
清野竜生, 斉藤直輝, 小川貴弘, 浅水仁, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2023, 2023

Acquisition of feature representation of record data via graph neural network to support determination of deterioration levels
山本一輝, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Distress detection using egocentric videos for increasing discovery rate of novel distress during subway tunnel inspection
櫻井慶悟, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Classification of Winter Road Surface Condition Based on Multi-modal Transformer Using Sequential Data
諸戸祐哉, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Canonical Correlation Analysis Introducing Label Dequantization for Visual Emotion Recognition
斉藤直輝, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀, 電子情報通信学会論文誌 D(Web), J106-D, 5, 2023

A Note on Multi-label Image Classification in Animation Illustration Considering Hierarchical Relationships of Attributes
LAM Ziwen, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Text Prompt Tuning in Cross-modal Image Retrieval for a Specific Database
ZHANG Huaying, 柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos
吉田将規, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Improvement of Supervised Latent Variable Model with Graph-Encoded Class Information
渡部航史, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A note on gaze-dependent image re-ranking for content-based image retrieval
FENG Yuhu, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Accurate Shoot Prediction Considering Players’ Spatio-temporal Relations in Soccer Videos-Introduction of Complete Bipartite Graph Based on Players’ Team Information-
五箇亮太, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Interpretable Visual Question Answering Model with Visual Representation Based on Object Detection Model
ZHU He, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

AI研究の動向と最先端データ駆動型研究~次世代インフラ維持管理・防災・減災に向けた産学官・地域連携による取組~
小川貴弘, 寒地土木技術研究, 842, 2023

知識蒸留を用いたFew-shot Learningに基づく画像の感情ラベル推定に関する検討—A Note on Visual Sentiment Prediction Based on Few-shot Learning Using Knowledge Distillation—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
叶 穎睿, 諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 171, 175, Feb. 2022
Japanese

Variational Autoencoderに基づく深層生成モデルを用いた潜在表現のDisentanglementに関する検討 : Disentanglement評価指標を含む正則化損失の導入—A Note on Disentanglement Using Deep Generative Model Based on Variational Autoencoder : Introduction of Regularization Losses Based on Metrics of Disentangled Representation—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
中川 真, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 97, 102, Feb. 2022
Japanese

Captioning特徴を利用したグラフ畳み込みネットワークに基づくアニメイラストのマルチラベル画像分類に関する検討—A Note on Multi-label Image Recognition in Anime Illustration Based on Graph Convolutional Networks Using Captioning Features—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
蘭 子文, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 161, 165, Feb. 2022
Japanese

ゴム材料開発のためのGenerative Adversarial Networkに基づく配合量および物性からの電子顕微鏡画像の生成に関する一検討—A Note on Electron Microscope Image Generation from Mix Proportion and Material Property via Generative Adversarial Network for Rubber Materials—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
柳 凛太郎, 藤後 廉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 187, 191, Feb. 2022
Japanese

セマンティックセグメンテーションに対するマルチソースモデル適応に関する検討 : 複数のソースモデルからの不変な特徴表現の学習による適応精度の向上—A note on multi-source model adaptation for semantic segmentation : Improving adaptation performance by learning model-invariant representation from multiple source models—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
李 宗曜, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 37, 41, Feb. 2022
Japanese

地下鉄トンネル点検時の技術者から取得される生体信号と技術者の点検行動の関連性分析—Relevance Analysis between Bio-signals of Engineers Inspecting Subway Tunnels and Their Inspection Behaviors—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平澤 魁人, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 365, 370, Feb. 2022
Japanese

高速道路の遮音壁画像を用いた物体検出手法による変状分類の高精度化に関する検討—A note on improvement of distress classification using noise barrier images on highway via object detection method—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
梁 鋆, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 359, 363, Feb. 2022
Japanese

道路構造物の維持管理効率化に向けた変状画像分類の高精度化に関する検討 : テキストデータに基づく類似事例の含有率の導入—A Note on Improvement of Accuracy in Classification of Destress Images for Efficient Inspection of Road Structures : Introduction of Ratio of Similar Cases Based on Text Data—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平川 泰成, 小川 直輝, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 43, 48, Feb. 2022
Japanese

地下鉄トンネルの維持管理支援のためのマルチスケール解析を導入した深層学習に基づく変状検出に関する検討—A Note on Distress Detection Based on Deep Learning with Hierarchical Multi-Scale Attention Mechanism for Supporting Maintenance of Subway Tunnels—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
高田 紗弥, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 377, 381, Feb. 2022
Japanese

A Note on Automatic Diagnosis of Helicobacter Pylori Infection Based on Self-Supervised Learning and Self-Knowledge Distillation: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, マベ カツヒロ, ニシダ シュンペイ, トモダ ヨシヒロ, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 49, 52, Feb. 2022
English

A Note on Realizing Adversarial Defense Based on Regularization of Multi-stage Squeeze-and-Excitation Features: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
マエダ ケイスケ, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 87, 90, Feb. 2022
English

ユーザの動作情報を用いたコンテンツの関心度推定に関する検討 : 複数ユーザを導入した特徴統合の有効性検証—A Note on Interest level Estimation Using Users' behavior Information : Validating the Effectiveness of Feature Integration with Multiple Users—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
上川 恭平, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 103, 107, Feb. 2022
Japanese

画像注視時の脳活動信号を用いた圧縮再構成ネットワークに基づく視覚認知内容の推定に関する検討—A Note on Perceived Visual Content Estimation Based on Compressed Reconstruction Network Using Brain Signals While Gazing on Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
東 孝明, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 349, 353, Feb. 2022
Japanese

Attention mapに対する確信度を考慮可能な深層学習を用いた変状分類の高精度化に関する検討—A Note on Accurate Distress Classification Using Deep Learning Considering Confidence in Attention map—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
小川 直輝, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 371, 376, Feb. 2022
Japanese

A Note on Transformer-based Scene Recognition in Soccer Videos Using Different Lengths of Clips: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 167, 170, Feb. 2022
English

画像中の物体情報を考慮したユーザ類似度に基づく個人に特化した注視領域の推定に関する検討—A Note on Personalized Saliency Prediction Based on User Similarity Considering Object Information in Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 181, 186, Feb. 2022
Japanese

橋梁点検時の技術者の一人称および三人称視点映像を用いた点検動作の分類に関する検討—A Note on Inspection Action Classification Using First and Third Person Video of Engineers Inspecting Bridges—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
増田 毅, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 177, 180, Feb. 2022
Japanese

ユーザの嗜好を考慮した強化学習と知識グラフに基づく楽曲プレイリスト生成に関する検討—A Note on User Preference-Aware Music Playlist Generation Based on Reinforcement Learning and Knowledge Graph—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
櫻井 慶悟, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 109, 112, Feb. 2022
Japanese

A note on multi-source model adaptation for semantic segmentation-Improving adaptation performance by learning model-invariant representation from multiple source models-
LI Zongyao, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on User Preference-Aware Music Playlist Generation Based on Reinforcement Learning and Knowledge Graph
櫻井慶悟, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Disentanglement Using Deep Generative Model Based on Variational Autoencoder-Introduction of Regularization Losses Based on Metrics of Disentangled Representation-
中川真, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

Initiatives on Data-driven AI Research through Inter-disciplinary Collaboration
小川貴弘, Imaging Conference Japan論文集, 2022, 2022

A Note of the Relationship between Popularity Bias and Embedding Representations of Latent Factor Models in Collaborative Filtering
岡村洋希, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Robust Recommendation System for Domain-dependent Preference Based on Domain-shared Network
山本一輝, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Image Retrieval Robust for Changing Camera Views Using Synthesized Images by pixelNeRF
江良勇輝, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Study on Non-reference Image Quality Assessment Considering Phaseless Components of Neural Radiance Fields
河合雅斗, 柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Uncertainty-based Shoot Event Prediction from Scouting Videos of Soccer by Considering Spatio-Temporal Relations between Players
五箇亮太, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Estimation of Viewed Images Using Brain Activity Data While Viewing Images-A Verification of Estimation Accuracy by Regression Model Used for fMRI Decoder-
七田亮, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Estimation of Diabetic Retinopathy Grades Based on Unsupervised Domain Adaptation Using Fundus Images
國枝翼, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Electron Microscope Image Generation from Mix Proportion and Material Property via Generative Adversarial Network for Rubber Materials
柳凜太郎, 藤後廉, 前田圭介, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Improvement of Accuracy in Classification of Destress Images for Efficient Inspection of Road Structures- Introduction of Ratio of Similar Cases Based on Text Data-
平川泰成, 小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Distress Detection Based on Deep Learning with Hierarchical Multi-Scale Attention Mechanism for Supporting Maintenance of Subway Tunnels
高田紗弥, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Perceived Visual Content Estimation Based on Compressed Reconstruction Network Using Brain Signals While Gazing on Images
東孝明, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Interest level Estimation Using Users’ behavior Information-Validating the Effectiveness of Feature Integration with Multiple Users-
上川恭平, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Visual Sentiment Prediction Based on Few-shot Learning Using Knowledge Distillation
YE Yingrui, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

Relevance Analysis between Bio-signals of Engineers Inspecting Subway Tunnels and Their Inspection Behaviors
平澤魁人, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Personalized Saliency Prediction Based on User Similarity Considering Object Information in Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Accurate Distress Classification Using Deep Learning Considering Confidence in Attention map
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

Recent Researches on Media Engineering
新井啓之, 田良島周平, 河村圭, 堀淳志, 多田昌裕, 望月貴裕, 小川貴弘, 映像情報メディア学会誌, 76, 1, 78, 85, Jan. 2022
映像情報メディア学会, Japanese

招待講演 マルチメディアAI技術に基づく異分野融合研究と実社会応用—Invited Talk : Interdisciplinary Research and Real-world Applications Based on Multimedia AI technologies—情報センシング メディア工学 映像・信号の処理・解析・AI技術とその多分野応用
小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 13, 73, 80, Jun. 2021
映像情報メディア学会, Japanese

A Note on Distress Classification of High-resolution Subway Tunnel Images based on HRNet: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 13, 16, Feb. 2021
English

地下鉄トンネルの維持管理支援を目的とした深層学習に基づく変状検出の高精度化に関する検討 : 壁面の施工方法に注目した精度検証—A Note on Improving Performance of Deep Learning-based Distress Detection for Supporting Maintenance of Subway Tunnels : Accuracy Verification Focusing on Tunnel Wall Characteristics—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
春山 知生, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 1, 6, Feb. 2021
映像情報メディア学会, Japanese

特別講演 実店舗の防犯カメラ映像データを用いた顧客の関心推定に関する検討 : 姿勢推定モデルから得られる特徴量を用いた異常検知モデルの精度検証—A Note on Customer's Interest Estimation Method Using Security Camera Video Data from the Real Store : Validation of the accuracy of an anomaly detection model using features obtained from a posture estimation model—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
本間 勇紀, 藤後 廉, 阿部 真育, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 193, 198, Feb. 2021
映像情報メディア学会, Japanese

地下鉄トンネル点検時の技術者から取得される視線およびモーションデータに基づく熟練度の推定に関する検討—A Note on Estimation of Expert-Novice Level Based on Eye Tracking and Motion Data from Engineers While Inspecting Subway Tunnel—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
赤松 祐亮, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 7, 12, Feb. 2021
映像情報メディア学会, Japanese

特別講演 ゴム材料開発のためのConditional StyleGANに基づく配合量からの電子顕微鏡画像の生成に関する一検討—A Note on Electron Microscope Image Generation from Mix Proportion via Conditional Style Generative Adversarial Network for Rubber Materials—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
柳 凜太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 171, 175, Feb. 2021
映像情報メディア学会, Japanese

特別講演 路面画像を用いた深層学習に基づく路面状態の分類に関する検討—A Note on Discrimination of Road Surface Conditions Based on Deep Learning Using Road Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 165, 169, Feb. 2021
映像情報メディア学会, Japanese

TCCに基づく自己教師学習による特徴表現を用いた映像中の人物動作の検出に関する検討 : 動作検出パラメータが与える影響に対する考察—A Note on Action Detection Using Feature Representation via Self-Supervised Learning based on TCC : Effectiveness evaluation of parameters for action detection—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
増田 毅, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 47, 51, Feb. 2021
映像情報メディア学会, Japanese

電子顕微鏡により撮像されたゴム材料からの異常検知に基づく劣化領域の推定に関する一検討 : 深層学習モデルにより抽出された特徴表現の有効性検証—A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images : Verification of Feature Representations Extracted from Deep Learning Models—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
松本 真直, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 43, 46, Feb. 2021
映像情報メディア学会, Japanese

サッカー映像における情報の提示位置の推定に関する検討 : 視聴者の注視位置および選手位置に基づいたオブジェクト配置の決定—A Note on Position Estimation of Superimposing Information in Soccer Videos : Decision of Object Placement Based on User's Gaze Position and Players' Positions—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 77, 81, Feb. 2021
映像情報メディア学会, Japanese

A Note on Automatic Diagnosis of Helicobacter Pylori Infection Based on EfficientNet with Flooding Loss: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, マベ カツヒロ, ニシダ シュンペイ, トモダ ヨシヒロ, シミズ ヒカリ, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 23, 26, Feb. 2021
English

Twitterと映像を用いたMVAEに基づく野球映像の重要シーン予測に関する検討—A Note on Prediction of Important Scenes in Baseball Videos via Multimodal Variational Autoencoder Using Tweets and Videos—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平澤 魁人, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 71, 75, Feb. 2021
映像情報メディア学会, Japanese

Detecting axillary lymph node metastasis of breast cancer with FDG-PET/CT images based on attention mechanism: FDG-PET/CT画像を用いたattention mechanismに基づく乳癌の腋窩リンパ節転移の検出: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, ヒラタ ケンジ, キタジマ カズヒロ, タケナカ ジュンキ, ミヨシ ヤスオ, クドウ コウスケ, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 33, 36, Feb. 2021
English

特別講演 ユーザの嗜好を反映可能なインテリアコーディネート検索技術の構築 : コーディネートを表現可能な特徴の抽出と実データへの適用—A Note on Interior Coordination Retrieval Reflecting User's Preferences : Extraction of Features Representing Coordination and Application to Real Data—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 187, 191, Feb. 2021
映像情報メディア学会, Japanese

A Note on Off-screen Sound Detection Based on Audio-visual Spatialization-Introduction of Audio-visual Features Based on Self-supervised Learning-
吉田将規, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Effect of Visual Representation Ability to Performance of World Models
大羽賀駿也, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Verification of Vulnerability of Graph-based Recommender Models under Shilling Attacks
小野寺望, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Multi-label Image Recognition in Anime Illustration Based on Graph Convolutional Networks
LAN Ziwen, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Introducing Self-supervised Learning to Latent Variable Model for Low-dimensional Feature Extraction
渡部航史, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

An Note on Use of Multiple Datasets in Visual Sentiment Prediction Using Few-shot Learning
YE Yingrui, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Accuracy Assessment Focusing on Manipulation Area by Text-Guided Image Manipulation
渡邉優宇人, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

The Clinical Application of Artificial Intelligence for Detecting Atrophic Gastritis by Upper Gastrointestinal Barium X-ray Radiography
竹内千尋, 藤後廉, 小川貴弘, 長谷山美紀, 山道信毅, 臨床消化器内科, 36, 8, 1062, 1065, 2021
日本メディカルセンター

A Proposal and Evaluation of a New Method Incorporating Indicators of Blood Flow and Resistance to Improve the Accuracy of Continuous Blood Pressure Estimation Using PPG
川上健, 住友和弘, 菅野厚博, 小川貴弘, 南重信, 長谷山美紀, 電気学会論文誌 E, 141, 6, 186, 196, 2021
This paper presents a new method which achieves accuracy like cuff-based BP (Blood pressure) measurement to improve the conventional cuff-less BP estimation method for a wrist-type single PPG sensor. The conventional method used the modified normalized pulse volume and the pulse rate for resistance and flow rate respectively to express BP. However, it is required to improve the performance in comparison with methods based on pulse wave velocity which involve elasticity rate relating to vascular pressure. The new method incorporates both resistance which is derived from relative velocity and peripheral arterial elasticity based on vascular function to elevate the accuracy of diastolic BP estimation. Systolic BP is also improved by the sum of estimated pulse pressure with resistance and the estimated diastolic BP value. The accuracy comparison between the conventional and proposal method was performed by cross validation for estimated BP values in 5 seconds each by using wrist-type PPG signals. It is found that the estimated SBP and DBP of the conventional method differed from the reference BP by 3.9 (mean absolute error) ±12 (standard deviation) mmHg and 4.7 ± 11 mmHg respectively. In the proposal method, the differences of estimated values are 2.5 ± 7.2 mmHg and 2.7 ± 7.8 mmHg. When compared to the AAMI requirement (5 ± 8 mmHg for both SBP and DBP estimation), the results suggest that the proposed method has great potential for a single PPG sensor device., The Institute of Electrical Engineers of Japan, Japanese

A Note on Estimation of Semantic Content Based on a Question Answering Model Using Brain Activity Data while Viewing Images-Improvement of Estimation Performance Based on fine-tuning of the VQA model-
高田紗弥, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 27, 31, 2021
映像情報メディア学会, Japanese

A Note on Improvement of Image Sentiment Analysis Based on Introduction of Image Captioning
LIANG Yun, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 65, 69, 2021
映像情報メディア学会, Japanese

A Note on Cross-domain Recommendation Based on Multi-layer Graph Analysis with Visual Features
平川泰成, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 59, 63, 2021
映像情報メディア学会, Japanese

A Note on Improving Performance of Deep Learning-based Distress Detection for Supporting Maintenance of Subway Tunnels-Accuracy Verification Focusing on Tunnel Wall Characteristics-
春山知生, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Customer’s Interest Estimation Method Using Security Camera Video Data from the Real Store-Validation of the accuracy of an anomaly detection model using features obtained from a posture estimation model-
本間勇紀, 本間勇紀, 藤後廉, 阿部真育, 小川貴弘, 長谷山美紀, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Estimating of Deteriorated Refions Based on Anomaly Detection from Rubber Material Electron Microscope Images-Verification of Feature Representations Extracted from Deep Learning Models-
松本真直, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Estimation of Expert-Novice Level Based on Eye Tracking and Motion Data from Engineers While Inspecting Subway Tunnel
赤松祐亮, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Action Detection Using Feature Representation via Self-Supervised Learning based on TCC-Effectiveness evaluation of parameters for action detection-
増田毅, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Discrimination of Road Surface Conditions Based on Deep Learning Using Road Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Position Estimation of Superimposing Information in Soccer Videos-Decision of Object Placement Based on User’s Gaze Position and Players’ Positions-
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Prediction of Important Scenes in Baseball Videos via Multimodal Variational Autoencoder Using Tweets and Videos
平澤魁人, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Interior Coordination Retrieval Reflecting User’s Preferences-Extraction of Features Representing Coordination and Application to Real Data-
藤後廉, 小川貴弘, 長谷山美紀, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Accurate Distress Image Classification of Road Structures Using Attention Map Based on Text Data
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 17, 21, 2021
映像情報メディア学会, Japanese

A Note on Electron Microscope Image Generation from Mix Proportion via Conditional Style Generative Adversarial Network for Rubber Materials
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

AIを中心とした医療デジタル技術基盤の構築へ向けた取り組み
藤後廉, 小川貴弘, 長谷山美紀, 日本消化器がん検診学会雑誌(Web), 59, Supplement 1, 2021

Interdisciplinary Research and Real-world Applications Based on Multimedia AI technologies
小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 13(IST2021 22-29/ME2021 53-60), 2021

Value Creation from Real Data via Human-Centric AI
OGAWA Takahiro, Journal of The Society of Instrument and Control Engineers, 59, 5, 295, 296, 10 May 2020
The Society of Instrument and Control Engineers, Japanese

IEEE GCCE 2019開催報告—IEEE GCCE 2019 Report
小川 貴弘, 映像情報メディア学会誌 = The journal of the Institute of Image Information and Television Engineers, 74, 2, 314, 318, Mar. 2020
映像情報メディア学会, Japanese

画像内の物体に着目した画像検索に関する検討 : RetinaNetを用いた物体認識に基づく高精度化—A Note on Image Retrieval Focusing on Objects in Images : Improving Retrieval Performance Based on Object Recognition Using RetinaNet—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
柳 凛太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 377, 381, Feb. 2020
映像情報メディア学会, Japanese

画像注視時の視線データを用いた関心度の推定に関する検討 : ラベルの逆量子化を導入したsMVCCAによる推定精度の高精度化—A Note on Eye Gaze Based User-specific Interest Estimation for Images : Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization—ITS : Intelligent Transport Systems Technology
松本 真直, 斉藤 直輝, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 73, 77, Feb. 2020
電子情報通信学会, Japanese

画像注視時の視線データを用いた関心度の推定に関する検討 : ラベルの逆量子化を導入したsMVCCAによる推定精度の高精度化—A Note on Eye Gaze Based User-specific Interest Estimation for Images : Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松本 真直, 斉藤 直輝, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 73, 77, Feb. 2020
映像情報メディア学会, Japanese

地下鉄トンネル点検時の生体信号に基づいた熟練および若手技術者の分類に関する検討—A Note on Classification of Experienced and Novice Inspectors Based on Bio-signals While Inspecting in Subway Tunnels—ITS : Intelligent Transport Systems Technology
九島 哲哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 101, 105, Feb. 2020
電子情報通信学会, Japanese

特別講演 社会インフラ維持管理効率化のための最先端AI技術の導入 : 点検データを用いた変状分類の精度向上に向けた取り組み—Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure : Performance Improvement of Distress Image Classification Utilizing Inspection Data—ITS : Intelligent Transport Systems Technology
前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 359, 362, Feb. 2020
電子情報通信学会, Japanese

道路構造物の変状評価における技術者の視線データと熟練度の分析に関する一考察—A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松井 太我, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 97, 100, Feb. 2020
映像情報メディア学会, Japanese

A note on detection of distress regions in subway tunnels by using U-net based network: ITS : Intelligent Transport Systems Technology
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 119, 421, 69, 72, Feb. 2020
English

道路構造物の維持管理効率化に向けた変状画像分類における信頼性の向上に関する検討—A Note on Improvement of Reliability of Distress Image Classification for Efficient Maintenance of Road Structures—ITS : Intelligent Transport Systems Technology
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 51, 56, Feb. 2020
電子情報通信学会, Japanese

画像内の物体に着目した画像検索に関する検討 : RetinaNetを用いた物体認識に基づく高精度化—A Note on Image Retrieval Focusing on Objects in Images Improving Retrieval Performance Based on Object Recognition Using RetinaNet—ITS : Intelligent Transport Systems Technology
柳 凜太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 377, 381, Feb. 2020
電子情報通信学会, Japanese

弱異常検知アルゴリズムに基づくCCTV映像を用いた河川利用者の異常行動の検出に関する検討—A Note on Abnormal Motion Detection of River Users on CCTV Videos Based on Weakly Supervised Anomaly Detection Algorithm—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
渡邊 はるな, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 367, 370, Feb. 2020
映像情報メディア学会, Japanese

A note on retrieval of visually similar distress regions in subway tunnel images : Introduction of deep features extracted by semantic segmentation network: ITS : Intelligent Transport Systems Technology
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 119, 421, 65, 68, Feb. 2020
English

画像注視時のヒトの感情推定のための視線特徴の推定に関する検討—A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 85, 89, Feb. 2020
映像情報メディア学会, Japanese

ゴム材料の配合量を用いたAC-GANに基づく電子顕微鏡画像の生成に関する一検討—A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions—ITS : Intelligent Transport Systems Technology
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 107, 111, Feb. 2020
電子情報通信学会, Japanese

ゴム材料の配合量を用いたAC-GANに基づく電子顕微鏡画像の生成に関する一検討—A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 107, 111, Feb. 2020
映像情報メディア学会, Japanese

道路構造物の変状評価における技術者の視線データと熟練度の分析に関する一考察—A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures—ITS : Intelligent Transport Systems Technology
松井 太我, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 97, 100, Feb. 2020
電子情報通信学会, Japanese

エッジの補完を導入したツイートクラスタリングの高精度化に関する検討—A Note on Tweet Clustering by Complementing Edges among Tweets—ITS : Intelligent Transport Systems Technology
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 61, 64, Feb. 2020
電子情報通信学会, Japanese

地下鉄トンネル維持管理支援を目的とした類似画像の検索に関する検討 : 技術者の評価を反映可能な距離計量学習の導入—A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels : Introduction of Distance Metric Learning Reflecting Inspectors' Evaluation—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松本 有衣, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 47, 50, Feb. 2020
映像情報メディア学会, Japanese

特別講演 社会インフラ維持管理効率化のための最先端AI技術の導入 : 点検データを用いた変状分類の精度向上に向けた取り組み—Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure : Performance Improvement of Distress Image Classification Utilizing Inspection Data—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 359, 362, Feb. 2020
映像情報メディア学会, Japanese

地下鉄トンネル維持管理支援を目的とした類似画像の検索に関する検討 : 技術者の評価を反映可能な距離計量学習の導入—A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels : Introduction of Distance Metric Learning Reflecting Inspectors' Evaluation—ITS : Intelligent Transport Systems Technology
松本 有衣, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 47, 50, Feb. 2020
電子情報通信学会, Japanese

穿孔データを用いたオンライン学習に基づく岩盤の圧縮強度指数推定に関する検討—A Note on Estimation of rock compressive strength index from drilling data based on online learning—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
山本 健太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 57, 60, Feb. 2020
映像情報メディア学会, Japanese

穿孔データを用いたオンライン学習に基づく岩盤の圧縮強度指数推定に関する検討—A Note on Estimation of rock compressive strength index from drilling data based on online learning—ITS : Intelligent Transport Systems Technology
山本 健太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 57, 60, Feb. 2020
電子情報通信学会, Japanese

サッカー映像視聴時の視線データを用いた情報の提示領域の算出に関する検討—Presentation Region Calculation of Information Based on Gaze Tracking Data in Soccer Videos—ITS : Intelligent Transport Systems Technology
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 91, 95, Feb. 2020
電子情報通信学会, Japanese

地下鉄トンネル点検時の生体信号に基づいた熟練および若手技術者の分類に関する検討—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
九島 哲哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 101, 105, Feb. 2020
映像情報メディア学会, Japanese

Bidirectional LSTMを用いたサッカー映像におけるイベント分類に関する検討—A Note on Event Classification in Soccer Videos Using Bidirectional LSTM—ITS : Intelligent Transport Systems Technology
春山 知生, 高橋 翔, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 371, 375, Feb. 2020
電子情報通信学会, Japanese

画像注視時のヒトの感情推定のための視線特徴の推定に関する検討—A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images—ITS : Intelligent Transport Systems Technology
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 85, 89, Feb. 2020
電子情報通信学会, Japanese

道路構造物の維持管理効率化に向けた変状画像分類における信頼性の向上に関する検討—A Note on Improvement of Reliability of Distress Image Classification for Efficient Maintenance of Road Structures—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 51, 56, Feb. 2020
映像情報メディア学会, Japanese

A Note on Tweet Clustering by Complementing Edges among Tweets
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 61, 64, Feb. 2020
映像情報メディア学会, Japanese

A Note on Estimation of Image Categories Using Brain Activity While Viewing Images Based on MVBGM-MS
赤松 祐亮, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 79, 83, Feb. 2020
映像情報メディア学会, Japanese

サッカー映像視聴時の視線データを用いた情報の提示領域の算出に関する検討 (マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス)
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 91, 95, Feb. 2020
映像情報メディア学会, Japanese

A Note on Event Classification in Soccer Videos Using Bidirectional LSTM
春山 知生, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 371, 375, Feb. 2020
映像情報メディア学会, Japanese

A Note on Tweet Clustering by Complementing Edges among Tweets
滝村祥司, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Event Classification in Soccer Videos Using Bidirectional LSTM
春山知生, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure-Performance Improvement of Distress Image Classification Utilizing Inspection Data-
前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

Presentation Region Calculation of Information Based on Gaze Tracking Data in Soccer Videos
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Improvement of Reliability of Dstress Image Classification for Efficient Maintenace of Road Structures
堀井風葉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels-Introduction of Distance Metric Learning Reflecting Inspectors’ Evaluation-
松本有衣, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of rock compressive strength index from drilling data based on online learning
山本健太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Classification of Experienced and Novice Inspectors Based on Bio-signals While Inspecting in Subway Tunnels
九島哲哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Abnormal Motion Detection of River Users on CCTV Videos Based on Weakly Supervised Anomaly Detection Algorithm
渡邊はるな, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Eye Gaze Based User-specific Interest Estimation for Images-Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization-
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures
松井太我, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of Image Categories Using Brain Activity While Viewing Images Based on MVBGM-MS
赤松祐亮, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Image Retrieval Focusing on Objects in Images-Improving Retrieval Performance Based on Object Recognition Using RetinaNet-
柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Music Playlist Generation Based on Reinforcement Learning Using Self-Organized Map
櫻井慶悟, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Discrimination of Road Surface Conditions Based on Anomaly Detection Using Road Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Cross-domain Recommendation Based on Multilayer Graph Analysis-Performance Evaluation with Change in Dimension of Embedding Feature-
平川泰成, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Detection of Human Actions Based on Temporal Cycle Consistency Learning
増田毅, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on User Interest Estimation Based on Feature Integration Using m-SimGP
上川恭平, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Image Sentiment Analysis based on Multi-level Deep Metric Net
LIANG Yun, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Dataset Complexity Assessment Based on nAULS
LI Guang, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Improvement of Distress Detection Performance in Subway Tunnel Images by Data Augmentation Based on RICAP
春山知生, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note of Estimation of Viewed Images Using fMRI Data While Viewing Images-Introduction of Shared Brain Responses of Multiple Subjects Based on Probabilistic Generative Model-
東孝明, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Disentangled Representation Learning via Pre-Trained Semantic Segmentation Model
中川真, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

Human-centered image classification via a neural network considering visual and biological features.
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, Multim. Tools Appl., 79, 7-8, 4395, 4415, 2020, [Peer-reviewed]
© 2019, Springer Science+Business Media, LLC, part of Springer Nature. In this paper, we propose a human-centered image classification via a neural network considering visual and biological features. The proposed method has two novelties. Firstly, we apply Group-Sparse Local Fisher Discriminant Analysis (GS-LFDA) to biological features. GS-LFDA realizes dimensionality reduction and noise elimination for biological features with consideration of local structures and class information. Secondly, we construct a Canonical Correlation Analysis (CCA)-based hidden layer via Discriminative Locality Preserving CCA (DLPCCA). DLPCCA transforms visual features into effective features by considering the relationships with biological information and class information. The CCA-based hidden layer enables transformation of visual features into effective features for image classification from a small number of training samples. Furthermore, once the projection can be obtained in the training phase, elimination of the need for biological data acquisition in the test phase is realized. This is another merit of our method.

胃バリウム検査におけるAIによるHelicobacter pylori診断 (Helicobacter Topics! : 「旬のHelicobacter」を知り,驚き,理解を深め,楽しむ)
藤後 廉, 間部 克裕, 山道 信毅, 大泉 晴史, 小川 貴弘, 長谷山 美紀, 加藤 元嗣, 坂本 直哉, Helicobacter research : Journal of helicobacter research, 23, 2, 92,133, 137, Nov. 2019
先端医学社, Japanese

A Note on User-specific Visual Attention Estimation Based on Visual and Spatial Information in Images : A Study on Relationship between Estimation Performance and the Similarity of Visual Features
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 13, 16, 19 Jul. 2019
電子情報通信学会, Japanese

A Note on Eye Gaze Based User-specific Interest Estimation for Images : Performance Improvement Based on Decision Level Fusion
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 17, 20, 19 Jul. 2019
電子情報通信学会, Japanese

A Note on Semantic Evaluation of Images Generated by Text-to-image Generative Adversarial Networks
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 21, 24, 19 Jul. 2019
電子情報通信学会, Japanese

Performance improvement of users' interest estimation using viewing behavior data obtained by sensors
長谷山 美紀, 小川 貴弘, 髙橋 翔, 原川 良介, 画像ラボ, 30, 7, 8, 12, Jul. 2019
日本工業出版, Japanese

IEEE GCCE 2018 Report : Places for creating new innovation
小川 貴弘, 映像情報メディア学会誌 = The journal of the Institute of Image Information and Television Engineers, 73, 2, 299, 302, Mar. 2019
映像情報メディア学会, Japanese

A Note on Performance Improvement of Distress Detection in Subway Tunnels Using CNN
WANG An, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 121, 124, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Estimation of Characteristic Curve for Rubber Materials Using Electron Microscope Images and Mix Proportions
斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 261, 264, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images
藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 265, 268, 19 Feb. 2019
電子情報通信学会, Japanese

Performance Verification of Stress Detection Using a High-performance Wristband with Multi-sensing
伊藤良起, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 269, 274, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Attribute Estimation for Improving Interpretability of Convolutional Neural Network
堀井風葉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 275, 279, 19 Feb. 2019
電子情報通信学会, Japanese

A note on estimation of inspectors' visual attention using distress images of subway tunnels : Trial introduction of deep learning-based saliency prediction methods
斉藤僚汰, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 281, 285, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Retrieval of Similar Inspection Records Based on Distance Metric Learning for Supporting Maintenance of Subway Tunnels
弦間奨, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 287, 290, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Estimation of Rock Drilling Energy Using Tunnel Working Face Images
山本健太郎, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 291, 294, 19 Feb. 2019
電子情報通信学会, Japanese

A note on detection of distress regions in subway tunnels from laser data : Visualization of distress regions with fully convolutional networks
豊田陽, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 295, 299, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Popularity Prediction of Artists on Music Streaming Services Based on Network Analysis Using Heterogeneous Features
松本有衣, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 301, 306, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Twitter Followee Recommendation Based on Adaptive Integration of Multiple Classification Results
滝村祥司, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 307, 310, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Automatic Malignant Tumor Candidate Detection Based on a 3D Deep Residual Network with FDG-PET/CT Images (ITS)
李 宗曜, 藤後 廉, 小川 貴弘, 平田 健司, 真鍋 治, 志賀 哲, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 311, 314, 19 Feb. 2019
電子情報通信学会, English

A Note on Gastritis Detection from Gastric X-ray Images via Transfer Learning Approach
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 315, 318, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Generation of Gastritis Images Based on Progressive Growing GAN for Gastritis Classification
渡邊はるな, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 319, 322, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Eye Gaze-based Estimation of Emotion Evoked by Watching Images : Performance Improvement via Feature Transformation Based on KDLPCCA
松井太我, 斉藤直輝, 小川貴弘, 浅水仁, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 323, 327, 19 Feb. 2019
電子情報通信学会, Japanese

A note on interest level estimation for videos using users' behavior based on OpenPose
九島哲哉, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 351, 354, 19 Feb. 2019
電子情報通信学会, Japanese

A Method for Predicting Importance of Attack Players based on Gaze Tracking Data in Soccer Videos
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 355, 359, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Accurate Estimation of Deterioration Levels on Transmission Towers via Deep Learning Using Heterogeneous Features
前田圭介, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 361, 364, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Action Classification of River Users on CCTV Videos
川本舜, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 365, 369, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Anomaly Detection of River Users on CCTV Videos
川本舜, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449, 371, 374, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Performance Improvement of Distress Detection in Subway Tunnels Using CNN
王 安, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 121, 124, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Characteristic Curve for Rubber Materials Using Electron Microscope Images and Mix Proportions
斉藤 直輝, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 261, 264, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images
藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 265, 268, Feb. 2019
映像情報メディア学会, Japanese

Performance Verification of Stress Detection Using a High-performance Wristband with Multi-sensing
伊藤 良起, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 269, 274, Feb. 2019
映像情報メディア学会, Japanese

A Note on Attribute Estimation for Improving Interpretability of Convolutional Neural Network
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 275, 279, Feb. 2019
映像情報メディア学会, Japanese

A note on estimation of inspectors' visual attention using distress images of subway tunnels : Trial introduction of deep learning-based saliency prediction methods
斉藤 僚汰, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 281, 285, Feb. 2019
映像情報メディア学会, Japanese

A Note on Retrieval of Similar Inspection Records Based on Distance Metric Learning for Supporting Maintenance of Subway Tunnels
弦間 奨, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 287, 290, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Rock Drilling Energy Using Tunnel Working Face Images
山本 健太郎, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 291, 294, Feb. 2019
映像情報メディア学会, Japanese

A note on detection of distress regions in subway tunnels from laser data : Visualization of distress regions with fully convolutional networks
豊田 陽, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 295, 299, Feb. 2019
映像情報メディア学会, Japanese

A Note on Popularity Prediction of Artists on Music Streaming Services Based on Network Analysis Using Heterogeneous Features
松本 有衣, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 301, 306, Feb. 2019
映像情報メディア学会, Japanese

A Note on Twitter Followee Recommendation Based on Adaptive Integration of Multiple Classification Results
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 307, 310, Feb. 2019
映像情報メディア学会, Japanese

A Note on Automatic Malignant Tumor Candidate Detection Based on a 3D Deep Residual Network with FDG-PET/CT Images (マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス)
李 宗曜, 藤後 廉, 小川 貴弘, 平田 健司, 真鍋 治, 志賀 哲, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 311, 314, Feb. 2019
映像情報メディア学会, English

A Note on Gastritis Detection from Gastric X-ray Images via Transfer Learning Approach
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 315, 318, Feb. 2019
映像情報メディア学会, Japanese

A Note on Generation of Gastritis Images Based on Progressive Growing GAN for Gastritis Classification
渡邊 はるな, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 319, 322, Feb. 2019
映像情報メディア学会, Japanese

A Note on Eye Gaze-based Estimation of Emotion Evoked by Watching Images : Performance Improvement via Feature Transformation Based on KDLPCCA
松井 太我, 斉藤 直輝, 小川 貴弘, 浅水 仁, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 323, 327, Feb. 2019
映像情報メディア学会, Japanese

A note on interest level estimation for videos using users' behavior based on OpenPose
九島 哲哉, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 351, 354, Feb. 2019
映像情報メディア学会, Japanese

A Method for Predicting Importance of Attack Players based on Gaze Tracking Data in Soccer Videos
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, 2019 IEEE 8th Global Conference on Consumer Electronics, GCCE 2019, 43, 5, 355, 359, Feb. 2019
This paper presents a new method for predicting importance of attack players based on multiple gaze tracking data in soccer videos. In order to understand the game situation in the soccer game, experienced soccer players look at the movement, position, and space which are other players more often and faster than inexperienced players. Also, these features are different from each experienced player. For this reason, the gaze tracking data of experienced soccer player is useful for tactical analysis. Therefore, by introducing the gaze tracking data of multiple experienced soccer players into the importance prediction, more robust prediction than a method which use only one player's gaze data can be expected. Since this predicted importance is obtained from gaze data of the experienced soccer players, the obtained one is more useful data for understanding soccer game situation. Therefore, the proposed method can contribute to automatically data generation for developing novel services of video/data distribution that supports to easily understanding of video contents for many viewers. Experimental results using actual soccer videos showed the effectiveness of our method., IEEE, Japanese

A Note on Accurate Estimation of Deterioration Levels on Transmission Towers via Deep Learning Using Heterogeneous Features
前田 圭介, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 361, 364, Feb. 2019
映像情報メディア学会, Japanese

A Note on Action Classification of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 365, 369, Feb. 2019
映像情報メディア学会, Japanese

A Note on Anomaly Detection of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 371, 374, Feb. 2019
映像情報メディア学会, Japanese

人工知能が医療を変える!医療分野におけるAI研究開発最前線2019 II 領域別・画像診断におけるAI研究開発の最前線 7.その他 3)核医学におけるディープラーニングを用いた画像診断,画像処理
平田健司, 藤後廉, 小川貴弘, 長谷山美紀, 志賀哲, Innervision, 34, 7, 2019

ユーザの好みを反映した潜在特徴を用いたGCNに基づくTumblrポストの推薦
大友一馬, 原川良介, 小川貴弘, 長谷山美紀, 岩橋政宏, 信号処理シンポジウム講演論文集(CD-ROM), 34th, 2019

変状分類におけるGrad-CAM++に基づいたCNNの注目領域の可視化に関する検討
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

サッカー映像視聴時の視線データを用いた周辺視に基づく攻撃選手の重要度算出に関する検討
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

画像注視時の視線データを用いた関心の識別に関する検討-LDCCAに基づく特徴量の射影による高精度化-
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Sparse Bayesian Learningに基づく注視領域の時間変化を考慮したヒトの感情推定に関する検討
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

複数の特徴量から得られる類似度の統合に基づくサッカー映像における類似場面検索の高精度化に関する検討
春山知生, 高橋翔, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

複数変量の生成モデルに基づく画像注視時の脳活動データを用いた画像カテゴリの推定に関する検討
赤松祐亮, 原川良介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Text-to-imageGANに基づくスタイル変換に関する検討-Image-to-textモデル導入による高精度化-
古寺恵, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Semantic Segmentationに基づく地下鉄トンネルにおける変状検出に関する検討
WANG An, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

脳活動データを用いた注視画像の再構成における用いる視覚野の領域に関する検討
高田紗弥, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Twitterを用いた異常検知に基づく野球映像の重要シーン検出に関する検討
平澤魁人, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

生体情報解析を導入した次世代AI技術の構築
長谷山美紀, 小川貴弘, 電子情報通信学会大会講演論文集(CD-ROM), 2019, 2019

Classification of tourism categories based on heterogeneous features considering existence of reliable results
Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents a classification method of tourism categories based on heterogeneous features considering existence of reliable results. The proposed method performs estimation of existence of reliable results based on one-versus-one scheme from three kinds of classification results obtained from tourism images, geotags and textual tags, separately. Then if the reliable result is included in the above results, this result is regarded as a final result. Otherwise, the final result is obtained by the multiple annotator logistic regression. The proposed method realizes accurate classification by estimating the existence of reliable results from more than two kinds of results.

Field position estimation in soccer videos using convolutional neural network-based image features
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents a novel estimation method of field positions in soccer videos using Convolutional Neural Network (CNN)-based image features. CNN-based features have been well known to be ei€ective for tasks in machine learning. Therefore, the proposed method adopts CNN-based image features. By using these image features, the proposed method enables accurate estimation of soccer field positions than handcrafted features, i.e., Hough transform-based features. We show the ei€ectiveness of our method via experiment results using actual soccer videos.

Gaze-based visual feature extraction via DLPCCA for visual sentiment estimation
Taiga Matsui, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents gaze-based visual feature extraction via Discriminative Locality Preserving Canonical Correlation Analysis (DLPCCA) for visual sentiment estimation. The proposed method calculates novel visual features reflecting users' visual sentiment by applying DLPCCA to gaze and original visual features. Consequently, accurate visual sentiment estimation becomes feasible by utilizing the novel visual features derived by the proposed method.

Automatic Metastatic Bone Tumor Classification with DCNN-based Features Using Treatment-planning CT Images
Haruna Watanabe, Ren Togo, Takahiro Ogawa, Miki Haseyama, Koichi Yasuda, Khin Khin Tha, Kohsuke Kudo, Hiroki Shirato, INTERNATIONAL FORUM ON MEDICAL IMAGING IN ASIA 2019, 11050, 2019
In this paper, we propose a method to classify metastatic bone tumors using treatment-planning computed tomography images. The proposed method utilizes pre-trained deep convolutional neural network (DCNN) models as feature extractors and enables the metastatic bone tumor classification by using the obtained features. Performance of several state-of-the-art DCNN-based features was compared and evaluated in our experiment., SPIE-INT SOC OPTICAL ENGINEERING, English

The iSpect DIA-10 Dynamic Particle Image Analysis System
赤坂 剛, 深井 秋博, 小川 貴弘, 島津評論, 76, 1, 15, 25, 2019
島津評論編集部 ; 1940-, Japanese

Automatic detection of fish sounds based on multi-stage classification including logistic regression via adaptive feature weighting
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, Tomonari Akamatsu, Journal of the Acoustical Society of America, 144, 5, 2709, 2718, 01 Nov. 2018
© 2018 Acoustical Society of America. This paper presents a method for automatic detection of fish sounds in an underwater environment. There exist two difficulties: (i) features and classifiers that provide good detection results differ depending on the underwater environment and (ii) there are cases where a large amount of training data that is necessary for supervised machine learning cannot be prepared. A method presented in this paper (the proposed hybrid method) overcomes these difficulties as follows. First, novel logistic regression (NLR) is derived via adaptive feature weighting by focusing on the accuracy of classification results by multiple classifiers, support vector machine (SVM), and k-nearest neighbors (k-NN). Although there are cases where SVM or k-NN cannot work well due to divergence of useful features, NLR can produce complementary results. Second, the proposed hybrid method performs multi-stage classification with consideration of the accuracy of SVM, k-NN, and NLR. The multi-stage acquisition of reliable results works adaptively according to the underwater environment to reduce performance degradation due to diversity of useful classifiers even if abundant training data cannot be prepared. Experiments on underwater recordings including sounds of Sciaenidae such as silver croakers (Pennahia argentata) and blue drums (Nibea mitsukurii) show the effectiveness of the proposed hybrid method.

International Conference on Multimedia Retrieval (ICMR2018) : With the organizers' viewpoint
相澤 清晴, 佐藤 真一, 柳井 啓司, 井手 一郎, 山﨑 俊彦, 入江 豪, 小川 貴弘, 望月 貴裕, 新田 直子, 篠田 浩一, 呉 志鵬, 松井 勇佑, 牛久 祥孝, 内田 祐介, 映像情報メディア学会誌 = The journal of the Institute of Image Information and Television Engineers, 72, 6, 888, 895, Nov. 2018
映像情報メディア学会, Japanese

画像注視時の注視領域の時間変化を考慮したテンソル解析に基づく感情推定に関する検討
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.92, 27 Oct. 2018
Japanese

地下鉄トンネルの点検における視線データを用いた熟練度の分析に関する一考察
斉藤僚汰, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.11, 27 Oct. 2018
Japanese

深層学習に基づく画像特徴量を利用した放射線治療用CT画像における転移性骨腫瘍の検出に関する検討
渡邊はるな, 藤後廉, 小川貴弘, 長谷山美紀, 安田耕一, THA Khin Khin, 工藤與亮, 白土博樹, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.87, 27 Oct. 2018
Japanese

AC‐GANに基づく胃炎識別のための画像生成に関する検討
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.89, 27 Oct. 2018
Japanese

Tri‐trainingに基づく胃X線画像を用いた胃炎の識別に関する検討
LI Zongyao, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.88, 27 Oct. 2018
Japanese

AttnGANを用いたシーン検索に関する検討―再検索の導入による高精度化―
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.10, 27 Oct. 2018
Japanese

Inpainting Based on Low-dimensional Image Approximation and Its Applications
小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 147, 1, 4, 20 Jul. 2018
電子情報通信学会, Japanese

PRIMARY SJOGREN'S SYNDROME MISDIAGNOSED AS ORTHOSTATIC DYSREGULATION : A CASE REPORT
岡本 奈央子, 岡本 義久, 石川 琢也, 大貫 裕太, 小川 玲, 児玉 雅彦, 渡邊 常樹, 西岡 貴弘, 池田 裕一, 磯山 恵一, 昭和学士会雑誌 = Journal of the Showa University Society, 78, 3, 275, 281, Jun. 2018
原発性シェーグレン症候群（Sj?gren's syndrome, SS）は慢性唾液腺炎と乾燥性角膜炎を主徴とする自己免疫疾患である．小児膠原病疫学調査において若年性特発性関節炎，全身性エリテマトーデスに次ぐ高頻度な疾患であることが明らかにされた．小児では目や口の乾燥を呈する腺症状ではなく，非特異的な症状で受診する例が多い．今回，日常生活に支障をきたすほどの全身倦怠感や発熱の反復などの症状を認めながら，長期間にわたり起立性調節障害（Orthostatic Dysregulation, OD）と診断されていたSSの1例を経験したので報告する．症例は14歳女児．12歳時にめまいと一時的な両下肢の筋力低下を自覚したのを契機にODと診断された．その後も頻回な倦怠感を訴え，複数の診療科・施設でODと診断されていた．14歳時に発熱，嘔吐，腹痛，下腿紫斑を主訴に前医に入院し，アレルギー性紫斑病の疑いでプレドニゾロン（PSL）の投与を開始された．しかし，症状改善せず，多様な所見が加わったため当科に紹介となった．抗核抗体，抗SS-A/Ro抗体，抗SS-B/La抗体が陽性，ガム試験と口唇小唾液腺生検がSSの診断基準を満たし，原発性SSと診断した．PSLは漸減中止しNSAIDsの内服を開始したが，低血圧および全身倦怠感が増悪した．その後，昇圧剤やPSLの内服を再開するも症状のコントロールが困難だったため，免疫抑制剤を投与した．現在はPSL，シクロスポリン，昇圧剤の内服を併用し加療中である．倦怠感や原因不明の発熱や皮疹などの症状を繰り返す症例では，腺症状がなくともSSの可能性を考慮し，精査する必要性があると考えられた．, 昭和大学学士会, Japanese

A note on accurate retrieval of similar inspection records based on canonical correlation between eye tracking data and inspection records
斉藤僚汰, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 117, 431, 195, 200, 15 Feb. 2018
電子情報通信学会, Japanese

A Note on Image Retrieval Based on LRGA Algorithm for Insect Identification : Verification of Retrieval Accuracy Improvement Based on Data Augmentation
弦間奨, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 117, 431, 167, 171, 15 Feb. 2018
電子情報通信学会, Japanese

A Note on Estimation of Users' Emotion Evoked During Listening to Music : Performance Improvement Based on Deep Learning Methods
段博瀟, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 117, 431, 201, 206, 15 Feb. 2018
電子情報通信学会, Japanese

Accuracy Improvement of Preference Estimation for Video Using SFEM-GS
伊藤良起, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 117, 431, 315, 318, 15 Feb. 2018
電子情報通信学会, Japanese