Oct. 2023, Bronze Prize GCCE2023 Excellent Student Poster Award　　　　　　　　　　　　　　　
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama

Oct. 2023, Silver Prize GCCE2023 Excellent Paper Award　　　　　　　　　　　　　　　
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

Jul. 2023, Best Paper Award Honorable Mention　　　　　　　　　　　　　　　
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

Jan. 2023, The 2022 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Nao Nakagawa;Ren Togo;Takahiro Ogawa;Miki Haseyama

Jan. 2023, The 2022 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Naoki Ogawa;Keisuke Maeda;Takahiro Ogawa;Miki Haseyama

Jan. 2023, The 2022 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　
Kyohei Kamikawa;Keisuke Maeda;Takahiro Ogawa;Miki Haseyama

Jan. 2023, Best Paper Award　　　　　　　　　　　　　　　
Teruhisa Yamashiro;Yuki Honma;Ren Togo;Takahiro Ogawa;Miki Haseyama;International;Workshop on;Advanced Image Technology;IWAIT

Dec. 2022, 優秀研究発表賞　　　　　　　　　　　　　　　
ユーザの嗜好を考慮した強化学習と知識グラフに基づく楽曲プレイリスト生成に関する検討, 映像情報メディア学会技術報告, vol.46, no.6, pp.109-112
櫻井 慶悟, 藤後 廉, 小川 貴弘, 長谷山 美紀

Dec. 2022, 若手優秀論文発表賞　　　　　　　　　　　　　　　
五箇 亮太;諸戸 祐哉;前田 圭介;小川 貴弘;長谷山 美紀

Dec. 2022, 若手優秀論文発表賞　　　　　　　　　　　　　　　
七田 亮, 藤後 廉, 前田 圭介, 小川 貴弘, 長谷山 美紀

Dec. 2022, 若手優秀論文発表賞　　　　　　　　　　　　　　　
山本一輝;前田 圭介;藤後 廉;小川 貴弘;長谷山 美紀

Nov. 2022, Bronze Prize GCCE2022 Excellent Student Paper Award　　　　　　　　　　　　　　　
Cross-platform recommendation considering common users’ preferences based on preference propagation graphnet, 2022 IEEE 11th Global Conference on Consumer Electronics (GCCE 2022), pp.133-134
Kazuki Yamamoto;Keisuke Maeda;Ren Togo;Takahiro Ogawa;Miki Haseyama

Nov. 2022, 第1回 北海道大学医療AIシンポジウム 優秀研究賞　　　　　　　　　　　　　　　
李 広;藤後 廉;小川 貴弘;長谷山 美紀

Oct. 2022, Silver Prize GCCE2022 Excellent Poster Award　　　　　　　　　　　　　　　
Free-viewpoint sports video generation based on dynamic NeRF considering time series, 2022 IEEE 11th Global Conference on Consumer Electronics (GCCE 2022), pp.419-420
Masato Kawai;Rintaro Yanagi;Togo Ren;Takahiro Ogawa;Miki Haseyama

Oct. 2022, Silver Prize GCCE2022 Excellent Student Poster Award　　　　　　　　　　　　　　　
Content-based image retrieval using effective synthesized images from different camera views via pixelNeRF, 2022 IEEE 11th Global Conference on Consumer Electronics (GCCE 2022), pp.415-416
Yuki Era;Ren Togo;Keisuke Maeda;Takahiro Ogawa;Miki Haseyama

Sep. 2022, 土木学会 土木情報学システム開発賞　　　　　　　　　　　　　　　

Aug. 2022, MIRU 2022 学生奨励賞2件　　　　　　　　　　　　　　　

Mar. 2022, IEEE LifeTech 2022 WIE Excellent Poster Award　　　　　　　　　　　　　　　

Feb. 2022, 2021 IEEE Sapporo Section Student Paper Contest, Best Presentation Award　　　　　　　　　　　　　　　

Feb. 2022, 2021 IEEE Sapporo Section Encouragement Award 2件　　　　　　　　　　　　　　　

Jan. 2022, International Workshop on Advanced Image Technology (IWAIT2022) Best Paper Award　　　　　　　　　　　　　　　

Dec. 2021, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Dec. 2021, 令和3年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 3件　　　　　　　　　　　　　　　

Oct. 2021, The 1st Hokkaido Young Professionals Workshop Best Student Presentation Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Gold Prize GCCE2021 Excellent Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Gold Prize GCCE2021 Excellent Student Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, Silver Prize GCCE2021 Excellent Student Poster Award　　　　　　　　　　　　　　　

Oct. 2021, 2021 IEEE 10th Global Conference on Consumer Electronics, GCCE2021 Outstanding Paper Award　　　　　　　　　　　　　　　

Jun. 2021, 映像情報メディア学会丹羽高柳賞論文賞　　　　　　　　　　　　　　　

Mar. 2021, ACM Multimedia Asia 2020, Best Paper Runner-up Award　　　　　　　　　　　　　　　

Mar. 2021, 2021 IEEE 3rd Global Conference on Life Sciences and Technologies, Excellent Poster (On-site) Award Winners: Bronze Prize　　　　　　　　　　　　　　　

Mar. 2021, IEEE LifeTech 2021 Excellent Student Paper Award for Oral Presentation, 2nd Prize　　　　　　　　　　　　　　　

Feb. 2021, 2020 IEEE Sapporo Section Student Paper Awards, Encouragement Paper Award　　　　　　　　　　　　　　　

Feb. 2021, 2020 IEEE Sapporo Section Student Paper Awards, Best Paper Award　　　　　　　　　　　　　　　

Nov. 2020, 令和2年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 3件　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Bronze Prize GCCE2020 Excellent Paper Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE2020 Excellent Paper Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize IEEE GCCE2020 Excellent Demo! Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize GCCE2020 Excellent Poster Award　　　　　　　　　　　　　　　

Oct. 2020, 2020 IEEE 9th Global Conference on Consumer Electronics, Gold Prize IEEE GCCE2020 Excellent Student Paper Award　　　　　　　　　　　　　　　

Jun. 2020, 映像情報メディア学会丹羽高柳賞論文賞　　　　　　　　　　　　　　　

May 2020, 2020 ICCE-TW Best Paper Award Honorable Metion　　　　　　　　　　　　　　　

Feb. 2020, The 2019 IEEE Sapporo Section Student Paper Contest Encouraging Prize 3件　　　　　　　　　　　　　　　

Feb. 2020, The 2019 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　

Dec. 2019, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Dec. 2019, 令和元年度電気・情報関係学会北海道支部連合大会 若手優秀論文発表賞 2件　　　　　　　　　　　　　　　

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE 2019 Excelent Paper Award　　　　　　　　　　　　　　　

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Silver Prize IEEE GCCE 2019 Excelent Poster Award　　　　　　　　　　　　　　　

Oct. 2019, 2019 IEEE 8th Global Conference on Consumer Electronics, Outstanding Prize IEEE GCCE 2019 Excelent Demo! Award　　　　　　　　　　　　　　　

Mar. 2019, 2019 IEEE 1st Global Conference on Life Sciences and Technologies, 2nd Prize IEEE Lifetech 2019 Excellent Paper Award　　　　　　　　　　　　　　　

Feb. 2019, The 2018 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　

Feb. 2019, The 2018 IEEE Sapporo Section Encouragement Award 2件　　　　　　　　　　　　　　　

Jan. 2019, The 2019 joint International Workshop on Advanced Image Technology & International Forum on Medical Imaging in Asia IWAIT Best Paper Award　　　　　　　　　　　　　　　

Dec. 2018, 平成30年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　

Dec. 2018, 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Oct. 2018, 2018 IEEE 7th Global Conference on Consumer Electronics, IEEE GCCE 2018 Outstanding Paper Award　　　　　　　　　　　　　　　

Oct. 2018, 2018 IEEE 7th Global Conference on Consumer Electronics, 1st Prize IEEE GCCE 2018 Excellent Poster Award　　　　　　　　　　　　　　　

2018, 平成29年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　

2018, The 2017 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　

2018, The 2017 IEEE Sapporo Section Encouragement Award (2件)　　　　　　　　　　　　　　　

2017, International Workshop on Advanced Image Technology (IWAIT2017) Best Paper Award　　　　　　　　　　　　　　　

2017, 平成28年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　

2017, The 2016 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　

2017, The 2016 IEEE Sapporo Section Encouragement Award　　　　　　　　　　　　　　　

2017, 電子情報通信学会 学術奨励賞　　　　　　　　　　　　　　　

2017, 精密工学会画像応用技術専門委員会・映像情報メディア学会メディア工学研究委員会合同サマーセミナー 優秀発表賞　　　　　　　　　　　　　　　

2017, 2017 IEEE 6th Global Conference on Consumer Electronics, IEEE GCCE 2017 Outstanding Poster Award　　　　　　　　　　　　　　　

2016, 2016 IEEE 5th Global Conference on Consumer Electronics 1st Prize IEEE GCCE 2016 Excellent Poster Award　　　　　　　　　　　　　　　

2016, 平成27年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞 (2件)　　　　　　　　　　　　　　　

2016, The 2015 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　

2016, The 2015 IEEE Sapporo Section Encouragement Award (2件)　　　　　　　　　　　　　　　

May 2015, 映像情報メディア学会, 丹羽高柳賞　　　　　　　　　　　　　　　
業績賞
長谷山 美紀

2015, International Workshop on Advanced Image Technology (IWAIT2015) Best Paper Award　　　　　　　　　　　　　　　

2015, IEEE GCCE 2015 Excellent Poster Award　　　　　　　　　　　　　　　

2015, IEEE GCCE 2015 Outstanding Poster Award　　　　　　　　　　　　　　　

2015, The 2014 IEEE Sapporo Section Student Paper Contest Best Presentation Award　　　　　　　　　　　　　　　

2015, 平成27年度 映像情報メディア学会 優秀研究発表賞　　　　　　　　　　　　　　　

Jun. 2014, 総務省北海道総合通信局, 平成26年度情報通信月間 北海道総合通信局長表彰　　　　　　　　　　　　　　　
長谷山 美紀

2014, IEEE GCCE 2014 Undergraduate Poster Award　　　　　　　　　　　　　　　

2013, 平成25年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　

2011, 平成23年度信号処理学生奨励賞 (2件)　　　　　　　　　　　　　　　

2011, 平成23年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　

2011, 映像情報メディア学会 学生優秀発表賞　　　　　　　　　　　　　　　

2011, SIP学生奨励賞　　　　　　　　　　　　　　　
Japan

2011, 平成23年度電気・情報関係学会北海道支部 優秀論文発表賞　　　　　　　　　　　　　　　
Japan

2011, 映像情報メディア学会年次大会 学生優秀発表賞　　　　　　　　　　　　　　　
Japan

2010, 平成22年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　

2010, 2010 IEEE Sapporo Section Student Member Best Presentation Award　　　　　　　　　　　　　　　

2009, 電子情報通信学会論文賞　　　　　　　　　　　　　　　

2009, 平成21年度電気・情報関係学会北海道支部連合大会 優秀論文発表賞　　　　　　　　　　　　　　　
Japan

2009, 平成20年度電子情報通信学会論文賞　　　　　　　　　　　　　　　
Japan

2008, IEEE CE-Society 日本支部 若手論文賞　　　　　　　　　　　　　　　

2008, 平成20年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　

2008, 2008 IEEE Sapporo Section Student Member Encouraging Prize　　　　　　　　　　　　　　　

2007, 平成19年度電気関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　

2007, IEEE International Conference on Consumer Electronics, IEEE Consumer Electronics Society Japan Chapter Young Scientist Paper Award　　　　　　　　　　　　　　　

2006, 2006 IEEE Sapporo Section Student Paper Contest Award　　　　　　　　　　　　　　　

2005, 精密工学会画像応用技術専門委員会・映像情報メディア学会メディア工学研究委員会合同サマーセミナー優秀発表賞　　　　　　　　　　　　　　　

2005, 平成17年度電気情報関係学会北海道支部連合大会 若手優秀論文発表賞　　　　　　　　　　　　　　　

2005, 映像情報メディア学会 研究奨励賞　　　　　　　　　　　　　　　

The 2022 IEEE Sapporo Section Student Paper Contest Encouraging Prize　　　　　　　　　　　　　　　
河合 雅斗;柳 凜太郎;藤後 廉;小川 貴弘;長谷山 美紀

Graph Convolutional Network-based Sports Skill-level Recognition via Deep Metric Learning
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2024 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC), 15, 1, 4, IEEE, 02 Jul. 2024
International conference proceedings

Multimodal Transformer Model Using Time-Series Data to Classify Winter Road Surface Conditions.
Yuya Moroto, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 11, 3440, 3440, Jun. 2024
Scientific journal

Expert–Novice Level Classification Using Graph Convolutional Network Introducing Confidence-Aware Node-Level Attention Mechanism
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
Sensors, 24, 10, 3033, 3033, MDPI AG, 10 May 2024
Scientific journal, In this study, we propose a classification method of expert–novice levels using a graph convolutional network (GCN) with a confidence-aware node-level attention mechanism. In classification using an attention mechanism, highlighted features may not be significant for accurate classification, thereby degrading classification performance. To address this issue, the proposed method introduces a confidence-aware node-level attention mechanism into a spatiotemporal attention GCN (STA-GCN) for the classification of expert–novice levels. Consequently, our method can contrast the attention value of each node on the basis of the confidence measure of the classification, which solves the problem of classification approaches using attention mechanisms and realizes accurate classification. Furthermore, because the expert–novice levels have ordinalities, using a classification model that considers ordinalities improves the classification performance. The proposed method involves a model that minimizes a loss function that considers the ordinalities of classes to be classified. By implementing the above approaches, the expert–novice level classification performance is improved.

Analysis of Continual Learning Techniques for Image Generative Models with Learned Class Information Management.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 10, 3087, 3087, May 2024
Scientific journal

A Novel Frame-Selection Metric for Video Inpainting to Enhance Urban Feature Extraction.
Yuhu Feng, Jiahuan Zhang, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 10, 3035, 3035, May 2024
Scientific journal

Algal Bed Region Segmentation Based on a ViT Adapter Using Aerial Images for Estimating CO2 Absorption Capacity.
Guang Li 0008, Ren Togo, Keisuke Maeda, Akinori Sako, Isao Yamauchi, Tetsuya Hayakawa, Shigeyuki Nakamae, Takahiro Ogawa 0001, Miki Haseyama
Remote. Sens., 16, 10, 1742, 1742, May 2024
Scientific journal

Confidence-Aware Spatial-Temporal Attention Graph Convolutional Network for Skeleton-Based Expert-Novice Level Classification
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 14 Apr. 2024
International conference proceedings

Parameter-efficient tuning of cross-modal retrieval for a specific database via trainable textual and visual prompts.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Int. J. Multim. Inf. Retr., 13, 1, 14, 14, Mar. 2024
Scientific journal

Flexibly manipulating popularity bias for tackling trade-offs in recommendation.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Inf. Process. Manag., 61, 2, 103606, 103606, Mar. 2024
Scientific journal

Text-Guided Image Editing Based on Post Score for Gaining Attention on Social Media.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 24, 3, 921, 921, Feb. 2024
Scientific journal

AUTOMATIC RECOGNITION OF ALGAL BED AREAS BASED ON A LARGE-SCALE SEMANTIC SEGMENTATION MODEL FOR ESTIMATING CO₂ ABSORPTION BY BLUE CARBON
LI Guang, TOGO Ren, MAEDA Keisuke, SAKO Akinori, YAMAUCHI Isao, HAYAKAWA Tetsuya, NAKAMAE Shigeyuki, OGAWA Takahiro, HASEYAMA Miki
Japanese Journal of JSCE, 80, 17, n/a, Japan Society of Civil Engineers, 2024
Japanese, Measuring the CO₂ absorption of algal beds is one of the key issues for achieving carbon neutrality, but identifying the area of algal beds from UAV images requires a great deal of labor and experience. In this study, we propose a method for automatic recognition of algal beds using UAV images. The proposed method uses a model that enables semantic domain segmentation at the pixel level, and employs ViT-Adapter, one of the latest models. The advantage of this technique is that it effectively utilizes the knowledge of a trained large-scale model to recognize algal beds, and it can identify algal beds at the pixel level by adjusting the parameters of the model. In this study, we conducted learning using mask images of visually identified algal beds from aerial photographs, and further examined data expansion and other processing to adapt the learning to UAV images. The effectiveness of this method was verified through a demonstration using UAV images of the Erimo coast of Hokkaido.

Generative Dataset Distillation: Balancing Global Structure and Local Details.
Longzhen Li, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CVPR Workshops, 7664, 7671, 2024
International conference proceedings

Cross-domain Few-shot In-context Learning for Enhancing Traffic Sign Recognition.
Yaozong Gan, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2407.05814, 2024
Scientific journal

Zero-shot Composed Image Retrieval Considering Query-target Relationship Leveraging Masked Image-text Pairs.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2406.18836, 2024
Scientific journal

Prompt-based Personalized Federated Learning for Medical Visual Question Answering.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2402.09677, 2024
Scientific journal

Importance-Aware Adaptive Dataset Distillation.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2401.15863, 2024
Scientific journal

Multi-Object Editing in Personalized Text-To-Image Diffusion Model Via Segmentation Guidance.
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 8140, 8144, 2024
International conference proceedings

Caption Unification for Multi-View Lifelogging Images Based on In-Context Learning with Heterogeneous Semantic Contents.
Masaya Sato, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 8085, 8089, 2024
International conference proceedings

Enhancing Noisy Label Learning Via Unsupervised Contrastive Loss with Label Correction Based on Prior Knowledge.
Masaki Kashiwagi, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 6235, 6239, 2024
International conference proceedings

Prompt-Based Personalized Federated Learning for Medical Visual Question Answering.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1821, 1825, 2024
International conference proceedings

Importance-aware adaptive dataset distillation.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Neural Networks, 172, 106154, 106154, 2024
Scientific journal

Dataset Distillation Using Parameter Pruning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 107, 6, 936, 940, 2024
Scientific journal

Expert-novice level classification using engineers’ motion data in subway tunnel inspections - Introduction of explainable graph convolutional network -
SEINO Tatsuki, SAITO Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 5, 1, 101, 109, Japan Society of Civil Engineers, 2024
Japanese, Skill transfer to young engineers from senior engineers is a very important task in infrastructure equipment inspection. To support the skill transfer, an analysis method of the key factors of senior engineers skill is needed. However, conventional research has been limited to skill-level classification or analysis of the relationship between the skill level and biological data such as eye gaze and motion obtained from the engineers. This paper presents a method of classifying the skill level and visualization of its key factors to support the skill transfer. The proposed method employs a graph convolutional network introducing a novel attention mechanism for the classification and visualization.

Accident risk estimation with depth information in construction site videos
GOKA Ryota, MAEDA Keisuke, TOGO Ren, OGAWA Takahiro, HASEYAMA Miki
Proceedings of the Annual Conference of JSAI, JSAI2024, 2C6GS701, 2C6GS701, The Japanese Society for Artificial Intelligence, 2024
Japanese, In the construction industry, reducing accident risk and improving safety is one of the high-priority tasks. Recently, several methods have been proposed to estimate the contact accident risk with heavy machinery on construction sites for enhancing safety. Conventional studies based on deep learning estimate the risk by using relations within the image space of detected workers and machinery captured in construction site videos. However, these approaches focus on the distance between detected objects in the image, leading to the problem that accident risk is overestimated even when there is distance between objects in the real world. In this study, to consider 3D spatial information in videos, we propose a method for estimating the accident risk with visual features regarding depth information. Experimental results show that the proposed method performs better than existing methods in estimating the contact accident risk.

Reinforcing Pre-trained Models Using Counterfactual Images.
Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2406.13316, 2024
Scientific journal

Generative Dataset Distillation: Balancing Global Structure and Local Details.
Longzhen Li, Guang Li 0008, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2404.17732, 2024
Scientific journal

Enhancing Generative Class Incremental Learning Performance with Model Forgetting Approach.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2403.18258, 2024
Scientific journal

Automatic Findings Generation for Distress Images Using In-Context Few-Shot Learning of Visual Language Model Based on Image Similarity and Text Diversity.
Yuto Watanabe, Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
J. Robotics Mechatronics, 36, 2, 353, 364, 2024
Scientific journal

Individual Persistence Adaptation for User-Centric Evaluation of User Satisfaction in Recommender Systems.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 12, 23626, 23635, 2024
Scientific journal

Zero-Shot Traffic Sign Recognition Based on Midlevel Feature Matching
Yaozong Gan, Guang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 23, 23, 9607, 9607, MDPI AG, 04 Dec. 2023
Scientific journal, Traffic sign recognition is a complex and challenging yet popular problem that can assist drivers on the road and reduce traffic accidents. Most existing methods for traffic sign recognition use convolutional neural networks (CNNs) and can achieve high recognition accuracy. However, these methods first require a large number of carefully crafted traffic sign datasets for the training process. Moreover, since traffic signs differ in each country and there is a variety of traffic signs, these methods need to be fine-tuned when recognizing new traffic sign categories. To address these issues, we propose a traffic sign matching method for zero-shot recognition. Our proposed method can perform traffic sign recognition without training data by directly matching the similarity of target and template traffic sign images. Our method uses the midlevel features of CNNs to obtain robust feature representations of traffic signs without additional training or fine-tuning. We discovered that midlevel features improve the accuracy of zero-shot traffic sign recognition. The proposed method achieves promising recognition results on the German Traffic Sign Recognition Benchmark open dataset and a real-world dataset taken from Sapporo City, Japan.

Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 22, 9287, 9287, Nov. 2023
Scientific journal

Skill Level Classification Using Motion Data via Spatial Temporal Graph Convolutional Network
Tatsuki Seino, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2023 IEEE 12th Global Conference on Consumer Electronics (GCCE), IEEE, 10 Oct. 2023
International conference proceedings

Visual Emotion Recognition Through Multimodal Cyclic-Label Dequantized Gaussian Process Latent Variable Model.
Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
Journal of Robotics and Mechatronics, 35, 5, 1321, 1330, Oct. 2023
Scientific journal

Self-supervised learning for gastritis detection with gastric X-ray images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
International Journal of Computer Assisted Radiology and Surgery, 18, 10, 1841, 1848, Oct. 2023
Scientific journal

Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding.
Yusuke Akamatsu, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 15, 6903, 6903, Aug. 2023
Scientific journal

Off-Screen Sound Separation Based on Audio-visual Pre-training Using Binaural Audio
Masaki Yoshida, Ren Togo, Takahiro Ogawa, Miki Haseyama
SENSORS, 23, 9, 4540, 4540, MDPI, May 2023
English, Scientific journal, This study proposes a novel off-screen sound separation method based on audio-visual pre-training. In the field of audio-visual analysis, researchers have leveraged visual information for audio manipulation tasks, such as sound source separation. Although such audio manipulation tasks are based on correspondences between audio and video, these correspondences are not always established. Specifically, sounds coming from outside a screen have no audio-visual correspondences and thus interfere with conventional audio-visual learning. The proposed method separates such off-screen sounds based on their arrival directions using binaural audio, which provides us with three-dimensional sensation. Furthermore, we propose a new pre-training method that can consider the off-screen space and use the obtained representation to improve off-screen sound separation. Consequently, the proposed method can separate off-screen sounds irrespective of the direction from which they arrive. We conducted our evaluation using generated video data to circumvent the problem of difficulty in collecting ground truth for off-screen sounds. We confirmed the effectiveness of our methods through off-screen sound detection and separation tasks.

Multimodal Natural Language Explanation Generation for Visual Question Answering Based on Multiple Reference Data
He Zhu, Ren Togo, Takahiro Ogawa, Miki Haseyama
ELECTRONICS, 12, 10, MDPI, May 2023
English, Scientific journal, As deep learning research continues to advance, interpretability is becoming as important as model performance. Conducting interpretability studies to understand the decision-making processes of deep learning models can improve performance and provide valuable insights for humans. The interpretability of visual question answering (VQA), a crucial task for human-computer interaction, has garnered the attention of researchers due to its wide range of applications. The generation of natural language explanations for VQA that humans can better understand has gradually supplanted heatmap representations as the mainstream focus in the field. Humans typically answer questions by first identifying the primary objects in an image and then referring to various information sources, both within and beyond the image, including prior knowledge. However, previous studies have only considered input images, resulting in insufficient information that can lead to incorrect answers and implausible explanations. To address this issue, we introduce multiple references in addition to the input image. Specifically, we propose a multimodal model that generates natural language explanations for VQA. We introduce outside knowledge using the input image and question and incorporate object information into the model through an object detection module. By increasing the information available during the model generation process, we significantly improve VQA accuracy and the reliability of the generated explanations. Moreover, we employ a simple and effective feature fusion joint vector to combine information from multiple modalities while maximizing information preservation. Qualitative and quantitative evaluation experiments demonstrate that the proposed method can generate more reliable explanations than state-of-the-art methods while maintaining answering accuracy.

Boosting automatic COVID-19 detection performance with self-supervised learning and batch knowledge ensembling.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Comput. Biol. Medicine, 158, 106877, 106877, May 2023
Scientific journal

Canonical Correlation Analysis Introducing Label Dequantization for Visual Emotion Recognition
SAITO Naoki, MAEDA Keisuke, OGAWA Takahiro, ASAMIZU Satoshi, HASEYAMA Miki
電子情報通信学会論文誌D 情報・システム, J106-D, 5, 337, 348, The Institute of Electronics, Information and Communication Engineers, 01 May 2023
Japanese, Supervised multi-view canonical correlation analysis via cyclic label dequantization (sMVCCA-CLD) for visual emotion recognition is presented in this paper. In the CCA approach, the dimension of latent common space is limited to the minimum dimension among those of all features. The dimension of label features i.e., the number of classes for label information, tends to be lower than those of the other features. Then the dimension of the latent common space constructed by CCA becomes lower. Therefore, there is a possibility of misssing important information that is necessary for the estimation from the latent common space due to the dimensionality constraint. To overcome this constraint, sMVCCA-CLD increases the dimension of the label features by the label dequantization process, and estimates the canonical correlation between multi-view features. In addition, sMVCCA-CLD performs the label dequantization considering that the emotions are represented by a cyclic model, e.g., Plutchik's and Mikel's wheels. Consequently, the construction of the latent common space for the accurate recognition of emotions becomes feasible.

COVID-19 detection based on self-supervised transfer learning using chest X-ray images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Int. J. Comput. Assist. Radiol. Surg., 18, 4, 715, 722, Apr. 2023
Scientific journal

Estimation of Degradation Degree in Road Infrastructure Based on Multi-Modal ABN Using Contrastive Learning.
Takaaki Higashi, Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 3, 1657, 1657, Feb. 2023
Scientific journal

Diversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 3, 1057, 1057, MDPI, Feb. 2023
English, Scientific journal, Auxiliary clinical diagnosis has been researched to solve unevenly and insufficiently distributed clinical resources. However, auxiliary diagnosis is still dominated by human physicians, and how to make intelligent systems more involved in the diagnosis process is gradually becoming a concern. An interactive automated clinical diagnosis with a question-answering system and a question generation system can capture a patient's conditions from multiple perspectives with less physician involvement by asking different questions to drive and guide the diagnosis. This clinical diagnosis process requires diverse information to evaluate a patient from different perspectives to obtain an accurate diagnosis. Recently proposed medical question generation systems have not considered diversity. Thus, we propose a diversity learning-based visual question generation model using a multi-latent space to generate informative question sets from medical images. The proposed method generates various questions by embedding visual and language information in different latent spaces, whose diversity is trained by our newly proposed loss. We have also added control over the categories of generated questions, making the generated questions directional. Furthermore, we use a new metric named similarity to accurately evaluate the proposed model's performance. The experimental results on the Slake and VQA-RAD datasets demonstrate that the proposed method can generate questions with diverse information. Our model works with an answering model for interactive automated clinical diagnosis and generates datasets to replace the process of annotation that incurs huge labor costs.

Automatic generation of findings for distress images using visual language model—Introduction of few-shot learning based on similar image retrieval—
WATANABE Yuto, OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 223, 232, Japan Society of Civil Engineers, 2023
Japanese, In this study, we propose a novel method for automatic generation of findings using a visual language model to support the efficient creation of findings in inspection records for infrastructure facilities. It is essential for the creation of inspection records to write findings, which are sentences that include judgments and opinions of engineers in addition to what can be recognized from the distress image. However, there has been little discussion on the direct automatic generation of findings, and it is expected to realize generation methods to support the efficient creation of findings. With this background, in this paper, we introduce few-shot learning based on the similarity of distress images to the visual language model, which is an application of large language models attracted much attention in recent years and enables text output with a highly accurate understanding of both vision and language. By using past inspection records including images similar to the distress images, we can efficiently consider the relationship between the distress images and findings from a small number of pairs of them. In the last part of this paper, we confirm the effectiveness of the proposed method through experiments generating findings from the distress images included in the inspection records of bridges.

Distress estimation of road attachments based on attention-based multiple instance learning considering the diversity of background of images
WATANABE Koshi, OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 482, 489, Japan Society of Civil Engineers, 2023
Japanese, In this paper, we propose a distress estimation method for road attachments. Road attachments, such as signs and lighting, are equipped over a huge number and wide area, and therefore it is desired to achieve automatic inspection by using drones to reduce the burden on the inspectors. While captured images by drones include the diversity of the background including ground, sky, and road surfaces, the previous methods did not consider the diversity of the background of captured images of road attachments. This study proposes the distress estimation method via attention-based multiple instance learning to address this issue. We input patches of the images into the estimation model to distinguish between the background area and road attachment area and assign importance weight, or attention, to each patch. By performing this strategy, we realize the distress estimation method considering the diversity of the background area of images. In the experiment, we achieve a classification accuracy of about 70 % using images of actual road attachments confirming the effectiveness of this research approach.

Multi-task classification of distress types and deterioration levels for infrastructure maintenance
OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 807, 814, Japan Society of Civil Engineers, 2023
Japanese, This paper proposes a multi-task classification method that classifies both the distress type and deterioration level at the same time. Conventionally, classifying the deterioration level has been conducted for each distress type by using multiple models. In contrast, the proposed method enables classification of the deterioration level without assigning a distress type to the distress image in advance, by training a single model through loss minimization considering the distress type and deterioration level. In the last part of the paper, it is verified that the proposed method can achieve classification performance equivalent to models constructed for each distress type with a single model by using images of actual distress that have occurred on infrastructure.

Advanced AI research for enhancing the efficiency of infrastructure maintenance and management
MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 982, 989, Japan Society of Civil Engineers, 2023
Japanese, With the development and advancement of AI technology, research on the application to the field of infrastructure maintenance is actively progressing. Many of these studies focus on the development of learning theories that consider the characteristics of images obtained in infrastructure maintenance. The effectiveness of AI has been demonstrated in various tasks such as crack detection, classification of defect types, and estimation of degradation levels. On the other hand, to truly enhance the operational efficiency through AI, it is necessary to construct AI systems considering the practical business. Furthermore, to improve AI and continuously utilize AI, it is necessary to acquire images suitable for AI development in operations. Therefore, this paper introduces the learning theories that have been developed for images obtained in infrastructure maintenance, previous research on AI with essential function for the practical business and the authors’ idea on efficient image acquisition.

Automatic detection of dead trees using in-vehicle video based on semantic segmentation
OGAWA Naoki, MAEDA Keisuke, OGAWA Takahiro, HASEYAMA Miki
Artificial Intelligence and Data Science, 4, 3, 686, 693, Japan Society of Civil Engineers, 2023
Japanese, This paper proposes a method for automatic detection of dead trees using in-vehicle videos. The proposed method extracts vegetation regions from videos containing various objects based on semantic segmentation. Then, it detects dead trees from the extracted vegetation regions using color information. By presenting the dead tree regions detected by the proposed method to engineers, they can find dead trees efficiently. In the last part of this paper, the effectiveness of the proposed method is verified through experiments using actual in-vehicle videos.

Feature Integration via Back-Projection Ordering Multi-Modal Gaussian Process Latent Variable Model for Rating Prediction.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 3125, 3129, 2023
International conference proceedings

Multi-View Variational Recurrent Neural Network for Human Emotion Recognition Using Multi-Modal Biological Signals.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2925, 2929, 2023
International conference proceedings

Video-Music Retrieval with Fine-Grained Cross-Modal Alignment.
Yuki Era, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2005, 2009, 2023
International conference proceedings

Text-Guided Facial Image Manipulation for Wild Images via Manipulation Direction-Based Loss.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 361, 365, 2023
International conference proceedings

Text-to-image Diffusion Model Suppressing Catastrophic Forgetting via Elastic Weight Consolidation.
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 831, 832, 2023
International conference proceedings

Deterioration Level Estimation for Infrastructures Considering Noisy Labels via DivideMix.
Masaki Kashiwagi, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 829, 830, 2023
International conference proceedings

Novel Feature Extraction for Classification of Auditory-visual Stimuli from fNIRS Signals.
Taro Togo, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 759, 760, 2023
International conference proceedings

A Controllable Recoloring Method for Novel Views Using Segment Anything Model.
Haoyang Wang, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 587, 588, 2023
International conference proceedings

Caption Unification for Multiple Viewpoint Lifelogging Images and Its Verification.
Masaya Sato, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 415, 416, 2023
International conference proceedings

Improving Visual Counterfactual Explanation Models for Image Classification via CLIP.
Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 390, 391, 2023
International conference proceedings

Canonical Correlation Analysis Introducing Label Dequantization for Visual Emotion Recognition
斉藤直輝, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀
電子情報通信学会論文誌 D(Web), J106-D, 5, 2023

Few-Shot Personalized Saliency Prediction Using Tensor Regression for Preserving Structural Global Information.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2307.02799, 2023
Scientific journal

Personalized Content Recommender System via Non-verbal Interaction Using Face Mesh and Facial Expression.
Yuya Moroto, Rintaro Yanagi, Naoki Ogawa, Kyohei Kamikawa, Keigo Sakurai, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ACM Multimedia, 9399, 9401, 2023
International conference proceedings

Gromov-Wasserstein Autoencoders.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICLR, 2023
International conference proceedings

Proficiency-level Estimation Using Heterogeneous Features via Label Dequantized CCA.
Tatsuki Seino, Naoki Saito 0006, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
ICCE-Taiwan, 813, 814, 2023
International conference proceedings

Parameter-efficient Tuning of a Pre-trained Model via Prompt Learning in Cross-modal Retrieval.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 811, 812, 2023
International conference proceedings

Binaural Audio Generation with Data Augmentation from 360° Videos.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 795, 796, 2023
International conference proceedings

Prediction of Shoot Events by Considering Spatio-temporal Relations of Multimodal Features.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 793, 794, 2023
International conference proceedings

Shoot Event Prediction in Soccer Considering Expected Goals Based on Players' Positions.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 449, 450, 2023
International conference proceedings

Estimation of Amyloid-β Positivity Using QSM Images Considering Age Information.
Tsubasa Kunieda, Ren Togo, Noriko Nishioka, Yukie Shimizu, Shiro Watanabe, Kenji Hirata, Keisuke Maeda, Takahiro Ogawa 0001, Kohsuke Kudo, Miki Haseyama
ICCE-Taiwan, 165, 166, 2023
International conference proceedings

A Medical Domain Visual Question Generation Model via Large Language Model.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-Taiwan, 163, 164, 2023
International conference proceedings

Defense Against Black-Box Adversarial Attacks Via Heterogeneous Fusion Features.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Binauralization Robust To Camera Rotation Using 360° Videos.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Learning Graph Laplacian from Intrinsic Patterns via Gaussian Process.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Estimation of Visual Contents from Human Brain Signals via VQA Based on Brain-Specific Attention.
Ryo Shichida, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Class-Aware Shared Gaussian Process Dynamic Model.
Ryosuke Sawata, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Improving Dropout in Graph Convolutional Networks for Recommendation via Contrastive Loss.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1, 5, 2023
International conference proceedings

Multi-Label Classification in Anime Illustrations Based on Hierarchical Attribute Relationships.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 10, 4798, 4798, 2023
Scientific journal

Prediction of Shooting Events in Soccer Videos Using Complete Bipartite Graphs and Players' Spatial-Temporal Relations.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 23, 9, 4506, 4506, 2023
Scientific journal

Material Compound-Property Retrieval Using Electron Microscope Images for Rubber Material Development.
Rintaro Yanagi, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 88258, 88264, 2023
Scientific journal

Hierarchical Multi-Label Attribute Classification With Graph Convolutional Networks on Anime Illustration.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 35447, 35456, 2023
Scientific journal

SpectralMAP: Approximating Data Manifold With Spectral Decomposition
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE ACCESS, 11, 31530, 31540, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, Dimensionality reduction is widely used to visualize complex high-dimensional data. This study presents a novel method for effective data visualization. Previous methods depend on local distance measurements for data manifold approximation. This leads to unreliable results when a data manifold locally oscillates because of some undesirable effects, such as noise effects. In this study, we overcome this limitation by introducing a dual approximation of a data manifold. We roughly approximate a data manifold with a neighborhood graph and prune it with a global filter. This dual scheme results in local oscillation robustness and yields effective visualization with explicit global preservation. We consider a global filter based on principal component analysis frameworks and derive it with the spectral information of the original high-dimensional data. Finally, we experiment with multiple datasets to verify our method, compare its performance to that of state-of-the-art methods, and confirm the effectiveness of our novelty and results.

Summarizing Data Structures with Gaussian Process and Robust Neighborhood Preservation
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2022, PT V, 13717, 157, 173, SPRINGER INTERNATIONAL PUBLISHING AG, 2023
English, International conference proceedings, Latent variable models summarize high-dimensional data while preserving its many complex properties. This paper proposes a locality-aware and low-rank approximated Gaussian process latent variable model (LolaGP) that can preserve the global relationship and local geometry in the derivation of the latent variables. We realize the global relationship by imitating the sample similarity non-linearly and the local geometry based on our newly constructed neighborhood graph. Formally, we derive LolaGP from GP-LVM and implement a locality-aware regularization to reflect its adjacency relationship. The neighborhood graph is constructed based on the latent variables, making the local preservation more resistant to noise disruption and the curse of dimensionality than the previous methods that directly construct it from the high-dimensional data. Furthermore, we introduce a new lower bound of a log-posterior distribution based on low-rank matrix approximation, which allows LolaGP to handle larger datasets than the conventional GP-LVM extensions. Our contribution is to preserve both the global and local structures in the derivation of the latent variables using the robust neighborhood graph and introduce the scalable lower bound of the log-posterior distribution. We conducted an experimental analysis using synthetic as well as images with and without highly noise disrupted datasets. From both qualitative and quantitative standpoint, our method produced successful results in all experimental settings.

Recallable Question Answering-Based Re-Ranking Considering Semantic Region for Cross-Modal Retrieval
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE OPEN JOURNAL OF SIGNAL PROCESSING, 4, 1, 11, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, Question answering (QA)-based re-ranking methods for cross-modal retrieval have been recently proposed to further narrow down similar candidate images. The conventional QA-based re-ranking methods provide questions to users by analyzing candidate images, and the initial retrieval results are re-ranked based on the user's feedback. Contrary to these developments, only focusing on performance improvement makes it difficult to efficiently elicit the user's retrieval intention. To realize more useful QA-based re-ranking, considering the user interaction for eliciting the user's retrieval intention is required. In this paper, we propose a QA-based re-ranking method with considering two important factors for eliciting the user's retrieval intention: query-image relevance and recallability. Considering the query-image relevance enables to only focus on the candidate images related to the provided query text, while, focusing on the recallability enables users to easily answer the provided question. With these procedures, our method can efficiently and effectively elicit the user's retrieval intention. Experimental results using Microsoft Common Objects in Context and computationally constructed dataset including similar candidate images show that our method can improve the performance of the cross-modal retrieval methods and the QA-based re-ranking methods.

Interpretable Visual Question Answering Referring to Outside Knowledge.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2303.04388, 2023
Scientific journal

Text-Guided Image Manipulation via Generative Adversarial Network With Referring Image Segmentation-Based Guidance.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 42534, 42545, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, This study proposes a novel text-guided image manipulation method that introduces referring image segmentation into a generative adversarial network. The proposed text-guided image manipulation method aims to manipulate images containing multiple objects while preserving text-unrelated regions. The proposed method assigns the task of distinguishing between text-related and unrelated regions in an image to segmentation guidance based on referring image segmentation. With this architecture, the adversarial generative network can focus on generating new attributes according to the text description and reconstructing text-unrelated regions. For the challenging input images with multiple objects, the experimental results demonstrate that the proposed method outperforms conventional methods in terms of image manipulation precision.

Cross-Modal Image Retrieval Considering Semantic Relationships With Many-to-Many Correspondence Loss.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 11, 10675, 10686, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2023
English, Scientific journal, A cross-modal image retrieval that explicitly considers semantic relationships between images and texts is proposed. Most conventional cross-modal image retrieval methods retrieve the target images by directly measuring the similarities between the candidate images and query texts in a common semantic embedding space. However, such methods tend to focus on a one-to-one correspondence between a predefined image-text pair during the training phase, and other semantically similar images and texts are ignored. By considering the many-to-many correspondences between semantically similar images and texts, a common embedding space is constructed to assure semantic relationships, which allows users to accurately find more images that are related to the input query texts. Thus, in this paper, we propose a cross-modal image retrieval method that considers semantic relationships between images and texts. The proposed method calculates the similarities between texts as semantic similarities to acquire the relationships. Then, we introduce a loss function that explicitly constructs the many-to-many correspondences between semantically similar images and texts from their semantic relationships. We also propose an evaluation metric to assess whether each method can construct an embedding space considering the semantic relationships. Experimental results demonstrate that the proposed method outperforms conventional methods in terms of this newly proposed metric.

Similar interior coordination image retrieval with multi-view features
Ren Togo, Yuki Honma, Maiku Abe, Takahiro Ogawa, Miki Haseyama
International Journal of Multimedia Information Retrieval, 11, 4, 731, 740, Springer Science and Business Media LLC, 26 Aug. 2022
English, Scientific journal, This paper presents a novel similar image retrieval method for interior coordination. Interior coordination is very familiar; however, it is still an abstract and difficult concept. Even if we are involved in coordination every day, it does not mean we can become professional coordinators. By realizing the retrieval that can provide similar interior coordination images from a query room image, inspiring users' ideas for interior coordination becomes feasible. In the proposed method, we extract image features specialized for interior coordination and realize similar interior coordination image retrieval. We employ multi-view features: object-based, color-based, and semantic-based features, in the feature extraction phase. The extracted features are used to calculate similarity between the query image and the database images for the retrieval. We conducted experiments using a sophisticated real-world interior coordination image dataset. Furthermore, we qualitatively and quantitatively evaluated the effectiveness of the proposed method.

Brain Decoding of Multiple Subjects for Estimating Visual Information Based on a Probabilistic Generative Model
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
SENSORS, 22, 16, 6148, 6148, MDPI, Aug. 2022
English, Scientific journal, Brain decoding is a process of decoding human cognitive contents from brain activities. However, improving the accuracy of brain decoding remains difficult due to the unique characteristics of the brain, such as the small sample size and high dimensionality of brain activities. Therefore, this paper proposes a method that effectively uses multi-subject brain activities to improve brain decoding accuracy. Specifically, we distinguish between the shared information common to multi-subject brain activities and the individual information based on each subject's brain activities, and both types of information are used to decode human visual cognition. Both types of information are extracted as features belonging to a latent space using a probabilistic generative model. In the experiment, an publicly available dataset and five subjects were used, and the estimation accuracy was validated on the basis of a confidence score ranging from 0 to 1, and a large value indicates superiority. The proposed method achieved a confidence score of 0.867 for the best subject and an average of 0.813 for the five subjects, which was the best compared to other methods. The experimental results show that the proposed method can accurately decode visual cognition compared with other existing methods in which the shared information is not distinguished from the individual information.

Defect Detection of Subway Tunnels Using Advanced U-Net Network
An Wang, Ren Togo, Takahiro Ogawa, Miki Haseyama
SENSORS, 22, 6, 2330, 2330, MDPI, Mar. 2022
English, Scientific journal, In this paper, we present a novel defect detection model based on an improved U-Net architecture. As a semantic segmentation task, the defect detection task has the problems of background-foreground imbalance, multi-scale targets, and feature similarity between the background and defects in the real-world data. Conventionally, general convolutional neural network (CNN)-based networks mainly focus on natural image tasks, which are insensitive to the problems in our task. The proposed method has a network design for multi-scale segmentation based on the U-Net architecture including an atrous spatial pyramid pooling (ASPP) module and an inception module, and can detect various types of defects compared to conventional simple CNN-based methods. Through the experiments using a real-world subway tunnel image dataset, the proposed method showed a better performance than that of general semantic segmentation including state-of-the-art methods. Additionally, we showed that our method can achieve excellent detection balance among multi-scale defects.

Microscopy and biomimetics: the NanoSuit® method and image retrieval platform
Takahiko Hariyama, Yasuharu Takaku, Hideya Kawasaki, Masatsugu Shimomura, Chiyo Senoh, Yumi Yamahama, Atsushi Hozumi, Satoru Ito, Naoto Matsuda, Satoshi Yamada, Toshiya Itoh, Miki Haseyama, Takahiro Ogawa, Naoki Mori, Shuhei So, Hidefumi Mitsuno, Masahiro Ohara, Shuhei Nomura, Masao Hirasaka
Microscopy, 71, 1, 1, 12, Oxford University Press (OUP), 29 Jan. 2022
Scientific journal, Abstract

This review aims to clarify a suitable method towards achieving next-generation sustainability. As represented by the term ‘Anthropocene’, the Earth, including humans, is entering a critical era; therefore, science has a great responsibility to solve it. Biomimetics, the emulation of the models, systems and elements of nature, especially biological science, is a powerful tool to approach sustainability problems. Microscopy has made great progress with the technology of observing biological and artificial materials and its techniques have been continuously improved, most recently through the NanoSuit® method. As one of the most important tools across many facets of research and development, microscopy has produced a large amount of accumulated digital data. However, it is difficult to extract useful data for making things as biomimetic ideas despite a large amount of biological data. Here, we would like to find a way to organically connect the indispensable microscopic data with the new biomimetics to solve complex human problems.

Popularity-Aware Graph Social Recommendation for Fully Non-Interaction Users.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Proceedings of the 4th ACM International Conference on Multimedia in Asia(MMAsia), 30, 5, ACM, 2022
International conference proceedings

Affective Embedding Framework with Semantic Representations from Tweets for Zero-Shot Visual Sentiment Prediction.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Proceedings of the 4th ACM International Conference on Multimedia in Asia(MMAsia), 6, 7, ACM, 2022
International conference proceedings

Visual Sentiment Prediction Using Cross-Way Few-Shot Learning Based on Knowledge Distillation.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3838, 3842, IEEE, 2022
International conference proceedings

Human-Centric Image Retrieval with Gaze-Based Image Captioning.
Yuhu Feng, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3828, 3832, IEEE, 2022
International conference proceedings

Few-Shot Personalized Saliency Prediction with Similarity of Gaze Tendency Using Object-Based Structural Information.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3823, 3827, IEEE, 2022
International conference proceedings

Gaussian Distributed Graph Constrained Multi-Modal Gaussian Process Latent Variable Model for Ordinal Labeled Data.
Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 3798, 3802, IEEE, 2022
International conference proceedings

GCN-Based Multi-Modal Multi-Label Attribute Classification in Anime Illustration Using Domain-Specific Semantic Features.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
2022 IEEE International Conference on Image Processing(ICIP), 2021, 2025, IEEE, 2022
International conference proceedings

Trend Prediction of Students' Mock Examination Results Using Matrix Completion.
Yutaka Yamada, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 891, 892, IEEE, 2022
International conference proceedings

Shoot Event Prediction from Soccer Videos by Considering Players' Spatio-Temporal Relations.
Ryota Goka, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 406, 407, IEEE, 2022
International conference proceedings

Refinement of Gaze-based Image Caption for Image Retrieval.
Yuhu Feng, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
11th IEEE Global Conference on Consumer Electronics(GCCE), 272, 273, IEEE, 2022
International conference proceedings

Boosting Automatic COVID-19 Detection Performance with Self-Supervised Learning and Batch Knowledge Ensembling.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.09281, 2022
Scientific journal

COVID-19 Detection Based on Self-Supervised Transfer Learning Using Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.09276, 2022
Scientific journal

Union-set Multi-source Model Adaptation for Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2212.02785, 2022
Scientific journal

RGMIM: Region-Guided Masked Image Modeling for COVID-19 Detection.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2211.00313, 2022
Scientific journal

Rubber Material Retrieval System using Electron Microscope Images for Rubber Material Development.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MMAsia, 44, 3, ACM, 2022
International conference proceedings

Disentangled Image Attribute Editing in Latent Space via Mask-Based Retention Loss.
Shunya Ohaga, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MMAsia, 25, 7, ACM, 2022
International conference proceedings

Assessment of Image Manipulation Using Natural Language Description: Quantification of Manipulation Direction.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1046, 1050, IEEE, 2022
International conference proceedings

Improving Model Adaptation for Semantic Segmentation by Learning Model-Invariant Features with Multiple Source-Domain Models.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 421, 425, IEEE, 2022
International conference proceedings

A Multimodal Interpretable Visual Question Answering Model Introducing Image Caption Processor.
He Zhu, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 777, 778, IEEE, 2022
International conference proceedings

Cross-modal Image Retrieval Considering Semantic Relationships with Object Information.
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 775, 776, IEEE, 2022
International conference proceedings

Free-viewpoint Sports Video Generation Based on Dynamic NeRF Considering Time Series.
Masato Kawai, Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 408, 409, IEEE, 2022
International conference proceedings

Content-based Image Retrieval Using Effective Synthesized Images from Different Camera Views via pixelNeRF.
Yuki Era, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 404, 405, IEEE, 2022
International conference proceedings

Analysis of Relationships between Visual Cognitive Contents and Response of Each Brain Region via Visual Question Answering.
Ryo Shichida, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 402, 403, IEEE, 2022
International conference proceedings

GCN-based Collaborative Filtering Considering Personality Bias.
Hiroki Okamura, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 278, 279, IEEE, 2022
International conference proceedings

Prediction of Amyloid-β Positivity Using QSM Images Based on Bootstrap Your Own Latent.
Tsubasa Kunieda, Ren Togo, Noriko Nishioka, Yukie Shimizu, Shiro Watanabe, Kenji Hirata, Keisuke Maeda, Takahiro Ogawa 0001, Kohsuke Kudo, Miki Haseyama
GCCE, 137, 138, IEEE, 2022
International conference proceedings

Cross-platform Recommendation Considering Common Users' Preferences Based on Preference Propagation GraphNet.
Kazuki Yamamoto, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 135, 136, IEEE, 2022
International conference proceedings

Union-Set Multi-source Model Adaptation for Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ECCV (29), 579, 595, Springer, 2022
International conference proceedings

Trial Analysis of the Relationship between Taste and Biological Information Obtained While Eating Strawberries for Sensory Evaluation.
Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Shin-ichi Adachi, Fumiaki Yoshizawa, Miki Haseyama
Sensors, 22, 23, 9496, 9496, 2022
Scientific journal

Distress Detection in Subway Tunnel Images via Data Augmentation Based on Selective Image Cropping and Patching.
Keisuke Maeda, Saya Takada, Tomoki Haruyama, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 22, 8932, 8932, 2022
Scientific journal

Compressed gastric image generation based on soft-label dataset distillation for medical data sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Comput. Methods Programs Biomed., 227, 107189, 107189, 2022
Scientific journal

Dataset Complexity Assessment Based on Cumulative Maximum Scaled Area Under Laplacian Spectrum.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14743, 2022
Scientific journal

Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14635, 2022
Scientific journal

Dataset Distillation using Parameter Pruning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14609, 2022
Scientific journal

Dataset Distillation for Medical Dataset Sharing.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.14603, 2022
Scientific journal

Gromov-Wasserstein Autoencoders.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2209.07007, 2022
Scientific journal

TriBYOL: Triplet BYOL for Self-Supervised Representation Learning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2206.03012, 2022
Scientific journal

Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
CoRR, abs/2206.03009, 2022
Scientific journal

Generating Captions of Imagined content from Human Brain Activities Applying An Image Captioning Model.
Saya Takada, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
4th IEEE Global Conference on Life Sciences and Technologies(LifeTech), 614, 615, IEEE, 2022
International conference proceedings

Knowledge-Guided Sequential Recommendation with Reinforcement Learning Using Empirical Distribution Function.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
4th IEEE Global Conference on Life Sciences and Technologies(LifeTech), 187, 188, IEEE, 2022
International conference proceedings

Transformer Based Multimodal Scene Recognition in Soccer Videos.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICME Workshops, 1, 6, IEEE, 2022
International conference proceedings

Scene Retrieval in Soccer Videos by Spatial-temporal Attention with Video Vision Transformer.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 453, 454, IEEE, 2022
International conference proceedings

Multi-scale Defect Detection from Subway Tunnel Images with Spatial Attention Mechanism.
An Wang, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 305, 306, IEEE, 2022
International conference proceedings

Action Classification Based on LSTM Using First and Third Person Videos of Engineers Inspecting Bridges.
Tsuyoshi Masuda, Keisuke Maeda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICCE-TW, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 303, 304, IEEE, 2022

TriBYOL: Triplet BYOL for Self-Supervised Representation Learning.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 3458, 3462, IEEE, 2022
International conference proceedings

Divergence-Guided Feature Alignment for Cross-Domain Object Detection.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 2240, 2244, IEEE, 2022
International conference proceedings

Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 1371, 1375, IEEE, 2022
International conference proceedings

Regularization Meets Enhanced Multi-Stage Fusion Features: Making CNN More Robust against White-Box Adversarial Attacks.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 14, 5431, 5431, MDPI, 2022
English, Scientific journal, Regularization has become an important method in adversarial defense. However, the existing regularization-based defense methods do not discuss which features in convolutional neural networks (CNN) are more suitable for regularization. Thus, in this paper, we propose a multi-stage feature fusion network with a feature regularization operation, which is called Enhanced Multi-Stage Feature Fusion Network (EMSF(2)Net). EMSF(2)Net mainly combines three parts: multi-stage feature enhancement (MSFE), multi-stage feature fusion (MSF2), and regularization. Specifically, MSFE aims to obtain enhanced and expressive features in each stage by multiplying the features of each channel; MSF2 aims to fuse the enhanced features of different stages to further enrich the information of the feature, and the regularization part can regularize the fused and original features during the training process. EMSF(2)Net has proved that if the regularization term of the enhanced multi-stage feature is added, the adversarial robustness of CNN will be significantly improved. The experimental results on extensive white-box attacks on the CIFAR-10 dataset illustrate the robustness and effectiveness of the proposed method.

Controllable Music Playlist Generation Based on Knowledge Graph and Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 10, 3722, 3722, 2022
Scientific journal

Learning intra-domain style-invariant representation for unsupervised domain adaptation of semantic segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Pattern Recognit., 132, 108911, 108911, 2022
Scientific journal

Dataset complexity assessment based on cumulative maximum scaled area under Laplacian spectrum.
Guang Li 0008, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
Multimedia Tools and Applications, 81, 22, 32287, 32303, 2022
Scientific journal

User-centric multimodal feature extraction for personalized retrieval of tumblr posts.
Kazuma Ohtomo, Ryosuke Harakawa, Takahiro Ogawa 0001, Miki Haseyama, Masahiro Iwahashi
Multimedia Tools and Applications, 81, 2, 2979, 3003, 2022
Scientific journal

Chain centre loss: A psychology inspired loss function for image sentiment analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Neurocomputing, 495, 118, 128, 2022
Scientific journal

Generative Adversarial Network Including Referring Image Segmentation For Text-Guided Image Manipulation.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4818, 4822, IEEE, 2022
English, International conference proceedings, This paper proposes a novel generative adversarial network to improve the performance of image manipulation using natural language descriptions that contain desired attributes. Text-guided image manipulation aims to semantically manipulate an image aligned with the text description while preserving text-irrelevant regions. To achieve this, we newly introduce referring image segmentation into the generative adversarial network for image manipulation. The referring image segmentation aims to generate a segmentation mask that extracts the text-relevant region. By utilizing the feature map of the segmentation mask in the network, the proposed method explicitly distinguishes the text-relevant and irrelevant regions and has the following two contributions. First, our model can pay attention only to the text-relevant region and manipulate the region aligned with the text description. Second, our model can achieve an appropriate balance between the generation of accurate attributes in the text-relevant region and the reconstruction in the text-irrelevant regions. Experimental results show that the proposed method can significantly improve the performance of image manipulation.

Human Emotion Recognition Using Multi-Modal Biological Signals Based On Time Lag-Considered Correlation Maximization.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4683, 4687, IEEE, 2022
International conference proceedings

Distributed Label Dequantized Gaussian Process Latent Variable Model for Multi-View Data Integration.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 4643, 4647, IEEE, 2022
English, International conference proceedings, In this paper, we present a novel method for multi-view data analysis, distributed label dequantized Gaussian process latent variable model (DLDGP). DLDGP can integrate multi-view data and class information into a common latent space. In the previous multi-view methods, the dimension of label features transformed from the class information is much smaller than those of the other modalities, which causes a dimensionality-limitation problem in the latent space. DLDGP extends the dimension of the label features by a distributed label dequantization scheme. Additionally, DLDGP calculates correlation between different classes by encoding class information into distributed features. DLDGP can correctly capture the relationship between multi-view data and obtain the latent features with high expression ability. Experimental results show the effectiveness of our method by using the open dataset.

Variational Bayesian Graph Convolutional Network for Robust Collaborative Filtering.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICASSP, 3908, 3912, IEEE, 2022
International conference proceedings

Time-Lag Aware Latent Variable Model for Prediction of Important Scenes Using Baseball Videos and Tweets.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 7, 2465, 2465, 2022
Scientific journal

Refining Graph Representation for Cross-Domain Recommendation Based on Edge Pruning in Latent Space.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
IEEE Access, 10, 12503, 12509, 2022
Scientific journal

Interactive Re-ranking via Object Entropy-Guided Question Answering for Cross-Modal Image Retrieval.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
ACM Trans. Multim. Comput. Commun. Appl., 18, 3, 68, 17, ASSOC COMPUTING MACHINERY, 2022
English, Scientific journal, Cross-modal image-retrieval methods retrieve desired images from a query text by learning relationships between texts and images. Such a retrieval approach is one of the most effective ways of achieving the easiness of query preparation. Recent cross-modal image-retrieval methods are convenient and accurate when users input a query text that can be used to uniquely identify the desired image. However, in reality, users frequently input ambiguous query texts, and these ambiguous queries make it difficult to obtain desired images. To overcome these difficulties, in this study, we propose a novel interactive cross-modal image-retrieval method based on question answering. The proposed method analyzes candidate images and asks users questions to obtain information that can narrow down retrieval candidates. By only answering questions generated by the proposed method, users can reach their desired images, even when using an ambiguous query text. Experimental results show the proposed method's effectiveness.

Deterioration Level Estimation Based on Convolutional Neural Network Using Confidence-Aware Attention Mechanism for Infrastructure Inspection.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
Sensors, 22, 1, 382, 382, 2022
Scientific journal

Domain Adaptive Cross-Modal Image Retrieval via Modality and Domain Translations
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E104A, 6, 866, 875, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Jun. 2021
English, Scientific journal, Various cross-modal retrieval methods that can retrieve images related to a query sentence without text annotations have been proposed. Although a high level of retrieval performance is achieved by these methods, they have been developed for a single domain retrieval setting. When retrieval candidate images come from various domains, the retrieval performance of these methods might be decreased. To deal with this problem, we propose a new domain adaptive cross-modal retrieval method. By translating a modality and domains of a query and candidate images, our method can retrieve desired images accurately in a different domain retrieval setting. Experimental results for clipart and painting datasets showed that the proposed method has better retrieval performance than that of other conventional and state-of-the-art methods.

Database-adaptive Re-ranking for Enhancing Cross-modal Image Retrieval.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
MM '21: ACM Multimedia Conference, 3816, 3825, ACM, 2021
International conference proceedings

Interpretable Representation Learning on Natural Image Datasets via Reconstruction in Visual-Semantic Embedding Space.
Nao Nakagawa, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
2021 IEEE International Conference on Image Processing(ICIP), 2473, 2477, IEEE, 2021
International conference proceedings

Cross-Domain Recommendation Method Based On Multi-Layer Graph Analysis With Visual Information.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
ICIP, 2688, 2692, IEEE, 2021
International conference proceedings

Time-Lag Aware Multi-Modal Variational Autoencoder Using Baseball Videos And Tweets For Prediction Of Important Scenes.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2678, 2682, IEEE, 2021
International conference proceedings

Segmentation-Aware Text-Guided Image Manipulation.
Tomoki Haruyama, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 2433, 2437, IEEE, 2021
International conference proceedings

Few-Shot Personalized Saliency Prediction using Person Similarity based on Collaborative Multi-Output Gaussian Process Regression.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1469, 1473, IEEE, 2021
International conference proceedings

Interest Level Estimation via Multi-Modal Gaussian Process Latent Variable Factorization.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1209, 1213, IEEE, 2021
International conference proceedings

Deep Metric Network Via Heterogeneous Semantics for Image Sentiment Analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1039, 1043, IEEE, 2021
International conference proceedings

Correlation-Aware Attention Branch Network Using Multi-Modal Data For Deterioration Level Estimation Of Infrastructures.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
ICIP, 1014, 1018, IEEE, 2021
International conference proceedings

User Background Information-Aware Music Recommendation with Reinforcement Learning-Based Knowledge Graph Exploration.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Degradation Level Estimation of Road Structures via Attention Branch Network with Text Data.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Cross-view Self-supervised Learning via Momentum Statistics in Batch Normalization.
Guang Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
IEEE International Conference on Consumer Electronics-Taiwan(ICCE-TW), 1, 2, IEEE, 2021
International conference proceedings

Triplet Self-Supervised Learning for Gastritis Detection with Scarce Annotations.
Guang Li, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 787, 788, IEEE, 2021
International conference proceedings

Defense Against Image Captioning Attacks via A Robust and Stable Recurrent Neural Network.
Jiahuan Zhang, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 785, 786, IEEE, 2021
International conference proceedings

Text-Guided Image Manipulation for Desired Region Using Referring Image Segmentation.
Yuto Watanabe, Ren Togo, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 661, 662, IEEE, 2021
International conference proceedings

Multi-label Image Recognition Based on Multi-modal Graph Convolutional Networks Using Captioning Features.
Ziwen Lan, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 273, 274, IEEE, 2021

A Trial of Fine-grained Classification of Expert-novice Level Using Bio-signals While Inspecting Subway Tunnels.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 204, 205, IEEE, 2021
International conference proceedings

Listener Recommendation for Artist Based on Knowledge Graph and Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 202, 203, IEEE, 2021
International conference proceedings

Movie Rating Estimation Based on Weakly Supervised Multi-modal Latent Variable Model.
Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 195, 196, IEEE, 2021
International conference proceedings

Detection of Off-screen Sound Based on Loss Function of Self-supervised Audio-visual Spatialization.
Masaki Yoshida, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 193, 194, IEEE, 2021
International conference proceedings

Visual Sentiment Prediction Using Few-shot Learning via Distribution Relations of Visual Features.
Yingrui Ye, Yuya Moroto, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 191, 192, IEEE, 2021
International conference proceedings

Multi-class Similar Scene Retrieval in Soccer Videos: A Scene Confusion Reduction Method Based on Combination of Long and Short Frame Sequences.
Yaozong Gan, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 117, 118, IEEE, 2021
International conference proceedings

Sports Action Detection Based on Self-Supervised Feature Learning and Object Detection.
Tsuyoshi Masuda, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 54, 55, IEEE, 2021
International conference proceedings

Analysis of Social Trends Related to COVID-19 Pandemic Utilizing Social Media Data.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa 0001, Satoshi Asamizu, Miki Haseyama
GCCE, 43, 44, IEEE, 2021
International conference proceedings

Estimating Imagined Images from Brain Activities via Visual Question Answering.
Saya Takada, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 35, 36, IEEE, 2021
International conference proceedings

Action Classification from Egocentric Videos Using Reinforcement Learning-based Pose Estimation.
Shunya Ohaga, Ren Togo, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 9, 10, IEEE, 2021
International conference proceedings

Graph Analysis-based Recommendation via Entity Embeddings Using Wikipedia.
Nozomu Onodera, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
GCCE, 5, 6, IEEE, 2021
International conference proceedings

Reliable Estimation of Deterioration Levels via Late Fusion Using Multi-View Distress Images for Practical Inspection.
Keisuke Maeda, Naoki Ogawa, Takahiro Ogawa 0001, Miki Haseyama
Journal of Imaging, 7, 12, 273, 273, 2021
Scientific journal

Feature Integration Through Semi-Supervised Multimodal Gaussian Process Latent Variable Model With Pseudo-Labels for Interest Level Estimation.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa 0001, Miki Haseyama
IEEE Access, 9, 163843, 163850, 2021
Scientific journal

[Paper] Personalized Recommendation of Tumblr Posts Using Graph Convolutional Networks with Preference-aware Multimodal Features
Kazuma Ohtomo, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, Masahiro Iwahashi
ITE Transactions on Media Technology and Applications, 9, 1, 54, 61, Institute of Image Information and Television Engineers, 2021
Scientific journal

A Proposal and Evaluation of a New Method Incorporating Indicators of Blood Flow and Resistance to Improve the Accuracy of Continuous Blood Pressure Estimation Using PPG
川上健, 川上健, 川上健, 住友和弘, 菅野厚博, 小川貴弘, 南重信, 長谷山美紀
電気学会論文誌 E, 141, 6, 2021

User-selectable Event Summarization in Unedited Raw Soccer Video via Multimodal Bidirectional LSTM
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE TRANSACTIONS ON MEDIA TECHNOLOGY AND APPLICATIONS, 9, 1, 42, 53, INST IMAGE INFORMATION & TELEVISION ENGINEERS, 2021
English, Scientific journal, A new method that generates user-selectable event summaries from unedited raw soccer videos is presented in this paper. Since there are more unedited raw soccer videos than broadcasted/distributed soccer videos and unedited videos have various viewers, it is necessary to analyze these videos for meeting the demands of various viewers. The proposed method introduces a multimodal CNN-BiLSTM architecture for analyzing unedited raw soccer videos. This architecture extracts candidate scenes for event summarization from unedited soccer videos and classifies these scenes into typical events. Finally, our method generates user-selectable event summaries by simultaneously considering the importance of candidate scenes and the event classification results. Experimental results using real unedited raw soccer videos show the effectiveness of our method.

Cross-domain Recommendation Based on Multilayer Graph Analysis Using Subgraph Representation
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
INTERNATIONAL WORKSHOP ON ADVANCED IMAGING TECHNOLOGY (IWAIT) 2021, 11766, SPIE-INT SOC OPTICAL ENGINEERING, 2021
English, International conference proceedings, This paper presents cross-domain recommendation based on multilayer graph analysis using subgraph representation. The proposed method constructs two graphs in source and target domains utilizing user-item embedding and trains link relationships between the users' embedding features on each above graph via graph convolutional networks considering subgraph representation. Thus, the proposed method can obtain features with high representation ability, and this is the main contribution of this paper. Then the proposed method can estimate the user's embedding features in the target domain from those in the source domain and recommend items to users by using the estimated features. Experiments on real-world e-commerce datasets verify the effectiveness of the proposed method.

Interest Estimation Method Based on 2D Pose Features on Security Camera
Yuki Honma, Ren Togo, Maiku Abe, Takahiro Ogawa, Miki Haseyama
INTERNATIONAL WORKSHOP ON ADVANCED IMAGING TECHNOLOGY (IWAIT) 2021, 11766, SPIE-INT SOC OPTICAL ENGINEERING, 2021
English, International conference proceedings, This paper proposes a customer interest estimation method using security camera to meet the demand of the retail industry. In the field of retail industry, it is considered that the understanding of customers' interests in the real store can be used for various marketing activities such as the product development and the layout of the store. Then, it is important to pay attention to customers' behavior in the real store. Their behavior is often recorded by the cameras installed in the store for security purposes. A method for estimating their interests from the videos of the security camera is presented in this paper. The novelty of our method is three-fold. Firstly, the experimental data of subjects in our group were taken by using the security camera already installed in the real store. Secondly, we used a pre-trained posture estimation model and treated the results as the features to be trained by a two-layer neural network model. Finally, a professional have annotated the subjects' interests. The effectiveness of our method was confirmed by comparing with benchmark supervised machine learning models.

Preliminary study of AI-assisted diagnosis using FDG-PET/CT for axillary lymph node metastasis in patients with breast cancer
Zongyao Li, Kazuhiro Kitajima, Kenji Hirata, Ren Togo, Junki Takenaka, Yasuo Miyoshi, Kohsuke Kudo, Takahiro Ogawa, Miki Haseyama
EJNMMI RESEARCH, 11, 1, SPRINGER, Jan. 2021
English, Scientific journal, Background To improve the diagnostic accuracy of axillary lymph node (LN) metastasis in breast cancer patients using 2-[F-18]FDG-PET/CT, we constructed an artificial intelligence (AI)-assisted diagnosis system that uses deep-learning technologies. Materials and methods Two clinicians and the new AI system retrospectively analyzed and diagnosed 414 axillae of 407 patients with biopsy-proven breast cancer who had undergone 2-[F-18]FDG-PET/CT before a mastectomy or breast-conserving surgery with a sentinel lymph node (LN) biopsy and/or axillary LN dissection. We designed and trained a deep 3D convolutional neural network (CNN) as the AI model. The diagnoses from the clinicians were blended with the diagnoses from the AI model to improve the diagnostic accuracy. Results Although the AI model did not outperform the clinicians, the diagnostic accuracies of the clinicians were considerably improved by collaborating with the AI model: the two clinicians' sensitivities of 59.8% and 57.4% increased to 68.6% and 64.2%, respectively, whereas the clinicians' specificities of 99.0% and 99.5% remained unchanged. Conclusions It is expected that AI using deep-learning technologies will be useful in diagnosing axillary LN metastasis using 2-[F-18]FDG-PET/CT. Even if the diagnostic performance of AI is not better than that of clinicians, taking AI diagnoses into consideration may positively impact the overall diagnostic accuracy.

A Note on Detection of Sports Action Based on Temporal Cycle Consistency Learning
Tsuyoshi Masuda, Ren Togo, Takahiro Ogawa, Miki Haseyama
INTERNATIONAL WORKSHOP ON ADVANCED IMAGING TECHNOLOGY (IWAIT) 2021, 11766, SPIE-INT SOC OPTICAL ENGINEERING, 2021
English, International conference proceedings, This paper presents a method for action detection based on Temporal Cycle Consistency(TCC) Learning. The proposed method realizes the action detection of flexible length segments based on a frame-level action prediction technique. We enable calculation of similarities for spatio-temporal features based on TCC to detect target actions from input videos. Finally, our method determines temporal segments by smoothing the frame-level action detection result. Experimental results show the validity of the proposed method.

Interior Coordination Image Retrieval with Object-Detection-Based and Color Features
Ren Togo, Takahiro Ogawa, Miki Haseyama
INTERNATIONAL WORKSHOP ON ADVANCED IMAGING TECHNOLOGY (IWAIT) 2021, 11766, SPIE-INT SOC OPTICAL ENGINEERING, 2021
English, International conference proceedings, This paper presents a new interior coordination image retrieval method using object-detection-based and color features. Interior coordination requires consideration of objects' positional information and the overall atmosphere of the room simultaneously. However, similar image retrieval methods considering the coordination characteristics have not been proposed. In the proposed method, we extract different types of features from interior coordination images and realize the similar interior coordination image retrieval based on our newly derived features.

Self-Supervised Learning for Gastritis Detection with Gastric X-Ray Images.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
CoRR, abs/2104.02864, 2021
Scientific journal

Soft-Label Anonymous Gastric X-ray Image Distillation.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
CoRR, abs/2104.02857, 305, 309, IEEE, 2021
Scientific journal

IR Questioner: QA-based Interactive Retrieval System.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
ICMR '21: International Conference on Multimedia Retrieval(ICMR), 611, 614, ACM, 2021
English, International conference proceedings, Image retrieval from a given text query (text-to-image retrieval) is one of the most essential systems, and it is effectively utilized for databases (DBs) on the Web. To make them more versatile and familiar, a retrieval system that is adaptive even for personal DBs such as images in smartphones and lifelogging devices should be considered. In this paper, we present a novel text-to-image retrieval system that is specialized for personal DBs. With the cross-modal scheme and the question-answering scheme, the developed system enables users to obtain the desired image effectively even from personal DBs. Our demo is available at https://sites.google.com/view/ir-questioner/.

Human Emotion Estimation Using Multi-Modal Variational AutoEncoder with Time Changes.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2021 IEEE 3RD GLOBAL CONFERENCE ON LIFE SCIENCES AND TECHNOLOGIES (IEEE LIFETECH 2021), 67, 68, IEEE, 2021
English, International conference proceedings, A human emotion estimation method via feature integration using multi-modal variational autoencoder (MVAE) with time changes is presented in this paper. To utilize multi-modal information such as gaze and brain activity data including some noises, the proposed method newly introduces MVAE into the human emotion estimation. Furthermore, the proposed MVAE can consider the changes in bio-signals with time and reduce the effect of noises caused in bio-signals by using the probabilistic variation. Experimental results with that of some state-of-the-art methods indicate that the proposed method is effective.

Music Playlist Generation Based on Graph Exploration Using Reinforcement Learning.
Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama
2021 IEEE 3RD GLOBAL CONFERENCE ON LIFE SCIENCES AND TECHNOLOGIES (IEEE LIFETECH 2021), 2021, 53, 54, IEEE, 2021
English, Spreading of music streaming platforms that use playlists to make recommendations, automatic playlist generation has been actively researched. Recently, it has been reported that playlists that have high diversity and smooth track transitions increase user satisfaction. Our previous method that used a two-dimensional space as a reinforcement learning environment has achieved these demands, but there remains the problem that the content of multi-dimensional acoustic features cannot be retained accurately. To solve this problem, in this paper, we present a new method of music playlist generation based on reinforcement learning using a graph structure constructed from multi-dimensional acoustic features directly. The new playlist generation provides greater diversity and smoother track transitions than the previous method. Experimental results are shown for verifying the effectiveness of the proposal method.

Question Answering from Brain Activity Data via Decoder Based on Neural Networks.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
2021 IEEE 3RD GLOBAL CONFERENCE ON LIFE SCIENCES AND TECHNOLOGIES (IEEE LIFETECH 2021), 51, 52, IEEE, 2021
English, International conference proceedings, We build a model that can estimate what subjects recognize from functional magnetic resonance imaging (fMRI) data via a visual question answering (VQA) model. The VQA model can generate an answer to a question about an image. We convert fMRI signals into image features via an fMRI decoder based on the relationship between the fMRI signals and the image features extracted from the gazed image. Then this allows the VQA model to answer a visual question from the fMRI signals measured while the subject is gazing at the image. Though brain decoding, which interprets what humans recognize, has become overwhelmingly popular in neuroscience, they often suffer from the small datasets of brain activity data. To overcome the small size of datasets of fMRI signals, we introduce an fMRI decoder based on neural networks that have a high expressive ability. Even when we do not have enough fMRI signals, the proposed method derives the answer to what a person is looking at from fMRI signals. Experimental results on several datasets show that our method allows us to answer a question about gazed images from fMRI signals.

Cross-Domain Semi-Supervised Deep Metric Learning for Image Sentiment Analysis.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 4150, 4154, IEEE, 2021
International conference proceedings

Feature Integration via Semi-Supervised Ordinally Multi-Modal Gaussian Process Latent Variable Model.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 4130, 4134, IEEE, 2021
International conference proceedings

Multi-Modal Label Dequantized Gaussian Process Latent Variable Model for Ordinal Label Estimation.
Masanao Matsumoto, Keisuke Maeda, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 3985, 3989, IEEE, 2021
International conference proceedings

Semantic-Aware Unpaired Image-to-Image Translation for Urban Scene Images.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2150, 2154, IEEE, 2021
International conference proceedings

Classification of Expert-Novice Level Using Eye Tracking And Motion Data via Conditional Multimodal Variational Autoencoder.
Yusuke Akamatsu, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 1360, 1364, IEEE, 2021
International conference proceedings

Estimation of Visual Features of Viewed Image From Individual and Shared Brain Information Based on FMRI Data Using Probabilistic Generative Model.
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 1335, 1339, IEEE, 2021
English, International conference proceedings, This paper presents a method for estimation of visual features based on brain responses measured when subjects view images. The proposed method estimates visual features of viewed images by using both individual and shared brain information from functional magnetic resonance imaging (fMRI) data when subjects view images. To extract an effective latent space shared by multiple subjects from high dimensional fMRI data, a probabilistic generative model that can provide a prior distribution to the space is introduced into the proposed method. Also, the extraction of a robust feature space with respect to noise for the individual information becomes feasible via the proposed probabilistic generative model. This is the first contribution of our method. Furthermore, the proposed method constructs a decoder transforming brain information into visual features based on collaborative use of both estimated spaces for individual and shared brain information. This is the second contribution of our method. Experimental results show that the proposed method improves the estimation accuracy of the visual features of viewed images.

Human-Centered Favorite Music Classification Using EEG-Based Individual Music Preference Via Deep Time-Series CCA.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 1320, 1324, IEEE, 2021
International conference proceedings

Rubber Material Property Prediction Using Electron Microscope Images of Internal Structures Taken under Multiple Conditions.
Ren Togo, Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 21, 6, 2088, 2088, MDPI, 2021
English, Scientific journal, A method for prediction of properties of rubber materials utilizing electron microscope images of internal structures taken under multiple conditions is presented in this paper. Electron microscope images of rubber materials are taken under several conditions, and effective conditions for the prediction of properties are different for each rubber material. Novel approaches for the selection and integration of reliable prediction results are used in the proposed method. The proposed method enables selection of reliable results based on prediction intervals that can be derived by the predictors that are each constructed from electron microscope images taken under each condition. By monitoring the relationship between prediction results and prediction intervals derived from the corresponding predictors, it can be determined whether the target prediction results are reliable. Furthermore, the proposed method integrates the selected reliable results based on Dempster-Shafer (DS) evidence theory, and this integration result is regarded as a final prediction result. The DS evidence theory enables integration of multiple prediction results, even if the results are obtained from different imaging conditions. This means that integration can even be realized if electron microscope images of each material are taken under different conditions and even if these conditions are different for target materials. This nonconventional approach is suitable for our application, i.e., property prediction. Experiments on rubber material data showed that the evaluation index mean absolute percent error (MAPE) was under 10% by the proposed method. The performance of the proposed method outperformed conventional comparative property estimation methods. Consequently, the proposed method can realize accurate prediction of the properties with consideration of the characteristic of electron microscope images described above.

Detection of Important Scenes in Baseball Videos via a Time-Lag-Aware Multimodal Variational Autoencoder.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 21, 6, 2045, 2045, MDPI, 2021
English, Scientific journal, A new method for the detection of important scenes in baseball videos via a time-lag-aware multimodal variational autoencoder (Tl-MVAE) is presented in this paper. Tl-MVAE estimates latent features calculated from tweet, video, and audio features extracted from tweets and videos. Then, important scenes are detected by estimating the probability of the scene being important from estimated latent features. It should be noted that there exist time-lags between tweets posted by users and videos. To consider the time-lags between tweet features and other features calculated from corresponding multiple previous events, the feature transformation based on feature correlation considering such time-lags is newly introduced to the encoder in MVAE in the proposed method. This is the biggest contribution of the Tl-MVAE. Experimental results obtained from actual baseball videos and their corresponding tweets show the effectiveness of the proposed method.

Deterioration level estimation via neural network maximizing category-based ordinally supervised multi-view canonical correlation.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Multim. Tools Appl., 80, 15, 23091, 23112, SPRINGER, 2021
English, Scientific journal, A deterioration level estimation method via neural network maximizing category-based ordinally supervised multi-view canonical correlation is presented in this paper. This paper focuses on real world data such as industrial applications and has two contributions. First, a novel neural network handling multi-modal features transforms original features into features effectively representing deterioration levels in transmission towers, which are one of the infrastructures, with consideration of only correlation maximization. It can be realized by setting projection matrices maximizing correlations between multiple features into weights of hidden layers. That is, since the proposed network has only a few hidden layers, it can be trained from a small amount of training data. Second, since there exist diverse characteristics and an ordinal scale in deterioration levels, the proposed method newly derives category-based ordinally supervised multi-view canonical correlation analysis (Co-sMVCCA). Co-sMVCCA enables estimation of effective projection considering both within-class divergence and the ordinal scale between classes. Experimental results showed that the proposed method realizes accurate deterioration level estimation.

Disentangled Representation Learning in Real-World Image Datasets via Image Segmentation Prior.
Nao Nakagawa, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 110880, 110888, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, We propose a novel method that can learn easy-to-interpret latent representations in real-world image datasets using a VAE-based model by splitting an image into several disjoint regions. Our method performs object-wise disentanglement by exploiting image segmentation and alpha compositing. With remarkable results obtained by unsupervised disentanglement methods for toy datasets, recent studies have tackled challenging disentanglement for real-world image datasets. However, these methods involve deviations from the standard VAE architecture, which has favorable disentanglement properties. Thus, for disentanglement in images of real-world image datasets with preservation of the VAE backbone, we designed an encoder and a decoder that embed an image into disjoint sets of latent variables corresponding to objects. The encoder includes a pre-trained image segmentation network, which allows our model to focus only on representation learning while adopting image segmentation as an inductive bias. Evaluations using real-world image datasets, CelebA and Stanford Cars, showed that our method achieves improved disentanglement and transferability.

Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 84971, 84981, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only "tweets and corresponding multiple previous events" but also "events and corresponding multiple following posted tweets", the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.

Distress Image Retrieval for Infrastructure Maintenance via Self-Trained Deep Metric Learning Using Experts' Knowledge.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 65234, 65245, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, Distress image retrieval for infrastructure maintenance via self-trained deep metric learning using experts' knowledge is proposed in this paper. Since engineers take multiple images of a single distress part for inspection of road structures, it is necessary to construct a similar distress image retrieval method considering the input of multiple images to support determination of the level of deterioration. Thus, the construction of an image retrieval method while selecting an effective input from multiple images is described in this paper. The proposed method performs deep metric learning by using a small number of effective images labeled by experts' knowledge with information about their effectiveness and a large number of unlabeled images via a self-training approach. Specifically, an end-to-end learning approach that performs retraining of the model by assigning pseudo-labels to these unlabeled images according to the output confidence of the model is achieved. Thus, the proposed method can select an effective image from multiple images that are input at the retrieval as a query image. This is the main contribution of this paper. As a result, the proposed method realizes highly accurate retrieval of similar distress images considering the actual situation of inspection in which multiple images of a distress part are input.

Text-Guided Style Transfer-Based Image Manipulation Using Multimodal Generative Models.
Ren Togo, Megumi Kotera, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 64860, 64870, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, A new style transfer-based image manipulation framework combining generative networks and style transfer networks is presented in this paper. Unlike conventional style transfer tasks, we tackle a new task, text-guided image manipulation. We realize style transfer-based image manipulation that does not require any reference style images and generate a style image from the user's input sentence. In our method, since an initial reference input sentence for a content image can automatically be given by an image-to-text model, the user only needs to update the reference sentence. This scheme can help users when they do not have any images representing the desired style. Although this text-guided image manipulation is a new challenging task, quantitative and qualitative comparisons showed the superiority of our method.

Perceived Image Decoding From Brain Activity Using Shared Information of Multi-Subject fMRI Data.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 26593, 26606, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, Decoding a person's cognitive contents from evoked brain activity is becoming important in the field of brain-computer interaction. Previous studies have decoded a perceived image from functional magnetic resonance imaging (fMRI) activity by constructing brain decoding models that were trained with a single subject's fMRI data. However, accurate decoding is still challenging since fMRI data acquired from only a single subject have several disadvantageous characteristics such as small sample size, noisy nature, and high dimensionality. In this article, we propose a method to decode categories of perceived images from fMRI activity using shared information of multi-subject fMRI data. Specifically, by aggregating fMRI data of multiple subjects that contain a large number of samples, we extract a low-dimensional latent representation shared by multi-subject fMRI data. Then the latent representation is nonlinearly transformed into visual features and semantic features of the perceived images to identify categories from various candidate categories. Our approach leverages rich information obtained from multi-subject fMRI data and improves the decoding performance. Experimental results obtained by using two public fMRI datasets showed that the proposed method can more accurately decode categories of perceived images from fMRI activity than previous approaches using a single subject's fMRI data.

Supervised Fractional-Order Embedding Multiview Canonical Correlation Analysis via Ordinal Label Dequantization for Image Interest Estimation.
Masanao Matsumoto, Naoki Saito 0006, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 9, 21810, 21822, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2021
English, Scientific journal, Supervised fractional-order embedding multiview canonical correlation analysis via ordinal label dequantization (SFEMCCA-OLD) for image interest estimation is presented in this paper. SFEMCCA-OLD is a CCA method that realizes accurate integration of features including low-dimensional ordinal label features. In general, since information is lost due to a limitation of the number of classes, i.e., the dimension of ordinal label information is smaller than those of other features, derivation of highly accurate integration of features is difficult. In SFEMCCA-OLD, the dimension of the ordinal label information can be increased by estimation of the canonical correlation between multiview features. We call this approach ordinal label dequantization. In addition, by introducing a fractional-order technique, our method can calculate optimal projections for noisy data such as real data. Experimental results show that the accuracy of SFEMCCA-OLD for image interest estimation is better than that of recent CCA-based methods.

Chronic atrophic gastritis detection with a convolutional neural network considering stomach regions
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
WORLD JOURNAL OF GASTROENTEROLOGY, 26, 25, 3650, 3659, BAISHIDENG PUBLISHING GROUP INC, Jul. 2020
English, Scientific journal, BACKGROUND The risk of gastric cancer increases in patients withHelicobacter pylori-associated chronic atrophic gastritis (CAG). X-ray examination can evaluate the condition of the stomach, and it can be used for gastric cancer mass screening. However, skilled doctors for interpretation of X-ray examination are decreasing due to the diverse of inspections. AIM To evaluate the effectiveness of stomach regions that are automatically estimated by a deep learning-based model for CAG detection. METHODS We used 815 gastric X-ray images (GXIs) obtained from 815 subjects. The ground truth of this study was the diagnostic results in X-ray and endoscopic examinations. For a part of GXIs for training, the stomach regions are manually annotated. A model for automatic estimation of the stomach regions is trained with the GXIs. For the rest of them, the stomach regions are automatically estimated. Finally, a model for automatic CAG detection is trained with all GXIs for training. RESULTS In the case that the stomach regions were manually annotated for only 10 GXIs and 30 GXIs, the harmonic mean of sensitivity and specificity of CAG detection were 0.955 +/- 0.002 and 0.963 +/- 0.004, respectively. CONCLUSION By estimating stomach regions automatically, our method contributes to the reduction of the workload of manual annotation and the accurate detection of the CAG.

胃X線画像を用いたAIによるH.pylori感染識別と今後の展望　　　　　　　　　　　　　　　
藤後 廉, 小川 貴弘, 間部 克裕, 加藤 元嗣, 長谷山 美紀
日本消化器がん検診学会雑誌, 58, 2, 127, 127, (一社)日本消化器がん検診学会, Mar. 2020
Japanese

Multimodal Important Scene Detection in Far-view Soccer Videos Based on Single Deep Neural Architecture
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE TRANSACTIONS ON MEDIA TECHNOLOGY AND APPLICATIONS, 8, 2, 89, 99, INST IMAGE INFORMATION & TELEVISION ENGINEERS, 2020
English, Scientific journal, The details of the matches of soccer can be estimated from visual and audio sequences, and they correspond to the occurrence of important scenes. Therefore, the use of these sequences is suitable for important scene detection. In this paper, a new multimodal method for important scene detection from visual and audio sequences in far-view soccer videos based on a single deep neural architecture is presented. A unique point of our method is that multiple classifiers can be realized by a single deep neural architecture that includes a Convolutional Neural Network-based feature extractor and a Support Vector Machine-based classifier. This approach provides a solution to the problem of not being able to simultaneously optimize different multiple deep neural architectures from a small amount of training data. Then we monitor confidence measures output from this architecture for the multimodal data and enable their integration to obtain the final classification result.

Important Scene Detection Based on Anomaly Detection using Long Short-Term Memory for Baseball Highlight Generation
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - TAIWAN (ICCE-TAIWAN), 1, 2, IEEE, 2020
English, International conference proceedings, This paper presents an important scene detection method based on anomaly detection using a Long Short-Term Memory (LSTM) for baseball highlight generation. In order to deal with multi-view time series features calculated from tweets and videos, we adopt an anomaly detection method using LSTM. LSTM which can maintain a long-term memory is effective for training such features. Introduction of LSTM into important scene detection of baseball videos is the biggest contribution of this paper. Experimental results show high detection performance by our method.

Interpretable Convolutional Neural Network Including Attribute Estimation for Image Classification
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
ITE TRANSACTIONS ON MEDIA TECHNOLOGY AND APPLICATIONS, 8, 2, 111, 124, INST IMAGE INFORMATION & TELEVISION ENGINEERS, 2020
English, Scientific journal, An interpretable convolutional neural network (CNN) including attribute estimation for image classification is presented in this paper. Although CNNs perform highly accurate image classification, the reason for the classification results obtained by the neural networks is not clear. In order to provide interpretation of CNNs, the proposed method estimates attributes, which explain elements of objects, in an intermediate layer of the network. This enables improvement of the interpretability of CNNs, and it is the main contribution of this paper. Furthermore, the proposed method uses the estimated attributes for image classification in order to enhance its accuracy. Consequently, the proposed method not only provides interpretation of CNNs but also realizes improvement in the performance of image classification.

Image Retrieval Based on Supervised Local Regression and Global Alignment with Relevance Feedback for Insect Identification
Keisuke Maeda, Susumu Genma, Takahiro Ogawa, Miki Haseyama
ITE TRANSACTIONS ON MEDIA TECHNOLOGY AND APPLICATIONS, 8, 3, 140, 150, INST IMAGE INFORMATION & TELEVISION ENGINEERS, 2020
English, Scientific journal, A method for image retrieval based on supervised local regression and global alignment (sLRGA) with relevance feedback for insect identification is presented in this paper. Based on the novel sLRGA, which is an extended version of LRGA, the proposed method estimates ranking scores for image retrieval in such a way that the neighborhood structure of a feature space of the database can be optimally preserved with consideration of class information. This is the main contribution of this paper. By measuring the relevance between all of the images and the query image in the database, sLRGA realizes accurate image retrieval. Furthermore, when positive/negative labels to retrieved images are given by users, the proposed method can improve image retrieval performance considering the query relevance information via use of both relevance feedback and sLRGA. This is the second contribution of this paper. Experimental results show the effectiveness of the proposed method.

Estimation of Person-Specific Visual Attention via Selection of Similar Persons
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - TAIWAN (ICCE-TAIWAN), 1, 2, IEEE, 2020
English, International conference proceedings, This paper presents a method for estimation of person-specific visual attention based on estimated similar persons' visual attention. For improving the estimation performance of person-specific visual attention, the proposed method uses the dataset including the large number of images and corresponding gaze data of many persons not including the target person and trains an estimation model based on deep learning. By using the estimated visual attention of similar persons for the target image, the proposed method estimates the visual attention of the target person with the small amount of gaze data. Experimental results show that the proposed method is effective for estimation of person-specific visual attention.

A Method for Player Importance Prediction from Player Network Using Gaze Position Estimated by LSTM
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
ITE TRANSACTIONS ON MEDIA TECHNOLOGY AND APPLICATIONS, 8, 3, 151, 160, INST IMAGE INFORMATION & TELEVISION ENGINEERS, 2020
English, Scientific journal, A novel method for player importance prediction from a player network using gaze positions estimated by Long Short-Term Memory (LSTM) in soccer videos is presented in this paper. By newly using an estimation model of gaze positions trained by gaze tracking data of experienced persons, it is expected that the importance of each player can be predicted. First, we generate a player network by utilizing the estimated gaze positions and first-arrival regions representing players' connections, e.g., passes between players. The gaze positions are estimated by LSTM that is newly trained from the gaze tracking data of experienced persons. Second, the proposed method predicts the importance of each player by applying the Hypertext Induced Topic Selection (HITS) algorithm to the constructed network. Consequently, prediction of the importance of each player based on soccer tactic knowledge of experienced persons can be realized without constantly obtaining gaze tracking data.

An Estimation Method of Candidate Region for Superimposing Information Based on Gaze Tracking Data in Soccer Videos
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - TAIWAN (ICCE-TAIWAN), 1, 2, IEEE, 2020
English, International conference proceedings, A novel method estimating candidate regions for superimposing information in soccer videos based on gaze tracking data is presented in this paper. The proposed method generates a likelihood map based on visual attention regions based on the gaze tracking data and detection results of objects such as players and soccer goals in soccer videos. Candidate regions for superimposing information are estimated by using the likelihood map. Experimental results show that the proposed method realizes effective candidate region estimation.

Image Retrieval with Data Augmentation of Sentence Labels Based on Paraphrasing
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - TAIWAN (ICCE-TAIWAN), 1, 2, IEEE, 2020
English, International conference proceedings, Text-based image retrieval is a fundamental study in the field of information retrieval. Recent text-based image retrieval methods employ deep neural networks (here-inafter referred to as deep neural TBIR) to retrieve a desired image from a sentence query and achieve the state-of-the-art performance in TBIR. To improve the retrieval performance of the deep neural TBIR method further, it is essential to prepare diverse sentence labels in training data. However, it takes a lot of effort to prepare diverse sentence labels in training data. To address this problem, we propose a novel deep neural TBIR method with data augmentation of the sentence labels in training data. Experimental results show the effectiveness of the proposed method.

Interactive re-ranking for cross-modal retrieval based on object-wise question answering.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
MMAsia 2020: ACM Multimedia Asia(MMAsia), 37, 7, ACM, 2020
International conference proceedings

Similar scene retrieval in soccer videos with weak annotations by multimodal use of bidirectional LSTM.
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
MMAsia 2020: ACM Multimedia Asia(MMAsia), 27, 8, ACM, 2020
International conference proceedings

Quantitative Analysis of Engineer's Skill Using Wearable Sensor Data while Inspecting Highway Bridge.
Genki Suzuki, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2nd IEEE Global Conference on Life Sciences and Technologies(LifeTech), 111, 112, IEEE, 2020
International conference proceedings

Estimating Viewed Images with Natural Language Question Answering from fMRI Data.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
2nd IEEE Global Conference on Life Sciences and Technologies(LifeTech), 99, 100, IEEE, 2020
International conference proceedings

Distress Level Classification of Road Infrastructures via CNN Generating Attention Map.
Naoki Ogawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2nd IEEE Global Conference on Life Sciences and Technologies(LifeTech), 97, 98, IEEE, 2020
International conference proceedings

Interest Estimation for Images Using Eye Gaze-based Visual and Text Features via DLPCCA.
Masanao Matsumoto, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
2nd IEEE Global Conference on Life Sciences and Technologies(LifeTech), 3, 4, IEEE, 2020
International conference proceedings

Mvgan Maximizing Time-Lag Aware Canonical Correlation for Baseball Highlight Generation.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO WORKSHOPS (ICMEW), 1, 6, IEEE, 2020
English, International conference proceedings, This paper presents multi-view unsupervised generative adversarial network maximizing time-lag aware canonical correlation (MvGAN) for baseball highlight generation. MvGAN has the following two contributions. First, MvGAN utilizes textual, visual and audio features calculated from tweets and videos as multi-view features. MvGAN which adopts these multi-view features is the effective work for highlight generation of baseball videos. Second, since there is a temporal difference between posted tweets and the corresponding events, MvGAN introduces a novel feature embedding scheme considering a time-lag between textual features and other features. Specifically, the proposed method newly derives the timelag aware canonical correlation maximization of these multi-view features. This is the biggest contribution of this paper. Furthermore, since MvGAN is an unsupervised method for highlight generation, a large amount of training data with annotation is not needed. Thus, the proposed method has high applicability to the real world.

Generation of Viewed Image Captions From Human Brain Activity Via Unsupervised Text Latent Space.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2521, 2525, IEEE, 2020
English, International conference proceedings, Generation of human cognitive contents based on the analysis of functional magnetic resonance imaging (fMRI) data has been actively researched. Cognitive contents such as viewed images can be estimated by analyzing the relationship between fMRI data and semantic information of viewed images. In this paper, we propose a new method generating captions for viewed images from human brain activity via a novel robust regression scheme. Unlike conventional generation methods using image feature representations, the proposed method makes use of more semantic text feature representations, which are more suitable for the caption generation. We construct a text latent space with unlabeled images not used for the training, and the fMRI data are regressed to the text latent space. Besides, we newly make use of unlabeled images not used for the training phase to improve caption generation performance. Finally, the proposed method can generate captions from the fMRI data measured while subjects are viewing images. Experimental results show that the proposed method enables accurate caption generation for viewed images.

Multimodal Image-to-Image Translation for Generation of Gastritis Images.
Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2466, 2470, IEEE, 2020
English, International conference proceedings, We present a new multimodal image-to-image translation model for the generation of gastritis images using X-ray and blood inspection results. In clinical situations, clinicians estimate the prognosis of the target disease by considering multiple inspection results. Similarly, we take a multimodal approach in the task of gastric cancer risk prediction. Visual characteristics of the gastric X-ray image and blood index values are highly related in the evaluation of gastric cancer risk. If we can generate a prediction image from blood index values, it contributes to the clinicians' sophisticated and integrated diagnosis. Hence, we learn a model that can map non-gastritis images to gastritis images based on the blood index values. Although this is a challenging multimodal task in medical image analysis, experimental results showed the effectiveness of our model.

Image Retrieval With Lingual And Visual Paraphrasing Via Generative Models.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2431, 2435, IEEE, 2020
English, International conference proceedings, A new approach that improves text-based image retrieval (hereinafter referred to as TBIR) performance is proposed in this paper. TBIR methods aim to retrieve a desired image related to a query text. Especially, recent TBIR methods allow us to retrieve images considering word relationships by using a sentence as a query. In these TBIR methods, it is necessary to uniquely identify a desired image from similar images using a single query sentence. However, the diverse expressive styles for a query sentence make it difficult to uniquely identify a desired image. In this paper, we propose a novel TBIR method with paraphrasing on multiple representation spaces. Specifically, by paraphrasing a query sentence on lingual and visual representation spaces, the proposed method can retrieve a desired image from various perspectives and then it can uniquely identify a desired image from similar images. Comprehensive experimental results show the effectiveness of the proposed method.

Variational Autoencoder Based Unsupervised Domain Adaptation For Semantic Segmentation.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2426, 2430, IEEE, 2020
English, International conference proceedings, Unsupervised domain adaptation, which transfers supervised knowledge from a labeled domain to an unlabeled domain, remains a tough problem in the field of computer vision, especially for semantic segmentation. Some methods inspired by adversarial learning and semi-supervised learning have been developed for unsupervised domain adaptation in semantic segmentation and achieved outstanding performances. In this paper, we propose a novel method for this task. Like adversarial learning-based methods using a discriminator to align the feature distributions from different domains, we employ a variational autoencoder to get to the same destination but in a non-adversarial manner. Since the two approaches are compatible, we also integrate an adversarial loss into our method. By further introducing pseudo labels, our method can achieve state-of-the-art performances on two benchmark adaptation scenarios, GTA5-to-CITYSCAPES and SYNTHIA-to-CITYSCAPES.

Important Scene Detection Of Baseball Videos Via Time-Lag Aware Deep Multiset Canonical Correlation Maximization.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1236, 1240, IEEE, 2020
English, International conference proceedings, This paper presents a new important scene detection method of baseball videos based on correlation maximization between heterogeneous modalities via time-lag aware deep multiset canonical correlation analysis (Tl-dMCCA). The technical contributions of this paper are twofold. First, textual, visual and audio features calculated from tweets and videos are adopted as multi-view time series features. Since Tl-dMCCA which utilizes these features includes the unsupervised embedding scheme via deep networks, the proposed method can flexibly express the relationship between heterogeneous features. Second, since there is the time-lag between posted tweets and the corresponding multiple previous events, Tl-dMCCA considers the time-lag relationships between them. Specifically, we newly introduce the representation of such time-lags into the derivation of their covariance matrices. By considering time-lags via Tl-dMCCA, the proposed method correctly detects important scenes.

Soft-Label Anonymous Gastric X-Ray Image Distillation.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 305, 309, IEEE, 2020
English, International conference proceedings, This paper presents a soft-label anonymous gastric X-ray image distillation method based on a gradient descent approach. The sharing of medical data is demanded to construct high-accuracy computer-aided diagnosis (CAD) systems. However, the large size of the medical dataset and privacy protection are remaining problems in medical data sharing, which hindered the research of CAD systems. The idea of our distillation method is to extract the valid information of the medical dataset and generate a tiny distilled dataset that has a different data distribution. Different from model distillation, our method aims to find the optimal distilled images, distilled labels and the optimized learning rate. Experimental results show that the proposed method can not only effectively compress the medical dataset but also anonymize medical images to protect the patient's private information. The proposed approach can improve the efficiency and security of medical data sharing.

Estimation Of Visual Contents Based On Question Answering From Human Brain Activity.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 61, 65, IEEE, 2020
English, International conference proceedings, We propose an estimation method for free-form Visual Question Answering (VQA) from human brain activity, brain decoding VQA. The task of VQA in the field of computer vision is generating an answer given an image and a question about its contents. The proposed method can realize answering arbitrary visual questions about images from brain activity measured by functional Magnetic Resonance Imaging (fMRI) while viewing the same images. We enable estimating various information from brain activity via a unique VQA model, which can realize a more detailed understanding of images and complex reasoning. In addition, we newly make use of un-labeled images not used in the training phase to improve the performance of the transformation, since fMRI datasets are generally small. The proposed method can answer a visual question from a little amount of fMRI data measured while subjects are viewing images.

Feature Integration Via Geometrical Supervised Multi-View Multi-Label Canonical Correlation For Incomplete Label Assignment.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 46, 50, IEEE, 2020
English, International conference proceedings, This paper presents feature integration via geometrical supervised multi-view multi-label canonical correlation analysis (GSM2CCA) for incomplete label assignment. The problem of incomplete labels is frequently encountered in the multi-label classification problem where the training labels are obtained via crowd-sourcing. In such a situation, consideration of only the label correlation, which is the basic approach, is not suitable for improvement of representation ability of features. For dealing with the incomplete label assignment, GSM2CCA constructs effective feature embedding space providing the discriminant ability by introducing both the multi-label correlation and feature similarity of the original feature space into its objective function. Since novel integrated features with high discriminant ability can be calculated by our GSM2CCA, performance improvement of multi-label classification with the incomplete label assignment is realized. The main contribution of this paper is the realization of the effective feature integration via the adoption of the combination use of label similarity and locality preserving projection of heterogeneous features for solving the problem of the incomplete label assignment. The effectiveness of GSM2CCA by applying GSM2CCA-based feature integration to heterogeneous features calculated from various convolutional neural network models is verified via experimental results.

Unsupervised Domain Adaptation for Semantic Segmentation with Symmetric Adaptation Consistency.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 2263, 2267, IEEE, 2020
English, International conference proceedings, Unsupervised domain adaptation, which leverages label information from other domains to solve tasks on a domain without any labels, can alleviate the problem of the scarcity of labels and expensive labeling costs faced by supervised semantic segmentation. In this paper, we utilize adversarial learning and semi-supervised learning simultaneously to solve the task of unsupervised domain adaptation in semantic segmentation. We propose a new approach that trains two segmentation models with the adversarial learning symmetrically and further introduces the consistency between the outputs of the two models into the semi-supervised learning to improve the accuracy of pseudo labels which significantly affect the final adaptation performance. We achieve state-of-the-art semantic segmentation performance on the GTA5-to-Cityscapes scenario, a widely used benchmark setting in unsupervised domain adaptation.

Multi-View Bayesian Generative Model for Multi-Subject FMRI Data on Brain Decoding of Viewed Image Categories.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2020 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 1215, 1219, IEEE, 2020
English, International conference proceedings, Brain decoding studies have demonstrated that viewed image categories can be estimated from human functional magnetic resonance imaging (fMRI) activity. However, there are still limitations with the estimation performance because of the characteristics of fMRI data and the employment of only one modality extracted from viewed images. In this paper, we propose a multi-view Bayesian generative model for multi-subject fMRI data to estimate viewed image categories from fMRI activity. The proposed method derives effective representations of fMRI activity by utilizing multi-subject fMRI data. In addition, we associate fMRI activity with multiple modalities, i:e:, visual features and semantic features extracted from viewed images. Experimental results show that the proposed method outperforms existing state-of-the-art methods of brain decoding.

Interest Level Estimation Based on Feature Integration Considering Distribution of Partially Paired User's Behavior, Videos and Posters.
Kyohei Kamikawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 944, 945, IEEE, 2020
International conference proceedings

Music Playlist Generation Based on Reinforcement Learning Using Acoustic Feature Map.
Keigo Sakurai, Ren Togo, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 942, 943, IEEE, 2020
International conference proceedings

Estimation of Images Matched with Audio-Induced Brain Activity via Modified DGCCA.
Yun Liang 0014, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 940, 941, IEEE, 2020
International conference proceedings

Estimation of User-Specific Visual Attention Considering Individual Tendency toward Gazed Objects.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 745, 746, IEEE, 2020
International conference proceedings

Estimation of Viewed Images Using Individual and Shared Brain Responses.
Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 716, 717, IEEE, 2020
International conference proceedings

Cross-domain Recommendation via Multi-layer Graph Analysis Using User-item Embedding.
Taisei Hirakawa, Keisuke Maeda, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 714, 715, IEEE, 2020
International conference proceedings

Question Answering for Estimation of Seen Image Contents from Multi-subject fMRI Responses.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 712, 713, IEEE, 2020
International conference proceedings

Face Synthesis via User Manipulation of Disentangled Latent Representation.
Nao Nakagawa, Ren Togo, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 692, 693, IEEE, 2020
International conference proceedings

Complexity Evaluation of Medical Image Data for Classification Problem Based on Spectral Clustering.
Guang Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 667, 669, IEEE, 2020
International conference proceedings

Important Scene Prediction of Baseball Videos Using Twitter and Video Analysis Based on LSTM.
Kaito Hirasawa, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
9th IEEE Global Conference on Consumer Electronics(GCCE), 636, 637, IEEE, 2020
International conference proceedings

Brain Decoding of Viewed Image Categories via Semi-Supervised Multi-View Bayesian Generative Model.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Trans. Signal Process., 68, 5769, 5781, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, Brain decoding has shown that viewed image categories can be estimated from evoked functional magnetic resonance imaging (fMRI) activity. Recent studies attempted to estimate viewed image categories that were not used for training previously. Nevertheless, the estimation performance is limited since it is difficult to collect a large amount of fMRI data for training. This paper presents a method to accurately estimate viewed image categories not used for training via a semi-supervised multi-view Bayesian generative model. Our model focuses on the relationship between fMRI activity and multiple modalities, i.e., visual features extracted from viewed images and semantic features obtained from viewed image categories. Furthermore, in order to accurately estimate image categories not used for training, our semi-supervised framework incorporates visual and semantic features obtained from additional image categories in addition to image categories of training data. The estimation performance of the proposed model outperforms existing state-of-the-art models in the brain decoding field and achieves more than 95% identification accuracy. The results also have shown that the incorporation of additional image category information is remarkably effective when the number of training samples is small. Our semi-supervised framework is significant for the brain decoding field where brain activity patterns are insufficient but visual stimuli are sufficient.

Few-Shot Personalized Saliency Prediction Based on Adaptive Image Selection Considering Object and Visual Attention.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 20, 8, 2170, 2170, MDPI, 2020
English, Scientific journal, A few-shot personalized saliency prediction based on adaptive image selection considering object and visual attention is presented in this paper. Since general methods predicting personalized saliency maps (PSMs) need a large number of training images, the establishment of a theory using a small number of training images is needed. To tackle this problem, although finding persons who have visual attention similar to that of a target person is effective, all persons have to commonly gaze at many images. Thus, it becomes difficult and unrealistic when considering their burden. On the other hand, this paper introduces a novel adaptive image selection (AIS) scheme that focuses on the relationship between human visual attention and objects in images. AIS focuses on both a diversity of objects in images and a variance of PSMs for the objects. Specifically, AIS selects images so that selected images have various kinds of objects to maintain their diversity. Moreover, AIS guarantees the high variance of PSMs for persons since it represents the regions that many persons commonly gaze at or do not gaze at. The proposed method enables selecting similar users from a small number of images by selecting images that have high diversities and variances. This is the technical contribution of this paper. Experimental results show the effectiveness of our personalized saliency prediction including the new image selection scheme.

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Sensors, 20, 7, 2146, 2146, MDPI, 2020
English, Scientific journal, The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

Chronic gastritis classification using gastric X-ray images with a semi-supervised learning method based on tri-training.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
Medical Biol. Eng. Comput., 58, 6, 1239, 1250, SPRINGER HEIDELBERG, 2020
English, Scientific journal, High-quality annotations for medical images are always costly and scarce. Many applications of deep learning in the field of medical image analysis face the problem of insufficient annotated data. In this paper, we present a semi-supervised learning method for chronic gastritis classification using gastric X-ray images. The proposed semi-supervised learning method based on tri-training can leverage unannotated data to boost the performance that is achieved with a small amount of annotated data. We utilize a novel learning method named Between-Class learning (BC learning) that can considerably enhance the performance of our semi-supervised learning method. As a result, our method can effectively learn from unannotated data and achieve high diagnostic accuracy for chronic gastritis.

Multi-Task Convolutional Neural Network Leading to High Performance and Interpretability via Attribute Estimation.
Keisuke Maeda, Kazaha Horii, Takahiro Ogawa, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1609, 1612, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, 2020
English, Scientific journal, A multi-task convolutional neural network leading to high performance and interpretability via attribute estimation is presented in this letter. Our method can provide interpretation of the classification results of CNNs by outputting attributes that explain elements of objects as a judgement reason of CNNs in the middle layer. Furthermore, the proposed network uses the estimated attributes for the following prediction of classes. Consequently, construction of a novel multi-task CNN with improvements in both of the interpretability and classification performance is realized.

Inpainting via Sparse Representation Based on a Phaseless Quality Metric.
Takahiro Ogawa, Keisuke Maeda, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1541, 1551, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, 2020
English, Scientific journal, An inpainting method via sparse representation based on a new phaseless quality metric is presented in this paper. Since power spectra, phaseless features, of local regions within images enable more successful representation of their texture characteristics compared to their pixel values, a new quality metric based on these phaseless features is newly derived for image representation. Specifically, the proposed method enables spare representation of target signals, i.e., target patches, including missing intensities by monitoring errors converged by phase retrieval as the novel phaseless quality metric. This is the main contribution of our study. In this approach, the phase retrieval algorithm used in our method has the following two important roles: (1) derivation of the new quality metric that can be derived even for images including missing intensities and (2) conversion of phaseless features, i.e., power spectra, to pixel values, i.e., intensities. Therefore, the above novel approach solves the existing problem of not being able to use better features or better quality metrics for inpainting. Results of experiments showed that the proposed method using sparse representation based on the new phaseless quality metric outperforms previously reported methods that directly use pixel values for inpainting.

Heterogeneous-Graph-Based Video Search Reranking Using Topic Relevance.
Soh Yoshida, Mitsuji Muneyasu, Takahiro Ogawa, Miki Haseyama
IEICE Trans. Fundam. Electron. Commun. Comput. Sci., 103-A, 12, 1529, 1540, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, 2020
English, Scientific journal, In this paper, we address the problem of analyzing topics, included in a social video group, to improve the retrieval performance of videos. Unlike previous methods that focused on an individual visual aspect of videos, the proposed method aims to leverage the "mutual reinforcement" of heterogeneous modalities such as tags and users associated with video on the Internet. To represent multiple types of relationships between each heterogeneous modality, the proposed method constructs three subgraphs: user-tag, video-video, and video-tag graphs. We combine the three types of graphs to obtain a heterogeneous graph. Then the extraction of latent features, i.e., topics, becomes feasible by applying graph-based soft clustering to the heterogeneous graph. By estimating the membership of each grouped cluster for each video, the proposed method defines a new video similarity measure. Since the understanding of video content is enhanced by exploiting latent features obtained from different types of data that complement each other, the performance of visual reranking is improved by the proposed method. Results of experiments on a video dataset that consists of YouTube-8M videos show the effectiveness of the proposed method, which achieves a 24.3% improvement in terms of the mean normalized discounted cumulative gain in a search ranking task compared with the baseline method.

Deep convolutional neural network-based anomaly detection for organ classification in gastric X-ray examination.
Ren Togo, Haruna Watanabe, Takahiro Ogawa, Miki Haseyama
Comput. Biol. Medicine, 123, 103903, 103903, PERGAMON-ELSEVIER SCIENCE LTD, 2020
English, Scientific journal, Aim: The aim of this study was to determine whether our deep convolutional neural network-based anomaly detection model can distinguish differences in esophagus images and stomach images obtained from gastric X-ray examinations.Methods: A total of 6012 subjects were analyzed as our study subjects. Since the number of esophagus X-ray images is much smaller than the number of gastric X-ray images taken in X-ray examinations, we took an anomaly detection approach to realize the task of organ classification. We constructed a deep autoencoding gaussian mixture model (DAGMM) with a convolutional autoencoder architecture. The trained model can produce an anomaly score for a given test X-ray image. For comparison, the original DAGMM, AnoGAN, and a One-Class Support Vector Machine (OCSVM) that were trained with features obtained by a pre-trained Inception-v3 network were used.Results: Sensitivity, specificity, and the calculated harmonic mean of the proposed method were 0.956, 0.980, and 0.968, respectively. Those of the original DAGMM were 0.932, 0.883, and 0.907, respectively. Those of AnoGAN were 0.835, 0.833, and 0.834, respectively, and those of OCSVM were 0.932, 0.935, and 0.934, respectively. Experimental results showed the effectiveness of the proposed method for an organ classification task.Conclusion: Our deep convolutional neural network-based anomaly detection model has shown the potential for clinical use in organ classification.

Human-Centric Emotion Estimation Based on Correlation Maximization Considering Changes With Time in Visual Attention and Brain Activity.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 203358, 203368, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, A human-centric emotion estimation method based on correlation maximization with consideration of changes with time in visual attention and brain activity when viewing images is proposed in this paper. Owing to the recent developments of many kinds of biological sensors, many researchers have focused on multimodal emotion estimation using both eye gaze data and brain activity data for improving the quality of emotion estimation. In this paper, a novel method that focuses on the following two points is introduced. First, in order to reduce the burden on users, we obtain brain activity data from users only in the training phase by using a projection matrix calculated by canonical correlation analysis (CCA) between gaze-based visual features and brain activity-based features. Next, for considering the changes with time in both visual attention and brain activity, we obtain novel features based on CCA-based projection in each time unit. In order to include these two points, the proposed method analyzes a fourth-order gaze and image tensor for which modes are pixel location, color channel and the changes with time in visual attention. Moreover, in each time unit, the proposed method performs CCA between gaze-based visual features and brain activity-based features to realize human-centric emotion estimation with a high level of accuracy. Experimental results show that accurate human emotion estimation is achieved by using our new human-centric image representation.

Estimation of Interest Levels From Behavior Features via Tensor Completion Including Adaptive Similar User Selection.
Keisuke Maeda, Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 126109, 126118, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, A method for estimating interest levels from behavior features via tensor completion including adaptive similar user selection is presented in this paper. The proposed method focuses on a tensor that is suitable for data containing multiple contexts and constructs a third-order tensor in which three modes are "products", "users" and "user behaviors and interest levels" for these products. By complementing this tensor, unknown interest level estimation of a product for a target user becomes feasible. For further improving the estimation performance, the proposed method adaptively selects similar users for the target user by focusing on converged estimation errors between estimated interest levels and known interest levels in the tensor completion. Furthermore, the proposed method can adaptively estimate the unknown interest from the similar users. This is the main contribution of this paper. Therefore, the influence of users having different interests is reduced, and accurate interest level estimation can be realized. In order to verify the effectiveness of the proposed method, we show experimental results obtained by estimating interest levels of users holding books.

Supervised Fractional-Order Embedding Geometrical Multi-View CCA (SFGMCCA) for Multiple Feature Integration.
Keisuke Maeda, Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 114340, 114353, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, Techniques for integrating different types of multiple features effectively have been actively studied in recent years. Multiset canonical correlation analysis (MCCA), which maximizes the sum of pairwise correlations of inter-view (i.e., between different features), is one of the powerful methods for integrating different types of multiple features, and various MCCA-based methods have been proposed. This work focuses on a supervised MCCA variant in order to construct a novel effective feature integration framework. In this paper, we newly propose supervised fractional-order embedding geometrical multi-view CCA (SFGMCCA). This method constructs not only the correlation structure but also two types of geometrical structures of intra-view (i.e., within each feature) and inter-view simultaneously, thereby realizing more precise feature integration. This method also supports the integration of small sample and high-dimensional data by using the fractional-order technique. We conducted experiments using four types of image datasets, i.e., MNIST, COIL-20, ETH-80 and CIFAR-10. Furthermore, we also performed an fMRI dataset containing brain signals to verify the robustness. As a result, it was confirmed that accuracy improvements using SFGMCCA were statistically significant at the significance level of 0.05 compared to those using conventional representative MCCA-based methods.

Enhancing Cross-Modal Retrieval Based on Modality-Specific and Embedding Spaces.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 96777, 96786, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020
English, Scientific journal, A new approach that drastically improves cross-modal retrieval performance in vision and language (hereinafter referred to as & x201C;vision and language retrieval & x201D;) is proposed in this paper. Vision and language retrieval takes data of one modality as a query to retrieve relevant data of another modality, and it enables flexible retrieval across different modalities. Most of the existing methods learn optimal embeddings of visual and lingual information to a single common representation space. However, we argue that the forced embedding optimization results in loss of key information for sentences and images. In this paper, we propose an effective utilization of representation spaces in a simple but robust vision and language retrieval method. The proposed method makes use of multiple individual representation spaces through text-to-image and image-to-text models. Experimental results showed that the proposed approach enhances the performance of existing methods that embed visual and lingual information to a single common representation space.

Context-Aware Network Analysis of Music Streaming Services for Popularity Estimation of Artists.
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 8, 48673, 48685, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2020, [Peer-reviewed]
English, Scientific journal, A novel trial for estimating popularity of artists in music streaming services (MSS) is presented in this paper. The main contribution of this paper is to improve extensibility for using multi-modal features to accurately analyze latent relationships between artists. In the proposed method, a novel framework to construct a network is derived by collaboratively using social metadata and multi-modal features via canonical correlation analysis. Different from conventional methods that do not use multi-modal features, the proposed method can construct a network that can capture social metadata and multi-modal features, i.e., a context-aware network. For effectively analyzing the context-aware network, a novel framework to realize popularity estimation of artists is developed based on network analysis. The proposed method enables effective utilization of the network structure by extracting node features via a node embedding algorithm. By constructing an estimator that can distinguish differences between the node features, the proposed method can archive accurate popularity estimation of artists. Experimental results using multiple real-world datasets that contain artists in various genres in Spotify, one of the largest MSS, are presented. Quantitative and qualitative evaluations show that our method is effective for both classifying and regressing the popularity.

Retrieval of similar scenes based on multimodal distance metric learning in soccer videos
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
MMSports 2019 - Proceedings of the 2nd International Workshop on Multimedia Content Analysis in Sports, co-located with MM 2019, 10, 15, ACM, 15 Oct. 2019, [Peer-reviewed]
International conference proceedings, © 2019 Association for Computing Machinery. This paper presents a new method for retrieval of similar scenes based on multimodal distance metric learning in far-view soccer videos that broadly capture soccer fields and are not edited. We extract visual features and audio features from soccer video clips, and we extract text features from text data corresponding to these soccer video clips. In addition, distance metric learning based on Laplacian Regularized Metric Learning is performed to calculate the distances for each kind of features. Finally, by determining the final rank by integrating these distances, we realize successful multimodal retrieval of similar scenes from query scenes of soccer video clips. Experimental results show the effectiveness of our retrieval method.

Semantic Shot Classification in Baseball Videos Based on Similarities of Visual Features
K. Hirasawa, K. Maeda, T. Ogawa, M. Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 663, 664, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings

Region-based Distress Classification of Road Infrastructures via CNN Without Region Annotation
N. Ogawa, K. Maeda, T. Ogawa, M. Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 764, 765, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings

Aesthetic style transfer through text-to-image synthesis and image-to-image translation
Megumi Kotera, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 492, 493, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings

Voice-input multimedia information retrieval system based on text-to-image GAN
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 943, 944, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings

Estimation of drilling energy from tunnel cutting face image based on online learning
Kentaro Yamamoto, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 77, 1, 794, 795, IEEE, Oct. 2019, [Peer-reviewed]
English

Detection of distress region from subway tunnel images via U-net-based deep semantic segmentation
An Wang, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 766, 767, IEEE, Oct. 2019, [Peer-reviewed]
English, International conference proceedings

Estimation of Emotion Labels via Tensor-Based Spatiotemporal Visual Attention Analysis
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 4105, 4109, IEEE, Sep. 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. This paper presents emotion label estimation via tensor-based spatiotemporal visual attention analysis. It has been reported in the fields of psychology and neuroscience that human emotions are related to two elements, their visual attention change and objects included in a target image. Therefore, the proposed method focuses on the spatiotemporal change of visual attention of human gazing at objects in the target image and constructs two neural networks which enable the emotion label estimation considering both of the above two elements. Specifically, the proposed method newly constructs a fourth-order tensor, gaze and image tensor (GIT) whose modes correspond to the width, the height and the color channel of the target image and the time axis of visual attention which is used for representing the time change. Then the first network, which consists of general tensor discriminant analysis (GTDA) and extreme learning machine (ELM), estimates the emotion label from the fourth-order GIT with concerning their visual attention change. Furthermore, the second network, which consists of pre-trained convolutoinal neural network-based feature extraction, GTDA and ELM, enables the estimation from the second-order GIT including visual features obtained from objects focused at each time. Finally, the proposed method estimates emotion labels based on decision fusion of the outputs from the two networks. Experimental results show the effectiveness of the proposed method.

Scene Retrieval for Video Summarization Based on Text-to-Image gan
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 1825, 1829, IEEE, Sep. 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. We present a new scene retrieval method based on text-to-image Generative Adversarial Network (GAN) and its application to query-based video summarization. Text-to-image GAN is a deep learning method that can generate images from their corresponding sentences. In this paper, we reveal a characteristic that deep learning-based visual features extracted from images generated by text-to-image GAN include semantic information sufficiently. By utilizing the generated images as queries, the proposed method achieves higher scene retrieval performance than those of the stateof-the-art methods. In addition, we introduce a novel architecture that can consider order relationship of the input sentences to our method for realizing a target video summarization. Specifically, the proposed method generates multiple images thorough text-to-image GAN from multiple sentences summarizing target videos. Their summarized video can be obtained by performing the retrieval of corresponding scenes from the target videos according to the generated images with considering the order relationship. Experimental results show the effectiveness of the proposed method in the retrieval and summarization performance.

Gastritis Detection from Gastric X-Ray Images Via Fine-Tuning of Patch-Based Deep Convolutional Neural Network
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 1371, 1375, IEEE, Sep. 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. This paper presents a method for gastritis detection from gastric X-ray images via fine-tuning techniques using a deep convolutional neural network (DCNN). DCNNs can learn parameters to capture high-dimensional features which express semantic contents of images by training on a large number of labeled images. However, lack of gastric X-ray images for training often occurs. To realize accurate detection with a small number of gastric X-ray images, the proposed method adopts fine-tuning techniques and newly introduces simple annotation of stomach regions to gastric X-ray images used for training. The proposed method fine-tunes a pre-trained DCNN with patches and three kinds of patch-level class labels considering not only the image-level ground truth ('gastritis'/'non-gastritis') but also the regions of a stomach since the outside of the stomach is not related to the image-level ground truth. In the test phase, by estimating the patch-level class labels with the fine-tuned DCNN, the proposed method enables the image-level class label estimation which excludes the effect of the unnecessary regions. Experimental results show the effectiveness of the proposed method.

Neural Network Maximizing Ordinally Supervised Multi-View Canonical Correlation for Deterioration Level Estimation
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Proceedings - International Conference on Image Processing, ICIP, 2019-September, 919, 923, IEEE, Sep. 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. This paper presents a neural network maximizing ordinally supervised multi-view canonical correlation for deterioration level estimation. The contributions of this paper are twofold. First, in order to calculate features representing deterioration levels on transmission towers, which is one of the infrastructures, a novel neural network handling multi-modal features is constructed from a small amount of training data. Specifically, in our method, effective transformation to features with high discriminant ability without using many hidden layers is realized by setting projection matrices maximizing correlation between multiple features into hidden layer's weights. Second, since there exists ordinal scale in deterioration levels, the proposed method newly derives ordinally supervised multi-view canonical correlation analysis (OsMVCCA). OsMVCCA enables estimation of the effective projection considering not only label information but also their ordinal scales. Experimental results show that the proposed method realizes accurate deterioration level estimation.

Convolutional sparse coding-based deep random vector functional link network for distress classification of road structures
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Computer-Aided Civil and Infrastructure Engineering, 34, 8, 654, 676, WILEY, Aug. 2019, [Peer-reviewed]
English, Scientific journal, © 2019 Computer-Aided Civil and Infrastructure Engineering This paper presents a convolutional sparse coding (CSC)-based deep random vector functional link network (CSDRN) for distress classification of road structures. The main contribution of this paper is the introduction of CSC into a feature extraction scheme in the distress classification. CSC can extract visual features representing characteristics of target images because it can successfully estimate optimal convolutional dictionary filters and sparse features as visual features by training from a small number of distress images. The optimal dictionaries trained from distress images have basic components of visual characteristics such as edge and line information of distress images. Furthermore, sparse feature maps estimated on the basis of the dictionaries represent both strength of the basic components and location information of regions having their components, and these maps can represent distress images. That is, sparse feature maps can extract key components from distress images that have diverse visual characteristics. Therefore, CSC-based feature extraction is effective for training from a limited number of distress images that have diverse visual characteristics. The construction of a novel neural network, CSDRN, by the use of a combination of CSC-based feature extraction and the DRN classifier, which can also be trained from a small dataset, is shown in this paper. Accurate distress classification is realized via the CSDRN.

Bilingual Lexicon Learning Using Tagged Images via Graph Trilateral Filter-based Feature Refinement
Yui Matsumoto, Shota Hamano, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2019 IEEE International Conference on Consumer Electronics - Taiwan, ICCE-TW 2019, May 2019
International conference proceedings, A novel method to realize bilingual lexicon learning (BLL) using tagged images is presented in this paper. Different from existing methods that require parallel corpora, the proposed method enables extraction of semantically similar words by utilizing not such corpora but tagged images on image sharing services. The main contribution of this paper is derivation of a novel framework to refine visual features of tagged images based on graph trilateral filter-based smoothing. This enables reduction of the influence of noisy tags that are irrelevant to contents of images. As a result, accurate BLL becomes feasible by nearest neighbor search using the refined visual features.

Convolutional Sparse Coding-based Anomalous Event Detection in Surveillance Videos
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama
2019 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-TW), IEEE, May 2019
International conference proceedings

User-Specific Visual Attention Estimation Based on Visual Similarity and Spatial Information in Images　　　　　　　　　　　　　　　
Y. Moroto, K. Maeda, T. Ogawa, M. Haseyama
IEEE International Conference on Consumer Electronics – Taiwan (ICCE-TW), 479, 480, May 2019, [Peer-reviewed]
English, International conference proceedings

Estimating Viewed Image Categories from Human Brain Activity via Semi-supervised Fuzzy Discriminative Canonical Correlation Analysis
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2019-May, 1105, 1109, IEEE, May 2019, [Peer-reviewed]
English, International conference proceedings, © 2019 IEEE. This paper presents a method to estimate viewed image categories from human brain activity via newly derived semi-supervised fuzzy discriminative canonical correlation analysis (Semi-FDCCA). The proposed method can estimate image categories from functional magnetic resonance imaging (fMRI) activity measured while subjects view images by making fMRI activity and visual features obtained from images comparable through Semi-FDCCA. To realize Semi-FDCCA, we first derive a new supervised CCA called FDCCA that can consider fuzzy class information based on image category similarities obtained from WordNet ontology. Second, we adopt SemiCCA that can utilize additional unpaired visual features in addition to pairs of fMRI activity and visual features in order to prevent overfitting to the limited pairs. Furthermore, Semi-FDCCA can be derived by combining FDCCA with SemiCCA. Experimental results show that Semi-FDCCA enables accurate estimation of viewed image categories.

Extraction of regions related to cardiac sarcoidosis in polar map images
Ren Togo, Takahiro Ogawa, Osamu Manabe, Kenji Hirata, Tohru Shiga, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 237, 238, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for extracting important regions for deep learning models in the identification of cardiac sarcoidosis using polar map images. Although deep learning-based detection methods have widely studied, they are still often called black boxes. Since high reliability for provided results from computer-aided diagnosis systems is important toward clinical applications, this problem should be solved. In this paper, we try to visualize important regions for deep learning-based models for improvement of understanding to clinicians. We monitor the variance of confidence of a model constructed with a deep learning-based feature and define it as a contribution value toward the estimated label. We visualize important regions for models based on the contribution value.

Estimation of emotions evoked by images based on multiple gaze-based CNN features
Taiga Matsui, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 194, 195, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for estimating emotions evoked by watching images based on multiple visual features considering relationship with gaze information. The proposed method obtains multiple visual features from multiple middle layers of a Convolutional Neural Network. Then the proposed method newly derives their gaze-based visual features maximizing correlation with gaze information by using Discriminative Locality Preserving Canonical Correlation Analysis. The final estimation result is calculated by integrating multiple estimation results obtained from these gaze-based visual features. Consequently, successful emotion estimation becomes feasible by using such multiple estimation results which correspond to different semantic levels of target images.

Estimation of users' interest levels using tensor completion with SemiCCA
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 239, 240, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a new method for estimation of users' interest levels using tensor completion with SemiCCA. The proposed method extracts new features maximizing correlation between features calculated from partially paired users' behavior and contents with semi-supervised canonical correlation analysis (SemiCCA). By this approach, we can successfully use the contents that users have not viewed for the interest level estimation. Moreover, our method utilizes the tensor completion to estimate unknown interest levels. Consequently, in the proposed method, accurate estimation of interest levels using SemiCCA and the tensor completion is realized. Experimental results are shown to verify the effectiveness of the proposed method by using actual data.

Classification of subcellular protein patterns in human cells with transfer learning
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 273, 274, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. In this paper, we present a deep learning method for classifying subcellular protein patterns in human cells. Our method is mainly based on transfer learning and utilizes a newly proposed loss function named focal loss to deal with the problem of severe class imbalance existing in the task. The performance of our method is evaluated by a MacroF1 score of total 28 classes, and the final MacroF1 score of our method is 0.706.

Estimation of visual attention via canonical correlation between visual and gaze-based features
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 229, 230, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method for estimating visual attention via canonical correlation between visual and gaze-based features. The proposed method estimates user-specific visual attention by comparing a test image with training images including their corresponding individual eye gaze data in a common space. Specifically, canonical correlation analysis can derive projections which enable comparison between visual and gaze-based features in the common space. Therefore, given the new test image, our method projects its visual features to the common space and can estimate visual attention. Experimental results show the effectiveness of the proposed method.

Chronic gastritis detection from gastric X-ray images via deep autoencoding Gaussian mixture models
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 231, 232, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a detection method of chronic gastritis from gastric X-ray images. The conventional method cannot detect chronic gastritis accurately since the number of non-gastritis images is overwhelmingly larger than the number of gastritis images. To deal with this problem, the proposed method performs the detection of chronic gastritis by using Deep Autoencoding Gaussian Mixture Models (DAGMM) which is an anomaly detection approach. DAGMM enables construction of chronic gastritis detection model using only non-gastritis images. In addition, DAGMM is superior to conventional anomaly detection methods since the models of dimensionality reduction and density estimation can be learned simultaneously. Therefore, the proposed method realizes accurate detection of chronic gastritis by utilizing DAGMM.

Fine-tuning of pre-trained DCNN for gastritis detection from gastric X-ray images
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 196, 197, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a detection method of gastritis from gastric X-ray images using fine-tuning techniques. With the development of deep convolutional neural networks (DCNNs), DCNN-based methods have achieved more accurate performance than conventional machine learning methods using hand-crafted features in the field of medical image analysis. However, lack of training images often occurs in clinical situations even though DCNNs require a large amount of training images to avoid overfitting. Therefore, the proposed method aims to consider the clinical situations that a limited amount of the training images are available. By fine-tuning a DCNN pre-trained with a large amount of annotated natural images, we avoid overfitting and realize accurate detection of the gastritis with a small amount of the training images.

Bone metastatic tumor detection based on AnoGAN using CT images
Haruna Watanabe, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 235, 236, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. In this paper, we propose a method to detect bone metastatic tumors using computed tomography (CT) images. Bone metastatic tumors spread from primary cancer to other organs, and they can cause severe pain. Therefore, it is important to detect metastatic tumors earlier in addition to primary cancer. However, since metastatic tumors are very small, and they emerge from unpredictable regions in the body, collecting metastatic tumor images is difficult compared to primary cancer. In such a case, it can be considered that the idea of anomaly detection is suitable. The proposed method based on a generative adversarial network model trains with only non-metastatic bone tumor images and detects bone metastatic tumor in an unsupervised manner. Then the anomaly score is defined for each test CT image. Experimental results show the anomaly scores between non-metastatic bone tumor images and metastatic bone tumor images are clearly different. The anomaly detection approach may be effective for the detection of bone metastatic tumors in CT images.

Semi-supervised discriminative CCA for estimating viewed image categories from fMRI data
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 233, 234, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method that estimates viewed image categories from functional magnetic resonance imaging (fMRI) data via semi-supervised discriminative canonical correlation analysis (Semi-DCCA). We newly derive Semi-DCCA that enables direct comparison of fMRI data and visual features extracted from viewed images while taking into account the class information and additional visual features to avoid overfitting. The proposed method enables estimation of image categories from fMRI data measured when subjects view images by comparing fMRI data with visual features through Semi-DCCA. Experimental results show that Semi-DCCA can improve estimation performance of the viewed image categories.

Video classification based on user preferences with soft-bag multiple instance learning
Akira Toyoda, Takahiro Ogawa, Miki Haseyama
2019 IEEE 1st Global Conference on Life Sciences and Technologies, LifeTech 2019, 198, 199, IEEE, Mar. 2019, [Peer-reviewed]
International conference proceedings, © 2019 IEEE. This paper presents a method to classify videos based on user preferences with soft-bag multiple instance learning (MIL). Our method classifies videos that a user has watched into two classes (preferred and not-preferred) with two-modal features extracted from the videos and brain signals measured while the user is watching the videos. Our method splits videos and brain signals into fixed-length segments and computes features used for classification from only a fixed-number of segments selected based on the idea of soft-bag MIL. By using the features computed from the selected segments, our method makes it possible to classify videos in the case that some videos that a user prefers contain some scenes the user does not prefer, and vice versa. Our main contribution allows methods classifying videos based on user preferences to treat such a case unlike conventional methods.

The Friction Properties of Firebrat Scales
Yuji Hirai, Naoto Okuda, Naoki Saito, Takahiro Ogawa, Ryuichiro Machida, Shuhei Nomura, Masahiro Ohara, Miki Haseyama, Masatsugu Shimomura
BIOMIMETICS, 4, 1, MDPI, Jan. 2019
English, Scientific journal, Friction is an important subject for sustainability due to problems that are associated with energy loss. In recent years, micro- and nanostructured surfaces have attracted much attention to reduce friction; however, suitable structures are still under consideration. Many functional surfaces are present in nature, such as the friction reduction surfaces of snake skins. In this study, we focused on firebrats, Thermobia domestica, which temporary live in narrow spaces, such as piled papers, so their body surface (integument) is frequently in contact with surrounding substrates. We speculate that, in addition to optical, cleaning effects, protection against desiccation and enemies, their body surface may be also adapted to reduce friction. To investigate the functional effects of the firebrat scales, firebrat surfaces were observed using a field-emission scanning electron microscope (FE-SEM) and a colloidal probe atomic force microscope (AFM). Results of surface observations by FE-SEM revealed that adult firebrats are entirely covered with scales, whose surfaces have microgroove structures. Scale groove wavelengths around the firebrat's head are almost uniform within a scale but they vary between scales. At the level of single scales, AFM friction force measurements revealed that the firebrat scale reduces friction by decreasing the contact area between scales and a colloidal probe. The heterogeneity of the scales' groove wavelengths suggests that it is difficult to fix the whole body on critical rough surfaces and may result in a "fail-safe" mechanism.

Multimodal Retrieval of Similar Soccer Videos Based on Optimal Combination of Multiple Distance Measures.
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE 8th Global Conference on Consumer Electronics(GCCE), 665, 666, IEEE, 2019
International conference proceedings

Similarity Calculation Based on Pass Regions in Soccer Videos.
Sho Takahashi, Marco Bertini, Alberto Del Bimbo, Miki Haseyama, Toru Hagiwara
IEEE 8th Global Conference on Consumer Electronics(GCCE), 515, 516, IEEE, 2019
International conference proceedings

Interest Estimation for Images Based on Eye Gaze-based Visual and Text Features.
Masanao Matsumoto, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
IEEE 8th Global Conference on Consumer Electronics(GCCE), 481, 482, IEEE, 2019
International conference proceedings

Performance Prediction Method of Examinees Based on Matrix Completion.
Yutaka Yamada, Takahiro Ogawa, Miki Haseyama
IEEE 8th Global Conference on Consumer Electronics(GCCE), 229, 230, IEEE, 2019, [Peer-reviewed]
International conference proceedings

Estimating Viewed Image Categories from fMRI Activity via Multi-view Bayesian Generative Model.
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE 8th Global Conference on Consumer Electronics(GCCE), 127, 128, IEEE, 2019, [Peer-reviewed]
International conference proceedings

The Extraction of Individual Music Preference Based on Deep Time-series CCA.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
IEEE 8th Global Conference on Consumer Electronics(GCCE), 15, 16, IEEE, 2019, [Peer-reviewed]
International conference proceedings

Estimation of User-Specific Visual Attention Based on Gaze Information of Similar Users.
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 477, 478, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings

Effectiveness Evaluation of Deep Features for Image Reconstruction from fMRI Signals.
Saya Takada, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 479, 480, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings

Scene Retrieval Using Text-to-image GAN-based Visual Similarities and Image-to-text Model-based Textual Similarities.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Global Conference on Consumer Electronics (GCCE), 13, 14, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings

Query is GAN: Scene Retrieval with Attentional Text-To-Image Generative Adversarial Network
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 153183, 153193, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, © 2013 IEEE. Scene retrieval from input descriptions has been one of the most important applications with the increasing number of videos on the Web. However, this is still a challenging task since semantic gaps between features of texts and videos exist. In this paper, we try to solve this problem by utilizing a text-To-image Generative Adversarial Network (GAN), which has become one of the most attractive research topics in recent years. The text-To-image GAN is a deep learning model that can generate images from their corresponding descriptions. We propose a new retrieval framework, 'Query is GAN', based on the text-To-image GAN that drastically improves scene retrieval performance by simple procedures. Our novel idea makes use of images generated by the text-To-image GAN as queries for the scene retrieval task. In addition, unlike many studies on text-To-image GANs that mainly focused on the generation of high-quality images, we reveal that the generated images have reasonable visual features suitable for the queries even though they are not visually pleasant. We show the effectiveness of the proposed framework through experimental evaluation in which scene retrieval is performed from real video datasets.

Synthetic gastritis image generation via loss function-based conditional pggan
Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 87448, 87457, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, © 2013 IEEE. In this paper, a novel synthetic gastritis image generation method based on a generative adversarial network (GAN) model is presented. Sharing medical image data is a crucial issue for realizing diagnostic supporting systems. However, it is still difficult for researchers to obtain medical image data since the data include individual information. Recently proposed GAN models can learn the distribution of training images without seeing real image data, and individual information can be completely anonymized by generated images. If generated images can be used as training images in medical image classification, promoting medical image analysis will become feasible. In this paper, we targeted gastritis, which is a risk factor for gastric cancer and can be diagnosed by gastric X-ray images. Instead of collecting a large amount of gastric X-ray image data, an image generation approach was adopted in our method. We newly propose loss function-based conditional progressive growing generative adversarial network (LC-PGGAN), a gastritis image generation method that can be used for a gastritis classification problem. The LC-PGGAN gradually learns the characteristics of gastritis in gastric X-ray images by adding new layers during the training step. Moreover, the LC-PGGAN employs loss function-based conditional adversarial learning so that generated images can be used as the gastritis classification task. We show that images generated by the LC-PGGAN are effective for gastritis classification using gastric X-ray images and have clinical characteristics of the target symptom.

Semi-Supervised Learning Based on Tri-Training for Gastritis Classification using Gastric X-ray Images.
Zongyao Li, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE International Symposium on Circuits and Systems, ISCAS 2019, Sapporo, Japan, May 26-29, 2019, 1, 5, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents a method of semi-supervised learning based on tri-training for gastritis classification using gastric X-ray images. The proposed method is constructed based on the tri-training architecture, and the strategies of label smoothing regularization and random erasing augmentation are utilized in the method to enhance the performance. Although the task of gastritis classification is challenging, we report that the proposed semi-supervised learning method using only a small number of labeled data achieves 0.888 harmonic mean of sensitivity and specificity on test data composed of 615 patients.

Synthetic Image Generation for Gastritis Detection Based on Auxiliary Classifier Generative Adversarial Network.
Misaki Kanai, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE International Symposium on Circuits and Systems, ISCAS 2019, Sapporo, Japan, May 26-29, 2019, 1, 5, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, With the development of convolutional neural networks (CNNs), CNN-based methods for medical image analysis have achieved more accurate performance than conventional machine learning methods using hand-crafted features. Although these methods utilize a large number of training images and realize high performance, lack of the training images often occurs in medical image analysis due to several reasons. This paper presents a novel image generation method to construct a dataset for gastritis detection from gastric X-ray images. The proposed method effectively utilizes two kinds of training images (gastritis and non-gastritis images) to generate images of each domain by introducing label conditioning into a generative model. Experimental results using real-world gastric X-ray images show the effectiveness of the proposed method.

Text-to-Image GAN-Based Scene Retrieval and Re-Ranking Considering Word Importance.
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 169920, 169930, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a novel scene retrieval and re-ranking method based on a text-to-image Generative Adversarial Network (GAN). The proposed method generates an image from an input query sentence based on the text-to-image GAN and then retrieves a scene that is the most similar to the generated image. By utilizing the image generated from the input query sentence as a query, we can control semantic information of the query image at the text level. Furthermore, we introduce a novel interactive re-ranking scheme to our retrieval method. Specifically, users can consider the importance of each word within the first input query sentence. Then the proposed method re-generates the query image that reflects the word importance provided by users. By updating the generated query image based on the word importance, it becomes feasible for users to revise retrieval results through this re-ranking process. In experiments, we showed that our retrieval method including the re-ranking scheme outperforms recently proposed retrieval methods.

Estimating Regions of Deterioration in Electron Microscope Images of Rubber Materials via a Transfer Learning-Based Anomaly Detection Model.
Ren Togo, Naoki Saito 0006, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 162395, 162404, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, A method for estimating regions of deterioration in electron microscope images of rubber materials is presented in this paper. Deterioration of rubber materials is caused by molecular cleavage, external force, and heat. An understanding of these characteristics is essential in the field of material science for the development of durable rubber materials. Rubber material deterioration can be observed by using on electron microscope but it requires much effort and specialized knowledge to find regions of deterioration. In this paper, we propose an automated deterioration region estimation method based on deep learning and anomaly detection techniques to support such material development. Our anomaly detection model, called Transfer Learning-based Deep Autoencoding Gaussian Mixture Model (TL-DAGMM), uses only normal regions for training since obtaining training data for regions of deterioration is difficult. TL-DAGMM makes use of extracted high representation features from a pre-trained deep learning model and can automatically learn the characteristics of normal rubber material regions. Regions of deterioration are estimated at the pixel level by calculated anomaly scores. Experiments on real rubber material electron microscope images demonstrated the effectiveness of our model.

Multi-feature Fusion Based on Supervised Multi-view Multi-label Canonical Correlation Projection.
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, 3936, 3940, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, This paper presents multi-feature fusion based on supervised multi view multi-label canonical correlation projection (sM2CP). The proposed method applies sM2CP-based feature fusion to multiple features obtained from various convolutional neural networks (CNNs) whose characteristics are different. Since new fused features with high representation ability can be obtained, performance improvement of multi-label classification is realized. Specifically, in order to tackle the multi-label problem, sM2CP introduces a label similarity information of label vectors into the objective function of supervised multi-view canonical correlation analysis. Thus, sM2CP can deal with complex label information such as multi-label annotation. The main contribution of this paper is the realization of feature fusion of multiple CNN features for the multi-label problem by introducing multi-label similarity information into the canonical correlation analysis-based feature fusion approach. Experimental results show the effectiveness of sM2CP, which enables effective fusion of multiple CNN features.

Team Tactics Estimation in Soccer Videos Based on a Deep Extreme Learning Machine and Characteristics of the Tactics.
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 153238, 153248, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, A novel method for estimating team tactics in soccer videos based on a Deep Extreme Learning Machine (DELM) and unique characteristics of tactics is presented in this paper. The proposed method estimates the tactics of each team from players formations and enables successful training from a limited amount of training data. Specifically, the estimation of tactics consists of two stages. First, by utilizing two DELMs corresponding to the two teams, the proposed method estimates the provisional tactics of each team. Second, the proposed method updates the team tactics based on unique characteristics of soccer tactics, the relationship between tactics of the two teams and information on ball possession. Consequently, since the proposed method estimates the team tactics that satisfy these characteristics, accurate estimation results can be obtained. In an experiment, the proposed method is applied to actual soccer videos to verify its effectiveness.

Interest Level Estimation Based on Tensor Completion via Feature Integration for Partially Paired User's Behavior and Videos.
Tetsuya Kushima, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 148576, 148585, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, A novel method for interest level estimation based on tensor completion via feature integration for partially paired users' behavior and videos is presented in this paper. The proposed method defines a novel canonical correlation analysis (CCA) framework that is suitable for interest level estimation, which is a hybrid version of semi-supervised CCA (SemiCCA) and supervised locality preserving CCA (SLPCCA) called semi-supervised locality preserving CCA (S2LPCCA). For partially paired users' behavior and videos in actual shops and on the Internet, new integrated features that maximize the correlation between partially paired samples by the principal component analysis (PCA)-mixed CCA framework are calculated. Then videos that users have not watched can be used for the estimation of users' interest levels. Furthermore, local structures of partially paired samples in the same class are preserved for accurate estimation of interest levels. Tensor completion, which can be applied to three contexts, videos, users and "canonical features and interest levels," is used for estimation of interest levels. Consequently, the proposed method realizes accurate estimation of users' interest levels based on S2LPCCA and the tensor completion from partially paired training features of users' behavior and videos. Experimental results obtained by applying the proposed method to actual data show the effectiveness of the proposed method.

Consensus Clustering of Tweet Networks via Semantic and Sentiment Similarity Estimation.
Ryosuke Harakawa, Shoji Takimura, Takahiro Ogawa, Miki Haseyama, Masahiro Iwahashi
IEEE Access, 7, 116207, 116217, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, Although Twitter has become an important source of information, the number of accessible tweets is too large for users to easily find their desired information. To overcome this difficulty, a method for tweet clustering is proposed in this paper. Inspired by the reports that network representation is useful for multimedia content analysis including clustering, a network-based approach is employed. Specifically, a consensus clustering method for tweet networks that represent relationships among the tweets' semantics and sentiment are newly derived. The proposed method integrates multiple clustering results obtained by applying successful clustering methods to the tweet networks. By integrating complementary clustering results obtained based on semantic and sentiment features, the accurate clustering of tweets becomes feasible. The contribution of this work can be found in the utilization of the features, which differs from existing network-based consensus clustering methods that target only the network structure. Experimental results for a real-world Twitter dataset, which includes 65 553 tweets of 25 datasets, verify the effectiveness of the proposed method.

Music Video Recommendation Based on Link Prediction Considering Local and Global Structures of a Network.
Yui Matsumoto, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE Access, 7, 104155, 104167, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2019, [Peer-reviewed]
English, Scientific journal, A novel method for music video recommendation is presented in this paper. The contributions of this paper are two-fold. (i) The proposed method constructs a network, which not only represents relationships between music videos and users but also captures multi-modal features of music videos. This enables collaborative use of multi-modal features such as audio, visual, and textual features, and multiple social metadata that can represent relationships between music videos and users on video hosting services. (ii) A novel scheme for link prediction considering local and global structures of the network (LP-LGSN) is newly derived by fusing multiple link prediction scores based on both local and global structures. By using the LP-LGSN to predict the degrees to which users desire music videos, the proposed method can recommend users' desired music videos. The experimental results for a real-world dataset constructed from YouTube-8M show the effectiveness of the proposed method.

Scene Retrieval from Multiple Resolution Generated Images Based on Text-to-image GAN
Rintaro Yanagi, Ren Togo, Takahiro Ogawa, Miki Haseyama
2019 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), 1, 5, IEEE, 2019, [Peer-reviewed]
English, International conference proceedings, Text-to-image Generative Adversarial Network (GAN) is a deep learning model that generates an image from an input sentence. It is expressly attracting attentions because of its applicability of the generated images. However, many existing studies have still focused on generation of high-quality images, and there are few studies focusing on application of the generated images since text-to-image GANs still cannot produce visually pleasing images in the complicated tasks. In this paper, we apply a text-to-image GAN as a generator of query images for a scene retrieval task to show availability of the visually non-pleasant images. The proposed method utilizes a low-resolution generated image that focuses on a sentence and a high-resolution generated image that focuses on each word of the sentence to retrieve a desired scene. With this mechanism, the proposed method realizes a high-accuracy scene retrieval from a sentence input. Experimental results show the effectiveness of our method.

Cardiac sarcoidosis classification with deep convolutional neural network-based features using polar maps.
Ren Togo, Kenji Hirata, Osamu Manabe, Hiroshi Ohira, Ichizo Tsujino, Keiichi Magota, Takahiro Ogawa, Miki Haseyama, Tohru Shiga
Computers in biology and medicine, 104, 81, 86, PERGAMON-ELSEVIER SCIENCE LTD, Jan. 2019, [Peer-reviewed], [International Magazine]
English, Scientific journal, AIMS: The aim of this study was to determine whether deep convolutional neural network (DCNN)-based features can represent the difference between cardiac sarcoidosis (CS) and non-CS using polar maps. METHODS: A total of 85 patients (33 CS patients and 52 non-CS patients) were analyzed as our study subjects. One radiologist reviewed PET/CT images and defined the left ventricle region for the construction of polar maps. We extracted high-level features from the polar maps through the Inception-v3 network and evaluated their effectiveness by applying them to a CS classification task. Then we introduced the ReliefF algorithm in our method. The standardized uptake value (SUV)-based classification method and the coefficient of variance (CoV)-based classification method were used as comparative methods. RESULTS: Sensitivity, specificity and the harmonic mean of sensitivity and specificity of our method with the ReliefF algorithm were 0.839, 0.870 and 0.854, respectively. Those of the SUVmax-based classification method were 0.468, 0.710 and 0.564, respectively, and those of the CoV-based classification method were 0.655, 0.750 and 0.699, respectively. CONCLUSION: The DCNN-based high-level features may be more effective than low-level features used in conventional quantitative analysis methods for CS classification.

Distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
Advanced Engineering Informatics, 37, 79, 87, Elsevier Ltd, 01 Aug. 2018, [Peer-reviewed]
English, Scientific journal, This paper presents distress classification of class-imbalanced inspection data via correlation-maximizing weighted extreme learning machine (CMWELM). For distress classification, it is necessary to extract semantic features that can effectively distinguish multiple kinds of distress from a small amount of class-imbalanced data. In recent machine learning techniques such as general deep learning methods, since effective feature transformation from visual features to semantic features can be realized by using multiple hidden layers, a large amount of training data are required. However, since the amount of training data of civil structures becomes small, it becomes difficult to perform successful transformation by using these multiple hidden layers. On the other hand, CMWELM consists of two hidden layers. The first hidden layer performs feature transformation, which can directly extract the semantic features from visual features, and the second hidden layer performs classification with solving the class-imbalanced problem. Specifically, in the first hidden layer, the feature transformation is realized by using projections obtained by maximizing the canonical correlation between visual and text features as weight parameters of the hidden layer without designing multiple hidden layers. Furthermore, the second hidden layer enables successful training of our classifier by using weighting factors concerning the class-imbalanced problem. Consequently, CMWELM realizes accurate distress classification from a small amount of class-imbalanced data.

Strategy to develop convolutional neural network-based classifier for diagnosis of whole-body FDG PET images
Keisuke Kawauchi, Kenji Hirata, Seiya Ichikawa, Osamu Manabe, Kentaro Kobayashi, Shiro Watanabe, Miki Haseyama, Takahiro Ogawa, Ren Togo, Tohru Shiga, Chietsugu Katoh
Society of Nuclear Medicine and Molecular Imaging Annual Meeting (SNMMI), 59, SOC NUCLEAR MEDICINE INC, Jun. 2018, [Peer-reviewed]
English, International conference proceedings

Use of deep convolutional neural network-based features for detection of cardiac sarcoidosis from polar map
Ren Togo, Kenji Hirata, Osamu Manabe, Hiroshi Ohira, Ichizo Tsujino, Takahiro Ogawa, Miki Haseyama, Tohru Shiga
Society of Nuclear Medicine and Molecular Imaging Annual Meeting (SNMMI), 59, SOC NUCLEAR MEDICINE INC, Jun. 2018, [Peer-reviewed]
English, International conference proceedings

Estimating the quality of fractal compressed images using lacunarity
Megumi Takezawa, Hirofumi Sanada, Takahiro Ogawa, Miki Haseyama
IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E101A, 6, 900, 903, Institute of Electronics, Information and Communication, Engineers, IEICE, 01 Jun. 2018, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose a highly accurate method for estimating the quality of images compressed using fractal image compression. Using an iterated function system, fractal image compression compresses images by exploiting their self-similarity, thereby achieving high levels of performance
however, we cannot always use fractal image compression as a standard compression technique because some compressed images are of low quality. Generally, sufficient time is required for encoding and decoding an image before it can be determined whether the compressed image is of low quality or not. Therefore, in our previous study, we proposed a method to estimate the quality of images compressed using fractal image compression. Our previous method estimated the quality using image features of a given image without actually encoding and decoding the image, thereby providing an estimate rather quickly
however, estimation accuracy was not entirely sufficient. Therefore, in this paper, we extend our previously proposed method for improving estimation accuracy. Our improved method adopts a new image feature, namely lacunarity. Results of simulation showed that the proposed method achieves higher levels of accuracy than those of our previous method.

Graph-Based Video Search Reranking with Local and Global Consistency Analysis
Soh Yoshida, Takahiro Ogawa, Miki Haseyama, Mitsuji Muneyasu
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E101D, 5, 1430, 1440, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, May 2018, [Peer-reviewed]
English, Scientific journal, Video reranking is an effective way for improving the retrieval performance of text-based video search engines. This paper proposes a graph-based Web video search reranking method with local and global consistency analysis. Generally, the graph-based reranking approach constructs a graph whose nodes and edges respectively correspond to videos and their pairwise similarities. A lot of reranking methods are built based on a scheme which regularizes the smoothness of pairwise relevance scores between adjacent nodes with regard to a user's query. However, since the overall consistency is measured by aggregating only the local consistency over each pair, errors in score estimation increase when noisy samples are included within query-relevant videos' neighbors. To deal with the noisy samples, the proposed method leverages the global consistency of the graph structure, which is different from the conventional methods. Specifically, in order to detect this consistency, the propose method introduces a spectral clustering algorithm which can detect video groups, in which videos have strong semantic correlation, on the graph. Furthermore, a new regularization term, which smooths ranking scores within the same group, is introduced to the reranking framework. Since the score regularization is performed by both local and global aspects simultaneously, the accurate score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

映像情報メディア年報2017シリーズ(第8回)メディア工学の研究動向
長谷山 美紀, 河村 圭, 田良島 周平, 新井 啓之
映像情報メディア学会誌, Vol.72, No.2, pp.241-246, Mar. 2018
Japanese, Scientific journal, 本稿では、(1) 拡張現実・仮想現実の研究動向、(2) 深層学習の研究動向と実用展開、(3) IoT とメディア処理、等、メディア工学分野の研究動向を紹介する。(1)については、デプスセンサやヘッドマウントディスプレイ等のデバイスの登場と性能向上を踏まえつつ、これらを活用した研究開発の進展と動向を紹介する。(2) については、加速度的に進む画像映像メディアへの深層学習技術の適用について、具体的に画像分類や物体検出、さらには画像検索から教師データの検討、実用展開に至るまでを解説する。(3)では、(2) を受けてメディア工学の分野でAI 技術と相乗効果を生み出すIoT 技術について解説する。

A Novel Framework for Estimating Viewer Interest by Unsupervised Multimodal Anomaly Detection
Yuma Sasaka, Takahiro Ogawa, Miki Haseyama
IEEE Access, 6, 8340, 8350, Institute of Electrical and Electronics Engineers Inc., 09 Feb. 2018, [Peer-reviewed]
English, Scientific journal, A reliable method to estimate viewer interest is highly sought after for human-centered video information retrieval. A method that estimates viewer interest while users are watching Web videos is presented in this paper. The method uses a framework for anomaly detection based on collaborative use of facial expression and biological signals such as electroencephalogram (EEG) signals. To the best of our knowledge, there have been no studies that have taken into account two actual mechanisms of the behavior of users while they are watching Web videos. First, whereas most Web videos garner very little attention, a small number attract millions of views. Therefore, a framework for anomaly detection is newly applied to facial expression and EEG in order to model the imbalanced distribution of popularity. Second, since the number of Web videos that are labeled by users as interesting/not interesting is generally too small to estimate viewer interest by a supervised approach, the proposed method utilizes parametric techniques for anomaly detection, which estimates viewer interest in an unsupervised way. Unlike some related studies for estimating viewer interest, our method takes into account actual mechanisms of the behavior of users while they are watching Web videos by utilizing parametric techniques for anomaly detection. Then viewer interest can be estimated on the basis of an anomaly score calculated from our proposed method. Consequently, successful estimation of viewer interest based on a framework for anomaly detection, via collaborative use of facial expression and biological signals, becomes feasible.

Accurate estimation of personalized video preference using multiple users' viewing behavior
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
IEICE Transactions on Information and Systems, E101D, 2, 481, 490, Institute of Electronics, Information and Communication, Engineers, IEICE, 01 Feb. 2018, [Peer-reviewed]
English, Scientific journal, A method for accurate estimation of personalized video preference using multiple users' viewing behavior is presented in this paper. The proposed method uses three kinds of features: a video, user's viewing behavior and evaluation scores for the video given by a target user. First, the proposed method applies Supervised Multiview Spectral Embedding (SMSE) to obtain lower-dimensional video features suitable for the following correlation analysis. Next, supervised Multi-View Canonical Correlation Analysis (sMVCCA) is applied to integrate the three kinds of features. Then we can get optimal projections to obtain new visual features, "canonical video features" reflecting the target user's individual preference for a video based on sMVCCA. Furthermore, in our method, we use not only the target user's viewing behavior but also other users' viewing behavior for obtaining the optimal canonical video features of the target user. This unique approach is the biggest contribution of this paper. Finally, by integrating these canonical video features, Support Vector Ordinal Regression with Implicit Constraints (SVORIM) is trained in our method. Consequently, the target user's preference for a video can be estimated by using the trained SVORIM. Experimental results show the effectiveness of our method.

Media engineering
Miki Haseyama, Kei Kawamura, Shuhei Tarashima, Hiroyuki Arai
Kyokai Joho Imeji Zasshi/Journal of the Institute of Image Information and Television Engineers, 72, 3, 241, 246, Inst. of Image Information and Television Engineers, 2018
Japanese, Scientific journal

Sentiment-aware personalized tweet recommendation through multimodal FFM
Ryosuke Harakawa, Daichi Takehara, Takahiro Ogawa, Miki Haseyama
Multimedia Tools and Applications, 77, 14, 18741, 18759, SPRINGER, 2018, [Peer-reviewed]
English, Scientific journal, For realizing quick and accurate access to desired information and effective advertisements or election campaigns, personalized tweet recommendation is highly demanded. Since multimedia contents including tweets are tools for users to convey their sentiment, users' interest in tweets is strongly influenced by sentiment factors. Therefore, successful personalized tweet recommendation can be realized if sentiment in tweets can be estimated. However, sentiment factors were not taken into account in previous works and the performance of previous methods may be limited. To overcome the limitation, a method for sentiment-aware personalized tweet recommendation through multimodal Field-aware Factorization Machines (FFM) is newly proposed in this paper. Successful personalized tweet recommendation becomes feasible through the following three contributions: (i) sentiment factors are newly introduced into personalized tweet recommendation, (ii) users' interest is modeled by deriving multimodal FFM that enables collaborative use of multiple factors in a tweet, i.e., publisher, topic and sentiment factors, and (iii) the effectiveness of using sentiment factors as well as publisher and topic factors is clarified from results of experiments using real-world datasets related to worldwide hot topics, "#trump", "#hillaryclinton" and "#ladygaga". In addition to showing the effectiveness of the proposed method, the applicability of the proposed method to other tasks such as advertisement and social analysis is discussed as a conclusion and future work of this paper.

Favorite Video Estimation Based on Multiview Feature Integration via KMvLFDA.
Array,Array,Miki Haseyama
IEEE Access, 6, 63833, 63842, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2018, [Peer-reviewed]
English, Scientific journal, This paper presents a novel method for favorite video estimation based on multiview feature integration via kernel multiview local fisher discriminant analysis (KMvLFDA). The proposed method first extracts electroencephalogram (EEG) features from users' EEG signals recorded while watching videos and multiple visual features from videos. Then, multiple EEG-based visual features are obtained by applying locality preserving canonical correlation analysis to EEG features and each visual feature. Next, KMvLFDA, which is newly derived in this paper, explores the complementary properties of different features and integrates the multiple EEG-based visual features. In addition, by using KMvLFDA, between-class scatter is maximized and within-class scatter is minimized in the integrated feature space. Consequently, it can be expected that the new features that are obtained by the above integration are more effective than each of the EEG-based visual features for the estimation of users' favorite videos. The main contribution of this paper is the new derivation of KMvLFDA. Successful estimation of users' favorite videos becomes feasible by using the new features obtained via KMvLFDA.

Favorite Video Classification Based on Multimodal Bidirectional LSTM.
Array,Array, Keisuke Maeda, Miki Haseyama
IEEE Access, 6, 61401, 61409, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2018, [Peer-reviewed]
English, Scientific journal, Video classification based on the user's preference (information of what a user likes: WUL) is important for realizing human-centered video retrieval. A better understanding of the rationale of WUL would greatly contribute to the support for successful video retrieval. However, a few studies have shown the relationship between information of what a user watches and WUL. A new method that classifies videos on the basis of WUL using video features and electroencephalogram (EEG) signals collaboratively with a multimodal bidirectional Long Short-Term Memory (Bi-LSTM) network is presented in this paper. To the best of our knowledge, there has been no study on WUL-based video classification using video features and EEG signals collaboratively with LSTM. First, we newly apply transfer learning to the WUL-based video classification since the number of labels (liked or not liked) attached to videos by users is small, and it is difficult to classify videos based on WUL. Furthermore, we conduct a user study for showing that the representation of psychophysiological signals calculated from Bi-LSTM is effective for the WUL-based video classification. Experimental results showed that our deep neural network feature representations can distinguish WUL for each subject.

Selection of Significant Brain Regions Based on MvGTDA and TS-DLF for Emotion Estimation.
Array,Array,Miki Haseyama
IEEE Access, 6, 32481, 32492, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2018, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a novel method for estimating human emotion using functional brain images. The final goal of our study is contribution to affective brain computer interfaces (aBCIs), which use neuropsychological signals. In the proposed method, we newly derive multiview general tensor discriminant analysis (MvGTDA) in order to reveal significant brain regions and accurately estimate human emotion evoked by visual stimuli. This is because it is important to find activation of multiple brain regions for estimating emotional states. Since we regard a Brodmann area as a view and introduce -norm regularization for these views, MvGTDA can eliminate non-crucial Brodmann areas and select significant ones. Moreover, in general studies on functional brain images based on machine learning methodologies, there is an overfitting problem caused by a small sample size. Therefore, revealing significant Brodmann areas based on MvGTDA has another important role, i.e., solving the overfitting problem. By inputting estimation results respectively obtained from the significant areas and the MvGTDA-based feature, tensor-based supervised decision-level fusion (TS-DLF) integrates them and outputs the final estimation result of the users emotion. In experiments, we showed the effectiveness of our method by using actual functional brain images and we revealed the significant brain regions in emotional states.

A Language-Independent Ontology Construction Method Using Tagged Images in Folksonomy.
Array,Takahiro Ogawa, Miki Haseyama
IEEE Access, 6, 2930, 2942, 2018, [Peer-reviewed]
Scientific journal

Tracking topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
Multimedia Tools and Applications, 77, 16, 1, 28, Springer New York LLC, 06 Dec. 2017, [Peer-reviewed]
English, Scientific journal, A method to track topic evolution via salient keyword matching with consideration of semantic broadness for Web video discovery is presented in this paper. The proposed method enables users to understand the evolution of topics over time for discovering Web videos in which they are interested. A framework that enables extraction and tracking of the hierarchical structure, which contains Web video groups with various degrees of semantic broadness, is newly derived as follows: Based on network analysis using multimodal features, i.e., features of video contents and metadata, our method extracts the hierarchical structure and salient keywords that represent contents of each Web video group. Moreover, salient keyword matching, which is newly developed by considering salient keyword distribution, semantic broadness of each Web video group and initial topic relevance, is applied to each hierarchical structure obtained in different time stamps. Unlike methods in previous works, by considering the semantic broadness as well as the salient keyword distribution, our method can overcome the problem of the desired semantic broadness of topics being different depending on each user. Also, the initial topic relevance enables correction of the gap from an initial topic at the start of tracking. Consequently, it becomes feasible to track the evolution of topics over time for finding Web videos in which the users are interested. Experimental results for real-world datasets containing YouTube videos verify the effectiveness of the proposed method.

Extracting hierarchical structure of content groups from different social media platforms using multiple social metadata
Daichi Takehara, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
MULTIMEDIA TOOLS AND APPLICATIONS, 76, 19, 20249, 20272, SPRINGER, Oct. 2017, [Peer-reviewed]
English, Scientific journal, A novel scheme for retrieving users' desired contents, i.e., contents with topics in which users are interested, from multiple social media platforms is presented in this paper. In existing retrieval schemes, users first select a particular platform and then input a query into the search engine. If users do not specify suitable platforms for their information needs and do not input suitable queries corresponding to the desired contents, it becomes difficult for users to retrieve the desired contents. The proposed scheme extracts the hierarchical structure of content groups (sets of contents with similar topics) from different social media platforms, and it thus becomes feasible to retrieve desired contents even if users do not specify suitable platforms and do not input suitable queries. This paper has two contributions: (1) A new feature extraction method, Locality Preserving Canonical Correlation Analysis with multiple social metadata (LPCCA-MSM) that can detect content groups without the boundaries of different social media platforms is presented in this paper. LPCCA-MSM uses multiple social metadata as auxiliary information unlike conventional methods that only use content-based information such as textual or visual features. (2) The proposed novel retrieval scheme can realize hierarchical content structuralization from different social media platforms. The extracted hierarchical structure shows various abstraction levels of content groups and their hierarchical relationships, which can help users select topics related to the input query. To the best of our knowledge, an intensive study on such an application has not been conducted; therefore, this paper has strong novelty. To verify the effectiveness of the above contributions, extensive experiments for real-world datasets containing YouTube videos and Wikipedia articles were conducted.

Wiener-Based Inpainting Quality Prediction
Takahiro Ogawa, Akira Tanaka, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 10, 2614, 2626, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Oct. 2017, [Peer-reviewed]
English, Scientific journal, A Wiener-based inpainting quality prediction method is presented in this paper. The proposed method is the first method that can predict inpainting quality both before and after the intensities have become missing even if their inpainting methods are unknown. Thus, when the target image does not include any missing areas, the proposed method estimates the importance of intensities for all pixels, and then we can know which areas should not be removed. Interestingly, since this measure can be also derived in the same manner for its corrupted image already including missing areas, the expected difficulty in reconstruction of these missing pixels is predicted, i.e., we can know which missing areas can be successfully reconstructed. The proposed method focuses on expected errors derived from the Wiener filter, which enables least-squares reconstruction, to predict the inpainting quality. The greatest advantage of the proposed method is that the same inpainting quality prediction scheme can be used in the above two different situations, and their results have common trends. Experimental results show that the inpainting quality predicted by the proposed method can be successfully used as a universal quality measure.

Visualizing Web Images Using Fisher Discriminant Locality Preserving Canonical Correlation Analysis
Kohei Tateno, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 9, 2005, 2016, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Sep. 2017, [Peer-reviewed]
English, Scientific journal, A novel dimensionality reduction method, Fisher Discriminant Locality Preserving Canonical Correlation Analysis (FDLPCCA), for visualizing Web images is presented in this paper. FDLP-CCA can integrate two modalities and discriminate target items in terms of their semantics by considering unique characteristics of the two modalities. In this paper, we focus onWeb images with text uploaded on Social Networking Services for these two modalities. Specifically, text features have high discriminate power in terms of semantics. On the other hand, visual features of images give their perceptual relationships. In order to consider both of the above unique characteristics of these two modalities, FDLPCCA estimates the correlation between the text and visual features with consideration of the cluster structure based on the text features and the local structures based on the visual features. Thus, FDLP-CCA can integrate the different modalities and provide separated manifolds to organize enhanced compactness within each natural cluster.

Biomimetics Image Retrieval Platform
Miki Haseyama, Takahiro Ogawa, Sho Takahashi, Shuhei Nomura, Masatsugu Shimomura
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 8, 1563, 1573, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2017, [Peer-reviewed]
English, Scientific journal, Biomimetics is a new research field that creates innovation through the collaboration of different existing research fields. However, the collaboration, i.e., the exchange of deep knowledge between different research fields, is difficult for several reasons such as differences in technical terms used in different fields. In order to overcome this problem, we have developed a new retrieval platform, "Biomimetics image retrieval platform," using a visualization-based image retrieval technique. A biological database contains a large volume of image data, and by taking advantage of these image data, we are able to overcome limitations of text-only information retrieval. By realizing such a retrieval platform that does not depend on technical terms, individual biological databases of various species can be integrated. This will allow not only the use of data for the study of various species by researchers in different biological fields but also access for a wide range of researchers in fields ranging from materials science, mechanical engineering and manufacturing. Therefore, our platform provides a new path bridging different fields and will contribute to the development of biomimetics since it can overcome the limitation of the traditional retrieval platform.

Deterioration Level Estimation on Transmission Towers via Extreme Learning Machine based on Combination Use of Local Receptive Field and Principal Component Analysis　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), 457, 458, Jul. 2017, [Peer-reviewed]
English, International conference proceedings

Effectiveness Evaluation of Imaging Direction for Estimation of Gastritis Regions on Gastric X-ray Images　　　　　　　　　　　　　　　
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
International Technical Conference on Circuits, Systems, Computers, and Communications (ITC-CSCC), 459, 460, May 2017, [Peer-reviewed]
English, International conference proceedings

Helicobacter Pylori infection detection from gastric X-ray images based on feature fusion and decision fusion
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
COMPUTERS IN BIOLOGY AND MEDICINE, 84, 69, 78, PERGAMON-ELSEVIER SCIENCE LTD, May 2017, [Peer-reviewed]
English, Scientific journal, In this paper, a fully automatic method for detection of Helicobacter pylori (H. pylori) infection is presented with the aim of constructing a computer-aided diagnosis (CAD) system. In order to realize a CAD system with good performance for detection of H. pylori infection, we focus on the following characteristic of stomach X-ray examination. The accuracy of X-ray examination differs depending on the symptom of H. pylori infection that is focused on and the position from which X-ray images are taken. Therefore, doctors have to comprehensively assess the symptoms and positions. In order to introduce the idea of doctors' assessment into the CAD system, we newly propose a method for detection of H. pylori infection based on the combined use of feature fusion and decision fusion. As a feature fusion scheme, we adopt Multiple Kernel Learning (MKL). Since MKL can combine several features with determination of their weights, it can represent the differences in symptoms. By constructing an MKL classifier for each position, we can obtain several detection results. Furthermore, we introduce confidence-based decision fusion, which can consider the relationship between the classifier's performance and the detection results. Consequently, accurate detection of H. pylori infection becomes possible by the proposed method. Experimental results obtained by applying the proposed method to real X-ray images show that our method has good performance, close to the results of detection by specialists, and indicate that the realization of a CAD system for determining the risk of H. pylori infection is possible.

Tracking hierarchical structure of web video groups based on salient keyword matching including semantic broadness estimation
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2016 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2016 - Proceedings, 1238, 1242, Institute of Electrical and Electronics Engineers Inc., 19 Apr. 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method to track the hierarchical structure of Web video groups on the basis of salient keyword matching including semantic broadness estimation. To the best of our knowledge, this paper is the first work to perform extraction and tracking of the hierarchical structure simultaneously. Specifically, the proposed method first extracts the hierarchical structure of Web video groups and salient keywords of them on the basis of an improved scheme of our previously reported method. Moreover, to calculate similarities between Web video groups obtained in different time stamps, salient keyword matching is newly developed by considering both co-occurrences of the salient keywords and semantic broadness of each Web video group. Consequently, tracking of the hierarchical structure over time becomes feasible to easily understand popularity trends of many Web videos for realizing effective retrieval.

Distress Classification of Class Imbalanced Data for Maintenance Inspection of Road Structures in Express Way　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Conference on Civil and Building Engineering Informatics in conjunction with Conference on Computer Applications in Civil and Hydraulic Engineering (ICCBEI & CCACHE), 182, 185, Apr. 2017, [Peer-reviewed]
English, International conference proceedings

Human-Centered Video Feature Selection via mRMR-SCMMCCA for Preference Extraction
Takahiro Ogawa, Yoshiaki Yamaguchi, Satoshi Asamizu, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E100D, 2, 409, 412, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Feb. 2017, [Peer-reviewed]
English, Scientific journal, This paper presents human-centered video feature selection via mRMR-SCMMCCA (minimum Redundancy and Maximum Relevance-Specific Correlation Maximization Multiset Canonical Correlation Analysis) algorithm for preference extraction. The proposed method derives SCMMCCA, which simultaneously maximizes two kinds of correlations, correlation between video features and users' viewing behavior features and correlation between video features and their corresponding rating scores. By monitoring the derived correlations, the selection of the optimal video features that represent users' individual preference becomes feasible.

PERSONALIZED VIDEO PREFERENCE ESTIMATION BASED ON EARLY FUSION USING MULTIPLE USERS'VIEWING BEHAVIOR
Yoshiki Ito, Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 3006, 3010, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for personalized video preference estimation based on early fusion using multiple users' viewing behavior. The proposed method adopts supervised Multi-View Canonical Correlation Analysis (sMVCCA) to estimate correlation between different types of features. Specifically, we estimate optimal projections maximizing the correlation between three features of video, target user's viewing behavior and evaluation scores for video. Then novel video features (canonical video features), which reflect the target user's individual preference, are obtained by the estimated projections. Furthermore, our method computes sMVCCA-based canonical video features by using multiple users' viewing behavior and a target user's evaluation scores. This non-conventional approach using the multiple users' viewing behavior for the preference estimation of the target user is the biggest contribution of our method, and it enables early fusion of the canonical video features. Consequently, successful video recommendation that reflects the users' individual preference can be expected via the evaluation score prediction from the integrated canonical video features. Experimental results show the effectiveness of our method.

EXEMPLAR-BASED IMAGE COMPLETION VIA NEW QUALITY MEASURE BASED ON PHASELESS TEXTURE FEATURES
Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 1827, 1831, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents an exemplar-based image completion via a new quality measure based on phaseless texture features. The proposed method derives a new quality measure obtained by monitoring errors caused in power spectra, i.e., errors of phaseless texture features, converged through phase retrieval. Even if a target patch includes missing pixels, this measure enables selection of the best matched patch including the most similar texture features for realizing the exemplar-based image completion. Furthermore, since the phaseless texture features are robust to various changes such as spatial gaps and luminance changes, the new quality measure successfully provides the best matched patch from few training examples. Then, by solving an optimization problem that retrieves the phase of the target patch from the phaseless texture features of the best matched patch, its missing areas can be reconstructed. Consequently, accurate image completion using the new quality measure becomes feasible. Subjective and quantitative experimental results are shown to verify the effectiveness of our method using the new quality measure.

EMOTION ESTIMATION VIA TENSOR-BASED SUPERVISED DECISION-LEVEL FUSION FROM MULTIPLE BRODMANN AREAS
Kento Sugata, Takahiro Ogawa, Miki Haseyama
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 999, 1003, IEEE, 2017, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method that estimates human emotion based on tensor-based supervised decision-level fusion (TS-DLF) from multiple Brodmann areas (BAs). From multiple brain data corresponding to these BAs captured by functional magnetic resonance imaging (fMRI), our method performs general tensor discriminant analysis (GTDA) to obtain features which can reflect the user's emotion. Furthermore, since the dimension of the obtained features becomes lower, this can avoid overfitting in the following training procedure of estimators. Next, by separately using the transformed BA data obtained after GTDA, we obtain multiple estimation results of the user's emotion based on logistic tensor regression (LTR). Then our method realizes the decision of the final result based on TS-DLF from the multiple estimation results. This approach, i.e., the integration of the multiple BAs' results for the whole-brain data, is the biggest contribution of this paper. TS-DLF successfully integrates the multiple estimation results with considering the performance of the LTR-based estimator constructed for each BA. Experimental results show that our method outperforms state-of-the-art approaches, and the effectiveness of our method can be confirmed.

Extracting Hierarchical Structure of Web Video Groups Based on Sentiment-Aware Signed Network Analysis
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
IEEE ACCESS, 5, 16963, 16973, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2017, [Peer-reviewed]
English, Scientific journal, Sentiment in multimedia contents has an influence on their topics, since multimedia contents are tools for social media users to convey their sentiment. Performance of applications such as retrieval and recommendation will be improved if sentiment in multimedia contents can be estimated; however, there have been few works in which such applications were realized by utilizing sentiment analysis. In this paper, a novel method for extracting the hierarchical structure of Web video groups based on sentiment-aware signed network analysis is presented to realize Web video retrieval. First, the proposed method estimates latent links between Web videos by using multimodalfeatures of contents and sentiment features obtained from texts attached to Web videos. Thus, our method enables construction of a signed network that reflects not only similarities but also positive and negative relations between topics of Web videos. Moreover, an algorithm to optimize a modularity-based measure, which can adaptively adjust the balance between positive and negative edges, was newly developed. This algorithm detects Web video groups with similar topics at multiple abstraction levels; thus, successful extraction of the hierarchical structure becomes feasible. By providing the hierarchical structure, users can obtain an overview of many Web videos and it becomes feasible to successfully retrieve the desired Web videos. Results of experiments using a new benchmark dataset, YouTube-8M, validate the contributions of this paper, i.e., 1) the first attempt to utilize sentiment analysis for Web video grouping and 2) a novel algorithm for analyzing a weighted signed network derived from sentiment and multimodal features.

Retrieval of similar inspection records based on metric learning using experienced inspectors' evaluation
Ryota Saito, Sho Takahashi, Takahiro Ogawa, Miki Hasayama
2016 IEEE 5th Global Conference on Consumer Electronics, GCCE 2016, 1, 2, Institute of Electrical and Electronics Engineers Inc., 27 Dec. 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a retrieval method of similar inspection records in road structures based on metric learning using experienced inspectors' evaluation. Inspection records of road structures include images and text-based information such as category of distress, damaged parts and degree of damage. The proposed method calculates distances from query inspection records, and rank lists of retrieval results are obtained for each feature. In this approach, the distance quantification are updated on the basis of experienced inspectors' evaluation. Finally, the proposed method obtains retrieval results by integrating the multiple rank lists. The experimental results show the effectiveness of the proposed method.

Adaptive Subspace-Based Inverse Projections via Division Into Multiple Sub-Problems for Missing Image Data Restoration
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON IMAGE PROCESSING, 25, 12, 5971, 5986, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Dec. 2016, [Peer-reviewed]
English, Scientific journal, This paper presents adaptive subspace-based inverse projections via division into multiple sub-problems (ASIP-DIMSs) for missing image data restoration. In the proposed method, a target problem for estimating missing image data is divided into multiple sub-problems, and each sub-problem is iteratively solved with the constraints of other known image data. By projection into a subspace model of image patches, the solution of each sub-problem is calculated, where we call this procedure "subspace-based inverse projection" for simplicity. The proposed method can use higher dimensional subspaces for finding unique solutions in each sub-problem, and successful restoration becomes feasible, since a high level of image representation performance can be preserved. This is the biggest contribution of this paper. Furthermore, the proposed method generates several subspaces from known training examples and enables derivation of a new criterion in the above framework to adaptively select the optimal subspace for each target patch. In this way, the proposed method realizes missing image data restoration using ASIP-DIMS. Since our method can estimate any kind of missing image data, its potential in two image restoration tasks, image inpainting and super-resolution, based on several methods for multivariate analysis is also shown in this paper.

A Web video retrieval method using hierarchical structure of Web video groups
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
MULTIMEDIA TOOLS AND APPLICATIONS, 75, 24, 17059, 17079, SPRINGER, Dec. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a Web video retrieval method that uses hierarchical structure of Web video groups. Existing retrieval systems require users to input suitable queries that identify the desired contents in order to accurately retrieve Web videos; however, the proposed method enables retrieval of the desired Web videos even if users cannot input the suitable queries. Specifically, we first select representative Web videos from a target video dataset by using link relationships between Web videos obtained via metadata "related videos" and heterogeneous video features. Furthermore, by using the representative Web videos, we construct a network whose nodes and edges respectively correspond to Web videos and links between these Web videos. Then Web video groups, i.e., Web video sets with similar topics are hierarchically extracted based on strongly connected components, edge betweenness and modularity. By exhibiting the obtained hierarchical structure of Web video groups, users can easily grasp the overview of many Web videos. Consequently, even if users cannot write suitable queries that identify the desired contents, it becomes feasible to accurately retrieve the desired Web videos by selecting Web video groups according to the hierarchical structure. Experimental results on actual Web videos verify the effectiveness of our method.

Performance Improvement of Error-Resilient 3D DWT Video Transmission Using Invertible Codes
Kotoku Omura, Shoichiro Yamasaki, Tomoko K. Matsushima, Hirokazu Tanaka, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E99A, 12, 2256, 2265, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Dec. 2016, [Peer-reviewed]
English, Scientific journal, Many studies have applied the three-dimensional discrete wavelet transform (3D DWT) to video coding. It is known that corruptions of the lowest frequency sub-band (LL) coefficients of 3D DWT severely affect the visual quality of video. Recently, we proposed an error resilient 3D DWT video coding method (the conventional method) that employs dispersive grouping and an error concealment (EC). The EC scheme of our conventional method adopts a replacement technique of the lost LL coefficients. In this paper, we propose a new 3D DWT video transmission method in order to enhance error resilience. The proposed method adopts an error correction scheme using invertible codes to protect LL coefficients. We use half-rate Reed-Solomon (RS) codes as invertible codes. Additionally, to improve performance by using the effect of interleave, we adopt a new configuration scheme at the RS encoding stage. The evaluation by computer simulation compares the performance of the proposed method with that of other EC methods, and indicates the advantage of the proposed method.

Classifying Insects from SEM Images Based on Optimal Classifier Selection and D-S Evidence Theory
Takahiro Ogawa, Akihiro Takahashi, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E99A, 11, 1971, 1980, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, an insect classification method using scanning electron microphotographs is presented. Images taken by a scanning electron microscope (SEM) have a unique problem for classification in that visual features differ from each other by magnifications. Therefore, direct use of conventional methods results in inaccurate classification results. In order to successfully classify these images, the proposed method generates an optimal training dataset for constructing a classifier for each magnification. Then our method classifies images using the classifiers constructed by the optimal training dataset. In addition, several images are generally taken by an SEM with different magnifications from the same insect. Therefore, more accurate classification can be expected by integrating the results from the same insect based on Dempster-Shafer evidence theory. In this way, accurate insect classification can be realized by our method. At the end of this paper, we show experimental results to confirm the effectiveness of the proposed method.

Estimation of salient regions related to chronic gastritis using gastric X-ray images
Ren Togo, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
COMPUTERS IN BIOLOGY AND MEDICINE, 77, 9, 15, PERGAMON-ELSEVIER SCIENCE LTD, Oct. 2016, [Peer-reviewed]
English, Scientific journal, Since technical knowledge and a high degree of experience are necessary for diagnosis of chronic gastritis, computer-aided diagnosis (CAD) systems that analyze gastric X-ray images are desirable in the field of medicine. Therefore, a new method that estimates salient regions related to chronic gastritis/non-gastritis for supporting diagnosis is presented in this paper. In order to estimate salient regions related to chronic gastritis/non-gastritis, the proposed method monitors the distance between a target image feature and Support Vector Machine (SVM)-based hyperplane for its classification. Furthermore, our method realizes removal of the influence of regions outside the stomach by using positional relationships between the stomach and other organs. Consequently, since the proposed method successfully estimates salient regions of gastric X-ray images for which chronic gastritis and non-gastritis are unknown, visual support for inexperienced clinicians becomes feasible. (C) 2016 Elsevier Ltd. All rights reserved.

Category Classification of Tourism Images in Image Sharing Services
SAITO Naoki, OGAWA Takahiro, ASAMIZU Satoshi, HASEYAMA Miki
電子情報通信学会論文誌D 情報・システム, J99-D, 9, 848, 860, The Institute of Electronics, Information and Communication Engineers, 01 Sep. 2016, [Peer-reviewed]
Japanese, In this paper, we propose a category classification method of tourism images in image sharing services. First, the proposed method performs classification of tourism images by applying a fuzzy K-nearest neighbor algorithm to the location feature vectors and newly estimates their confidence measures. Second, if the confidence measures of the classification results are lower, the proposed method performs decision level fusion of related visual and textual tag feature vectors to obtain final classification results. Consequently, the proposed method enables successful integration of multiple classification results obtained from the several kinds of features based on the above two stage classification scheme.

NMF-Based Spectral Reflectance Estimation From Image Pairs Including Near-Infrared Components
Takahiro Ogawa, Yuta Igarashi, Miki Haseyama
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 26, 5, 855, 867, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, May 2016, [Peer-reviewed]
English, Scientific journal, In this paper, a novel spectral reflectance estimation method from image pairs including near-infrared (NIR) components based on nonnegative matrix factorization (NMF) is presented. The proposed method enables estimation of spectral reflectance from only two kinds of input images: 1) an image including both visible light components and NIR components and 2) an image including only NIR components. These two images can be easily obtained using a general digital camera without an infrared-cut filter and one with a visible light-cut filter, respectively. Since RGB values of these images are obtained according to spectral sensitivity of the image sensor, the spectrum power distribution of the light source and the spectral reflectance, we have to solve the inverse problem for estimating the spectral reflectance. Therefore, our method approximates spectral reflectance by a linear combination of several bases obtained by applying NMF to a known spectral reflectance data set. Then estimation of the optimal solution to the above problem becomes feasible based on this approximation. In the proposed method, NMF is used for obtaining the bases used in this approximation from a characteristic that the spectral reflectance is a nonnegative component. Furthermore, the proposed method realizes simple approximation of the spectrum power distribution of the light source with direct and scattered light components. Therefore, estimation of spectral reflectance becomes feasible using the spectrum power distribution of the light source in our method. In the last part of this paper, we show some simulation results to verify the performance of the proposed method. The effectiveness of the proposed method is also shown using the method for several applications that are closely related to spectral reflectance estimation. Although our method is based on a simple scheme, it is the first method that realizes the estimation of the spectral reflectance and the spectrum power distribution of the light source from the above two kinds of images taken by general digital cameras and provides breakthroughs to several fundamental applications.

A Most Resource-Consuming Disease Estimation Method from Electronic Claim Data Based on Labeled LDA
Yasutaka Hatakeyama, Takahiro Ogawa, Hironori Ikeda, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E99D, 3, 763, 768, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Mar. 2016, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a method to estimate the most resource-consuming disease from electronic claim data based on Labeled Latent Dirichlet Allocation (Labeled LDA). The proposed method models each electronic claim from its medical procedures as a mixture of resource-consuming diseases. Thus, the most resource-consuming disease can be automatically estimated by applying Labeled LDA to the electronic claim data. Although our method is composed of a simple scheme, this is the first trial for realizing estimation of the most resource-consuming disease.

Distress Classification of Road Structures via Multiple Classifier-based Bayesian Network　　　　　　　　　　　　　　　
K. Maeda, S. Takahashi, T. Ogawa, M. Haseyama
International Workshop on Advanced Image Technology (IWAIT), 1, 4, 2016, [Peer-reviewed]
English, International conference proceedings

6. Visualization Methods for Encourage Experience of Sports in Soccer Videos
Takahashi Sho, Haseyama Miki
The Journal of The Institute of Image Information and Television Engineers, 70, 9, 722, 724, 一般社団法人 映像情報メディア学会, 2016
Japanese

Multimodal Interest Level Estimation via Variational Bayesian Mixture of Robust CCA
Yuma Sasaka, Takhiro Ogawa, Miki Haseyama
MM'16: PROCEEDINGS OF THE 2016 ACM MULTIMEDIA CONFERENCE, 387, 391, ASSOC COMPUTING MACHINERY, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method which estimates interest level while watching videos, based on collaborative use of facial expression and biological signals such as electroencephalogram (EEG) and electrocardiogram (ECG). To the best of our knowledge, no studies have been carried out on the collaborative use of facial expression and biological signals for estimating interest level. Since training data, which is used for estimating interest level, is generally small and imbalanced, Variational Bayesian Mixture of Robust Canonical Correlation Analysis (VBMRCCA) is newly applied to facial expression and biological signals, which are obtained from users while they are watching the videos. Unlike some related works, VBMRCCA is used to obtain the posterior distributions which represent the latent correlation between facial expression and biological signals in our method. Then, the users' interest level can be estimated by comparing the posterior distributions of the positive class data with those of the negative. Consequently, successful interest level estimation, via collaborative use of facial expression and biological signals, becomes feasible.

Realization of Associative Image Search Development of Image Retrieval Platform for Enhancing Serendipity
Miki Haseyama
2016 IEEE 46TH INTERNATIONAL SYMPOSIUM ON MULTIPLE-VALUED LOGIC (ISMVL 2016), 56, 59, IEEE COMPUTER SOC, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents "Associative Image Search", a new image retrieval scheme and its specific engineering application, which enable value creation from big data. The main aim of the associative image search is the realization of information retrieval that enhances the potential for serendipities by providing users with new awareness. Thus, this paper presents the details of research for realizing associative image retrieval. Furthermore, as an example of its applications, a Biomimetics image retrieval platform is also introduced in this paper. By associatively and collaboratively using data accumulated in the fields of biology and material science, the Biomimetics image retrieval platform enables acceleration of their knowledge sharing in different research fields. From retrieval results actually obtained from this platform, there is discussion of the potential of serendipities such as new knowledge emergence

A Virtual Vital Signs Sensor "MIRUWS" for Visualization of Healthy to Illness Transition (HIT)
Shigenobu Minami, Miki Haseyama, Hirokazu Tanaka, Toru Takahashi, Tatsuya Komori
2016 10TH INTERNATIONAL SYMPOSIUM ON MEDICAL INFORMATION AND COMMUNICATION TECHNOLOGY (ISMICT), 1, 5, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper proposes a virtual Vital Signs Sensor (VSS) for visualization of half illness (so called "Mibyou") and sudden illness. Since both half and sudden illnesses, which are located in between wellness and illness, are categorized in healthy(H) to illness(I) transition(T) status. So, we name this status as HIT. Since HIT happens in an ordinary life, VSS for HIT visualization should work for 24 hours continuously and unconsciously.
The proposed virtual VSS is named as "MIRUWS" which means (M)multi-sensing, (I)integrated, (R)reliable, and (U)unconscious (V)virtual (V)vital (S) signs sensor. All these key words are meaningful and required for professional HIT (Pro-HIT) visualization which has real demand rather than purely personal one.
MIRUWS is a virtual VSS in a cyber space targeting this Pro-HIT visualization, and is a projection of actual VSSs in a physical space. There are wide varieties of physical sensors such as patch, touch, proximity, and remote types. To handle these wide varieties of VSSs efficiently and consistently, MIRUWS plays as a common and unique virtual VSS in a cyber space.
Unlikely to medical devices which need very high reliability rather than flexibility, MIRUWS is needed to satisfy both reliability and flexibility at the same time to cover wide range of Pro-HIT use-cases. To realize this, MIRUWS visualizes physical VSS's specifications throughout new API, which are dynamically determined by connected and released physical VSSs in a BAN.
To certify MIRUWS performance objectively, conformance testing is desired. This paper also presents MIRUWS test environment example, which measures basic vital signs performances, heart rate variation (HRV) and pulse wave transition time (PWTT) using two target physical VSSs at once.

An Accurate Mortality Prediction Method Based on Decision-level Fusion of Existing ICU Scoring Systems
Yasutaka Hatakeyama, Takahiro Ogawa, Hirokazu Tanaka, Miki Haseyama
PROCEEDINGS OF 2016 INTERNATIONAL SYMPOSIUM ON INFORMATION THEORY AND ITS APPLICATIONS (ISITA 2016), 126, 130, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose a mortality prediction method based on decision-level fusion (DLF) of existing intensive unit care (ICU) scoring systems. First, the proposed method obtains severity scores from the existing ICU scoring systems. Furthermore, we construct classifiers that categorize patients into survivors or non-survivors. Next, patient feature vectors are extracted based on the mortality rates that are estimated from the obtained severity scores by using a non-linear least squares method to obtain other types of classification results. In order to obtain the final severity score for each patient, we integrate the obtained multiple classification results based on DLF that can estimate the final severity scores. Finally, we performed the proposed method to actual ICU patient data and verified the effectiveness of the proposed method. Thus, the proposed method can realize accurate mortality prediction without any additional work by using the existing ICU scoring systems.

GRAPH-BASED WEB VIDEO SEARCH RERANKING THROUGH CONSISTENCY ANALYSIS USING SPECTRAL CLUSTERING
Soh Yoshida, Takahiro Ogawa, Miki Haseyaina
2016 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA & EXPO (ICME), 1, 6, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper proposes a graph-based Web video search reranking method through consistency analysis using spectral clustering. Graph-based reranking is effective for refining text-based video search results. Generally, this approach constructs a graph where the vertices are videos and the edges reflect their pairwise similarities. A lot of reranking methods are built based on a scheme which regularizes the smoothness of pairwise ranking scores between adjacent nodes. However, since the overall consistency is measured by aggregating the individual consistency over each pair, errors in score estimation increase when noisy samples are included within their neighbors. To deal with the noisy samples, different from the conventional methods, the proposed method models the global consistency of the graph structure. Specifically, in order to detect this consistency, the propose method introduces a spectral clustering algorithm which can detect video groups, whose videos have strong semantic correlation, on the graph. Furthermore, a new regularization term, which smooths ranking scores within the same group, is introduced to the reranking framework. Since score regularization is performed by both local and global aspect simultaneously, the accurate score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

HIERARCHICAL CONTENT GROUP DETECTION FROM DIFFERENT SOCIAL MEDIA PLATFORMS USING WEB LINK STRUCTURE
Daichi Takehara, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 479, 483, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for hierarchical content group detection from different social media platforms, which can reveal hierarchical structure of content groups. In this paper, content groups are defined as sets of contents with similar topics. Based on the revealed hierarchical structure, our method enables users to efficiently find the desired contents from large amount of contents placed in diversified social media platforms. The main contributions of this paper are twofold. First, effective latent features for comparing the contents placed in different social media platforms can be extracted by the combination use of the correlation between features obtained from different social media platform and the Web link structure. Second, the hierarchical structure of the content groups, which captures their various abstraction levels, can be revealed by hierarchically detecting their content groups. Experimental results on the real-world dataset containing YouTube videos and Wikipedia articles show the effectiveness of our method.

Distress Classification of Road Structures via Decision Level Fusion
Keisuke Maeda, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 589, 593, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, A distress classification method of road structures via decision level fusion is presented in this paper. In order to classify various kinds of distresses accurately, the proposed method integrates multiple classification results with considering their performance, and this is the biggest contribution of this paper. By introducing this approach, it becomes feasible to adaptively integrate the multiple classification results based on the accuracy of each classifier for a target sample. Consequently, realization of the accurate distress classification can be expected. Experimental results show that our method outperforms existing methods.

Improvement of video coding efficiency based on sparse contractive mapping approach
Zaixing He, Takahiro Ogawa, Sho Takahashi, Miki Haseyama, Xinyue Zhao
NEUROCOMPUTING, 173, 1898, 1907, ELSEVIER SCIENCE BV, Jan. 2016, [Peer-reviewed]
English, Scientific journal, This paper presents a new method for improving video coding efficiency based on a sparse contractive mapping approach. The proposed method introduces a new sparse contractive mapping approach to replace the traditional intra prediction in the video coding standards such as H.264/AVC. Specifically, the intra- and its following inter-frame are respectively approximated by the sparse representation, satisfying contractive mapping. Then these two frames can be reconstructed from an arbitraryinitial image by utilizing a few representation coefficients. With this advantage, the proposed method reduces the total amount of bits by removing MBs in the target I frame, whose approximation performance is higher than the others in the encoder. Furthermore, by transmitting the representation coefficients of the removed MBs, these MBs can be accurately reconstructed in the decoder. Since the reconstruction performance is better than that of the conventional approach, the proposed method can remove more MBs from the target video sequences, and reduction of total amount of bits can be feasible. Therefore, the proposed method realizes the improvement of the video coding efficiency. Some experimental results are shown to verify the superior performance of the proposed method to that of H.264/AVC. The results also demonstrate that the bit-saving performance of the proposed method is comparable to that of H.2651 HEVC. (C) 2015 Elsevier B.V. All rights reserved.

NOVEL FAVORITE MUSIC CLASSIFICATION USING EEG-BASED OPTIMAL AUDIO FEATURES SELECTED VIA KDLPCCA
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
2016 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING PROCEEDINGS, 759, 763, IEEE, 2016, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method of favorite music classification using EEG-based optimal audio features. To select audio features related to user's music preference, our method utilizes a relationship between EEG features obtained from the user's EEG signals during listening to music and their corresponding audio features since EEG signals of human reflect his/her music preference. Specifically, cross-loadings, whose components denote the degree of the relationship, are calculated based on Kernel Discriminative Locality Preserving Canonical Correlation Analysis (KDLPCCA) which is newly derived in the proposed method. In contrast with standard CCA, KDLPCCA can consider (1) non-linear correlation, (2) class information and (3) local structures of input EEG and audio features, simultaneously. Therefore, KDLPCCA-based cross-loadings can reflect best correlation between the user's EEG and corresponding audio signals. Then an optimal set of audio features related to his/her music preference can be obtained by employing the cross-loadings as novel criteria for feature selection. Consequently, our method realizes favorite music classification successfully by using the EEG-based optimal audio features.

Bregman pooling: Feature-space local pooling for image classification
Alameen Najjar, Takahiro Ogawa, Miki Haseyama
International Journal of Multimedia Information Retrieval, 4, 4, 247, 259, Springer London, 01 Dec. 2015, [Peer-reviewed]
English, Scientific journal, In this paper, we propose a novel feature-space local pooling method for the commonly adopted architecture of image classification. While existingmethods partition the feature space based on visual appearance to obtain pooling bins, learning more accurate space partitioning that takes semantics into account boosts performance even for a smaller number of bins. To this end, we propose partitioning the feature space over clusters of visual prototypes common to semantically similar images (i.e., images belonging to the same category). The clusters are obtained by Bregman co-clustering applied offline on a subset of training data. Therefore, being aware of the semantic context of the input image, our features have higher discriminative power than do those pooled from appearance-based partitioning. Testing on four datasets (Caltech-101, Caltech-256, 15 Scenes, and 17 Flowers) belonging to three different classification tasks showed that the proposed method outperforms methods in previous works on local pooling in he feature space for less feature dimensionality. Moreover, when implemented within a spatial pyramid, our method achieves comparable results on three of the datasets used.

Perceptually Optimized Missing Texture Reconstruction via Neighboring Embedding
Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E98A, 8, 1709, 1717, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2015, [Peer-reviewed]
English, Scientific journal, Perceptually optimized missing texture reconstruction via neighboring embedding (NE) is presented in this paper. The proposed method adopts the structural similarity (SSIM) index as a measure for representing texture reconstruction performance of missing areas. This provides a solution to the problem of previously reported methods not being able to perform perceptually optimized reconstruction. Furthermore, in the proposed method, a new scheme for selection of the known nearest neighbor patches for reconstruction of target patches including missing areas is introduced. Specifically, by monitoring the SSIM index observed by the proposed NE-based reconstruction algorithm, selection of known patches optimal for the reconstruction becomes feasible even if target patches include missing pixels. The above novel approaches enable successful reconstruction of missing areas. Experimental results show improvement of the proposed method over previously reported methods.

本格化するバイオミメティクス材料の開発〔解説‐2〕バイオミメティクス・画像検索システムの実現と活用
HASEYAMA MIKI
工業材料, 63, 8, 29, 33, 01 Aug. 2015
Japanese

A Biological Classification Method Using Scanning Electron Mircroscope Images via Taxonomy-Based Decision Tree
Piao Jun, Ogawa Takahiro, Haseyama Miki
電子情報通信学会論文誌 D, 情報・システム, 98, 5, 823, 834, 電子情報通信学会, 01 May 2015
Japanese, 本論文では，走査型電子顕微鏡(Scanning Electron Microscop，SEM)で撮像された画像を用いた生物の分類法を提案する．提案手法では，生物学者が構築した分類体系に注目し，各ノードにその下位のノードへの分類を行う分類器を割り当てることで，決定木を構築する．これにより，構築された決定木を用いて生物の分類が可能となる．このように，生物の分類に有用な分類体系の構造を導入することで，画像特徴のみに注目する生物の分類法と比較して，高精度な分類が期待できる．また，提案手法では，更なる精度向上のため，以下の二つの処理も導入する．まず，分類体系において，画像特徴が類似する異なるノードに注目した決定木の変更を行い，誤分類を抑制する．次に，同種の生物の異なる撮像倍率の画像に対し，モーフィングを施すことで学習データの充足を行う．以上によって，提案手法では，SEMで撮像された生物の高精度な分類が可能となる．本論文の最後では，提案手法の有効性を確認するための実験結果を示す．

ものづくりの発想を支援する―バイオミメティクス・画像検索基盤―
HASEYAMA MIKI, HASEYAMA MIKI
現代化学, 529, 31, 34, 01 Apr. 2015
Japanese

Biomimetics Image Retrieval Platform for Enhancing Serendipity
HASEYAMA MIKI
タクサ, 38, 22, 25, Japanese Society of Systematic Zoology, 28 Feb. 2015
Japanese, Biomimetics is a new research area that creates innovation through the collaboration of different existing research fields. Since biomimetics brings together expert researchers with deep knowledge of various research fields, there is a need to facilitate the mutual exchange of that knowledge in order to create new research areas. However, this exchange is difficult due to several reasons, e.g., differences in technical terms between different fields. In order to overcome this problem, we started the development of a new data retrieval platform based on the theory of associative image retrieval. A biological database contains many image data, and by taking advantage of these image data, we are able to overcome limitations of text-only information retrieval. If the development of such a retrieval platform that does not depend on text data can be realized, individual biological databases of various species (insects, fish, etc.) will be integrated. This will allow not only the use for the study of the various species by researchers in different biological fields, but also access for a wide range of researchers in fields ranging from materials science, mechanical engineering and manufacturing.

「バイオミメティクス」日本企業の研究開発の現況:「生物・博物・情報」分野 バイオミメティクス・画像検索システム―生物の微細構造から材料開発の発想を支援する―
長谷山美紀
OHM, 102, 1, 26, 27, 12 Jan. 2015
Japanese

Automatic Martian Dust Storm Detection from Multiple Wavelength Data Based on Decision Level Fusion
Maeda Keisuke, Ogawa Takahiro, Haseyama Miki
IMT, 10, 3, 473, 477, Information and Media Technologies Editorial Board, 2015
English, This paper presents automatic Martian dust storm detection from multiple wavelength data based on decision level fusion. In our proposed method, visual features are first extracted from multiple wavelength data, and optimal features are selected for Martian dust storm detection based on the minimal-Redundancy-Maximal-Relevance algorithm. Second, the selected visual features are used to train the Support Vector Machine classifiers that are constructed on each data. Furthermore, as a main contribution of this paper, the proposed method integrates the multiple detection results obtained from heterogeneous data based on decision level fusion, while considering each classifiers detection performance to obtain accurate final detection results. Consequently, the proposed method realizes successful Martian dust storm detection.

NOVEL IMAGE CLASSIFICATION BASED ON INTEGRATION OF EEG AND VISUAL FEATURES VIA MSLPCCA
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 957, 961, IEEE, 2015
English, International conference proceedings, This paper presents a novel image classification method based on integration of EEG and visual features. In the proposed method, we obtain classification results by separately using EEG and visual features. Furthermore, we merge the above classification results based on a kernelized version of Supervised learning from multiple experts and obtain the final classification result. In order to generate feature vectors used for the final image classification, we apply Multiset supervised locality preserving canonical correlation analysis (MSLPCCA), which is newly derived in the proposed method, to EEG and visual features. Our method realizes successful multimodal classification of images by the object categories that they contain based on MSLPCCA-based feature integration.

MISSING INTENSITY RESTORATION VIA ADAPTIVE SELECTION OF PERCEPTUALLY OPTIMIZED SUBSPACES
Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 1628, 1632, IEEE, 2015
English, International conference proceedings, A missing intensity restoration method via adaptive selection of perceptually optimized subspaces is presented in this paper. In order to realize adaptive and perceptually optimized restoration, the proposed method generates several subspaces of known textures optimized in terms of the structural similarity (SSIM) index. Furthermore, the SSIM-based missing intensity restoration is performed by a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Furthermore, the selection of the optimal subspace is realized by monitoring the SSIM index converged in the POCS algorithm, and the adaptive restoration becomes feasible. Experimental results show that our method outperforms existing methods.

Heterogeneous Graph-based Video Search Reranking using Web Knowledge via Social Media Network
Soh Yoshida, Takahiro Ogawa, Miki Haseyama
MM'15: PROCEEDINGS OF THE 2015 ACM MULTIMEDIA CONFERENCE, 871, 874, ASSOC COMPUTING MACHINERY, 2015, [Peer-reviewed]
English, International conference proceedings, Graph-based reranking is effective for refining text-based video search results by making use of the social network structure. Unlike previous works which only focus on an individual video graph, the proposed method leverages the mutual reinforcement of heterogeneous graphs, such as videos and their associated tags obtained by social influence mining. Specifically, propagation of information relevancy across different modalities is performed by exchanging information of inter- and intra-relations among heterogeneous graphs. The proposed method then formulates the video search reranking as an optimization problem from the perspective of Bayesian framework. Furthermore, in order to model the consistency over the modified video graph topology, a local learning regularization with a social community detection scheme is introduced to the framework. Since videos within the same social community have strong semantic correlation, the consistency score estimation becomes feasible. Experimental results obtained by applying the proposed method to a real-world video collection show its effectiveness.

HELICOBACTER PYLORI INFECTION DETECTION FROM MULTIPLE X-RAY IMAGES BASED ON COMBINATION USE OF SUPPORT VECTOR MACHINE AND MULTIPLE KERNEL LEARNING
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 4728, 4732, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a detection method of Helicobacter pylori (H. pylori) infection from multiple gastric X-ray images based on combination use of Support Vector Machine (SVM) and Multiple Kernel Learning (MKL). The proposed method firstly computes some types of visual features from multiple gastric X-ray images taken in several specific directions in order to represent the characteristics of X-ray images with H. pylori infection. Second, based on the minimal-Redundancy-Maximal-Relevance algorithm, we select the effective features for H. pylori infection detection from each type of visual feature and all visual features. The selected features are used to train the SVM classifier and the MKL classifier for each direction of gastric X-ray images. Finally, the proposed method integrates multiple detection results based on a late fusion scheme considering the detection performance of each classifier. Experimental results obtained by applying the proposed method to real X-ray images prove its effectiveness.

AUTOMATIC DETECTION OF MARTIAN DUST STORMS FROM HETEROGENEOUS DATA BASED ON DECISION LEVEL FUSION
Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2246, 2250, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents automatic detection of Martian dust storms from heterogeneous data (raw data, reflectance data and background subtraction data of the reflectance data) based on decision level fusion. Specifically, the proposed method first extracts image features from these data and selects optimal features for dust storm detection based on the minimal-Redundancy-Maximal-Relevance algorithm. Second, the selected image features are used to train the Support Vector Machine classifier that is constructed on each data. Furthermore, as a main contribution of this paper, the proposed method combines the multiple detection results obtained from the heterogeneous data based on decision level fusion with considering each classifier's detection performance to obtain accurate final detection results. Consequently, the proposed method realizes automatic and accurate detection of Martian dust storms.

EXTRACTION OF HIERARCHICAL STRUCTURE OF WEB COMMUNITIES INCLUDING SALIENT KEYWORD ESTIMATION FOR WEB VIDEO RETRIEVAL
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1021, 1025, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, In this paper, we present a method for extraction of hierarchical structure of Web communities including salient keyword estimation for Web video retrieval. The following two contributions of the proposed method enable retrieval of the desired Web videos even if users cannot input suitable queries that identify the desired contents. First, our method realizes the extraction of hierarchical structure of Web communities, i.e., Web video sets with similar topics by using heterogeneous features of Web videos and link relationships between Web videos obtained via metadata "related videos". Second, we can estimate salient keywords to identify the contents of each obtained Web community at a glance based on text attached to Web videos such as title, the heterogeneous features of Web videos and the link relationships between Web videos. Experimental results on actual Web videos verify that our method can realize accurate retrieval of the desired Web videos via the hierarchical structure of Web communities with their salient keywords.

Human-centered Favorite Music Estimation: EEG-based Extraction of Audio Features Reflecting Individual Preference
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 818, 822, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a human-centered method for favorite music estimation using EEG-based audio features. In order to estimate user's favorite musical pieces, our method utilizes his/her EEG signals for calculating new audio features suitable for representing the user's music preference. Specifically, projection, which transforms original audio features into the features reflecting the preference, is calculated by applying kernel Canonical Correlation Analysis (CCA) to the audio features and the EEG features which are extracted from the user's EEG signals during listening to favorite musical pieces. By using the obtained projection, the new EEG-based audio features can be derived since this projection provides the best correlation between the user's EEG signals and their corresponding audio signals. Thus, successful estimation of user's favorite musical pieces via a Support Vector Machine (SVM) classifier using the new audio features becomes feasible. Since our method does not need acquisition of EEG signals for obtaining new audio features from new musical pieces after calculating the projection, this indicates the high practicability of our method. Experimental results show that our method outperforms methods using original audio features or EEG features.

Algorithm for Sparse Representation Minimizing Mean Square Error of Power Spectrograms
Yuma Tanaka, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON DIGITAL SIGNAL PROCESSING (DSP), 618, 622, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, Sparse representation is an idea to approximate a target signal by a linear combination of a small number of sample signals, and it is utilized in various research fields. In this paper, we evaluate the approximation error of signals by the mean square error of power spectrograms (P-MSE). Specifically, we propose a P-MSE minimization algorithm for sparse representation. Our method minimizes the P-MSE by an iterative approach. Specifically, in each iteration, we find the optimal sample signal and optimize the corresponding coefficients by a gradient-based method. In this approach, our method can utilize the result of the previous iteration for fast and stable convergence in the optimization of the coefficients. Based on this algorithm, the sparse representation which minimizes the P-MSE becomes feasible. Experimental results show the effectiveness of our method in terms of the P-MSE minimization.

MISSING INTENSITY RESTORATION VIA ADAPTIVE SELECTION OF PERCEPTUALLY OPTIMIZED SUBSPACES
Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 1628, 1632, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, A missing intensity restoration method via adaptive selection of perceptually optimized subspaces is presented in this paper. In order to realize adaptive and perceptually optimized restoration, the proposed method generates several subspaces of known textures optimized in terms of the structural similarity (SSIM) index. Furthermore, the SSIM-based missing intensity restoration is performed by a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Furthermore, the selection of the optimal subspace is realized by monitoring the SSIM index converged in the POCS algorithm, and the adaptive restoration becomes feasible. Experimental results show that our method outperforms existing methods.

NOVEL IMAGE CLASSIFICATION BASED ON INTEGRATION OF EEG AND VISUAL FEATURES VIA MSLPCCA
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2015 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), 957, 961, IEEE, 2015, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel image classification method based on integration of EEG and visual features. In the proposed method, we obtain classification results by separately using EEG and visual features. Furthermore, we merge the above classification results based on a kernelized version of Supervised learning from multiple experts and obtain the final classification result. In order to generate feature vectors used for the final image classification, we apply Multiset supervised locality preserving canonical correlation analysis (MSLPCCA), which is newly derived in the proposed method, to EEG and visual features. Our method realizes successful multimodal classification of images by the object categories that they contain based on MSLPCCA-based feature integration.

Random combination for information extraction in compressed sensing and sparse representation-based pattern recognition
Zaixing He, Xinyue Zhao, Shuyou Zhang, Takahiro Ogawa, Miki Haseyama
NEUROCOMPUTING, 145, 160, 173, ELSEVIER SCIENCE BV, Dec. 2014, [Peer-reviewed]
English, Scientific journal, In compressed sensing and sparse representation-based pattern recognition, random projection with a dense random transform matrix is widely used for information extraction. However, the complicated structure makes dense random matrices computationally expensive and difficult in hardware implementation. This paper considers the simplification of the random projection method. First, we propose a simple random method, random combination, for information extraction to address the issues of dense random methods. The theoretical analysis and the experimental results show that it can provide comparable performance to those of dense random methods. Second, we analyze another simple random method, random choosing, and give its applicable occasions. The comparative analysis and the experimental results show that it works well in dense cases but worse in sparse cases. Third, we propose a practical method for measuring the effectiveness of the feature transform matrix in sparse representation-based pattern recognition. A matrix satisfying the Representation Residual Restricted Isometry Property can provide good recognition results. (C) 2014 Elsevier B.V. All rights reserved.

Adaptive missing texture reconstruction method based on kernel cross-modal factor analysis with a new evaluation criterion
Takahiro Ogawa, Mild Haseyama
SIGNAL PROCESSING, 103, 69, 83, ELSEVIER SCIENCE BV, Oct. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents an adaptive missing texture reconstruction method based on kernel cross-modal factor analysis (KCFA) with a new evaluation criterion. The proposed method estimates the latent relationship between two areas, which correspond to a missing area and its neighboring area, respectively, from known parts within the target image and realizes reconstruction of the missing textures. In order to obtain this relationship, KCFA is applied to each cluster containing similar known textures, and the optimal cluster is used for reconstructing each target missing area. Specifically, a new criterion obtained by monitoring errors caused in the latent space enables selection of the optimal cluster. Then each missing texture is adaptively estimated by the optimal cluster's latent relationship, which enables accurate reconstruction of similar textures. In our method, the above criterion is also used for estimating patch priority, which determines the reconstruction order of missing areas within the target image. Since patches, whose textures are accurately modeled by our KCFA-based method, can be selected by using the new criterion, it becomes feasible to perform successful reconstruction of the missing areas. Experimental results show improvements of our KCFA-based reconstruction method over previously reported methods. (C) 2013 Elsevier B.V. All rights reserved.

Player Tracking in Far-View Soccer Videos Based on Composite Energy Function
Kazuya Iwai, Sho Takahashi, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E97D, 7, 1885, 1892, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Jul. 2014, [Peer-reviewed]
English, Scientific journal, In this paper, an accurate player tracking method in far-view soccer videos based on a composite energy function is presented. In far-view soccer videos, player tracking methods that perform processing based only on visual features cannot accurately track players since each player region becomes small, and video coding causes color bleeding between player regions and the soccer field. In order to solve this problem, the proposed method performs player tracking on the basis of the following three elements. First, we utilize visual features based on uniform colors and player shapes. Second, since soccer players play in such a way as to maintain a formation, which is a positional pattern of players, we use this characteristic for player tracking. Third, since the movement direction of each player tends to change smoothly in successive frames of soccer videos, we also focus on this characteristic. Then we adopt three energies: a potential energy based on visual features, an elastic energy based on formations and a movement direction-based energy. Finally, we define a composite energy function that consists of the above three energies and track players by minimizing this energy function. Consequently, the proposed method achieves accurate player tracking in far-view soccer videos.

A new method for error degree estimation in numerical weather prediction via MKDA-based ordinal regression
Takahiro Ogawa, Shintaro Takahashi, Sho Takahashi, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2014, 115, 115, SPRINGER INTERNATIONAL PUBLISHING AG, Jul. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents a new method for estimating error degrees in numerical weather prediction via multiple kernel discriminant analysis (MKDA)-based ordinal regression. The proposed method tries to estimate how large prediction errors will occur in each area from known observed data. Therefore, ordinal regression based on KDA is used for estimating the prediction error degrees. Furthermore, the following points are introduced into the proposed approach. Since several meteorological elements are related to each other based on atmospheric movements, the proposed method merges such heterogeneous features in the target and neighboring areas based on a multiple kernel algorithm. This approach is based on the characteristics of actual meteorological data. Then, MKDA-based ordinal regression for estimating the prediction error degree of a target meteorological element in each area becomes feasible. Since the amount of training data obtained from known observed data becomes very large in the training stage of MKDA, the proposed method performs simple sampling of those training data to reduce the number of samples. We effectively use the remaining training data for determining the parameters of MKDA to realize successful estimation of the prediction error degree.

歌謡番組における映像の構造に注目したシーン分割手法
吉田壮, 小川貴弘, 長谷山美紀
電子情報通信学会論文誌 D(Web), J97-D, 7, WEB ONLY 1177-1188, 01 Jul. 2014
Japanese

バイオミメティクスエンジニアリング 発想を支援するバイオミメティクス・データ検索と技術革新
長谷山美紀
化学工学, 78, 6, 394, 397, 05 Jun. 2014
Japanese

A Cross-Modal Approach for Extracting Semantic Relationships Between Concepts Using Tagged Images
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON MULTIMEDIA, 16, 4, 1059, 1074, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Jun. 2014, [Peer-reviewed]
English, Scientific journal, This paper presents a cross-modal approach for extracting semantic relationships between concepts using tagged images. In the proposed method, we first project both text and visual features of the tagged images to a latent space using canonical correlation analysis (CCA). Then, under the probabilistic interpretation of CCA, we calculate a representative distribution of the latent variables for each concept. Based on the representative distributions of the concepts, we derive two types of measures: the semantic relatedness between the concepts and the abstraction level of each concept. Because these measures are derived from a cross-modal scheme that enables the collaborative use of both text and visual features, the semantic relationships can successfully reflect semantic and visual contexts. Experiments conducted on tagged images collected from Flickr show that our measures are more coherent to human cognition than the conventional measures that use either text or visual features, or the WordNet-based measures. In particular, a new measure of semantic relatedness, which satisfies the triangle inequality, obtains the best results among different distance measures in our framework. The applicability of our measures to multimedia-related tasks such as concept clustering, image annotation and tag recommendation is also shown in the experiments.

Trial Realization of Associative Search for Media Understanding
HASEYAMA Miki
Technical report of IEICE. PRMU, 113, 493, 73, 77, The Institute of Electronics, Information and Communication Engineers, 06 Mar. 2014
Japanese, This paper presents a new associative search system for enhancing serendipity which collaboratively use several unstructured data such as images and videos and extracts their latent similarities. In order to realize such systems, the multimodal signal processing is essential. By using several different kinds of data such as images, videos and users' log data, the multimodal signal processing provides a solution to problems of not being able to improve the performance when using only single modalities. Therefore, in this paper, we introduce video retrieval using the multimodal signal processing and also show trial realization of new associative retrieval systems for effectively providing desired contents.

映像の構造に注目したMCMC法に基づくシーン分割法
SONG Yan, 小川貴弘, 長谷山美紀
電子情報通信学会論文誌 D, J97-D, 3, 560, 573, Mar. 2014
Japanese

A Scene Segmentation Approach Based on the MCMC Method Using Video Structures
SONG Yan, OGAWA Takahiro, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edition), 97, 3, 560, 573, The Institute of Electronics, Information and Communication Engineers, 01 Mar. 2014
Japanese, 以前,我々は,映像の類似したショットが隣接せずに出現する構造(映像の構造)に注目したシーン分割手法を提案した.しかしながら,映像の構造に注目したシーン分割手法の共通の問題として,映像の構造を取得する際に用いられるシーンの最長時間幅の設定が困難であるという点が存在した.そこで,本論文では,その改良手法として映像の構造に注目したMCMC法に基づくシーン分割手法を提案する.提案手法では,シーンの最長時間幅を変化させて映像の構造に基づき取得された全ての境界をシーン境界候補とし,それらの中からMCMC法に基づいて最適なシーン境界を推定する.これにより,従来の手法を適用する際に設定が困難であったシーンの最長時間幅を一意に決定することなく,MCMC法により最適なシーン境界を求めることが可能となる.本論文の最後では,実際にテレビで放送された映像に対して提案手法を適用し,その有効性を確認する.

MFCC extraction in AAC domain for audio content analysis
Ai Haojun, Miki Haseyama, Wang Kang
INFORMATION SCIENCE AND MANAGEMENT ENGINEERING, VOLS 1-3, 46, 1413, 1420, WIT PRESS, 2014
English, International conference proceedings, We focus the attention on the extraction of Mel-frequency cepstral coefficients (MFCC) features from MDCT spectrum in AAC domain for audio content analysis. In particular, a MFCC extraction method is proposed, which is adaptive to the window switch in AAC encoding process, and independent of the audio sampling frequency. We discuss the fusion method of MFCC features from different window type in order to keep the balance of the frequency and temporal resolution. The audio scene segmentation and audio classification experimental results show that such approach based on compression domain can approach the performance of the system based on PCM audio, and the CPU overload decreased dramatically. It is meaningful to the real-time analysis of audio content.

HELICOBACTER PYLORI INFECTION DETECTION FROM MULTIPLE X-RAY IMAGES BASED ON DECISION LEVEL FUSION
Kenta Ishihara, Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2769, 2773, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents an automatic detection method of Helicobacter pylori (H. pylori) infection from multiple gastric X-ray images. As the biggest contribution of this paper, we combine multiple detection results based on a decision level fusion. In order to obtain multiple detection results, the proposed method first focuses on characteristics of gastric X-ray images with H. pylori infection and computes several visual features from multiple X-ray images taken in several specific directions. Second, we select effective features for H. pylori infection detection from all features based on the minimal-Redundancy-Maximal-Relevance algorithm, and the selected features are used to train the Support Vector Machine (SVM) classifiers that are constructed for each direction of gastric radiography. Therefore, the detection of H. pylori infection becomes feasible, and we can obtain multiple detection results from the SVM classifiers. Furthermore, we combine multiple detection results based on the decision level fusion scheme considering the detection performance of each SVM classifier. Experimental results obtained by applying the proposed method to real X-ray images prove the effectiveness of the proposed method.

2D SEMI-SUPERVISED CCA-BASED INPAINTING INCLUDING NEW PRIORITY ESTIMATION
Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1837, 1841, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents an inpainting method based on 2D semi-supervised canonical correlation analysis (2D semi-CCA) including new priority estimation. The proposed method estimates relationship, i.e., the optimal correlation, between missing area and its neighboring area from known parts within the target image by using 2D CCA. In this approach, we newly introduce a semi-supervised scheme into the 2D CCA for deriving the 2D semi-CCA which corresponds to a hybrid version of 2D CCA and 2D principle component analysis (2D PCA). This enables successful relationship estimation even if sufficient number of training pairs cannot be provided. Then, by using the obtained relationship, accurate estimation of the missing intensities can be realized. Furthermore, in the proposed method, errors caused in the new variate space obtained by the 2D semi-CCA are effectively used for deriving patch priority determining inpainting order of missing areas. Experimental results show our inpainting method can outperform previously reported methods.

NOVEL IMAGE CLASSIFICATION BASED ON DECISION-LEVEL FUSION OF EEG AND VISUAL FEATURES
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 5874, 5878, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel image classification based on decision-level fusion of EEG and visual features. In the proposed method, we extract the EEG features from EEG signals recorded while users stare at images, and the visual features are computed from these images. Then the classification of images is performed based on Support Vector Machine (SVM) by separately using the EEG and visual features. Furthermore, we merge the above classification results based on Supervised Learning from Multiple Experts to obtain the final classification result. This method focuses on the classification accuracy calculated from each classification result. Therefore, although classification accuracy based on EEG and visual features are different from each other, our method realizes effective integration of these classification results. In addition, we newly derive a kernelized version of the method in order to realize more accurate integration of the classification results. Consequently, our method realizes successful multimodal classification of images by the object categories that they contain.

MISSING INTENSITY RESTORATION VIA PERCEPTUALLY OPTIMIZED SUBSPACE PROJECTION BASED ON ENTROPY COMPONENT ANALYSIS
Takahiro Ogawa, Miki Haseyama
2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 175, 179, IEEE, 2014, [Peer-reviewed]
English, International conference proceedings, A missing intensity restoration method via perceptually optimized subspace projection based on entropy component analysis (ECA) is presented in this paper. The proposed method calculates the optimal subspace of known patches within a target image based on structural similarity (SSIM) index, and the optimal bases are determined based on ECA. Then missing intensity estimation whose results maximize the SSIM index is realized by using a projection onto convex sets (POCS) algorithm whose constraints are the obtained subspace and known intensities within the target image. In this approach, a non-convex maximization problem for calculating the projection onto the subspace is reformulated as a quasi-convex problem, and the restoration of the missing intensities becomes feasible. Experimental results show that our restoration method outperforms previously reported methods.

Image inpainting based on sparse representations with a perceptual metric
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2013, 179, 179, SPRINGER INTERNATIONAL PUBLISHING AG, Dec. 2013, [Peer-reviewed]
English, Scientific journal, This paper presents an image inpainting method based on sparse representations optimized with respect to a perceptual metric. In the proposed method, the structural similarity (SSIM) index is utilized as a criterion to optimize the representation performance of image data. Specifically, the proposed method enables the formulation of two important procedures in the sparse representation problem, 'estimation of sparse representation coefficients' and 'update of the dictionary', based on the SSIM index. Then, using the generated dictionary, approximation of target patches including missing areas via the SSIM-based sparse representation becomes feasible. Consequently, image inpainting for which procedures are totally derived from the SSIM index is realized. Experimental results show that the proposed method enables successful inpainting of missing areas.

Exploring and visualizing tag relationships in photo sharing websites based on distributional representations
Marie Katsurai, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 3617, 3621, IEEE, 18 Oct. 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for exploring and visualizing tag relationships in photo sharing websites based on distributional representations of tags. First, we find a representative distribution of a tag, which is summarized by the mean and covariance, using features of tagged photos. This distributional representation can jointly consider the semantic meaning of tags and their abstraction levels. Then, based on the representative distributions, we derive two kinds of semantic measures on tag relationships. The extracted information is visualized in a graphical network to facilitate the understanding of tag usage. Experiments conducted using tagged photos collected from Flickr show that our tag network is more coherent to human cognition than other networks constructed by conventional methods. © 2013 IEEE.

Vocal segment estimation in music pieces based on collaborative use of EEG and audio features
Takuya Kawakami, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 1197, 1201, IEEE, 18 Oct. 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel estimation method of segments including vocals in music pieces based on collaborative use of features extracted from electroencephalogram (EEG) signals recorded while users are listening to music pieces and features extracted from these audio signals. From extracted EEG features and audio features, we estimate segments including vocals based on Support Vector Machine (SVM) by separately utilizing these two features. Furthermore, the final classification results are obtained by integrating these estimation results based on supervised learning from multiple experts. Therefore, our method realizes multimodal estimation of segments including vocals in music pieces. Experimental results show the improvement of our method over the methods utilizing only EEG or audio features. © 2013 IEEE.

胃X線検査における胃背景粘膜の自動解析による胃がんリスク評価の検討
間部克裕, 長谷山美紀, 小川貴弘, 吉澤和哉, 大泉晴史, 中島滋美, 加藤元嗣
日本消化器がん検診学会雑誌, 51, 3, 96, 15 May 2013
Japanese

Player Tracking by Using Level-Set Method in Soccer Video
TAKAHASHI Sho, LIM Wonkuk, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edetion), 96, 3, 695, 703, 一般社団法人電子情報通信学会, Mar. 2013, [Peer-reviewed]
Japanese, 本論文では,サッカー映像からレベルセット法を用いて選手を追跡する手法を提案する.提案手法では,サッカー映像を各フレームが時間軸方向に重なるように連結した三次元データとして扱う.このデータに対して,レベルセット法を適用することで抽出される三次元の領域は,複数フレームに渡って存在する同一選手を包含する.提案手法では,この三次元の領域をサッカー映像から抽出することで,選手の追跡を実現する.したがって,提案手法では,フレームごとに選手を検出する必要がないため,フレームを個別に処理する従来手法における選手の検出と追跡それぞれの誤差によって精度が低下する問題を解決可能である.また,我々は,ユニフォームの色成分をサッカー映像から色コリログラムを用いて推定し,これをレベルセット法を用いて追跡する選手の特徴として導入する.これにより,提案手法では,追跡対象の特徴を事前に与えることなく,選手の頑健な追跡が可能となる.本文の最後では,実際にテレビで放送されたサッカー映像に対する実験により,提案手法の有効性を確認する.

Missing Texture Reconstruction Method Based on Error Reduction Algorithm Using Fourier Transform Magnitude Estimation Scheme
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON IMAGE PROCESSING, 22, 3, 1252, 1257, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Mar. 2013, [Peer-reviewed]
English, Scientific journal, A missing texture reconstruction method based on an error reduction (ER) algorithm, including a novel estimation scheme of Fourier transform magnitudes is presented in this brief. In our method, Fourier transform magnitude is estimated for a target patch including missing areas, and the missing intensities are estimated by retrieving its phase based on the ER algorithm. Specifically, by monitoring errors converged in the ER algorithm, known patches whose Fourier transform magnitudes are similar to that of the target patch are selected from the target image. In the second approach, the Fourier transform magnitude of the target patch is estimated from those of the selected known patches and their corresponding errors. Consequently, by using the ER algorithm, we can estimate both the Fourier transform magnitudes and phases to reconstruct the missing areas.

Super-resolution for simultaneous realization of resolution enhancement and motion blur removal based on adaptive prior settings
Takahiro Ogawa, Daisuke Izumi, Akane Yoshizaki, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2013, 1, 17, SPRINGER INTERNATIONAL PUBLISHING AG, Feb. 2013, [Peer-reviewed]
English, Scientific journal, A super-resolution method for simultaneously realizing resolution enhancement and motion blur removal based on adaptive prior settings are presented in this article. In order to obtain high-resolution (HR) video sequences from motion-blurred low-resolution video sequences, both of the resolution enhancement and the motion blur removal have to be performed. However, if one is performed after the other, errors in the first process may cause performance deterioration of the subsequent process. Therefore, in the proposed method, a new problem, which simultaneously performs the resolution enhancement and the motion blur removal, is derived. Specifically, a maximum a posterior estimation problem which estimates original HR frames with motion blur kernels is introduced into our method. Furthermore, in order to obtain the posterior probability based on Bayes' rule, a prior probability of the original HR frame, whose distribution can adaptively be set for each area, is newly defined. By adaptively setting the distribution of the prior probability, preservation of the sharpness in edge regions and suppression of the ringing artifacts in smooth regions are realized. Consequently, based on these novel approaches, the proposed method can perform successful reconstruction of the HR frames. Experimental results show impressive improvements of the proposed method over previously reported methods.

Randomized-Controlled Study on the Effects of Hand Bathing on Stroke Patients in Convalescence - Subjective Improvements in Hand Movement and Increased Positiveness in Talking
Rika Yano, Miki Haseyama
STROKE, 44, 2, LIPPINCOTT WILLIAMS & WILKINS, Feb. 2013
English

Active grid-based method for visualizing pass regions in soccer videos
Sho Takahashi, Miki Haseyama
Electronic Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2013, 1, 6, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a method for visualizing pass regions that have high probabilities of the pass succeeding from broadcast soccer videos. In soccer matches, players discover pass regions based on player position geometry and player velocities. Therefore, by using player position geometry and player velocities, which are obtained from a broadcast soccer video, we can visualize pass regions. The proposed method is realized by the following two steps. 1) Generation of new three-dimensional data (volume data) for analyzing pass regions, which are not visible. 2) Visualization of pass regions. In the first step, volume data are generated from player position geometry and player velocities. By generating the volume data, which express the player position geometry and the player velocities, analysis of invisible pass regions is enabled. In the second step, by applying Active grid to the generated volume data, pass regions are visualized. Specifically, lattice points of the Active grid converge to the pass regions. Therefore, positions of the pass regions on the pitch can be visualized from densities of the lattice points. In the experiment, the proposed method is applied to actual TV programs to verify its effectiveness. © 2013 IEEE.

An extraction method of hierarchical Web communities for Web video retrieval
Ryosuke Harakawa, Yasutaka Hatakeyama, Takahiro Ogawa, Miki Haseyama
2013 IEEE International Conference on Image Processing, ICIP 2013 - Proceedings, 4397, 4401, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents an extraction method of hierarchical Web communities for Web video retrieval. In the proposed method, Web communities containing Web videos whose topics are similar to each other are extracted by using hyperlinks between Web pages including Web videos and their video features. Furthermore, we focus on graph structure of hyperlinks between Web pages including Web videos which belong to the Web communities. Then, by using strongly connected components and betweenness centrality of the graph, hierarchical structure of the Web communities can be estimated. Consequently, users can easily find Web videos including related topics in each hierarchy, and desired Web videos can be effectively retrieved. © 2013 IEEE.

Insect classification using Scanning Electron Microphotographs considering magnifications
Akihiro Takahashi, Takahiro Ogawa, Miki Haseyama
2013 IEEE International Conference on Image Processing, ICIP 2013 - Proceedings, 3269, 3273, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a method of insect classification using images taken by Scanning Electron Microscope (SEM) considering magnifications. Generally, when images of the same insects are taken by SEM with different magnifications, visual features of these images are different from each other. Thus, the proposed method adopts a new scheme which groups images of different magnifications in such a way that the classification performance becomes the highest. Then a classifier is constructed for each group, and the insect classification becomes feasible based on a target image magnification. In addition, by integrating the classification results of several images obtained from the same sample, i.e., the same insect, performance improvement of the insect classification considering magnifications can be realized. Experimental results show the effectiveness of the proposed method. © 2013 IEEE.

SPECTRAL REFLECTANCE ESTIMATION FROM VISIBLE LIGHT COMPONENTS AND NEAR-INFRARED COMPONENTS
Yuta Igarashi, Takahiro Ogawa, Miki Haseyama
2013 20TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP 2013), 2388, 2392, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, This paper presents a novel method for estimating a spectral reflectance from two kinds of input images: an image including both visible light components and near-infrared (NIR) components, and an image including only NIR components. From these input images, we estimate the spectral reflectance based on the Non-negative Matrix Factorization algorithm using spectral sensitivities of a digital camera. The estimated spectral reflectance enables several important applications. In this paper, the e ff ectiveness of the proposed method is verified by using the estimated spectral reflectance in the two image processing applications.

KCFA-BASED MISSING AREA RESTORATION INCLUDING NEW PRIORITY ESTIMATION
Takahiro Ogawa, Miki Haseyama
2013 20TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP 2013), 704, 708, IEEE, 2013, [Peer-reviewed]
English, International conference proceedings, A kernel cross-modal factor analysis (KCFA) based missing area restoration method including a new priority estimation scheme is presented in this paper. The proposed method estimates latent relationship between missing areas and their neighboring areas by deriving projection matrices minimizing their errors in the latent space based on KCFA. This latent relationship represented by the derived projection matrices is optimal for accurately restoring missing areas within the target image. Furthermore, the proposed method adopts a new priority estimation scheme which determines the restoration order of missing areas. Specifically, this priority is estimated based on the criterion representing the restoration performance derived from KCFA, and it enables adaptive selection of missing areas successfully restored by our method. Consequently, it becomes feasible to accurately perform the restoration of missing areas by using the proposed KCFA-based method. Experimental results show subjective and quantitative improvements of the proposed method over previously reported restoration methods.

Trial Realization of Human-Centered Multimedia Navigation for Video Retrieval
Miki Haseyama, Takahiro Ogawa
INTERNATIONAL JOURNAL OF HUMAN-COMPUTER INTERACTION, 29, 2, 96, 109, TAYLOR & FRANCIS INC, Jan. 2013, [Peer-reviewed]
English, Scientific journal, A trial realization of human-centered navigation for video retrieval is presented in this article. This system consists of the following functions: (a) multimodal analysis for collaborative use of multimedia data, (b) preference extraction for the system to adapt to users' individual demands, and (c) adaptive visualization for users to be guided to their desired contents. By using these functions, users can find their desired video contents more quickly and accurately than with the conventional retrieval schemes since our system can provide new pathways to the desired contents. Experimental results verify the effectiveness of the proposed system.

Glare Detection for Night Wet Road Surfaces and Driver Visibility Improvement by Using Multiple Onboard Cameras
公文宏明, 長谷山美紀
映像情報メディア学会誌(Web), 67, 3, 2013

Performance of Spatial and Temporal Error Concealment Method for 3D DWT Video Coding in Packet Loss Channel
Hirokazu Tanaka, Sunmi Kim, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E95A, 11, 2015, 2022, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2012, [Peer-reviewed]
English, Scientific journal, A new spatial and temporal error concealment method for three-dimensional discrete wavelet transform (3D DWT) video coding is analyzed. 3D DWT video coding employing dispersive grouping (DG) and two-step error concealment is an efficient method in a packet loss channel [20], [21]. In the two-step error concealment method, the interpolations are only spatially applied however, higher efficiency of the interpolation can be expected by utilizing spatial and temporal similarities. In this paper, we propose an enhanced spatial and temporal error concealment method in order to achieve higher error concealment (EC) performance in packet loss networks. In the temporal error concealment method, structural similarity (SSIM) index is employed for inter group of pictures (GOP) EC and minimum mean square error (MMSE) is used for intra GOP EC. Experimental results show that the proposed method can obtain remarkable performance compared with the conventional methods.

Super-Resolution Reconstruction for Spatio-Temporal Resolution Enhancement of Video Sequences
Miki Haseyama, Daisuke Izumi, Makoto Takizawa
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E95D, 9, 2355, 2358, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Sep. 2012, [Peer-reviewed]
English, Scientific journal, A method for spatio-temporal resolution enhancement of video sequences based on super-resolution reconstruction is proposed. A new observation model is defined for accurate resolution enhancement, which enables subpixel motion in intermediate frames to be obtained. A modified optimization formula for obtaining a high-resolution sequence is also adopted.

Transmission and Vibration in Circuit Theory (7) Band-Pass Filters Having Impedance Transformation Ratio
永井 信夫, 任 捷, 長谷山 美紀
信号処理, 16, 5, 359, 368, [信号処理学会], Sep. 2012
Japanese

Transmission and Vibration in Circuit Theory (6) Reconsideration of Left-Handed Circuit Viewing form Circuit Theory
永井 信夫, 任 捷, 長谷山 美紀
信号処理, 16, 4, 263, 272, [信号処理学会], Jul. 2012
Japanese

A Novel Framework for Extracting Visual Feature-Based Keyword Relationships from an Image Database
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E95A, 5, 927, 937, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, May 2012, [Peer-reviewed]
English, Scientific journal, In this paper, a novel framework for extracting visual feature-based keyword relationships from an image database is proposed. From the characteristic that a set of relevant keywords tends to have common visual features, the keyword relationships in a target image database are extracted by using the following two steps. First, the relationship between each keyword and its corresponding visual features is modeled by using a classifier. This step enables detection of visual features related to each keyword. In the second step, the keyword relationships are extracted from the obtained results. Specifically, in order to measure the relevance between two keywords, the proposed method removes visual features related to one keyword from training images and monitors the performance of the classifier obtained for the other keyword. This measurement is the biggest difference from other conventional methods that focus on only keyword co-occurrences or visual similarities. Results of experiments conducted using an image database showed the effectiveness of the proposed method.

Transmission and Vibration in Circuit Theory (5) Generalization of Difference Equation and Left-handed Circuit
永井 信夫, 任 捷, 長谷山 美紀
信号処理, 16, 3, 187, 196, [信号処理学会], May 2012
Japanese

Transmission and Vibration in Circuit Theory (4) Characteristic Vibration and Resonance Created at Image Phase π/2
永井 信夫, 任 捷, 長谷山 美紀
信号処理, 16, 2, 101, 110, [信号処理学会], Mar. 2012
Japanese

Haseyama Laboratory at Hokkaido University
長谷山 美紀
信号処理, 16, 2, 121, 126, [信号処理学会], Mar. 2012
Japanese

An SVDD-Based Method for Taxonomy Estimation of Benthic Species from Microscopic Images
HASEGAWA TAKASHI, OGAWA TAKAHIRO, WATANABE HIDEMI, HASEYAMA MIKI
電子情報通信学会技術研究報告, 111, 442(IE2011 105-132), 73, 78, Feb. 2012
Japanese

A cross-modal approach for extracting semantic relationships of concepts from an image database
Marie Katsurai, Takahiro Ogawa, Miki Haseyama
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, 2373, 2376, IEEE, 2012, [Peer-reviewed]
English, International conference proceedings, This paper presents a cross-modal approach for extracting semantic relationships of concepts from an image database. First, canonical correlation analysis (CCA) is used to capture the cross-modal correlations between visual features and tag features in the database. Then, in order to measure inter-concept relationships and estimate semantic levels, the proposed method focuses on the distributions of images under the probabilistic interpretation of CCA. Results of experiments conducted by using an image database showed the improvement of the proposed method over existing methods. © 2012 IEEE.

PERCEPTUALLY OPTIMIZED SUBSPACE ESTIMATION FOR MISSING TEXTURE RECONSTRUCTION
Takahiro Ogawa, Miki Haseyama
2012 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 1141, 1144, IEEE, 2012, [Peer-reviewed]
English, International conference proceedings, This paper presents a perceptually optimized subspace estimation method for missing texture reconstruction. The proposed method calculates the optimal subspace of known patches within a target image based on structural similarity (SSIM) index instead of calculating mean square error (MSE)-based eigenspace. Furthermore, from the obtained subspace, missing texture reconstruction whose results maximize the SSIM index is performed. In this approach, the non-convex maximization problem is reformulated as a quasi convex problem, and the reconstruction of the missing textures becomes feasible. Experimental results show that our method overcomes previously reported MSE-based reconstruction methods.

Transmission and Vibration in Circuit Theory (3) Characteristic Vibration Related to Combinational Oscillation and Its Expansion
永井 信夫, 任 捷, 長谷山 美紀
信号処理, 16, 1, 13, 22, [信号処理学会], Jan. 2012
Japanese

8-3 A note on improvement of 3D pass region estimation method using player velocity in soccer videos
TAKAHASHI Sho, HASEYAMA Miki
PROCEEDINGS OF THE ITE WINTER ANNUAL CONVENTION, 2012, 8, 3-1, The Institute of Image Information and Television Engineers, 2012
Japanese, This paper realizes an improvement of 3D pass region estimation method by using player velocity in soccer videos. In the previous method, since the pass region was estimated regardless of player velocity, the accuracy was limited. Therefore, by introducing the player velocity to the pass region estimation, we improve the performance of the previous method.

Support Vector Data Dscription-Based Method for Finding New Benthic Species and Estimating Their Taxonomy Position from Microscopic Images.
長谷川尭史, 小川貴弘, 渡邉日出海, 長谷山美紀
映像情報メディア学会誌(Web), 66, 7, 2012

Missing Image Data Reconstruction Based on Adaptive Inverse Projection via Sparse Representation
Takahiro Ogawa, Miki Haseyama
IEEE TRANSACTIONS ON MULTIMEDIA, 13, 5, 974, 992, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, Oct. 2011, [Peer-reviewed]
English, Scientific journal, In this paper, a missing image data reconstruction method based on an adaptive inverse projection via sparse representation is proposed. The proposed method utilizes sparse representation for obtaining low-dimensional subspaces that approximate target textures including missing areas. Then, by using the obtained low-dimensional subspaces, inverse projection for reconstructing missing areas can be derived to solve the problem of not being able to directly estimate missing intensities. Furthermore, in this approach, the proposed method monitors errors caused by the derived inverse projection, and the low-dimensional subspaces optimal for target textures are adaptively selected. Therefore, we can apply adaptive inverse projection via sparse representation to target missing textures, i.e., their adaptive reconstruction becomes feasible. The proposed method also introduces some schemes for color processing into the calculation of subspaces on the basis of sparse representation and attempts to avoid spurious color caused in the reconstruction results. Consequently, successful reconstruction of missing areas by the proposed method can be expected. Experimental results show impressive improvement of our reconstruction method over previously reported reconstruction methods.

Cross Low-Dimension Pursuit for Sparse Signal Recovery from Incomplete Measurements Based on Permuted Block Diagonal Matrix
Zaixing He, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E94A, 9, 1793, 1803, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Sep. 2011, [Peer-reviewed]
English, Scientific journal, In this paper, a novel algorithm, Cross Low-dimension Pursuit, based on a new structured sparse matrix, Permuted Block Diagonal (PBD) matrix, is proposed in order to recover sparse signals from incomplete linear measurements. The main idea of the proposed method is using the PBD matrix to convert a high-dimension sparse recovery problem into two (or more) groups of highly low-dimension problems and crossly recover the entries of the original signal from them in an iterative way. By sampling a sufficiently sparse signal with a PBD matrix, the proposed algorithm can recover it efficiently. It has the following advantages over conventional algorithms: (1) low complexity, i.e., the algorithm has linear complexity, which is much lower than that of existing algorithms including greedy algorithms such as Orthogonal Matching Pursuit and (2) high recovery ability, i.e., the proposed algorithm can recover much less sparse signals than even l(1)-norm minimization algorithms. Moreover, we demonstrate both theoretically and empirically that the proposed algorithm can reliably recover a sparse signal from highly incomplete measurements.

Transmission and vibration in circuit theory (2) Resonance and one-dimensional vibration for two atoms crystal
永井 信夫, 任 捷, 長谷山 美紀
Journal of signal processing, 15, 5, 331, 340, 〔信号処理学会〕, Sep. 2011
Japanese

Adaptive Single Image Superresolution Approach Using Support Vector Data Description
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, An adaptive single image superresolution (SR) method using a support vector data description (SVDD) is presented. The proposed method represents the prior on high-resolution (HR) images by hyperspheres of the SVDD obtained from training examples and reconstructs HR images from low-resolution (LR) observations based on the following schemes. First, in order to perform accurate reconstruction of HR images containing various kinds of objects, training HR examples are previously clustered based on the distance from a center of a hypersphere obtained for each cluster. Furthermore, missing high-frequency components of the target image are estimated in order that the reconstructed HR image minimizes the above distances. In this approach, the minimized distance obtained for each cluster is utilized as a criterion to select the optimal hypersphere for estimating the high-frequency components. This approach provides a solution to the problem of conventional methods not being able to perform adaptive estimation of the high-frequency components. In addition, local patches in the target low-resolution (LR) image are utilized as the training HR examples from the characteristic of self-similarities between different resolution levels in general images, and our method can perform the SR without utilizing any other HR images.

ADAPTIVE KPCA-BASED MISSING TEXTURE RECONSTRUCTION APPROACH INCLUDING CLASSIFICATION SCHEME VIA DIFFERENCE SUBSPACES
Takahiro Ogawa, Miki Haseyama
2011 18TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 1133, 1136, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive kernel principal component analysis (KPCA) based missing texture reconstruction approach including a classification scheme via difference subspaces. The proposed method utilizes a KPCA-based nonlinear eigenspace, which is obtained from each kind of known texture within a target image, as a constraint for reconstructing missing textures with a constraint of known neighboring areas. Then since these two constraints are convex, we can estimate missing textures based on a projection onto convex sets (POCS) algorithm. Furthermore, in this approach, the proposed method derives a new criterion for selecting the optimal eigenspace by monitoring errors caused in the projection via a difference subspace of each kind of known texture. This provides a solution to conventional problems of not being able to perform accurate texture classification, and the adaptive reconstruction of missing textures can be realized by the proposed method. Experimental results show subjective and quantitative improvement of the proposed method over previously reported reconstruction methods.

LINEAR TIME DECODING OF REAL-FIELD CODES OVER HIGH ERROR RATE CHANNELS
Zaixing He, Takahiro Ogawa, Miki Haseyama
2011 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 3172, 3175, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel algorithm for decoding real-field codes over erroneous channels, where the encoded message is corrupted by sparse errors, i.e., impulsive noise. The main problem of decoding such a corrupted encoded message is to reconstruct the error vector; recently, a common way to reconstruct it is to find the sparsest solution to an underdetermined system that is constructed using a parity-check matrix. Unlike the conventional approaches reconstructing the high-dimensional error vector directly, the proposed method crossly recovers the elements of error vector from two (or several) groups of low-dimensional equations. Compared with the traditional algorithms, the proposed method can decode an encoded message with a much higher corruption rate. Furthermore, the complexity of our method is linear, which is much lower than those of the traditional methods. The experimental results verified the high error correction ability and speed of the proposed method.

ADAPTIVE RECONSTRUCTION METHOD OF MISSING TEXTURES BASED ON PERCEPTUALLY OPTIMIZED ALGORITHM
Takahiro Ogawa, Miki Haseyama
2011 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, 1157, 1160, IEEE, 2011, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive reconstruction method of missing textures based on structural similarity (SSIM) index. The proposed method firstly performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Furthermore, from the selected known textures, the missing texture reconstruction maximizing the SSIM index is performed. In this approach, the non-convex maximization problem is reformulated as a quasi convex problem, and the adaptive reconstruction of the missing textures becomes feasible. Experimental results show impressive improvement of the proposed method over previously reported reconstruction methods.

Adaptive example-based super-resolution using kernel PCA with a novel classification approach
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, 1, 29, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, An adaptive example-based super-resolution (SR) using kernel principal component analysis (PCA) with a novel classification approach is presented in this paper. In order to enable estimation of missing high-frequency components for each kind of texture in target low-resolution (LR) images, the proposed method performs clustering of high-resolution (HR) patches clipped from training HR images in advance. Based on two nonlinear eigenspaces, respectively, generated from HR patches and their corresponding low-frequency components in each cluster, an inverse map, which can estimate missing high-frequency components from only the known low-frequency components, is derived. Furthermore, by monitoring errors caused in the above estimation process, the proposed method enables adaptive selection of the optimal cluster for each target local patch, and this corresponds to the novel classification approach in our method. Then, by combining the above two approaches, the proposed method can adaptively estimate the missing high-frequency components, and successful reconstruction of the HR image is realized.

Music recommendation according to human motion based on kernel CCA-based relationship
Hiroyuki Ohkushi, Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2011, 121, 121, SPRINGER INTERNATIONAL PUBLISHING AG, 2011, [Peer-reviewed]
English, Scientific journal, In this article, a method for recommendation of music pieces according to human motions based on their kernel canonical correlation analysis (CCA)-based relationship is proposed. In order to perform the recommendation between different types of multimedia data, i.e., recommendation of music pieces from human motions, the proposed method tries to estimate their relationship. Specifically, the correlation based on kernel CCA is calculated as the relationship in our method. Since human motions and music pieces have various time lengths, it is necessary to calculate the correlation between time series having different lengths. Therefore, new kernel functions for human motions and music pieces, which can provide similarities between data that have different time lengths, are introduced into the calculation of the kernel CCA-based correlation. This approach effectively provides a solution to the conventional problem of not being able to calculate the correlation from multimedia data that have various time lengths. Therefore, the proposed method can perform accurate recommendation of best matched music pieces according to a target human motion from the obtained correlation. Experimental results are shown to verify the performance of the proposed method.

足跡を用いた男女識別　　　　　　　　　　　　　　　
画像ラボ, 22, 1, 17, 21, 2011

固有空間BPLP法の補間精度に関する解析
田中章, 小川貴弘, 長谷山美紀, 宮腰政明
電子情報通信学会 論文誌(A), J94-A, 2, 116, 126, 2011

Error-Resilient 3-D Wavelet Video Coding with Duplicated Lowest Sub-Band Coefficients and Two-Step Error Concealment Method
Sunmi Kim, Hirokazu Tanaka, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E93A, 11, 2173, 2183, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Nov. 2010, [Peer-reviewed]
English, Scientific journal, In this paper we propose a two step error concealment algorithm based on an error resilient three dimensional discrete wavelet transform (3 D DWT) video coding scheme. The proposed scheme consists of an error resilient encoder duplicating the lowest sub band bit streams for dispersive grouped frames and an error concealment decoder. The error concealment method of this decoder is decomposed of two steps the first step is replacement of erroneous coefficients in the lowest sub band by the duplicated coefficients and the second step is interpolation of the missing wavelet coefficients by minimum mean square error (MMSE) estimation. The proposed scheme can achieve robust transmission over unreliable channels. Experimental results provide performance comparisons in terms of peak signal to noise ratio (PSNR) and demonstrate increased performances compared to state of the art error concealment schemes.

Recent Trends of Image and Video Semantic Analysis and Retrieval Interfaces(Developments of Vision Computing based on Probabilistic Information Processing)
HASEYAMA Miki
The Journal of the Institute of Electronics, Information, and Communication Engineers, 93, 9, 764, 769, 一般社団法人電子情報通信学会, Sep. 2010
Japanese, 画像・映像意味理解の研究動向とその検索への応用について紹介する.更に,画像及び映像が持つ固有の多義性とあいまい性から検索結果の可視化システムの必要性を議論し,その実現の試みについて紹介するとともに今後の展開について考える.

Gender Identification Using Support Vector Machine Based on Shoe Print
ASAMIZU Satoshi, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edetion), 93, 5, 642, 646, The Institute of Electronics, Information and Communication Engineers, 01 May 2010
Japanese, 本論文では,施設や店舗などに出入りする人物の足跡を用いて男女識別する手法について検証する.取得した足跡から算出が可能な特徴量を用いてSVMにより男女を識別する.本手法を用いて被験者実験を行い,90%の識別率を実現した.

ADAPTIVE RECONSTRUCTION METHOD OF MISSING TEXTURES BASED ON INVERSE PROJECTION VIA SPARSE REPRESENTATION
Takahiro Ogawa, Miki Haseyama
2010 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME 2010), 352, 357, IEEE, 2010, [Peer-reviewed]
English, International conference proceedings, This paper presents an adaptive reconstruction method of missing textures based on an inverse projection via sparse representation. The proposed method approximates original and corrupted textures in lower-dimensional subspaces by using the sparse representation technique. Then, this approach effectively solves problems of not being able to directly estimate an inverse projection for reconstructing missing textures. Furthermore, even if target textures contain missing areas, the proposed method enables adaptive generation of the subspaces by monitoring errors caused in their known neighboring textures by the estimated inverse projection. Consequently, since the optimal inverse projection is adaptively estimated for each texture, successful reconstruction of the missing areas can be expected. Experimental results show impressive improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

THE SIMPLEST MEASUREMENT MATRIX FOR COMPRESSED SENSING OF NATURAL IMAGES
Zaixing He, Takahiro Ogawa, Miki Haseyama
2010 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, 4301, 4304, IEEE, 2010, [Peer-reviewed]
English, International conference proceedings, There exist two main problems in currently existing measurement matrices for compressed sensing of natural images, the difficulty of hardware implementation and low sensing efficiency. In this paper, we present a novel simple and efficient measurement matrix, Binary Permuted Block Diagonal (BPBD) matrix. The BPBD matrix is binary and highly sparse (all but one or several "1"s in each column are "0"s). Therefore, it can simplify the compressed sensing procedure dramatically. The proposed measurement matrix has the following advantages, which cannot be entirely satisfied by existing measurement matrices. (1) It has easy hardware implementation because of the binary elements; (2) It has high sensing efficiency because of the highly sparse structure; (3) It is incoherent with different popular sparsity basis' like wavelet basis and gradient basis; (4) It provides fast and nearly optimal reconstructions. Moreover, the simulation results demonstrate the advantages of the proposed measurement matrix.

Spatio-temporal resolution enhancement of video sequence based on super-resolution reconstruction.
Miki Haseyama, Daisuke Izumi, Makoto Takizawa
Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2010, 14-19 March 2010, Sheraton Dallas Hotel, Dallas, Texas, USA, 870, 873, IEEE, 2010, [Peer-reviewed]
International conference proceedings

A Genetic Algorithm for Generating Multiple Paths on Mesh Maps
Jun Inagaki, Tomoaki Shirakawa, Tetsuo Shimono, Miki Haseyama
2010 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC), 1, 4, IEEE, 2010, [Peer-reviewed]
English, International conference proceedings, Path generation is an optimization problem mainly performed on grid square maps that combines generation of paths with minimization of their cost. Several methods that belong to the class of exhaustive searches are available; however, these methods are only able to obtain a single path as a solution for each iteration of the search. Hence, this paper proposes a new method using genetic algorithms for this problem with the goal of simultaneously searching for multiple candidate paths.

Missing Texture Reconstruction Method Based on Perceptually Optimized Algorithm
Takahiro Ogawa, Miki Haseyama
EURASIP JOURNAL ON ADVANCES IN SIGNAL PROCESSING, 2010, HINDAWI PUBLISHING CORPORATION, 2010, [Peer-reviewed]
English, Scientific journal, This paper presents a simple and effective missing texture reconstruction method based on a perceptually optimized algorithm. The proposed method utilizes the structural similarity (SSIM) index as a new visual quality measure for reconstructing missing areas. Furthermore, in order to adaptively reconstruct target images containing several kinds of textures, the following two novel approaches are introduced into the SSIM-based reconstruction algorithm. First, the proposed method performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Secondly, missing texture reconstruction that maximizes the SSIM index in the known neighboring areas is performed. In this approach, the nonconvex maximization problem is reformulated as a quasi convex problem, and adaptive reconstruction of the missing textures based on the perceptually optimized algorithm becomes feasible. Experimental results show impressive improvements of the proposed method over previously reported reconstruction methods.

Erratum: Error-Resilient 3-D Wavelet Video Coding with Duplicated Lowest Sub-Band Coefficients and Two-Step Error Concealment Method [IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E93.A (2010) , No. 11 pp.2173-2183]
KIM Sunmi, TANAKA Hirokazu, OGAWA Takahiro, HASEYAMA Miki
IEICE Trans. Fundamentals, 93, 12, 2763_e1, 2763_e1, The Institute of Electronics, Information and Communication Engineers, 2010
English

Interdisciplinary Research Project on Biological Science(Next-Generation Information Technology Based on Knowledge Discovery and Knowledge Federation: Hokkaido University Global COE Program and Research Projects on ICT in Hokkaido)
WATANABE Hidemi, KANEKO Shun'ichi, HASEYAMA Miki
The Journal of the Institute of Electronics, Information, and Communication Engineers, 92, 10, 822, 827, 一般社団法人電子情報通信学会, Oct. 2009
Japanese

Adaptive Missing Texture Reconstruction Method Based on Kernel Canonical Correlation Analysis with a New Clustering Scheme
Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1950, 1960, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, a method for adaptive reconstruction of missing textures based on kernel canonical correlation analysis (CCA) with a new clustering scheme is presented. The proposed method estimates the correlation between two areas, which respectively correspond to a missing area and its neighboring area, from known parts within the target image and realizes reconstruction of the missing texture. In order to obtain this correlation, the kernel CCA is applied to each cluster containing the same kind of textures, and the optimal result is selected for the target missing area. Specifically, a new approach monitoring errors caused in the above kernel CCA-based reconstruction process enables selection of the optimal result. This approach provides a solution to the problem in traditional methods of not being able to perform adaptive reconstruction of the target textures due to missing intensities. Consequently, all of the missing textures are successfully estimated by the optimal cluster's correlation, which provides accurate reconstruction of the same kinds of textures. In addition, the proposed method can obtain the correlation more accurately than our previous works, and more successful reconstruction performance can be expected. Experimental results show impressive improvement of the proposed reconstruction technique over previously reported reconstruction techniques.

An ER Algorithm-Based Method for Removal of Adherent Water Drops from Images Obtained by a Rear View Camera Mounted on a Vehicle in Rainy Conditions
Tomoki Hiramatsu, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1939, 1949, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, an ER (Error-Reduction) algorithm-based method for removal of adherent water drops from images obtained by a rear view camera mounted on a vehicle in rainy conditions is proposed. Since Fourier-domain and object-domain constraints are needed for any ER algorithm-based method, the proposed method introduces the following two novel constraints for the removal of adherent water drops. The first one is the Fourier-domain constraint that utilizes the Fourier transform magnitude of the previous frame in the obtained images as that of the target frame. Noting that images obtained by the rear view camera have the unique characteristics of objects moving like ripples because the rear view camera is generally composed of a fish-eye lens for a wide view angle, the proposed method assumes that the Fourier transform magnitudes of the target frame and the previous frame are the same in the polar coordinate system. The second constraint is the object-domain constraint that utilizes intensities in an area of the target frame to which water drops have adhered. Specifically, the proposed method models a deterioration process of intensities that are corrupted by the water drop adhering to the rear view camera lens. By utilizing these novel constraints, the proposed ER algorithm can remove adherent water drops from images obtained by the rear view camera. Experimental results that verify the performance of the proposed method are represented.

Estimating Number of People Using Calibrated Monocular Camera Based on Geometrical Analysis of Surface Area
Hiroyuki Arai, Isao Miyagawa, Hideki Koike, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1932, 1938, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, We propose a novel technique for estimating the number of people in a video sequence; it has the advantages of being stable even in crowded situations and needing no ground-truth data. By analyzing the geometrical relationships between image pixels and their intersection volumes in the real world quantitatively, a foreground image directly indicates the number of people. Because foreground detection is possible even in crowded situations, the proposed method can be applied in such situations. Moreover, it can estimate the number of people in an a priori manner, so it needs no ground-truth data unlike existing feature-based estimation techniques. Experiments show the validity of the proposed method.

An Accurate Scene Segmentation Method Based on Graph Analysis Using Object Matching and Audio Feature
Makoto Yamamoto, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 8, 1883, 1891, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2009, [Peer-reviewed]
English, Scientific journal, A method for accurate scene segmentation using two kinds of directed graph obtained by object matching and audio features is proposed. Generally, in audiovisual materials, such as broadcast programs and movies, there are repeated appearances of similar shots that include frames of the same background, object or place, and such shots are included in a single scene. Many scene segmentation methods based on this idea have been proposed; however, since they use color information as visual features, they cannot provide accurate scene segmentation results if the color features change in different shots for which frames include the same object due to camera operations such as zooming and panning. In order to solve this problem, scene segmentation by the proposed method is realized by using two novel approaches. In the first approach, object matching is performed between two frames that are each included in different shots. By using these matching. results, repeated appearances of shots for which frames include the same object can be successfully found and represented as a directed graph. The proposed method also generates another directed graph that represents the repeated appearances of shots with similar audio features in the second approach. By combined use of these two directed graphs, degradation of scene segmentation accuracy, which results from using only one kind of graph, can be avoided in the proposed method and thereby accurate scene segmentation can be realized. Experimental results performed by applying the proposed method to actual broadcast programs are shown to verify the effectiveness of the proposed method.

Maintaining the Stability of Solutions in Dynamic Fuzzy CSPs
SUDO Yasuhiro, YANAGIDA Takuto, KURIHARA Masahito, HASEYAMA Miki
Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, 21, 3, 372, 380, Japan Society for Fuzzy Theory and Intelligent Informatics, 15 Jun. 2009
Japanese, A Fuzzy Constraint Satisfaction Problem (FCSP) is an extension of the classical CSP, a powerful tool for modeling various problems based on constraints among variables. Meanwhile, a Dynamic CSP (DCSP) is a framework for modeling the transformation of problems, and the differences between past solutions and new solutions should be as small as possible. These schemes are the techniques to formulate real world problems as CSPs more easily. The CSP model that combines these (Dynamic Fuzzy CSP) has already existing researches. However, as CSPs are NP-hard problems in general, no efficient and complete algorithms for solving CSPs exist and the increase in the worst-case computation time is exponential in the size of the problems. In the work reported in this paper we tested a hybrid approximate method, called the SRS algorithm. Moreover, we introduce a post-filtering method, called the SRSD algorithm. We empirically show that SRS and SRSD algorithms keep the stability of solutions better than other algorithms. They are able to quickly get good-quality approximate and stabile solutions to large problems.

Active Net-Based Non-interception Region Estimation in Soccer Videos
TAKAHASHI Sho, KON Hirofumi, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edetion), 92, 4, 501, 510, The Institute of Electronics, Information and Communication Engineers, 01 Apr. 2009
Japanese, 本論文では,チームスポーツ映像からアクティブネットを用いてパス可能領域を推定する手法を提案する.チームスポーツ映像の一つであるサッカー映像の意味内容解析を行うために重要なサッカーの戦術は,選手の移動とボール運びによって表現されるため,ボール運びを実現するパスを分析することは重要である.一般にパスコースはボール保持者と味方チームの選手へとつながる緩やかな曲線で表される.提案手法では,新たなエネルギーの定義とパス可能領域を推定するための画像生成により,アクティブネットを用いて前述の曲線が存在する領域を抽出する.また,パス可能領域は守備の選手から離れるほど,パスが成功する可能性が高いという特徴をもつ.提案手法では,格子点の密度に着眼することで,パスが成功する可能性をパス可能領域の推定と同時に得る.更に,アクティブネットの収束結果は多少の選手位置の誤差を許容するため,選手の動きを用いた従来手法における,選手位置の誤差の影響を受けやすいという問題点を解決することが可能である.したがって,提案手法はカメラワークが存在し,高精度な選手位置の推定が困難であるテレビ映像に対しても,高精度にパス可能領域の推定が可能である.

Kalman Filter-Based Error Concealment for Video Transmission
Shigeki Takahashi, Takahiro Ogawa, Hirokazu Tanaka, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 3, 779, 787, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Mar. 2009, [Peer-reviewed]
English, Scientific journal, A novel error concealment method using a Kalman filter is presented ill this paper, In order to successfully utilize the Kalman filter, its state transition and observation models that are suitable for the video error concealment are newly defined as follows. The state transition model represents the video decoding process by a notion-compensated prediction. Furthermore, the new observation model that represents all image blurring process is defined. and calculation of the Kalman gain becomes possible. The problem of the traditional methods is solved by using the Kalman filter in the proposed method, and accurate reconstruction of corrupted video frames, is achieved. Consequently. an effective error concealment method using the Kalman filter is realized. Experimental results showed that the proposed method has better performance than that of traditional methods.

An IFS-Based Image Enlargement Method Considering Edge Continuity
KAKUKOU Norihiro, OGAWA Takahiro, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edetion), 92, 3, 382, 392, The Institute of Electronics, Information and Communication Engineers, 01 Mar. 2009
Japanese, 本論文では,エッジの連続性を考慮した, Iterated Function System (IFS)に基づく画像拡大法を提案する.従来のIFS画像拡大法では,拡大後の画像において処理の最小単位となるブロックの境界で,本来存在しないはずの輝度値の変化が生じる.また,エッジの連続性を考慮しておらず,拡大後の画像におけるエッジが不連続となる問題が存在した.そこで提案手法では,まず,処理の最小単位となるブロックの重なりを許すことで, IFSによる高近似縮小写像を実現し,ブロック境界での輝度値の変化を抑制する.更に,連続性を保ったエッジの推定が可能であるラインプロセスを新たにIFS画像拡大法に導入する.このとき我々は,ラインプロセスを拡大後の画像のエッジを推定する手法に拡張することで,輝度値が未知である拡大後の画像に対しても,連続性を保ったエッジの推定を可能とする.また,提案手法では得られるエッジの存在を考慮した上でIFSに基づく画像拡大を行うことにより,従来法で発生していた拡大後の画像におけるエッジの不連続を解決し,高精細な拡大を実現する.本論文の最後では,提案手法の有効性を示すため比較実験を行い,その拡大性能を評価する.

A Kalman Filter-Based Method for Restoration of Images Obtained by an In-Vehicle Camera in Foggy Conditions
Tomoki Hiramatsu, Takahiro Ogawa, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E92A, 2, 577, 584, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Feb. 2009, [Peer-reviewed]
English, Scientific journal, In this paper, a Kalman filter-based method for restoration of video images acquired by an in-vehicle camera in foggy conditions is proposed. In order to realize Kalman filter-based restoration, the proposed method clips local blocks from the target frame by using a sliding window and regards the intensities in each block as elements of the state variable of the Kalman filter. Furthermore, the proposed method designs the following two models for restoration of foggy images. The first one is an observation model, which represents a fog deterioration model. The proposed method automatically determines all parameters of the fog deterioration model from only the foggy images to design the observation model. The second one is a non-linear state transition model, which represents the target frame in the original video image from its previous frame based on motion vectors. By utilizing the observation and state transition models, the correlation between successive frames can be effectively utilized for restoration, and accurate restoration of images obtained in foggy conditions can be achieved. Experimental results show that the proposed method has better performance than that of the traditional method based on the fog deterioration model.

"Common technologies" of information grand voyage project: Introduction to image and video processing technologies
Miki Haseyama, Toru Hisamitsu
Kyokai Joho Imeji Zasshi/Journal of the Institute of Image Information and Television Engineers, 63, 1, 42, 47, Inst. of Image Information and Television Engineers, 2009
Japanese, 2007年度に3年計画でスタートした情報大航海プロジェクトは,モデルサービスによる実証を通じて次世代の情報検索・解析技術を開発することを目的としている.同時に,実証を通じた制度的課題の洗い出しにより,市場創出に必要な環境整備を目指している.本稿では,開発された技術の中から,マルチメディア情報処理の中核を担う,画像・映像処理技術を紹介する.

Improvement of video coding performance using adaptive interpolation filter without filter selection information
Satoshi Kondor, Takaya Matsuno, Miki Haseyama
Kyokai Joho Imeji Zasshi/Journal of the Institute of Image Information and Television Engineers, 63, 11, 1592, 1597, The Institute of Image Information and Television Engineers, 2009
Japanese, Scientific journal, We propose a method to improve performance of video coding using an adaptive interpolation filter technique. The adaptive interpolation technique was based on clustering using the k-means method and did not need filter selection information. To improve the performance of the clustering, we introduced autocorrelation coefficients of the pixel values and the directions of the motion vectors as the new features and the Mahalanobis distance as the distance scale in the k-means method. We also used vector quantization to reduce the number of interpolation filter coefficients. In the simulation, our proposed method was implemented in the MPEG-2 based video codec. The simulation results show that the proposed method can reduce the bit rate by up to nearly 7 % compared to conventional adaptive interpolation filter methods.

Accurate Graph-Based Scene Segmentation Using Object Matching and Audio Feature
Makoto Yamamoto, Miki Haseyama
ISCE: 2009 IEEE 13TH INTERNATIONAL SYMPOSIUM ON CONSUMER ELECTRONICS, VOLS 1 AND 2, 670, 671, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, A method for accurate scene segmentation utilizing two kinds of directed graph obtained by object matching and by using audio features is proposed. Generally, in audiovisual materials, there are repeated appearances of shots that include frames of the same background, object or place. It should be assumed that such shots are included in a single scene. In the proposed method, by performing object matching between two frames that are each included in different shots, multiple shots for which frames include the same object can be successfully found and their repeated appearances are represented as a directed graph. The proposed method also generates another directed graph that represents the repeated appearances of shots with similar audio features. By the combined use of these two graphs, accurate scene segmentation can be realized. The effectiveness of the proposed method is verified by applying this method to news programs and another broadcast program.

Semantic Image Retrieval Based on POCS Algorithm Using Kernel PCA And Its Performance Verification
Takahiro Ogawa, Miki Haseyama
ISCE: 2009 IEEE 13TH INTERNATIONAL SYMPOSIUM ON CONSUMER ELECTRONICS, VOLS 1 AND 2, 342, 343, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, This paper presents a projection onto convex sets (POCS)-based semantic image retrieval method and its performance verification. The main contributions of the proposed method are twofold: introduction of nonlinear eigenspace of visual and semantic features into the constraint of the POCS-based semantic image retrieval algorithm and adaptive selection of the annotated images utilized for this algorithm. Then, by combining these two approaches., the semantic features of the query image are successfully estimated, and accurate image retrieval can be expected. Finally, relationship between the performance of the proposed method and the kinds of the kernel functions utilized for the kernel PICA is shown in this paper.

A new image retrieval interface and its practical use in "View Search Hokkaido"
Miki Haseyama, Toshifumi Murata, Hisashi Ukawa
ISCE: 2009 IEEE 13TH INTERNATIONAL SYMPOSIUM ON CONSUMER ELECTRONICS, VOLS 1 AND 2, 624, +, IEEE, 2009, [Peer-reviewed]
English, International conference proceedings, New image retrieval technology, which is used for a service demonstration project "View Search Hokkaido" in "Information Grand Voyage Project" conducted by Ministry of Economy, Trade and Industry, Japan, is presented in this paper. The new technology enables image retrieval based on low-level features without utilizing any tag-based scheme and realizes a 3D interface for image retrieval. This 3D interface can provide new user experience, which has not been provided by the conventional retrieval services. The effectiveness of the 3D interface has been verified in "View Search Hokkaido".

映像検索のための画像及び音響信号によるシーン間の相違度算出--正準相関分析とDynamic Time Warpingを用いたアプローチ
道山 大悟, 長谷山 美紀
画像ラボ, 19, 7, 1, 4, 日本工業出版, Jul. 2008
Japanese

Innovation of Video Technology and Visual Information Media
Haseyama Miki
The journal of the Institute of Image Information and Television Engineers, 62, 4, K12, K12, The Institute of Image Information and Television Engineers, 01 Apr. 2008
Japanese

Media distribution and processing
Kazuto Kamikura, Miki Haseyama, Kazuhito Murakami
Kyokai Joho Imeji Zasshi/Journal of the Institute of Image Information and Television Engineers, 62, 8, 1251, 1254, Inst. of Image Information and Television Engineers, 2008
Japanese, Scientific journal

Estimating the number of people in a video sequence via geometrical model.
Hiroyuki Arai, Isao Miyagawa, Hideki Koike, Miki Haseyama
19th International Conference on Pattern Recognition (ICPR 2008)(ICPR), 1, 4, IEEE Computer Society, 2008
International conference proceedings

KERNEL PCA-BASED SEMANTIC FEATURE ESTIMATION APPROACH FOR SIMILAR IMAGE RETRIEVAL
Takahiro Ogawa, Miki Haseyama
2008 IEEE International Conference on Image Processing, Proceedings, 969, 972, IEEE, 2008, [Peer-reviewed]
English, International conference proceedings, A kernel PCA-based semantic feature estimation approach for similar image retrieval is presented in this paper. Utilizing database images previously annotated by keywords, tire proposed method estimates unknown semantic features of a query image. First, our method performs semantic clustering of the database images and derives a new map from a nonlinear eigenspace of visual and semantic features in each c aster. This map accurately provides the semantic features for the images belonging to each cluster by using their visual features. Further, in order to select the optional cluster including the query image, the proposed method monitors errors of the visual features caused by the Semantic feature estimation process. Then, even if any semantics of the query image arc unknown, its semantic features are successfully estimated by tire optimal cluster. Experimental results verify the effectiveness of the proposed method for semantic image retrieval.

カルマンフィルタと適応信号処理, 谷萩隆嗣著, (コロナ社, Tel.03-3941-3131, 2005年, A5判, 282頁, 定価4,515円(税込))
長谷山 美紀
日本音響学会誌, 63, 8, 社団法人日本音響学会, 01 Aug. 2007
Japanese

Audio-based shot classification for audiovisual indexing using PCA, MGD and fuzzy algorithm
Naoki Nitanda, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E90A, 8, 1542, 1548, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2007, [Peer-reviewed]
English, Scientific journal, An audio-based shot classification method for audiovisual indexing is proposed in this paper. The proposed method mainly consists of two parts, an audio analysis part and a shot classification part. In the audio analysis part, the proposed method utilizes both principal component analysis (PCA) and Mahalanobis generalized distance (MGD). The effective features for the analysis can be automatically obtained by using PCA, and these features are analyzed based on MGD, which can take into account the correlations of the data set. Thus, accurate analysis results can be obtained by the combined use of PCA and MGD. In the shot classification part, the proposed method utilizes a fuzzy algorithm. By using the fuzzy algorithm, the mixing rate of the multiple audio sources can be roughly measured, and thereby accurate shot classification can be attained. Results of experiments performed by applying the proposed method to actual audiovisual materials are shown to verify the effectiveness of the proposed method.

Players clustering based on graph theory tor tactics analysis purpose in soccer videos
Hiroftuni Kon, Miki Haseyama
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E90A, 8, 1528, 1533, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Aug. 2007, [Peer-reviewed]
English, Scientific journal, In this paper, a new method for clustering of players in order to analyze games in soccer videos is proposed. The proposed method classifies players who are closely related in terms of soccer tactics into one group. Considering soccer tactics, the players in one group are located near each other. For this reason, the Euclidean distance between the players is an effective measurement for the clustering of players. However, the distance is not sufficient to extract tactics-based groups. Therefore, we utilize a modified version of the community extraction method, which finds community structure by dividing a non-directed graph. The use of this method in addition to the distance enables accurate clustering of players.

Visualization of relationship among academic papers in small database
二反田 直己, 鎌倉 純一, 長谷山 美紀
Journal of signal processing, 11, 2, 179, 185, 〔信号処理学会〕, Mar. 2007
Japanese

Phase retrieval based on a snake for image reconstruction
Keiko Kondo, Miki Haseyama, Hideo Kitajima
IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, E90D, 1, 283, 287, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Jan. 2007
English, Scientific journal, A new phase retrieval method using an active contour model (snake) for image reconstruction is proposed. The proposed method reconstructs a target image by retrieving the phase from the magnitude of its Fourier transform and the measured area of the image. In general, the measured area is different from the true area where the target image exists. Thus a snake, which can extract the shape of the target image, is utilized to renew the measured area. By processing this renewal iteratively, the area obtained by the snake converges to the true area and as a result the proposed method can accurately reconstruct a target image even when the measured area is different from the true area. Experimental results show the effectiveness of the proposed method.

Steady-state properties of a CORDIC-based adaptive ARMA lattice filter
Shin'ichi Shiraishi, Miki Haseyama, Hideo Kitajima
IEICE TRANSACTIONS ON FUNDAMENTALS OF ELECTRONICS COMMUNICATIONS AND COMPUTER SCIENCES, E89A, 12, 3724, 3729, IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, Dec. 2006
English, Scientific journal, This paper analyzes the steady-state properties of a CORDIC-based adaptive ARMA lattice filter. In our previous study, the convergence properties of the filter in the non-steady state were clarified; however, its behavior in the steady state was not discussed. Therefore, we develop a distinct analysis technique based on a Markov chain in order to investigate the steady-state properties of the filter. By using the proposed technique, the relationship between step size and coefficient estimation error is revealed.

基礎シリーズ 線形予測理論とラティスフィルタアルゴリズム(2)ARラティスフィルタの実現とその性質
長谷山 美紀
信号処理, 10, 5, 309, 315, 〔信号処理学会〕, Sep. 2006
Japanese

Image Recognition Method Using Intensity Gradient Vectors
HIRAMOTO Masao, OGAWA Takahiro, HASEYAMA Miki
The IEICE transactions on information and systems (Japanese edetion), 89, 6, 1348, 1358, The Institute of Electronics, Information and Communication Engineers, 01 Jun. 2006
Japanese, 本論文では,撮像素子の多面素化や高画質化の流れを踏まえ,画像の回転・移動等の幾何学的変換にも対応できる大局的な画像識別方法を提案している.提案手法は,ベクトルを利用した投票方式を用いかものであり,画像を輝度こう配を表すベクトルと位置を示すベクトルで表現し,識別のための投票ベクトルと類似度を定義している.また提案手法では,同一画像であれば得票場所が原点に集中し,得票結果が幾何学的な変換に影響されないという特徴がある.原画像に対してガウシアン,メジアンのフィルタリング処理,JPEG圧縮処理を施した画像も含め,自然画像の識別実験を行ったところ,類似性において明確な差が現れ,画像に対して人工的な処理を加えても識別可能であることが分かった.更に提案手法の応用として,最多得票点を利用した画像の識別について検討したところ,識別能力が高く,1画像内に含まれる部分画像の識別も可能であることを示すことができた.

基礎シリーズ 線形予測理論とラティスフィルタアルゴリズム(1)線形予測とARモデル同定
長谷山 美紀
信号処理, 10, 3, 153, 159, 〔信号処理学会〕, May 2006
Japanese

A report on video retrieval for intelligent information access
長谷山 美紀
回路とシステム軽井沢ワークショップ論文集, 19, 199, 203, [電子情報通信学会], 24 Apr. 2006
Japanese

Improvements on Watershed-Based Image Segmentation Using Color Edges and Parallel Region Merging
ZHAO Yanjun, HASEYAMA Miki, KITAJIMA Hideo
The IEICE transactions on information and systems (Japanese edetion), 89, 4, 836, 849, The Institute of Electronics, Information and Communication Engineers, 01 Apr. 2006
Japanese, 画像内容に基づく画像処理では,画素ではなく,画像中の物体を構成する領域を対象とし,画像の圧縮,検索,認識等様々な処理を行う.そのため,領域を検出し,ラベルを付与する画像分割が必要になる.本論文では画像における輝度値の空間分布を地形とみなすWatershedアルゴリズムによる領域分割を検討する.Watershedアルゴリズムにより,閉じた単一の領域分割線が得られるが,分割結果には領域の未分割及び過剰分割という問題が発生する.本論文においてはこれらの問題を解決するために,原画像を直接処理することではなく,連続エッジと均質部分の両方が強調できる色エッジの強度画像を分割対象とする.更に,未分割と過剰分割とを防ぐために,色エッジの強度画像に対する分割結果を初期領域として,ノイズ抑制を考慮した並列実行の領域統合を行う.提案手法を各種の画像へ適用した結果によりその有効性を示す.

A multi-objective service restoration method for power distribution systems
Jun Inagaki, Jun Nakajima, Miki Haseyama
2006 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, VOLS 1-11, PROCEEDINGS, 1784, +, IEEE, 2006, [Peer-reviewed]
English, International conference proceedings, Service restoration problem in distribution systems is formulated as a multi-objective optimization problem which is demanded not only for minimizing the amount of unrestored total loads but also for minimizing the number of the switching operations. The solution of the multi-objective optimization problem is usually obtained with a set of Pareto optimal solutions. The Pareto optimal solutions for the service restoration problem are useful for users to obtain their desired restoration by comparing a Pareto optimal solution with the others. However, the conventional methods cannot obtain several Pareto optimal solutions in one trial. Therefore, this paper proposes a method for obtaining the Pareto optimal set for the service restoration problem with a genetic algorithm. The genetic algorithm produces many possible solutions in its search process. By utilizing this feature, the proposed method can obtain the Pareto optimal set.

Restoration method of missing areas in still images using GMRF model
T Ogawa, M Haseyama, H Kitajima
2005 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), VOLS 1-6, CONFERENCE PROCEEDINGS, 4931, 4934, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, This paper proposes a GMRF-model based restoration method of missing areas in still images. The GMRF model used in the proposed method is realized by a new assumption that reasonably holds for an image source. This model can express important image features such as edges because of the use of the new assumption. Therefore, the proposed method restores the missing areas using the modified GMRF model and can correctly reconstruct the missing edges. Consequently, the proposed method achieves more accurate restoration than those of the traditional methods on both objective and subjective measures. Extensive experimental results demonstrate the improvement of the proposed method over the previous methods.

GA-based applications for routing with an upper bound constraint
J Inagaki, M Haseyama
2005 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), VOLS 1-6, CONFERENCE PROCEEDINGS, 2239, 2242, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, This paper presents a method of searching for the shortest route via the most designated points among the routes whose lengths are less than the upper bound using a genetic algorithm (GA). If chromosomes whose route lengths exceed the upper bound are simply screened out in the GA process, the optimization probability gets worse. For the purpose of solving this problem, this paper proposes a new fitness function including an upper bound constraint which can be flexibly changed during the searching process. By using this function, the optimum is efficiently obtained and the optimization probability can be raised. Furthermore, the effectiveness of the proposed method is verified by the experiments applying it to the actual map data.

Audio signal segmentation and classification for scene-cut detection
N Nitanda, M Haseyama, H Kitajima
2005 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), VOLS 1-6, CONFERENCE PROCEEDINGS, 4030, 4033, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, A scene is regarded as a basic unit of audiovisual material, and thereby the boundaries between two adjacent scenes, which are called scene-cuts, must be detected in advance for audiovisual indexing. This paper proposes a scene-cut detection method. Since scene-cuts are associated with a simultaneous change of visual and audio characteristics, both audio and visual analyses are required for the scene-cut detection. For the audio signal analysis, the proposed method utilizes an audio signal segmentation and classification method using fuzzy c-means clustering, which has been proposed by the authors. For the visual signal analysis, the proposed method utilizes some visual segmentation methods. By using these methods simultaneously, the proposed method can accurately detect the scene-cuts, and thereby it is highly valuable for the preprocessing of the audiovisual indexing. Experimental results performed by applying the proposed method to real audiovisual material are shown to verify its high performance.

Quality improvement technique for JPEG images with fractal image coding
M Takezawa, H Sanada, K Watanabe, M Haseyama
2005 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), VOLS 1-6, CONFERENCE PROCEEDINGS, 6320, 6323, IEEE, 2005, [Peer-reviewed]
English, International conference proceedings, This paper proposes a quality improvement technique for JPEG images by using fractal image coding. JPEG coding is a commonly used standard method of compressing images. However, in its decoded images, quantization noise is sometimes visible in high frequency regions, such as the edges of objects. Hence, in order for the JPEG coding to become a more powerful image-coding method, the JPEG image quality must be improved. Therefore, our method solves this problem by adding the obtained codes by the fractal image coding to improve the image quality. Some simulation results verify that the proposed method achieved higher coding-performance than the traditional JPEG coding.

A GA-based fast search algorithm for realizing efficiency motion compensation
S Asamizu, M Haseyama
8TH WORLD MULTI-CONFERENCE ON SYSTEMICS, CYBERNETICS AND INFORMATICS, VOL VI, PROCEEDINGS, 185, 190, INT INST INFORMATICS & SYSTEMICS, 2004, [Peer-reviewed]
English, International conference proceedings, This paper proposes a fast search technique using a genetic algorithm (GA) [1] for realizing efficiency motion compensation. Previously proposed method[2] that the motion compensation based on table lookup refers to two or more frames with one motion table. Therefore, dispersion of the prediction error of each frame is not only dispersion of the image quality of each prediction image but also influences the search speed of the motion table designing. In this proposed method, the prediction error of each frame is obtained by using the block matching method[3] before the motion table is designed. The weight of each frame is put from the obtained prediction error to the fitness function of the GA, and the GA searches for the optimal motion table. Based on this strategy, the proposed method can search for a motion table more effectively.

New JPEG coding algorithm including fractal image coding for color images
M Takezawa, M Haseyama
8TH WORLD MULTI-CONFERENCE ON SYSTEMICS, CYBERNETICS AND INFORMATICS, VOL VI, PROCEEDINGS, 333, 336, INT INST INFORMATICS & SYSTEMICS, 2004, [Peer-reviewed]
English, International conference proceedings, This paper proposes an effective JPEG coding algorithm using fractal image coding for color images. In the JPEG images, quantization noise is sometimes visible in high frequency regions, such as the edges of objects. Hence, in order for the JPEG coding to become a more powerful image-coding method, the JPEG image quality must be improved. Therefore, in this paper, a quality improvement method for the color JPEG images is proposed by using the fractal image coding. Some simulation results verify that the proposed method can improve the image quality. in the high frequency regions and provide the decoded image with 0.4 dB higher quality than the traditional JPEG.

A trainable retrieval system for cartoon character images
M Haseyama, A Matsumura
2003 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL III, PROCEEDINGS, 673, 676, IEEE, 2003, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel method to retrieve cartoon character images in a database or network. In this method, partial features of an image, defined as Regions and Aspects, are used as keys to identify cartoon character images. The similarities between a query cartoon character image and the images in the database are computed by using these features. Based on the similarities the cartoon images same or similar to the query image are identified and retrieved from the database. Moreover, our method adopts a training scheme to reflect the user's subjectivity. The training emphasizes the signficant Regions or Aspects by assigning more weight based on the user's preferences and actions, such as selecting a desired image or an area of an image. These processes make the retrieval more effective and accurate. Experiment results verify the effectiveness and retrieval accuracy of the method.

2-D functional AR model for image identification
M Haseyama, Kondo, I
2003 INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, VOL II, PROCEEDINGS, 377, 380, IEEE, 2003, [Peer-reviewed]
English, International conference proceedings, This paper proposes a 2-D Functional AR Model for image identification. The definition of the proposed model includes functions that can exploit the self-similarity nature in images to throughly extract image features. By introducing the functional scheme into the model, only a few number of parameters, which are called 2-D Functional AR parameters, can describe the image features simply and accurately. These characteristics make the model suitable for image identification applications. Some experiments of image identification are performed, and the results verify that the proposed model accurately represents the image feature, and the image can be correctly, identified. The calculation time is fast enough for practical use in image retrieval.

E cient fixed-valued and random-valued impulse detection for accurate image restoration
K Kondo, M Haseyama, H Kitajima
ISPA 2003: PROCEEDINGS OF THE 3RD INTERNATIONAL SYMPOSIUM ON IMAGE AND SIGNAL PROCESSING AND ANALYSIS, PTS 1 AND 2, 1009, 1012, UNIV ZAGREB, FACULTY MECHANICAL ENGINEERING & NAVAL ARCHITECTURE, 2003, [Peer-reviewed]
English, International conference proceedings, This paper proposes a novel impulse detection method for the restoration of images corrupted by impulse noise. Conventional impulse detection methods tend to work well for fixed-valued impulse noise but poorly for random-valued impulse noise. The proposed method can accurately detect not only fixed-valued but also random-valued impulse noise by using two different systems. The first system detects impulse noise by considering the di erences between the intensity of a target pixel and the output of a median filter The second system verifies whether the impulse detection results obtained by the first system are correct. By using these systems, the proposed method can accurately detect both types of impulse noise even in highly corrupted images. Furthermore, the proposed method can be e ectively used as a preprocessor for noise reduction filtering. Experiments are presented to demonstrate the e ectiveness of the proposed method.

Two-dimensional analysis of magnetic microstructures in the DC-demagnetized state and magnetization fluctuations in the transition region using MFM images
Takekuma, I, M Haseyama, K Sueoka, K Mukasa
JOURNAL OF MAGNETISM AND MAGNETIC MATERIALS, 239, 1-3, 359, 362, ELSEVIER SCIENCE BV, Feb. 2002, [Peer-reviewed]
English, Scientific journal, This study analyzed the directional dependence of magnetic microstructures of media by focusing on the distribution of magnetic poles. Media, whose magnetic poles tend to be distributed along a specific direction in the DC-demagnetized state, have larger magnetization fluctuations in the transition region, larger medium noise and higher partial erasure (PE) probability than media with uniform magnetic pole distribution. (C) 2002 Elsevier Science B.V. All rights reserved.

A new approach with IFS for image restoration.
Miki Haseyama, Megumi Takezawa, Junichi Miura, Hideo Kitajima
10th European Signal Processing Conference(EUSIPCO), 1, 4, IEEE, 2000
International conference proceedings

A practical method to reduce a number of reference signals for the ANC system
M Akiho, M Haseyama, H Kitajima
ICASSP '99: 1999 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, PROCEEDINGS VOLS I-VI, 2387, 2390, IEEE, 1999, [Peer-reviewed]
English, International conference proceedings, In this paper, we propose a practical method to reduce a number of reference signals for the active noise cancellation (ANC) system and the filter characteristics to generate the reduced number of reference signals, which maintain the original value of the coherence function. This method finds the number of independent noise sources and provides the filter characteristics based on SVD (singular value decomposition) of the power spectrum matrix of the reference signals. Then, we also use the multiple coherence function analysis to select dominant components in the reference signals. The method contributes greatly in reducing the number of reference signals for the ANC system that uses the large number of reference signals. We also discuss the characteristics of the filters that synthesis the new set of reference signals. And an experimental test is performed to confirm the theory.

A genetic algorithm based image segmentation for image analysis
M Haseyama, M Kumagai, H Kitajima
ICASSP '99: 1999 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, PROCEEDINGS VOLS I-VI, 3445, 3448, IEEE, 1999, [Peer-reviewed]
English, International conference proceedings, In this paper a new genetic algorithm (GA) based image segmentation method is proposed for image analysis. This method using a mean square error (MSE) based criterion can segment an image into some regions, while estimating a suitable region representation. The criterion is defined as MSE caused by interpolating each region of an observed image with a parametric model. Since the criterion is expressed with not only the parameters of the model but also shape and location of the regions, the criterion can not be easily minimized by the usual optimization methods, the proposed method minimizes the criterion by a GA. The proposed method also includes a processor to eliminate fragile regions with the Markov random field (MRF) model. Though the thresholds of the existent methods negatively affect image segmentation results; since no thresholds are required in the proposed method, it segments images more accurately than the existent methods.

A realization method of an ARMAX lattice filter
M HASEYAMA, T HIROHKU, H KITAJIMA
1995 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, VOLS 1-3, 365, 368, I E E E, 1995, [Peer-reviewed]
English, International conference proceedings

FREQUENCY-WEIGHTING MODEL IDENTIFICATION WITH AN ADAPTIVE ARMA LATTICE FILTER
M HASEYAMA, N NAGAI, N MIKI
IEEE INTERNATIONAL CONFERENCE ON SYSTEMS ENGINEERING, 543, 546, I E E E, 1992, [Peer-reviewed]
English, International conference proceedings

A Note on Classification of Expert-novice Level Using Motion and Gaze Data via Spatial Temporal Attention GCN
清野竜生, 斉藤直輝, 小川貴弘, 浅水仁, 長谷山美紀, 人工知能学会全国大会論文集(Web), 38th, 2024

Estimation of contact accident risk based on recurrent neural network introducing spatial-temporal attention
五箇亮太, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

Zero-shot high-risk situation detection based on object detection and pose estimation using fixed camera at construction site
大羽賀駿也, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

Prediction of event locations from urgent call using large language models
吉田将規, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 5, 1, 2024

サッカー映像における時空間的関係を考慮したシュート予測の高精度化に関する検討 : 競技者のチーム情報に基づく完全二部グラフの導入—A Note on Accurate Shoot Prediction Considering Players' Spatio-temporal Relations in Soccer Videos : Introduction of Complete Bipartite Graph Based on Players' Team Information—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
五箇 亮太, 諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 23, 28, Feb. 2023
東京 : 映像情報メディア学会, Japanese

クラス情報を導入したグラフ表現による教師有り潜在変数モデルの高精度化に関する検討—A Note on Improvement of Supervised Latent Variable Model with Graph-Encoded Class Information—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
渡部 航史, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 29, 33, Feb. 2023
東京 : 映像情報メディア学会, Japanese

属性情報の階層関係を考慮したアニメイラストのマルチラベル分類に関する検討—A Note on Multi-label Image Classification in Animation Illustration Considering Hierarchical Relationships of Attributes—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
蘭 子文, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 5, 9, Feb. 2023
東京 : 映像情報メディア学会, Japanese

ユーザの視線情報を考慮したコンテンツベースの画像再検索に関する検討—A note on gaze-dependent image re-ranking for content-based image retrieval—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
馮 鈺虎, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 47, 6, 89, 93, Feb. 2023
東京 : 映像情報メディア学会, Japanese

A Note on Classification of Proficiency-level Using Motion Data though Spatial Temporal Attention Graph Convolutional Network
清野竜生, 斉藤直輝, 小川貴弘, 浅水仁, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2023, 2023

Acquisition of feature representation of record data via graph neural network to support determination of deterioration levels
山本一輝, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Distress detection using egocentric videos for increasing discovery rate of novel distress during subway tunnel inspection
櫻井慶悟, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Classification of Winter Road Surface Condition Based on Multi-modal Transformer Using Sequential Data
諸戸祐哉, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 3, 2023

Efforts of Hokkaido university to develop data-driven interdisciplinary research
長谷山美紀, 水谷祐輔, 田中晋吾, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

DEGRADATION DEGREE ESTIMATION OF DISTRESS IMAGE IN ROAD INFRASTRUCTURE USING MULTI-DATASET CONTRASTIVE LEARNING
東孝明, 小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 4, 2, 2023

A Note on Multi-label Image Classification in Animation Illustration Considering Hierarchical Relationships of Attributes
LAM Ziwen, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Text Prompt Tuning in Cross-modal Image Retrieval for a Specific Database
ZHANG Huaying, 柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Improvement of Binauralization Performance Based on Multi-view Learning on 360° Videos
吉田将規, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Improvement of Supervised Latent Variable Model with Graph-Encoded Class Information
渡部航史, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A note on gaze-dependent image re-ranking for content-based image retrieval
FENG Yuhu, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Accurate Shoot Prediction Considering Players’ Spatio-temporal Relations in Soccer Videos-Introduction of Complete Bipartite Graph Based on Players’ Team Information-
五箇亮太, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

A Note on Interpretable Visual Question Answering Model with Visual Representation Based on Object Detection Model
ZHU He, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 47, 6(MMS2023 1-34/ME2023 21-54/AIT2023 1-34), 2023

知識蒸留を用いたFew-shot Learningに基づく画像の感情ラベル推定に関する検討—A Note on Visual Sentiment Prediction Based on Few-shot Learning Using Knowledge Distillation—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
叶 穎睿, 諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 171, 175, Feb. 2022
Japanese

Variational Autoencoderに基づく深層生成モデルを用いた潜在表現のDisentanglementに関する検討 : Disentanglement評価指標を含む正則化損失の導入—A Note on Disentanglement Using Deep Generative Model Based on Variational Autoencoder : Introduction of Regularization Losses Based on Metrics of Disentangled Representation—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
中川 真, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 97, 102, Feb. 2022
Japanese

Captioning特徴を利用したグラフ畳み込みネットワークに基づくアニメイラストのマルチラベル画像分類に関する検討—A Note on Multi-label Image Recognition in Anime Illustration Based on Graph Convolutional Networks Using Captioning Features—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
蘭 子文, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 161, 165, Feb. 2022
Japanese

ゴム材料開発のためのGenerative Adversarial Networkに基づく配合量および物性からの電子顕微鏡画像の生成に関する一検討—A Note on Electron Microscope Image Generation from Mix Proportion and Material Property via Generative Adversarial Network for Rubber Materials—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
柳 凛太郎, 藤後 廉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 187, 191, Feb. 2022
Japanese

セマンティックセグメンテーションに対するマルチソースモデル適応に関する検討 : 複数のソースモデルからの不変な特徴表現の学習による適応精度の向上—A note on multi-source model adaptation for semantic segmentation : Improving adaptation performance by learning model-invariant representation from multiple source models—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
李 宗曜, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 37, 41, Feb. 2022
Japanese

地下鉄トンネル点検時の技術者から取得される生体信号と技術者の点検行動の関連性分析—Relevance Analysis between Bio-signals of Engineers Inspecting Subway Tunnels and Their Inspection Behaviors—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平澤 魁人, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 365, 370, Feb. 2022
Japanese

高速道路の遮音壁画像を用いた物体検出手法による変状分類の高精度化に関する検討—A note on improvement of distress classification using noise barrier images on highway via object detection method—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
梁 鋆, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 359, 363, Feb. 2022
Japanese

道路構造物の維持管理効率化に向けた変状画像分類の高精度化に関する検討 : テキストデータに基づく類似事例の含有率の導入—A Note on Improvement of Accuracy in Classification of Destress Images for Efficient Inspection of Road Structures : Introduction of Ratio of Similar Cases Based on Text Data—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平川 泰成, 小川 直輝, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 43, 48, Feb. 2022
Japanese

地下鉄トンネルの維持管理支援のためのマルチスケール解析を導入した深層学習に基づく変状検出に関する検討—A Note on Distress Detection Based on Deep Learning with Hierarchical Multi-Scale Attention Mechanism for Supporting Maintenance of Subway Tunnels—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
高田 紗弥, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 377, 381, Feb. 2022
Japanese

A Note on Automatic Diagnosis of Helicobacter Pylori Infection Based on Self-Supervised Learning and Self-Knowledge Distillation: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, マベ カツヒロ, ニシダ シュンペイ, トモダ ヨシヒロ, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 49, 52, Feb. 2022
English

A Note on Realizing Adversarial Defense Based on Regularization of Multi-stage Squeeze-and-Excitation Features: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
マエダ ケイスケ, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 87, 90, Feb. 2022
English

ユーザの動作情報を用いたコンテンツの関心度推定に関する検討 : 複数ユーザを導入した特徴統合の有効性検証—A Note on Interest level Estimation Using Users' behavior Information : Validating the Effectiveness of Feature Integration with Multiple Users—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
上川 恭平, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 103, 107, Feb. 2022
Japanese

画像注視時の脳活動信号を用いた圧縮再構成ネットワークに基づく視覚認知内容の推定に関する検討—A Note on Perceived Visual Content Estimation Based on Compressed Reconstruction Network Using Brain Signals While Gazing on Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
東 孝明, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 349, 353, Feb. 2022
Japanese

Attention mapに対する確信度を考慮可能な深層学習を用いた変状分類の高精度化に関する検討—A Note on Accurate Distress Classification Using Deep Learning Considering Confidence in Attention map—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
小川 直輝, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 371, 376, Feb. 2022
Japanese

A Note on Transformer-based Scene Recognition in Soccer Videos Using Different Lengths of Clips: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 46, 6, 167, 170, Feb. 2022
English

画像中の物体情報を考慮したユーザ類似度に基づく個人に特化した注視領域の推定に関する検討—A Note on Personalized Saliency Prediction Based on User Similarity Considering Object Information in Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 181, 186, Feb. 2022
Japanese

橋梁点検時の技術者の一人称および三人称視点映像を用いた点検動作の分類に関する検討—A Note on Inspection Action Classification Using First and Third Person Video of Engineers Inspecting Bridges—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
増田 毅, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 177, 180, Feb. 2022
Japanese

ユーザの嗜好を考慮した強化学習と知識グラフに基づく楽曲プレイリスト生成に関する検討—A Note on User Preference-Aware Music Playlist Generation Based on Reinforcement Learning and Knowledge Graph—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
櫻井 慶悟, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 46, 6, 109, 112, Feb. 2022
Japanese

AIの実社会応用とインフラ分野の将来像
長谷山美紀, 高速道路と自動車, 65, 12, 2022

DETERIORATION LEVEL CLASSIFICATION OF DISTRESS IMAGE CONSIDERING CORRELATION AMONG HETEROGENEOUS FEATURES AND ATTENTION MAP CONFIDENCE
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 3, J2, 2022

WINTER ROAD SURFACE CONDITION CLASSIFICATION USING DEEP LEARNING WITH FOCAL LOSS BASED ON TEXT AND IMAGE INFORMATION
諸戸祐哉, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 3, J2, 2022

DISTRESS DETECTION BASED ON VISION TRANSFORMER USING EGOCENTRIC VIDEOS WHILE INSPECTING IN SUBWEY TUNNELS
櫻井慶悟, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 3, J2, 2022

CLASSIFICATION OF INSPECTION ACTION OF ENGINEERS FOR EFFICIENT VIDEO PRESENTATION TO ASSIST ASSESSMENT OF INFRASTRUCTURE FACILITY DISTRESS
上川恭平, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, AI・データサイエンス論文集(Web), 3, J2, 2022

A note on multi-source model adaptation for semantic segmentation-Improving adaptation performance by learning model-invariant representation from multiple source models-
LI Zongyao, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on User Preference-Aware Music Playlist Generation Based on Reinforcement Learning and Knowledge Graph
櫻井慶悟, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Disentanglement Using Deep Generative Model Based on Variational Autoencoder-Introduction of Regularization Losses Based on Metrics of Disentangled Representation-
中川真, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note of the Relationship between Popularity Bias and Embedding Representations of Latent Factor Models in Collaborative Filtering
岡村洋希, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Robust Recommendation System for Domain-dependent Preference Based on Domain-shared Network
山本一輝, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Image Retrieval Robust for Changing Camera Views Using Synthesized Images by pixelNeRF
江良勇輝, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Study on Non-reference Image Quality Assessment Considering Phaseless Components of Neural Radiance Fields
河合雅斗, 柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Uncertainty-based Shoot Event Prediction from Scouting Videos of Soccer by Considering Spatio-Temporal Relations between Players
五箇亮太, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Estimation of Viewed Images Using Brain Activity Data While Viewing Images-A Verification of Estimation Accuracy by Regression Model Used for fMRI Decoder-
七田亮, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Estimation of Diabetic Retinopathy Grades Based on Unsupervised Domain Adaptation Using Fundus Images
國枝翼, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2022, 2022

A Note on Electron Microscope Image Generation from Mix Proportion and Material Property via Generative Adversarial Network for Rubber Materials
柳凜太郎, 藤後廉, 前田圭介, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Improvement of Accuracy in Classification of Destress Images for Efficient Inspection of Road Structures- Introduction of Ratio of Similar Cases Based on Text Data-
平川泰成, 小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Distress Detection Based on Deep Learning with Hierarchical Multi-Scale Attention Mechanism for Supporting Maintenance of Subway Tunnels
高田紗弥, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Perceived Visual Content Estimation Based on Compressed Reconstruction Network Using Brain Signals While Gazing on Images
東孝明, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Interest level Estimation Using Users’ behavior Information-Validating the Effectiveness of Feature Integration with Multiple Users-
上川恭平, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Visual Sentiment Prediction Based on Few-shot Learning Using Knowledge Distillation
YE Yingrui, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

Relevance Analysis between Bio-signals of Engineers Inspecting Subway Tunnels and Their Inspection Behaviors
平澤魁人, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Personalized Saliency Prediction Based on User Similarity Considering Object Information in Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

A Note on Accurate Distress Classification Using Deep Learning Considering Confidence in Attention map
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 46, 6(MMS2022 1-37/ME2022 26-62/AIT2022 1-37), 2022

AIの実社会応用と次世代インフラ維持管理
長谷山 美紀, 学士会会報, 2022, 1, 78, 83, Jan. 2022
学士会, Japanese

招待講演 マルチメディアAI技術に基づく異分野融合研究と実社会応用—Invited Talk : Interdisciplinary Research and Real-world Applications Based on Multimedia AI technologies—情報センシング メディア工学 映像・信号の処理・解析・AI技術とその多分野応用
小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 13, 73, 80, Jun. 2021
映像情報メディア学会, Japanese

A Note on Distress Classification of High-resolution Subway Tunnel Images based on HRNet: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 13, 16, Feb. 2021
English

地下鉄トンネルの維持管理支援を目的とした深層学習に基づく変状検出の高精度化に関する検討 : 壁面の施工方法に注目した精度検証—A Note on Improving Performance of Deep Learning-based Distress Detection for Supporting Maintenance of Subway Tunnels : Accuracy Verification Focusing on Tunnel Wall Characteristics—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
春山 知生, 前田 圭介, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 1, 6, Feb. 2021
映像情報メディア学会, Japanese

特別講演 実店舗の防犯カメラ映像データを用いた顧客の関心推定に関する検討 : 姿勢推定モデルから得られる特徴量を用いた異常検知モデルの精度検証—A Note on Customer's Interest Estimation Method Using Security Camera Video Data from the Real Store : Validation of the accuracy of an anomaly detection model using features obtained from a posture estimation model—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
本間 勇紀, 藤後 廉, 阿部 真育, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 193, 198, Feb. 2021
映像情報メディア学会, Japanese

地下鉄トンネル点検時の技術者から取得される視線およびモーションデータに基づく熟練度の推定に関する検討—A Note on Estimation of Expert-Novice Level Based on Eye Tracking and Motion Data from Engineers While Inspecting Subway Tunnel—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
赤松 祐亮, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 7, 12, Feb. 2021
映像情報メディア学会, Japanese

特別講演 ゴム材料開発のためのConditional StyleGANに基づく配合量からの電子顕微鏡画像の生成に関する一検討—A Note on Electron Microscope Image Generation from Mix Proportion via Conditional Style Generative Adversarial Network for Rubber Materials—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
柳 凜太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 171, 175, Feb. 2021
映像情報メディア学会, Japanese

特別講演 路面画像を用いた深層学習に基づく路面状態の分類に関する検討—A Note on Discrimination of Road Surface Conditions Based on Deep Learning Using Road Images—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 165, 169, Feb. 2021
映像情報メディア学会, Japanese

TCCに基づく自己教師学習による特徴表現を用いた映像中の人物動作の検出に関する検討 : 動作検出パラメータが与える影響に対する考察—A Note on Action Detection Using Feature Representation via Self-Supervised Learning based on TCC : Effectiveness evaluation of parameters for action detection—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
増田 毅, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 47, 51, Feb. 2021
映像情報メディア学会, Japanese

電子顕微鏡により撮像されたゴム材料からの異常検知に基づく劣化領域の推定に関する一検討 : 深層学習モデルにより抽出された特徴表現の有効性検証—A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images : Verification of Feature Representations Extracted from Deep Learning Models—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
松本 真直, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 43, 46, Feb. 2021
映像情報メディア学会, Japanese

サッカー映像における情報の提示位置の推定に関する検討 : 視聴者の注視位置および選手位置に基づいたオブジェクト配置の決定—A Note on Position Estimation of Superimposing Information in Soccer Videos : Decision of Object Placement Based on User's Gaze Position and Players' Positions—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 77, 81, Feb. 2021
映像情報メディア学会, Japanese

A Note on Automatic Diagnosis of Helicobacter Pylori Infection Based on EfficientNet with Flooding Loss: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, マベ カツヒロ, ニシダ シュンペイ, トモダ ヨシヒロ, シミズ ヒカリ, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 23, 26, Feb. 2021
English

Twitterと映像を用いたMVAEに基づく野球映像の重要シーン予測に関する検討—A Note on Prediction of Important Scenes in Baseball Videos via Multimodal Variational Autoencoder Using Tweets and Videos—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
平澤 魁人, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 71, 75, Feb. 2021
映像情報メディア学会, Japanese

Detecting axillary lymph node metastasis of breast cancer with FDG-PET/CT images based on attention mechanism: FDG-PET/CT画像を用いたattention mechanismに基づく乳癌の腋窩リンパ節転移の検出: マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
トウゴ レン, ヒラタ ケンジ, キタジマ カズヒロ, タケナカ ジュンキ, ミヨシ ヤスオ, クドウ コウスケ, オガワ タカヒロ, ハセヤマ ミキ, 45, 4, 33, 36, Feb. 2021
English

特別講演 ユーザの嗜好を反映可能なインテリアコーディネート検索技術の構築 : コーディネートを表現可能な特徴の抽出と実データへの適用—A Note on Interior Coordination Retrieval Reflecting User's Preferences : Extraction of Features Representing Coordination and Application to Real Data—マルチメディアストレージ メディア工学 映像表現&コンピュータグラフィックス
藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 45, 4, 187, 191, Feb. 2021
映像情報メディア学会, Japanese

A Note on Off-screen Sound Detection Based on Audio-visual Spatialization-Introduction of Audio-visual Features Based on Self-supervised Learning-
吉田将規, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Effect of Visual Representation Ability to Performance of World Models
大羽賀駿也, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Verification of Vulnerability of Graph-based Recommender Models under Shilling Attacks
小野寺望, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Multi-label Image Recognition in Anime Illustration Based on Graph Convolutional Networks
LAN Ziwen, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Introducing Self-supervised Learning to Latent Variable Model for Low-dimensional Feature Extraction
渡部航史, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

An Note on Use of Multiple Datasets in Visual Sentiment Prediction Using Few-shot Learning
YE Yingrui, 諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Accuracy Assessment Focusing on Manipulation Area by Text-Guided Image Manipulation
渡邉優宇人, 藤後廉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2021, 2021

A Note on Estimation of Semantic Content Based on a Question Answering Model Using Brain Activity Data while Viewing Images-Improvement of Estimation Performance Based on fine-tuning of the VQA model-
高田紗弥, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 27, 31, 2021
映像情報メディア学会, Japanese

A Note on Improvement of Image Sentiment Analysis Based on Introduction of Image Captioning
LIANG Yun, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 65, 69, 2021
映像情報メディア学会, Japanese

A Note on Cross-domain Recommendation Based on Multi-layer Graph Analysis with Visual Features
平川泰成, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 59, 63, 2021
映像情報メディア学会, Japanese

A Note on Improving Performance of Deep Learning-based Distress Detection for Supporting Maintenance of Subway Tunnels-Accuracy Verification Focusing on Tunnel Wall Characteristics-
春山知生, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Customer’s Interest Estimation Method Using Security Camera Video Data from the Real Store-Validation of the accuracy of an anomaly detection model using features obtained from a posture estimation model-
本間勇紀, 本間勇紀, 藤後廉, 阿部真育, 小川貴弘, 長谷山美紀, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Estimating of Deteriorated Refions Based on Anomaly Detection from Rubber Material Electron Microscope Images-Verification of Feature Representations Extracted from Deep Learning Models-
松本真直, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Estimation of Expert-Novice Level Based on Eye Tracking and Motion Data from Engineers While Inspecting Subway Tunnel
赤松祐亮, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Action Detection Using Feature Representation via Self-Supervised Learning based on TCC-Effectiveness evaluation of parameters for action detection-
増田毅, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Discrimination of Road Surface Conditions Based on Deep Learning Using Road Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Position Estimation of Superimposing Information in Soccer Videos-Decision of Object Placement Based on User’s Gaze Position and Players’ Positions-
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Prediction of Important Scenes in Baseball Videos via Multimodal Variational Autoencoder Using Tweets and Videos
平澤魁人, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Interior Coordination Retrieval Reflecting User’s Preferences-Extraction of Features Representing Coordination and Application to Real Data-
藤後廉, 小川貴弘, 長谷山美紀, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

A Note on Accurate Distress Image Classification of Road Structures Using Attention Map Based on Text Data
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 17, 21, 2021
映像情報メディア学会, Japanese

A Note on Electron Microscope Image Generation from Mix Proportion via Conditional Style Generative Adversarial Network for Rubber Materials
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 4(MMS2021 1-28/ME2021 1-28/AIT2021 1-28), 2021

AIを中心とした医療デジタル技術基盤の構築へ向けた取り組み
藤後廉, 小川貴弘, 長谷山美紀, 日本消化器がん検診学会雑誌(Web), 59, Supplement 1, 2021

Interdisciplinary Research and Real-world Applications Based on Multimedia AI technologies
小川貴弘, 長谷山美紀, 映像情報メディア学会技術報告, 45, 13(IST2021 22-29/ME2021 53-60), 2021

The Clinical Application of Artificial Intelligence for Detecting Atrophic Gastritis by Upper Gastrointestinal Barium X-ray Radiography
竹内千尋, 藤後廉, 小川貴弘, 長谷山美紀, 山道信毅, 臨床消化器内科, 36, 8, 1062, 1065, 2021
日本メディカルセンター

データサイエンスのあるべき姿とは : 高等教育・研究機関における取り組みを通じての意見交換会—特集 データサイエンス教育の潮流(その1)
木島 正明, 竹村 彰通, 長谷山 美紀, 椿 広計, 蓮池 隆, オペレーションズ・リサーチ = Communications of the Operations Research Society of Japan : 経営の科学, 65, 10, 525, 536, Oct. 2020
日本オペレーションズ・リサーチ学会 ; 1956-, Japanese

メディア工学研究委員会—Technical Group on Media Engineering—特集 学会創立70周年記念特集 ; 映像情報メディア学会の研究会活動 : 過去から未来へ
村上 和人, 長谷山 美紀, 田川 憲男, 田良島 周平, 石井 大祐, 映像情報メディア学会誌 = The journal of the Institute of Image Information and Television Engineers, 74, 5, 779, 783, Sep. 2020
映像情報メディア学会, Japanese

画像内の物体に着目した画像検索に関する検討 : RetinaNetを用いた物体認識に基づく高精度化—A Note on Image Retrieval Focusing on Objects in Images : Improving Retrieval Performance Based on Object Recognition Using RetinaNet—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
柳 凛太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 377, 381, Feb. 2020
映像情報メディア学会, Japanese

画像注視時の視線データを用いた関心度の推定に関する検討 : ラベルの逆量子化を導入したsMVCCAによる推定精度の高精度化—A Note on Eye Gaze Based User-specific Interest Estimation for Images : Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization—ITS : Intelligent Transport Systems Technology
松本 真直, 斉藤 直輝, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 73, 77, Feb. 2020
電子情報通信学会, Japanese

画像注視時の視線データを用いた関心度の推定に関する検討 : ラベルの逆量子化を導入したsMVCCAによる推定精度の高精度化—A Note on Eye Gaze Based User-specific Interest Estimation for Images : Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松本 真直, 斉藤 直輝, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 73, 77, Feb. 2020
映像情報メディア学会, Japanese

地下鉄トンネル点検時の生体信号に基づいた熟練および若手技術者の分類に関する検討—A Note on Classification of Experienced and Novice Inspectors Based on Bio-signals While Inspecting in Subway Tunnels—ITS : Intelligent Transport Systems Technology
九島 哲哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 101, 105, Feb. 2020
電子情報通信学会, Japanese

特別講演 社会インフラ維持管理効率化のための最先端AI技術の導入 : 点検データを用いた変状分類の精度向上に向けた取り組み—Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure : Performance Improvement of Distress Image Classification Utilizing Inspection Data—ITS : Intelligent Transport Systems Technology
前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 359, 362, Feb. 2020
電子情報通信学会, Japanese

道路構造物の変状評価における技術者の視線データと熟練度の分析に関する一考察—A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松井 太我, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 97, 100, Feb. 2020
映像情報メディア学会, Japanese

道路構造物の維持管理効率化に向けた変状画像分類における信頼性の向上に関する検討—A Note on Improvement of Reliability of Distress Image Classification for Efficient Maintenance of Road Structures—ITS : Intelligent Transport Systems Technology
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 51, 56, Feb. 2020
電子情報通信学会, Japanese

画像内の物体に着目した画像検索に関する検討 : RetinaNetを用いた物体認識に基づく高精度化—A Note on Image Retrieval Focusing on Objects in Images Improving Retrieval Performance Based on Object Recognition Using RetinaNet—ITS : Intelligent Transport Systems Technology
柳 凜太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 377, 381, Feb. 2020
電子情報通信学会, Japanese

弱異常検知アルゴリズムに基づくCCTV映像を用いた河川利用者の異常行動の検出に関する検討—A Note on Abnormal Motion Detection of River Users on CCTV Videos Based on Weakly Supervised Anomaly Detection Algorithm—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
渡邊 はるな, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 367, 370, Feb. 2020
映像情報メディア学会, Japanese

A note on retrieval of visually similar distress regions in subway tunnel images : Introduction of deep features extracted by semantic segmentation network: ITS : Intelligent Transport Systems Technology
トウゴ レン, オガワ タカヒロ, ハセヤマ ミキ, 119, 421, 65, 68, Feb. 2020
English

画像注視時のヒトの感情推定のための視線特徴の推定に関する検討—A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 85, 89, Feb. 2020
映像情報メディア学会, Japanese

ゴム材料の配合量を用いたAC-GANに基づく電子顕微鏡画像の生成に関する一検討—A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions—ITS : Intelligent Transport Systems Technology
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 107, 111, Feb. 2020
電子情報通信学会, Japanese

ゴム材料の配合量を用いたAC-GANに基づく電子顕微鏡画像の生成に関する一検討—A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 107, 111, Feb. 2020
映像情報メディア学会, Japanese

道路構造物の変状評価における技術者の視線データと熟練度の分析に関する一考察—A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures—ITS : Intelligent Transport Systems Technology
松井 太我, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 97, 100, Feb. 2020
電子情報通信学会, Japanese

エッジの補完を導入したツイートクラスタリングの高精度化に関する検討—A Note on Tweet Clustering by Complementing Edges among Tweets—ITS : Intelligent Transport Systems Technology
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 61, 64, Feb. 2020
電子情報通信学会, Japanese

地下鉄トンネル維持管理支援を目的とした類似画像の検索に関する検討 : 技術者の評価を反映可能な距離計量学習の導入—A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels : Introduction of Distance Metric Learning Reflecting Inspectors' Evaluation—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
松本 有衣, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 47, 50, Feb. 2020
映像情報メディア学会, Japanese

特別講演 社会インフラ維持管理効率化のための最先端AI技術の導入 : 点検データを用いた変状分類の精度向上に向けた取り組み—Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure : Performance Improvement of Distress Image Classification Utilizing Inspection Data—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 359, 362, Feb. 2020
映像情報メディア学会, Japanese

地下鉄トンネル維持管理支援を目的とした類似画像の検索に関する検討 : 技術者の評価を反映可能な距離計量学習の導入—A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels : Introduction of Distance Metric Learning Reflecting Inspectors' Evaluation—ITS : Intelligent Transport Systems Technology
松本 有衣, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 47, 50, Feb. 2020
電子情報通信学会, Japanese

穿孔データを用いたオンライン学習に基づく岩盤の圧縮強度指数推定に関する検討—A Note on Estimation of rock compressive strength index from drilling data based on online learning—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
山本 健太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 57, 60, Feb. 2020
映像情報メディア学会, Japanese

穿孔データを用いたオンライン学習に基づく岩盤の圧縮強度指数推定に関する検討—A Note on Estimation of rock compressive strength index from drilling data based on online learning—ITS : Intelligent Transport Systems Technology
山本 健太郎, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 57, 60, Feb. 2020
電子情報通信学会, Japanese

サッカー映像視聴時の視線データを用いた情報の提示領域の算出に関する検討—Presentation Region Calculation of Information Based on Gaze Tracking Data in Soccer Videos—ITS : Intelligent Transport Systems Technology
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 91, 95, Feb. 2020
電子情報通信学会, Japanese

地下鉄トンネル点検時の生体信号に基づいた熟練および若手技術者の分類に関する検討—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
九島 哲哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 101, 105, Feb. 2020
映像情報メディア学会, Japanese

Bidirectional LSTMを用いたサッカー映像におけるイベント分類に関する検討—A Note on Event Classification in Soccer Videos Using Bidirectional LSTM—ITS : Intelligent Transport Systems Technology
春山 知生, 高橋 翔, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 371, 375, Feb. 2020
電子情報通信学会, Japanese

画像注視時のヒトの感情推定のための視線特徴の推定に関する検討—A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images—ITS : Intelligent Transport Systems Technology
諸戸 祐哉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 119, 421, 85, 89, Feb. 2020
電子情報通信学会, Japanese

道路構造物の維持管理効率化に向けた変状画像分類における信頼性の向上に関する検討—A Note on Improvement of Reliability of Distress Image Classification for Efficient Maintenance of Road Structures—マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 51, 56, Feb. 2020
映像情報メディア学会, Japanese

A Note on Tweet Clustering by Complementing Edges among Tweets
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 61, 64, Feb. 2020
映像情報メディア学会, Japanese

A Note on Estimation of Image Categories Using Brain Activity While Viewing Images Based on MVBGM-MS
赤松 祐亮, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 79, 83, Feb. 2020
映像情報メディア学会, Japanese

サッカー映像視聴時の視線データを用いた情報の提示領域の算出に関する検討 (マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス)
鈴木 元樹, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 91, 95, Feb. 2020
映像情報メディア学会, Japanese

A Note on Event Classification in Soccer Videos Using Bidirectional LSTM
春山 知生, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 44, 6, 371, 375, Feb. 2020
映像情報メディア学会, Japanese

A Note on Tweet Clustering by Complementing Edges among Tweets
滝村祥司, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Event Classification in Soccer Videos Using Bidirectional LSTM
春山知生, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

Introduction of Advanced AI Technology for Efficient Maintenance Inspection of Social Infrastructure-Performance Improvement of Distress Image Classification Utilizing Inspection Data-
前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

Presentation Region Calculation of Information Based on Gaze Tracking Data in Soccer Videos
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Improvement of Reliability of Dstress Image Classification for Efficient Maintenace of Road Structures
堀井風葉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Retrieval of Similar Images for Supporting Maintenance of Subway Tunnels-Introduction of Distance Metric Learning Reflecting Inspectors’ Evaluation-
松本有衣, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of rock compressive strength index from drilling data based on online learning
山本健太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Classification of Experienced and Novice Inspectors Based on Bio-signals While Inspecting in Subway Tunnels
九島哲哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Abnormal Motion Detection of River Users on CCTV Videos Based on Weakly Supervised Anomaly Detection Algorithm
渡邊はるな, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Eye Gaze Based User-specific Interest Estimation for Images-Estimation Performance Improvement Based on sMVCCA including Ordinal Label Dequantization-
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

Efforts of Hokkaido University to Develop Human Resources and the Education of Mathematical and Data Science
長谷山美紀, 大本亨, 湧田雄基, 阿部真育, 長谷山美紀, 大本亨, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Generation of Electron Microscope Images via Auxiliary Classifier Generative Adversarial Network with Mix Proportions
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Analysis of Gaze Data and Skills of Inspectors in Distress Assessment of Road Structures
松井太我, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of Image Categories Using Brain Activity While Viewing Images Based on MVBGM-MS
赤松祐亮, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Image Retrieval Focusing on Objects in Images-Improving Retrieval Performance Based on Object Recognition Using RetinaNet-
柳凛太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

A Note on Estimation of Gaze Features for Human Emotion Estimation while Viewing Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 421(ITS2019 30-56), 2020

胃X線画像を用いたAIによるH.pylori感染識別と今後の展望
藤後廉, 小川貴弘, 間部克裕, 加藤元嗣, 長谷山美紀, 日本消化器がん検診学会雑誌(Web), 58, 2, 2020

SIMILAR INSPECTION DATA RETRIEVAL FOR ROAD STRUCTURE INSPECTION BASED ON CANONICAL CORRELATION BETWEEN EYE TRACKING DATA AND INSPECTION RECORDS
前田圭介, 斉藤僚汰, 高橋翔, 小川貴弘, 長谷山美紀, 土木学会論文集 F3(土木情報学)(Web), 76, 1, 2020

A Note on Music Playlist Generation Based on Reinforcement Learning Using Self-Organized Map
櫻井慶悟, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Discrimination of Road Surface Conditions Based on Anomaly Detection Using Road Images
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Cross-domain Recommendation Based on Multilayer Graph Analysis-Performance Evaluation with Change in Dimension of Embedding Feature-
平川泰成, 前田圭介, 小川貴弘, 浅水仁, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Detection of Human Actions Based on Temporal Cycle Consistency Learning
増田毅, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on User Interest Estimation Based on Feature Integration Using m-SimGP
上川恭平, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Image Sentiment Analysis based on Multi-level Deep Metric Net
LIANG Yun, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Dataset Complexity Assessment Based on nAULS
LI Guang, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Improvement of Distress Detection Performance in Subway Tunnel Images by Data Augmentation Based on RICAP
春山知生, 前田圭介, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note of Estimation of Viewed Images Using fMRI Data While Viewing Images-Introduction of Shared Brain Responses of Multiple Subjects Based on Probabilistic Generative Model-
東孝明, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

A Note on Disentangled Representation Learning via Pre-Trained Semantic Segmentation Model
中川真, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2020, 2020

Human-centered image classification via a neural network considering visual and biological features.
Kazaha Horii, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, Multimedia Tools Appl., 79, 7-8, 4395, 4415, 2020
© 2019, Springer Science+Business Media, LLC, part of Springer Nature. In this paper, we propose a human-centered image classification via a neural network considering visual and biological features. The proposed method has two novelties. Firstly, we apply Group-Sparse Local Fisher Discriminant Analysis (GS-LFDA) to biological features. GS-LFDA realizes dimensionality reduction and noise elimination for biological features with consideration of local structures and class information. Secondly, we construct a Canonical Correlation Analysis (CCA)-based hidden layer via Discriminative Locality Preserving CCA (DLPCCA). DLPCCA transforms visual features into effective features by considering the relationships with biological information and class information. The CCA-based hidden layer enables transformation of visual features into effective features for image classification from a small number of training samples. Furthermore, once the projection can be obtained in the training phase, elimination of the need for biological data acquisition in the test phase is realized. This is another merit of our method., SPRINGER, English

胃バリウム検査におけるAIによるHelicobacter pylori診断 (Helicobacter Topics! : 「旬のHelicobacter」を知り,驚き,理解を深め,楽しむ)
藤後 廉, 間部 克裕, 山道 信毅, 大泉 晴史, 小川 貴弘, 長谷山 美紀, 加藤 元嗣, 坂本 直哉, Helicobacter research : Journal of helicobacter research, 23, 2, 92,133, 137, Nov. 2019
先端医学社, Japanese

A Note on User-specific Visual Attention Estimation Based on Visual and Spatial Information in Images : A Study on Relationship between Estimation Performance and the Similarity of Visual Features
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 13, 16, 19 Jul. 2019
電子情報通信学会, Japanese

A Note on Eye Gaze Based User-specific Interest Estimation for Images : Performance Improvement Based on Decision Level Fusion
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 17, 20, 19 Jul. 2019
電子情報通信学会, Japanese

A Note on Semantic Evaluation of Images Generated by Text-to-image Generative Adversarial Networks
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 119, 131, 21, 24, 19 Jul. 2019
電子情報通信学会, Japanese

Performance improvement of users' interest estimation using viewing behavior data obtained by sensors
長谷山 美紀, 小川 貴弘, 髙橋 翔, 原川 良介, 画像ラボ, 30, 7, 8, 12, Jul. 2019
日本工業出版, Japanese

Detection of gastritis by a deep convolutional neural network from double-contrast upper gastrointestinal barium X-ray radiography
Ren Togo, Nobutake Yamamichi, Katsuhiro Mabe, Yu Takahashi, Chihiro Takeuchi, Mototsugu Kato, Naoya Sakamoto, Kenta Ishihara, Takahiro Ogawa, Miki Haseyama, Journal of Gastroenterology, 54, 4, 321, 329, Apr. 2019
© 2018, Japanese Society of Gastroenterology. Background: Deep learning has become a new trend of image recognition tasks in the field of medicine. We developed an automated gastritis detection system using double-contrast upper gastrointestinal barium X-ray radiography. Methods: A total of 6520 gastric X-ray images obtained from 815 subjects were analyzed. We designed a deep convolutional neural network (DCNN)-based gastritis detection scheme and evaluated the effectiveness of our method. The detection performance of our method was compared with that of ABC (D) stratification. Results: Sensitivity, specificity, and harmonic mean of sensitivity and specificity of our method were 0.962, 0.983, and 0.972, respectively, and those of ABC (D) stratification were 0.925, 0.998, and 0.960, respectively. Although there were 18 false negative cases in ABC (D) stratification, 14 of those 18 cases were correctly classified into the positive group by our method. Conclusions: Deep learning techniques may be effective for evaluation of gastritis/non-gastritis. Collaborative use of DCNN-based gastritis detection systems and ABC (D) stratification will provide more reliable gastric cancer risk information., SPRINGER JAPAN KK, English

A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images
藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 265, 268, 19 Feb. 2019
電子情報通信学会, Japanese

Performance Verification of Stress Detection Using a High-performance Wristband with Multi-sensing
伊藤 良起, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 269, 274, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Estimation of Rock Drilling Energy Using Tunnel Working Face Images
山本 健太郎, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 291, 294, 19 Feb. 2019
電子情報通信学会, Japanese

A note on detection of distress regions in subway tunnels from laser data : Visualization of distress regions with fully convolutional networks
豊田 陽, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 295, 299, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Twitter Followee Recommendation Based on Adaptive Integration of Multiple Classification Results
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 307, 310, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Automatic Malignant Tumor Candidate Detection Based on a 3D Deep Residual Network with FDG-PET/CT Images (ITS)
李 宗曜, 藤後 廉, 小川 貴弘, 平田 健司, 真鍋 治, 志賀 哲, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 311, 314, 19 Feb. 2019
電子情報通信学会, English

A Note on Generation of Gastritis Images Based on Progressive Growing GAN for Gastritis Classification
渡邊 はるな, 藤後 廉, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 319, 322, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Eye Gaze-based Estimation of Emotion Evoked by Watching Images : Performance Improvement via Feature Transformation Based on KDLPCCA
松井 太我, 斉藤 直輝, 小川 貴弘, 浅水 仁, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 323, 327, 19 Feb. 2019
電子情報通信学会, Japanese

A note on interest level estimation for videos using users' behavior based on OpenPose
九島 哲哉, 高橋 翔, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 351, 354, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Action Classification of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 365, 369, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Anomaly Detection of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報, 118, 449, 371, 374, 19 Feb. 2019
電子情報通信学会, Japanese

A Note on Accurate Estimation of Deterioration Levels on Transmission Towers via Deep Learning Using Heterogeneous Features
前田圭介, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 361‐364, 12 Feb. 2019
Japanese

A Note on Popularity Prediction of Artists on Music Streaming Services Based on Network Analysis Using Heterogeneous Features
松本有衣, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 301‐306, 12 Feb. 2019
Japanese

A Note on Retrieval of Similar Inspection Records Based on Distance Metric Learning for Supporting Maintenance of Subway Tunnels
弦間奨, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 287‐290, 12 Feb. 2019
Japanese

A Method for Predicting Importance of Attack Players based on Gaze Tracking Data in Soccer Videos
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 355‐359, 12 Feb. 2019
Japanese

Leading edge of Education for Mathematical and Data Science in Hokkaido University―Concept of MDS Programs in Undergraduate and Graduate Schools―
長谷山美紀, 長谷山美紀, 大本亨, 大本亨, 湧田雄基, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 3‐4, 12 Feb. 2019
Japanese

A Note on Gastritis Detection from Gastric X‐ray Images via Transfer Learning Approach
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 315‐318, 12 Feb. 2019
Japanese

A Note on Performance Improvement of Distress Detection in Subway Tunnels Using CNN
WANG An, 原川良介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 121‐124, 12 Feb. 2019
Japanese

A Note on Attribute Estimation for Improving Interpretability of Convolutional Neural Network
堀井風葉, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 275‐279, 12 Feb. 2019
Japanese

A Note on Estimation of Characteristic Curve for Rubber Materials Using Electron Microscope Images and Mix Proportions
斉藤直輝, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 261‐264, 12 Feb. 2019
Japanese

A note on estimation of inspectors’ visual attention using distress images of subway tunnels―Trial introduction of deep learning‐based saliency prediction methods―
斉藤僚汰, 前田圭介, 小川貴弘, 長谷山美紀, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 281‐285, 12 Feb. 2019
Japanese

A Note on Performance Improvement of Distress Detection in Subway Tunnels Using CNN
王 安, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 121, 124, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Characteristic Curve for Rubber Materials Using Electron Microscope Images and Mix Proportions
斉藤 直輝, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 261, 264, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Deteriorated Regions Based on Anomaly Detection from Rubber Material Electron Microscope Images
藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 265, 268, Feb. 2019
映像情報メディア学会, Japanese

Performance Verification of Stress Detection Using a High-performance Wristband with Multi-sensing
伊藤 良起, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 269, 274, Feb. 2019
映像情報メディア学会, Japanese

A Note on Attribute Estimation for Improving Interpretability of Convolutional Neural Network
堀井 風葉, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 275, 279, Feb. 2019
映像情報メディア学会, Japanese

A note on estimation of inspectors' visual attention using distress images of subway tunnels : Trial introduction of deep learning-based saliency prediction methods
斉藤 僚汰, 前田 圭介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 281, 285, Feb. 2019
映像情報メディア学会, Japanese

A Note on Retrieval of Similar Inspection Records Based on Distance Metric Learning for Supporting Maintenance of Subway Tunnels
弦間 奨, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 287, 290, Feb. 2019
映像情報メディア学会, Japanese

A Note on Estimation of Rock Drilling Energy Using Tunnel Working Face Images
山本 健太郎, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 291, 294, Feb. 2019
映像情報メディア学会, Japanese

A note on detection of distress regions in subway tunnels from laser data : Visualization of distress regions with fully convolutional networks
豊田 陽, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 295, 299, Feb. 2019
映像情報メディア学会, Japanese

A Note on Popularity Prediction of Artists on Music Streaming Services Based on Network Analysis Using Heterogeneous Features
松本 有衣, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 301, 306, Feb. 2019
映像情報メディア学会, Japanese

A Note on Twitter Followee Recommendation Based on Adaptive Integration of Multiple Classification Results
滝村 祥司, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 307, 310, Feb. 2019
映像情報メディア学会, Japanese

A Note on Automatic Malignant Tumor Candidate Detection Based on a 3D Deep Residual Network with FDG-PET/CT Images (マルチメディアストレージ ヒューマンインフォメーション メディア工学 映像表現&コンピュータグラフィックス)
李 宗曜, 藤後 廉, 小川 貴弘, 平田 健司, 真鍋 治, 志賀 哲, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 311, 314, Feb. 2019
映像情報メディア学会, English

A Note on Gastritis Detection from Gastric X-ray Images via Transfer Learning Approach
金井 美岬, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 315, 318, Feb. 2019
映像情報メディア学会, Japanese

A Note on Generation of Gastritis Images Based on Progressive Growing GAN for Gastritis Classification
渡邊 はるな, 藤後 廉, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 319, 322, Feb. 2019
映像情報メディア学会, Japanese

A Note on Eye Gaze-based Estimation of Emotion Evoked by Watching Images : Performance Improvement via Feature Transformation Based on KDLPCCA
松井 太我, 斉藤 直輝, 小川 貴弘, 浅水 仁, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 323, 327, Feb. 2019
映像情報メディア学会, Japanese

A note on interest level estimation for videos using users' behavior based on OpenPose
九島 哲哉, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 351, 354, Feb. 2019
映像情報メディア学会, Japanese

A Method for Predicting Importance of Attack Players based on Gaze Tracking Data in Soccer Videos
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, IEEE 8th Global Conference on Consumer Electronics(GCCE), 43, 5, 355, 359, Feb. 2019
IEEE, Japanese

A Note on Accurate Estimation of Deterioration Levels on Transmission Towers via Deep Learning Using Heterogeneous Features
前田 圭介, 高橋 翔, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 361, 364, Feb. 2019
映像情報メディア学会, Japanese

A Note on Action Classification of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 365, 369, Feb. 2019
映像情報メディア学会, Japanese

A Note on Anomaly Detection of River Users on CCTV Videos
川本 舜, 原川 良介, 小川 貴弘, 長谷山 美紀, 映像情報メディア学会技術報告 = ITE technical report, 43, 5, 371, 374, Feb. 2019
映像情報メディア学会, Japanese

Efforts of Hokkaido University to Develop Data Science Human Resources-Construction of a foundation for creative human resource development: attempts to implement new functions of university education-
長谷山美紀, 長谷山美紀, 大本亨, 大本亨, 湧田雄基, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 2019

Efforts for Doctoral Human Resource Development at Hokkaido University-Smart Infrastructure Management Human Resources Development Consortium-
長谷山美紀, 長谷山美紀, 大本亨, 大本亨, 湧田雄基, 電子情報通信学会技術研究報告, 118, 449(ITS2018 59-85), 2019

人工知能が医療を変える!医療分野におけるAI研究開発最前線2019 II 領域別・画像診断におけるAI研究開発の最前線 7.その他 3)核医学におけるディープラーニングを用いた画像診断,画像処理
平田健司, 藤後廉, 小川貴弘, 長谷山美紀, 志賀哲, Innervision, 34, 7, 2019

ユーザの好みを反映した潜在特徴を用いたGCNに基づくTumblrポストの推薦
大友一馬, 原川良介, 小川貴弘, 長谷山美紀, 岩橋政宏, 信号処理シンポジウム講演論文集(CD-ROM), 34th, 2019

変状分類におけるGrad-CAM++に基づいたCNNの注目領域の可視化に関する検討
小川直輝, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

サッカー映像視聴時の視線データを用いた周辺視に基づく攻撃選手の重要度算出に関する検討
鈴木元樹, 高橋翔, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

画像注視時の視線データを用いた関心の識別に関する検討-LDCCAに基づく特徴量の射影による高精度化-
松本真直, 斉藤直輝, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Sparse Bayesian Learningに基づく注視領域の時間変化を考慮したヒトの感情推定に関する検討
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

複数の特徴量から得られる類似度の統合に基づくサッカー映像における類似場面検索の高精度化に関する検討
春山知生, 高橋翔, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

複数変量の生成モデルに基づく画像注視時の脳活動データを用いた画像カテゴリの推定に関する検討
赤松祐亮, 原川良介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Text-to-imageGANに基づくスタイル変換に関する検討-Image-to-textモデル導入による高精度化-
古寺恵, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Semantic Segmentationに基づく地下鉄トンネルにおける変状検出に関する検討
WANG An, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

脳活動データを用いた注視画像の再構成における用いる視覚野の領域に関する検討
高田紗弥, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

Twitterを用いた異常検知に基づく野球映像の重要シーン検出に関する検討
平澤魁人, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2019, 2019

AI・IoT・ビッグデータ解析に基づいた次世代の社会インフラ維持管理技術
長谷山美紀, 電子情報通信学会大会講演論文集(CD-ROM), 2019, 2019

生体情報解析を導入した次世代AI技術の構築
長谷山美紀, 小川貴弘, 電子情報通信学会大会講演論文集(CD-ROM), 2019, 2019

Novel Audio Feature Projection Using KDLPCCA-Based Correlation with EEG Features for Favorite Music Classification.
Ryosuke Sawata, Takahiro Ogawa, Miki Haseyama, IEEE Trans. Affective Computing, 10, 3, 430, 444, 2019
IEEE A novel audio feature projection using Kernel Discriminative Locality Preserving Canonical Correlation Analysis (KDLPCCA)-based correlation with electroencephalogram (EEG) features for favorite music classification is presented in this paper. The projected audio features reflect individual music preference adaptively since they are calculated by considering correlations with the user & #x0027;s EEG signals during listening to musical pieces that the user likes/dislikes via a novel CCA proposed in this paper. The novel CCA, called KDLPCCA, can consider not only a non-linear correlation but also local properties and discriminative information of each class sample, namely, music likes/dislikes. Specifically, local properties reflect intrinsic data structures of the original audio features, and discriminative information enhances the power of the final classification. Hence, the projected audio features have an optimal correlation with individual music preference reflected in the user & #x0027;s EEG signals, adaptively. If the KDLPCCA-based projection that can transform original audio features into novel audio features is calculated once, our method can extract projected audio features from a new musical piece without newly observing individual EEG signals. Our method therefore has a high level of practicability. Consequently, effective classification of user & #x0027;s favorite musical pieces via a Support Vector Machine (SVM) classifier using the new projected audio features becomes feasible. Experimental results show that our method for favorite music classification using projected audio features via the novel CCA outperforms methods using original audio features, EEG features and even audio features projected by other state-of-the-art CCAs., IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, English

Classification of tourism categories based on heterogeneous features considering existence of reliable results
Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents a classification method of tourism categories based on heterogeneous features considering existence of reliable results. The proposed method performs estimation of existence of reliable results based on one-versus-one scheme from three kinds of classification results obtained from tourism images, geotags and textual tags, separately. Then if the reliable result is included in the above results, this result is regarded as a final result. Otherwise, the final result is obtained by the multiple annotator logistic regression. The proposed method realizes accurate classification by estimating the existence of reliable results from more than two kinds of results., SPIE-INT SOC OPTICAL ENGINEERING, English

Field position estimation in soccer videos using convolutional neural network-based image features
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents a novel estimation method of field positions in soccer videos using Convolutional Neural Network (CNN)-based image features. CNN-based features have been well known to be ei€ective for tasks in machine learning. Therefore, the proposed method adopts CNN-based image features. By using these image features, the proposed method enables accurate estimation of soccer field positions than handcrafted features, i.e., Hough transform-based features. We show the ei€ectiveness of our method via experiment results using actual soccer videos., SPIE-INT SOC OPTICAL ENGINEERING, English

Gaze-based visual feature extraction via DLPCCA for visual sentiment estimation
Taiga Matsui, Naoki Saito, Takahiro Ogawa, Satoshi Asamizu, Miki Haseyama, Proceedings of SPIE - The International Society for Optical Engineering, 11049, 01 Jan. 2019
© COPYRIGHT SPIE. This paper presents gaze-based visual feature extraction via Discriminative Locality Preserving Canonical Correlation Analysis (DLPCCA) for visual sentiment estimation. The proposed method calculates novel visual features reflecting users' visual sentiment by applying DLPCCA to gaze and original visual features. Consequently, accurate visual sentiment estimation becomes feasible by utilizing the novel visual features derived by the proposed method., SPIE-INT SOC OPTICAL ENGINEERING, English

Automatic Metastatic Bone Tumor Classification with DCNN-based Features Using Treatment-planning CT Images
Haruna Watanabe, Ren Togo, Takahiro Ogawa, Miki Haseyama, Koichi Yasuda, Khin Khin Tha, Kohsuke Kudo, Hiroki Shirato, INTERNATIONAL FORUM ON MEDICAL IMAGING IN ASIA 2019, 11050, 2019
In this paper, we propose a method to classify metastatic bone tumors using treatment-planning computed tomography images. The proposed method utilizes pre-trained deep convolutional neural network (DCNN) models as feature extractors and enables the metastatic bone tumor classification by using the obtained features. Performance of several state-of-the-art DCNN-based features was compared and evaluated in our experiment., SPIE-INT SOC OPTICAL ENGINEERING, English

Multi-classifier Decision: Integration of Multiple Brain Activity-based Classifications
Takahiro Ogawa, Kento Sugata, Ren Togo, Miki Haseyama, ITE Transactions on Media Technology and Applications, 7, 1, 36, 44, 01 Jan. 2019
Copyright © 2019 by ITE Transactions on Media Technology and Applications (MTA) A novel method that integrates brain activity-based classifications obtained from multiple users is presented in this paper. The proposed method performs decision-level fusion (DLF) of the classifications using a kernelized version of extended supervised learning from multiple experts (KESLME), which is newly derived in this paper. In this approach, feature-level fusion of multiuser electroencephalogram (EEG) features is performed by multiset supervised locality preserving canonical correlation analysis (MSLPCCA). In the proposed method, the multiple classification results are obtained by classifiers separately constructed for the multiuser EEG features. Then DLF of these classification results becomes feasible based on KESLME, which can provide the final decision with consideration of the relationship between the MSLPCCA-based integrated EEG features and each classifier’s performance. In this way, a new multi-classifier decision technique, which depends only on users’ brain activities, is realized, and the performance in an image classification task becomes comparable to that of Inception-v3, one of the state-of-the-art deep convolutional neural networks., INST IMAGE INFORMATION & TELEVISION ENGINEERS, English

User-centric Visual Attention Estimation Based on Relationship between Image and Eye Gaze Data
Yuya Moroto, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 388, 389, 12 Dec. 2018
© 2018 IEEE. This paper presents a method for estimating user-centric visual attention based on the relationship between image and eye gaze data. The proposed method focuses on relationship between visual features calculated from images and saliency values calculated from eye gaze data. Specifically, our method calculates the saliency map of each training image by using individual eye gaze data obtained from only these images. Furthermore, from the pairs of visual features and the gaze-based saliency, the estimation of user-centric saliency from a new test image becomes feasible. Our contribution is the construction of a simple but successful estimation model which can train the relationship from limited amount of individual eye gaze data. Experimental results show the effectiveness of the proposed method., IEEE, English

Missing Image Data Estimation via Iterative Convolutional Sparse Coding
Masanao Matsumoto, Naoki Saito, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 331, 333, 12 Dec. 2018
© 2018 IEEE. A novel method for missing image data estimation is presented in this paper. The proposed method realizes accurate estimation of missing image data by iterating dictionary learning and Convolutional Sparse Coding (CSC). Specifically, our method iterates estimation of missing image data via CSC by using a dictionary that is constructed from a target image, and reconstruction of the dictionary by using the obtained estimation results. As the main contribution of our paper, the proposed method enables the missing image data estimation by using more suitable dictionaries obtained by this iterative scheme. Experimental results show high missing image data estimation performance by the proposed method., IEEE, English

Detection of Deformed PPG Waveforms Based on a Ratio of Their Average Accelerations
Ken Kawakami, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 40, 44, 12 Dec. 2018
© 2018 IEEE. This paper presents a new method to detect deformed Photoplethysmogram (PPG) waveforms for sufficient accuracy of signal processing. The PPG waveforms have been applied in many health indicators, such as blood pressure, blood viscosity and blood vessel elasticity. Usually, the measurements using a sensitive signals require user awareness so that all the PPG waveforms are kept accurate. Namely, accuracy of the calculated indicators becomes lower when the PPG waveform is deformed due to motion artifacts. In particular, detection methods of deformed PPG waveforms are important for incorporating the health indicators into general fitness trackers to find the correct waveform or to remove deformed PPG waveforms from the measurement. Therefore, we propose a new method which detects a badly formed PPG waveform by monitoring a ratio of average accelerations. Experimental results show the effectiveness of the method for detecting a deformed PPG waveform., IEEE, English

Team Tactics Estimation in Soccer Videos via Deep Extreme Learning Machine Based on Players Formation
Genki Suzuki, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 289, 290, 12 Dec. 2018
© 2018 IEEE. A method of team tactics estimation in soccer videos is presented in this paper. Our method enables estimation of basic tactics in each team on the basis of the Deep-Extreme Learning Machine (DELM) by using features of players formation. In the soccer games, team tactics relate to each other team. Therefore, the proposed method obtains final estimation results by utilizing two DELMs of each team and their relationship. Since the proposed method takes into consideration the relevance of the estimated tactics in each team, we realize accurate tactics estibimation. Experimental results using actual soccer videos showed the effectiveness of our method., IEEE, English

Blood Circulation Based on PPG Signals for Thermal Comfort Evaluation
Ken Kawakami, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 198, 199, 12 Dec. 2018
© 2018 IEEE. This paper presents a new index for monitoring transition of blood circulation from Photoplethysmogram (PPG) signals for thermal comfort evaluation of users. Heat dissipation reaction through the dilation of blood vessels is person's intrinsic ability to control the thermal comfort. When body temperature is higher than normal temperature, blood circulation changes according to the dilation of blood vessels in distal end of the extremities. Blood circulation is often evaluated by an index of peripheral resistance corresponding to changes in blood flow velocity such as systolic/diastolic ratio S/D of flow velocities, resistance index and the pulsatility index. Unfortunately, such an index cannot be utilized in daily life for healthcare with using fitness trackers since the blood flow velocity is measured by either an ultrasonic Doppler blood flowmeter (UDF) or a Laser-Doppler flowmeter. Therefore, we propose a new index which is easily acquirable from PPG signals. First, a couple of variables correlating to the blood flow velocity is calculated from a rate of volumetric strain of Photoplethysmogram signals. Then the new index can be obtained as difference of these variables. Experimental results show the effectiveness of this index by confirming high correlation with S/D of UDF., IEEE, English

Estimation of Viewed Image Categories via CCA Using Human Brain Activity
Yusuke Akamatsu, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 159, 160, 12 Dec. 2018
© 2018 IEEE. This paper presents a method to estimate viewed image categories via canonical correlation analysis (CCA) using human brain activity measured by functional magnetic resonance imaging (fMRI). The proposed method enables estimation of image categories that a subject viewed by using only the subject's brain activity. Specifically, the proposed method calculates the projection matrices that enable direct comparison between human brain activity and images that subjects viewed through CCA. After projecting the human brain activity and the viewed images on the same latent space, k-Nearest Neighbor (k-NN) is performed to estimate the viewed image categories from only human brain activity. Through the projection matrices, the proposed method can increase training data for k-NN even if a large number of pairs of human brain activity and images cannot be prepared. Experimental results for ten subjects show the effectiveness of the proposed method., IEEE, English

Estimation of Important Scenes in Soccer Videos Based on Collaborative Use of Audio-Visual CNN Features
Tomoki Haruyama, Sho Takahashi, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 710, 711, 12 Dec. 2018
© 2018 IEEE. This paper presents a novel method for estimating important scenes in soccer videos based on collaborative use of audio-visual Convolutional Neural Network (CNN) features. In soccer games, since game situations influence not only players' movements but also audiences' cheers, analyses of their audio and visual sequences are useful for the estimation of important scenes. In our method, such scenes are estimated from audio and visual CNN features via support vector machine (SVM) in each feature. Furthermore, by applying weighted majority voting based on confidences defined from the SVM-based estimation results, accurate estimation of important scenes becomes feasible. Experimental results show the effectiveness of our method., IEEE, English

Twitter Followee Recommendation Based on Multimodal FFM Considering Social Relations
Shoji Takimura, Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, 2018 IEEE 7th Global Conference on Consumer Electronics, GCCE 2018, 196, 197, 12 Dec. 2018
© 2018 IEEE. A method for Twitter followee recommendation based on multimodal field-aware factorization machines considering social relations (MFFM-SR) is presented. MFFM-SR enables collaborative use of textual and visual features and social relations unlike conventional methods. Specifically, for distinguishing users' interest, visual features are extracted from images in their tweets and icons as well as textual features and social relations. Furthermore, to construct a model that accurately represents users' interest, MFFM-SR that enables calculation of high-level features via estimation of latent relationships among the obtained features and social relations is derived. By using the constructed model, successful followee recommendation becomes feasible., IEEE, English

Binary sparse representation based on arbitrary quality metrics and its applications
Takahiro Ogawa, Sho Takahashi, Naofumi Wada, Akira Tanaka, Miki Haseyama, IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, E101A, 11, 1776, 1785, 01 Nov. 2018
© 2018 The Institute of Electronics, Information and Communication Engineers. Binary sparse representation based on arbitrary quality metrics and its applications are presented in this paper. The novelties of the proposed method are twofold. First, the proposed method newly derives sparse representation for which representation coefficients are binary values, and this enables selection of arbitrary image quality metrics. This new sparse representation can generate quality metric-independent subspaces with simplification of the calculation procedures. Second, visual saliency is used in the proposed method for pooling the quality values obtained for all of the parts within target images. This approach enables visually pleasant approximation of the target images more successfully. By introducing the above two novel approaches, successful image approximation considering human perception becomes feasible. Since the proposed method can provide lower-dimensional subspaces that are obtained by better image quality metrics, realization of several image reconstruction tasks can be expected. Experimental results showed high performance of the proposed method in terms of two image reconstruction tasks, image inpainting and super-resolution., IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG, English

Automatic detection of fish sounds based on multi-stage classification including logistic regression via adaptive feature weighting
Ryosuke Harakawa, Takahiro Ogawa, Miki Haseyama, Tomonari Akamatsu, Journal of the Acoustical Society of America, 144, 5, 2709, 2718, 01 Nov. 2018
© 2018 Acoustical Society of America. This paper presents a method for automatic detection of fish sounds in an underwater environment. There exist two difficulties: (i) features and classifiers that provide good detection results differ depending on the underwater environment and (ii) there are cases where a large amount of training data that is necessary for supervised machine learning cannot be prepared. A method presented in this paper (the proposed hybrid method) overcomes these difficulties as follows. First, novel logistic regression (NLR) is derived via adaptive feature weighting by focusing on the accuracy of classification results by multiple classifiers, support vector machine (SVM), and k-nearest neighbors (k-NN). Although there are cases where SVM or k-NN cannot work well due to divergence of useful features, NLR can produce complementary results. Second, the proposed hybrid method performs multi-stage classification with consideration of the accuracy of SVM, k-NN, and NLR. The multi-stage acquisition of reliable results works adaptively according to the underwater environment to reduce performance degradation due to diversity of useful classifiers even if abundant training data cannot be prepared. Experiments on underwater recordings including sounds of Sciaenidae such as silver croakers (Pennahia argentata) and blue drums (Nibea mitsukurii) show the effectiveness of the proposed hybrid method., ACOUSTICAL SOC AMER AMER INST PHYSICS, English

画像注視時の注視領域の時間変化を考慮したテンソル解析に基づく感情推定に関する検討
諸戸祐哉, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.92, 27 Oct. 2018
Japanese

地下鉄トンネルの点検における視線データを用いた熟練度の分析に関する一考察
斉藤僚汰, 前田圭介, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.11, 27 Oct. 2018
Japanese

深層学習に基づく画像特徴量を利用した放射線治療用CT画像における転移性骨腫瘍の検出に関する検討
渡邊はるな, 藤後廉, 小川貴弘, 長谷山美紀, 安田耕一, THA Khin Khin, 工藤與亮, 白土博樹, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.87, 27 Oct. 2018
Japanese

AC‐GANに基づく胃炎識別のための画像生成に関する検討
金井美岬, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.89, 27 Oct. 2018
Japanese

Tri‐trainingに基づく胃X線画像を用いた胃炎の識別に関する検討
LI Zongyao, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.88, 27 Oct. 2018
Japanese

AttnGANを用いたシーン検索に関する検討―再検索の導入による高精度化―
柳凜太郎, 藤後廉, 小川貴弘, 長谷山美紀, 電気・情報関係学会北海道支部連合大会講演論文集(CD-ROM), 2018, ROMBUNNO.10, 27 Oct. 2018
Japanese