HOMMA Taiyu, a second-year student in the Department of Frontier Media Science, School of Interdisciplinary Mathematical Sciences, and Professor MIYASHITA Homei received the Interactive Presentation Award (public vote) at the Information Processing Society of Japan Symposium “Interaction 2024.”
HOMMA and Professor MIYASHITA gave a demonstration of the TasteColorizer system, which can reproduce the taste of food shown in videos, whether in black and white or animated, and was selected by voting participants. They were also selected as a Premium Presentation through screening by the program committee.
Since the invention of imaging technology, humans have recorded a variety of videos. And in recent years, with the development of “taste media,” not only videos but also tastes can be recorded and reproduced. However, it is not possible to record the taste of videos created in the past. Therefore, they proposed a system to estimate and assign taste to videos for which no taste has been recorded, similar to the technology that colorizes black-and-white videos. Taste is estimated for all scenes of the video by GPT4-Vision. The prompts instruct the system to first guess the name of the food and drink, its ingredients and their quantities, and then to estimate the taste for allowing for highly accurate estimation through step-by-step inference. The estimated taste data is recorded in an MP4-extended file, corresponding to each pixel along with the video. Viewers can specify food and drink items on the video during playback, and the corresponding tastes are output and tasted. Since the taste is recorded, no inference is required during playback. This system enables existing video media to become “tasteable videos.”
HOMMA Taiyu, MIYASHITA Homei. TasteColorizer: A system that makes existing video media "tasteable videos", Proceedings of Interaction 2024, pp.1260 - 1265, 2024
