Abstract

Ensuring the overall end-user experience is a challenging task in arbitrary style transfer (AST) due to the subjective nature of style transfer quality. A good practice is to provide users many instead of one AST result. However, existing approaches require to run multiple AST models or inference a diversified AST (DAST) solution multiple times, and thus they are either slow in speed or limited in diversity. In this paper, we propose a novel solution ensuring both efficiency and diversity for generating multiple user-controllable AST results by systematically modulating AST behavior at run-time. We begin with reformulating three prominent AST methods into a unified assign-and-mix problem and discover that the entropies of their assignment matrices exhibit a large variance. We then solve the unified problem in an optimal transport framework using the Sinkhorn-Knopp algorithm with a user input ε to control the said entropy and thus modulate stylization. Empirical results demonstrate the superiority of the proposed solution, with speed and stylization quality comparable to or better than existing AST and significantly more diverse than previous DAST works. Code is available at https://github. com/cplusx/eps-Assign-and-Mix.

Metadata

publication

Proceedings of the AAAI Conference on Artificial Intelligence 37 (1), 433-441, 2023

year

2023

publication date

2023/6/26

authors

Jiaxin Cheng, Yue Wu, Ayush Jaiswal, Xu Zhang, Pradeep Natarajan, Prem Natarajan

link

https://ojs.aaai.org/index.php/AAAI/article/view/25117

resource_link

https://ojs.aaai.org/index.php/AAAI/article/download/25117/24889

journal

Proceedings of the AAAI Conference on Artificial Intelligence

volume

37

issue

1

pages

433-441