Controlling StyleGANs Using Rough Scribbles via One-shot Learning

Yuki Endo and Yoshihiro Kanamori

University of Tsukuba

Computer Animation and Virtual Worlds (Computer Graphics International 2022)



Abstract:

This paper tackles the challenging problem of one-shot semantic image synthesis from rough sparse annotations, which we call "semantic scribbles." Namely, from only a single training pair annotated with semantic scribbles, we generate realistic and diverse images with layout control over, e.g., facial part layouts and body poses. We present a training strategy that performs pseudo labeling for semantic scribbles using the StyleGAN prior. Our key idea is to construct a simple mapping between StyleGAN features and each semantic class from a single example of semantic scribbles. With such mappings, we can generate an unlimited number of pseudo semantic scribbles from random noise to train an encoder for controlling a pre-trained StyleGAN generator. Even with our rough pseudo semantic scribbles obtained via one-shot supervision, our method can synthesize high-quality images thanks to our GAN inversion framework. We further offer optimization-based post-processing to refine the pixel alignment of synthesized images. Qualitative and quantitative results on various datasets demonstrate improvement over previous approaches in one-shot settings.


Video:


Publication:

  1. Yuki Endo, Yoshihiro Kanamori: "Controlling StyleGANs Using Rough Scribbles via One-shot Learning," Computer Animation and Virtual Worlds (Computer Graphics International 2022), 2022. [PDF(preprint) (22MB)][Code]

Related Publication:

  1. Yuki Endo, Yoshihiro Kanamori: "Few-shot Semantic Image Synthesis Using StyleGAN Prior," arXiv, 2021. [PDF][Code]

BibTeX Citation

@Article{endoCAVW2022,
Title = {Controlling StyleGANs Using Rough Scribbles via One-shot Learning},
Author = {Yuki Endo and Yoshihiro Kanamori},
Journal = {Computer Animation and Virtual Worlds},
volume = {},
number = {},
pages = {},
doi = {},
Year = {2022}
}
@Article{endoArXiv2021,
Title = {Few-shot Semantic Image Synthesis Using StyleGAN Prior},
Author = {Yuki Endo and Yoshihiro Kanamori},
journal = {CoRR},
volume = {abs/2103.14877},
year = {2021},
url = {https://arxiv.org/abs/2103.14877},
eprinttype = {arXiv},
eprint = {2103.14877}
}

Last modified: June 2022

[back]