GANs are an interesting frontier. Videos are much more engaging than photos. The next logical step is to make a real-time GAN, like a videogame you can walk around in.
Imagine using a Vive to explore a GAN interactively. You'd be able to control the GAN using vive controllers and by walking around your room.
Right now it takes 163ms to render a 1024x1024 frame on a K80 GPU. That's 6 FPS, which is within an order of magnitude of 60FPS.
I haven't timed a 256x256 GAN, but presumably it would be 16x faster to generate. If so, then you'd be able to achieve 98FPS.
Someone should train a 256x256 FFHQ and make a 90FPS interactive renderer for it.
Unfortunately it's not possible to take a large GAN like 1024x1024 FFHQ and only generate a 256x256 image. Each GAN is trained for a specific size, so you're stuck with 6 FPS at 1024x1024. I wish the FFHQ authors had saved a 256x256 checkpoint during training.
Training a 256x256 GAN from scratch costs somewhere in the range of $150 GCE credits. But you might be able to bootstrap a 256x256 FFHQ using the weights from the 1024x1024 FFHQ (aka transfer learning). That might train a lot faster.
There is also the recent NoGAN technique, which skips progressive growing by pretraining the generator: https://github.com/jantic/DeOldify/#what-is-nogan Supposedly it speeds up GAN training by a huge amount.
Imagine using a Vive to explore a GAN interactively. You'd be able to control the GAN using vive controllers and by walking around your room.
Right now it takes 163ms to render a 1024x1024 frame on a K80 GPU. That's 6 FPS, which is within an order of magnitude of 60FPS.
I haven't timed a 256x256 GAN, but presumably it would be 16x faster to generate. If so, then you'd be able to achieve 98FPS.
The above timings are based on the 1024x1024 FFHQ GAN model, which generates portraits of humans. https://github.com/pbaylies/stylegan-encoder
And indeed, it looks like the author uploaded an FFHQ music video 14 minutes ago! https://www.youtube.com/watch?v=3TLEfOMBbMw It looks cool.
Someone should train a 256x256 FFHQ and make a 90FPS interactive renderer for it.
Unfortunately it's not possible to take a large GAN like 1024x1024 FFHQ and only generate a 256x256 image. Each GAN is trained for a specific size, so you're stuck with 6 FPS at 1024x1024. I wish the FFHQ authors had saved a 256x256 checkpoint during training.
Training a 256x256 GAN from scratch costs somewhere in the range of $150 GCE credits. But you might be able to bootstrap a 256x256 FFHQ using the weights from the 1024x1024 FFHQ (aka transfer learning). That might train a lot faster.
There is also the recent NoGAN technique, which skips progressive growing by pretraining the generator: https://github.com/jantic/DeOldify/#what-is-nogan Supposedly it speeds up GAN training by a huge amount.