Our method enables interactive physically based rendering (PBR) with 3D generative adversarial networks (GAN) under arbitrary illumination conditions in the form of environment maps and unconstrained camera navigation, including 360-degree rotations. We achieve this with three main contributions: a method to estimate poses of a dataset of car images, a generative pipeline for PBR, and an improved generative network architecture and training solution. We only require a dataset of images of the desired object class (cars, in our case) and a dataset of environment maps for training. Our method learns a disentangled representation of shading, enabling relighting with high-frequency reflections on shiny car bodies.