Neural View Synthesis and Appearance Editing from Unstructured Images

Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP) 2021

Pulkit Gera1, Aakash KT1, Dhawal Sirikonda1, P J Narayanan1
1CVIT, IIIT Hyderabad

Paper Code

Abstract

We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene from multi-view images captured under known environment illumination. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without direct control over the scene’s appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we independently estimate the BRDF and use it to learn a lighting-only representation of the scene. Such disentanglement allows our approach to generalize to arbitrary changes in appearance while performing view synthesis. We show results of editing the appearance of a real scene, demonstrating that our approach produces plausible appearance editing. The performance of our view synthesisapproach is demonstrated to be at par with state-of-the-art approaches on both real and synthetic data.

Video

Pipeline

We propose a formulation that disentangles the Bi-directional Reflectance Distrubution Function (BRDF) and the captured object's local irradiance function (LIF). Both the BRDF and the LIF are estimated together, using a novel loss that ensures proper diffuse albedo estimation. This disentanglement allows the diffuse albedo to be edited or modified independently. The modified appearance is then combined with the learnt LIF to produce the final image for a novel viewpoint.

Results

Citation

	@inproceedings{10.1145/3490035.3490299,
	author = {Gera, Pulkit and T, Aakash K and Sirikonda, Dhawal and Narayanan, P. J.},
	title = {Neural View Synthesis with Appearance Editing from Unstructured Images},
	year = {2021},
	isbn = {9781450375962},
	publisher = {Association for Computing Machinery},
	address = {New York, NY, USA},
	url = {https://doi.org/10.1145/3490035.3490299},
	doi = {10.1145/3490035.3490299},
	booktitle = {Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing},
	articleno = {40},
	numpages = {9},
	keywords = {text tagging, ACM proceedings, LATEX},
	location = {Jodhpur, India},
	series = {ICVGIP '21}
	}