Blending Methods and Other Improvements for Exemplar-based Image Inpainting Techniques Maxime Daisy, Pierre Buyssens, David Tschumperlé and Olivier Lézoray GREYC - CNRS UMR 6072, Image team 9 th of April 2015
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 2 / 45
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 2 / 45
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 2 / 45
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 2 / 45
Context D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 2 / 45
Context Geometry-based methods [Masnou et al. 98, Bertalmio et al. 00, Chan et al. 01, Tschumperlé and Deriche 03,...] boundary geometry interpolation Good global geometry reconstruction Not able to create complex textures Time consuming D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 3 / 45
Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 4 / 45
Inpainted with Diffusion PDE s [Tschumperlé and Deriche 03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 4 / 45
Context Pattern-based methods [Efros and Leung 99, Criminisi et al. 04, Lemeur et al. 11,...] copy/paste of image chunks Large area reconstruction Quite fast Not enough local/global coherence e.g. block-effect artifacts Best compromise between time and quality D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 5 / 45
Context Criminisi et al. 2004, "Region Filling and Object Removal by Exemplar-Based Image Inpainting" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 6 / 45
Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 7 / 45
Inpainted with [Criminisi et al. 04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 7 / 45
Exemplar-based inpainting : Proposed improvements D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 8 / 45
Exemplar-based Inpainting Algorithm Proposed improvements 1 priority accuracy enhancement more global geometry consistency 2 better lookup statregy more local geometry consistency 3 spatial patch blending strongly reduced block-effect artifacts D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 9 / 45
A better data term accuracy Criminisi et al. data term D p = I p. n p (1) α "The gradient I p is computed as the maximum value of the image gradient in Ψ p I." I p = { I q arg max I q } (2) q ((I Ω) ψ p) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 10 / 45
A better data term accuracy Proposed data term D p = G p. n p (3) with G p = w p (q) I q I T q (4) q ψ p (I Ω) w p : normalized 2d gaussian centered at p D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 11 / 45
A better lookup strategy Criminisi et al. 2014 lookup strategy window search? faster much less global all over the image search? slow not so much good results D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 12 / 45
A better lookup strategy Window search Iteration 0 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 13 / 45
A better lookup strategy Window search Iteration n D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 13 / 45
A better lookup strategy Our lookup strategy inspired from [Ashikhmin 01, PatchMatch 09] local/global search scheme use search sites of surrounding inpainted patches more local coherence D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 14 / 45
A better lookup strategy D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 15 / 45
Joshua Tree - 512 384 Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 16 / 45
Joshua Tree - 512 384 Inpainted with windowed search (4.5s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 16 / 45
Joshua Tree - 512 384 Inpainted with smart search (3.2s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 16 / 45
Fake Owl - 800 450 Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 17 / 45
Fake Owl - 800 450 Inpainted with windowed search (1.7s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 17 / 45
Fake Owl - 800 450 Inpainted with smart search (0.8s) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 17 / 45
block-effect artifacts? D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 18 / 45
Spatial Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 19 / 45
Spatial Patch Blending Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 20 / 45
Spatial Patch Blending Inpainted with [Criminisi et al. 04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 20 / 45
Spatial Patch Blending Inpainted with diffusion PDE s [Tschumperlé and Deriche 03] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 20 / 45
Spatial Patch Blending Inpainted with [Criminisi et al. 04] + Our spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 20 / 45
Artifact Detection Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 21 / 45
Artifact Detection Inpainted with [Criminisi et al. 04] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 21 / 45
Artifact Detection 1 Map of the artifact location probabilities 2 hypothesis for artifact locations local sharp variations in the inpainted image source patches come from very different locations P A = I. div(φ) φ : inpainting correspondence map 2 Strongest artifact locations L A = {p P A (p) > τ} 3 Blending amplitude map σ(p) = ρ. w(p,q) q L A max r I q L A w(p,q) with w(p, q) = exp ) ( p q 2 P A (q) 2 (5) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 22 / 45
Artifact Detection Inpainted image. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 23 / 45
Artifact Detection Break field image. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 23 / 45
Artifact Detection Break points. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 23 / 45
Artifact Detection Blending amplitude map. D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 23 / 45
Patch Blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 24 / 45
Patch Blending Weighted sum of overlapping patches pixels values w(p, q). ψ q (p q) ψ q Ψ p J(p) = w(q, p) ψ q Ψ p ) w(p, q) = exp ( p q 2 σ(p) 2 Ψ p = {ψ q ψ q ψ p } (6) one gaussian function for each possible blending amplitude quite slow D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 25 / 45
A faster patch blending algorithm D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 26 / 45
Result Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 27 / 45
Result Inpainted image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 28 / 45
Result Inpainted Image + Spatial patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 29 / 45
Block-effect artifacts? Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 30 / 45
Block-effect artifacts? Inpainted Image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 30 / 45
Block-effect artifacts? Blended Image (isotropic) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 30 / 45
Geometry-guided patch blending : process Anisotropic image regularization D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 31 / 45
Geometry-guided patch blending : process Anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 32 / 45
Geometry-guided patch blending 1 Structure eigen values normalization ˆλ S(p)i = λ S(p) max p I λ S(p)i 2 Blending tensors eigen values computation 3 Blending tensor building 1 λ Bi = (1 + ˆλ S1 + ˆλ S2 ) γ i B = λ σb1 e S1.T e S1 + λ σb2e S2.T e S2 (7) e Si : structure tensor eigen vectors λ σbi = σ B λ Bi σ B : maximum blending bandwidth D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 33 / 45
Geometry-guided patch blending Weighted sum of overlapping patches pixels values using Eq. (6) with ) exp ( p q 2 isotropic σ(p) w(p, q) = ( 2 ) X exp T B(p) 1 X anisotropic where X = q p 2σ 2 B D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 34 / 45
block-effect artifacts? Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 35 / 45
block-effect artifacts? Inpainted image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 35 / 45
block-effect artifacts? Blended image (isotropic) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 35 / 45
block-effect artifacts? Blended image (geometry-guided) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 35 / 45
Woman Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 36 / 45
Woman Without blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 36 / 45
Woman Isotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 36 / 45
Woman Geometry-guided patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 36 / 45
Copter Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 37 / 45
Copter Copter + mask D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 37 / 45
Copter Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 37 / 45
Opera Masked image D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 38 / 45
Opera Inpainted + anisotropic patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 38 / 45
Opera Inpainted without blending [Lemeur et al 11] D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 38 / 45
Video Inpainting D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 39 / 45
Adaptation to videos priorities computed frame by frame cobblestone patches (e.g. of size 5 5 3) and lookup windows Shown to be working in the state of the art Wexler et al. TPAMI 2007, "Space-time completion of Videos" Newson et al. SIAM J. Imaging sciences 2014, "Video inpainting of complex scenes" D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 40 / 45
Space-time artifacts Space-time patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 41 / 45
Adaptation to videos Space-time patch blending D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 42 / 45
Masked video : (x,y) and (x,t) plans D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 43 / 45
Inpainted video (without blending) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 43 / 45
Blended Image (geometry-guided) D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 43 / 45
Conclusion Contributions 1 Improvements of a reference inpainting method 2 Method to reduce block-effect artifacts 3 Adaptation of (1) and (2) to video data Recent advances Depth-aware patch blending for stereoscopic inpainted results More results at https://daisy.users.greyc.fr/@research D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 44 / 45
References and publications References Wexler et al., Space-Time Video Completion, IEEE TPAMI 2007 Newson et al., Video Inpainting of Complex Scenes, SIAM J. IMAGING SCIENCES 2014 Publications Daisy M., Tschumperlé D. and Lézoray O., Spatial Patch Blending for Artefact Reduction in Pattern-Based Inpainting Techniques, CAIP 13 Daisy M., Tschumperlé D. and Lézoray O., A Fast Spatial Patch Blending Algorithm for Artefact Reduction in Pattern-based Image Inpainting, SIGGRAPH-ASIA 13 Daisy M., Buyssens P., Tschumperlé D. and Lézoray O., A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence, ICIP 14 D. Tschumperlé (GREYC, Caen - FRANCE) Journée Imagerie Rouen 9 th of April 2015 45 / 45