Skip to main content
SpringerLink
Log in

A Benchmark for Inpainting of Clothing Images with Irregular Holes

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 Workshops (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12538))

Included in the following conference series:

Abstract

Fashion image understanding is an active research field with a large number of practical applications for the industry. Despite its practical impacts on intelligent fashion analysis systems, clothing image inpainting has not been extensively examined yet. For that matter, we present an extensive benchmark of clothing image inpainting on well-known fashion datasets. Furthermore, we introduce the use of a dilated version of partial convolutions, which efficiently derive the mask update step, and empirically show that the proposed method reduces the required number of layers to form fully-transparent masks. Experiments show that dilated partial convolutions (DPConv) improve the quantitative inpainting performance when compared to the other inpainting strategies, especially it performs better when the mask size is 20% or more of the image.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/karfly/qd-imd.

  2. 2.

    https://github.com/MingtaoGuo/PatchMatch.

  3. 3.

    https://github.com/avalonstrel/GatedConvolution_pytorch.

  4. 4.

    https://github.com/NVIDIA/partialconv.

References

  1. Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001). https://doi.org/10.1109/83.935036

    Article  MathSciNet  MATH  Google Scholar 

  2. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)

    Article  Google Scholar 

  3. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 417–424. ACM Press/Addison-Wesley Publishing Co. (2000). https://doi.org/10.1145/344779.344972

  4. Chen, K., et al.: MMDetection: open MMLab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)

  5. Di, W., Wah, C., Bhardwaj, A., Piramuthu, R., Sundaresan, N.: Style finder: fine-grained clothing style detection and retrieval. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2013

    Google Scholar 

  6. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, pp. 341–346. Association for Computing Machinery, New York (2001). https://doi.org/10.1145/383259.383296

  7. Ge, Y., Zhang, R., Wu, L., Wang, X., Tang, X., Luo, P.: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images (2019)

    Google Scholar 

  8. Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision (ICCV), pp. 1440–1448, December 2015. https://doi.org/10.1109/ICCV.2015.169

  9. Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P., He, K.: Detectron (2018). https://github.com/facebookresearch/detectron

  10. Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates, Inc. (2014). http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

  11. Gunel, M., Erdem, E., Erdem, A.: Language guided fashion image manipulation with feature-wise transformations. In: First Workshop on Computer Vision in Art, Fashion and Design - in conjunction with ECCV 2018 (2018)

    Google Scholar 

  12. Han, X., Wu, Z., Wu, Z., Yu, R., Davis, L.S.: VITON: an image-based virtual try-on network. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7543–7552, June 2018. https://doi.org/10.1109/CVPR.2018.00787

  13. Han, X., Wu, Z., Huang, W., Scott, M.R., Davis, L.S.: FiNet: compatible and diverse fashion image inpainting. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4481–4491 (2019)

    Google Scholar 

  14. Hays, J., Efros, A.A.: Scene completion using millions of photographs. Commun. ACM 51(10), 87–94 (2008). https://doi.org/10.1145/1400181.1400202

    Article  Google Scholar 

  15. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988, October 2017. https://doi.org/10.1109/ICCV.2017.322

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016. https://doi.org/10.1109/CVPR.2016.90

  17. Hsiao, W.L., Grauman, K.: Creating capsule wardrobes from fashion images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018. https://doi.org/10.1109/cvpr.2018.00748

  18. Huang, J., Feris, R.S., Chen, Q., Yan, S.: Cross-domain image retrieval with a dual attribute-aware ranking network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1062–1070 (2015)

    Google Scholar 

  19. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073659

  20. Inoue, N., Simo-Serra, E., Yamasaki, T., Ishikawa, H.: Multi-label fashion image classification with minimal human supervision. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2017

    Google Scholar 

  21. Jae Lee, H., Lee, R., Kang, M., Cho, M., Park, G.: LA-VITON: a network for looking-attractive virtual try-on. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019

    Google Scholar 

  22. Jagadeesh, V., Piramuthu, R., Bhardwaj, A., Di, W., Sundaresan, N.: Large scale visual recommendations from street fashion images. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2014, pp. 1925–1934. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2623330.2623332

  23. Ji, W., et al.: Semantic locality-aware deformable network for clothing segmentation. In: IJCAI, pp. 764–770 (2018)

    Google Scholar 

  24. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=Hk99zCeAb

  25. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. https://doi.org/10.1109/cvpr.2019.00453

  26. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations, December 2014

    Google Scholar 

  27. Kinli, F., Ozcan, B., Kirac, F.: Fashion image retrieval with capsule networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)

    Google Scholar 

  28. Kınlı, F., Özcan, B., Kıraç, F.: Description-aware fashion image inpainting with convolutional neural networks in coarse-to-fine manner. In: Proceedings of the 2020 6th International Conference on Computer and Technology Applications, ICCTA 2020, pp. 74–79 (2020). https://doi.org/10.1145/3397125.3397155

  29. Korneliusson, M., Martinsson, J., Mogren, O.: Generative modelling of semantic segmentation data in the fashion domain. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019

    Google Scholar 

  30. Kubo, S., Iwasawa, Y., Suzuki, M., Matsuo, Y.: UVTON: UV mapping to consider the 3D structure of a human in image-based virtual try-on network. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019

    Google Scholar 

  31. Kınlı, F., Kıraç, F.: FashionCapsNet: clothing classification with capsule networks. J. Inf. Technol. 13, 87–96 (2020). https://doi.org/10.17671/gazibtd.580222

    Article  Google Scholar 

  32. Liang, X., Lin, L., Yang, W., Luo, P., Huang, J., Yan, S.: Clothes co-parsing via joint image segmentation and labeling with application to clothing retrieval. IEEE Trans. Multimedia 18(6), 1175–1186 (2016)

    Article  Google Scholar 

  33. Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6

    Chapter  Google Scholar 

  34. Liu, H., Jiang, B., Xiao, Y., Yang, C.: Coherent semantic attention for image inpainting. In: The IEEE International Conference on Computer Vision (ICCV), October 2019

    Google Scholar 

  35. Liu, L., Zhang, H., Ji, Y., Wu, Q.M.J.: Toward AI fashion design: an attribute-GAN model for clothing match. Neurocomputing 341, 156–167 (2019)

    Article  Google Scholar 

  36. Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: DeepFashion: powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  37. Martinsson, J., Mogren, O.: Semantic segmentation of fashion images using feature pyramid networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)

    Google Scholar 

  38. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: EdgeConnect: structure guided image inpainting using edge prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, October 2019

    Google Scholar 

  39. Opitz, M., Waltner, G., Possegger, H., Bischof, H.: Bier - boosting independent embeddings robustly. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5199–5208, October 2017. https://doi.org/10.1109/ICCV.2017.555

  40. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  41. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting. In: Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  42. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003). https://doi.org/10.1145/882262.882269

    Article  Google Scholar 

  43. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  44. Rostamzadeh, N., et al.: Fashion-Gen: the Generative Fashion Dataset and Challenge. ArXiv e-prints, June 2018

    Google Scholar 

  45. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)

    Google Scholar 

  46. Sbai, O., Elhoseiny, M., Bordes, A., LeCun, Y., Couprie, C.: DesIGN: design inspiration from generative networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 37–44. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_5

    Chapter  Google Scholar 

  47. Song, Y., et al.: Contextual-based image inpainting: infer, match, and translate. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 3–18. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_1

    Chapter  Google Scholar 

  48. Telea, A.: An image inpainting technique based on the fast marching method. J. Graph. GPU Game Tools 9, 23–34 (2004)

    Article  Google Scholar 

  49. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5998–6008. Curran Associates, Inc. (2017). http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

  50. Wang, W., Xu, Y., Shen, J., Zhu, S.C.: Attentive fashion grammar network for fashion landmark detection and clothing category classification. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  51. Wang, Z., Gu, Y., Zhang, Y., Zhou, J., Gu, X.: Clothing retrieval with visual attention model. In: 2017 IEEE Visual Communications and Image Processing (VCIP), December 2017. https://doi.org/10.1109/vcip.2017.8305144

  52. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  53. Yamaguchi, K., Okatani, T., Sudo, K., Murasaki, K., Taniguchi, Y.: Mix and match: joint model for clothing and attribute recognition. In: BMVC, vol. 1, p. 4 (2015)

    Google Scholar 

  54. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4076–4084, July 2017. https://doi.org/10.1109/CVPR.2017.434

  55. Yildirim, G., Jetchev, N., Vollgraf, R., Bergmann, U.: Generating high-resolution fashion model images wearing custom outfits. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019

    Google Scholar 

  56. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)

    Google Scholar 

  57. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2018. https://doi.org/10.1109/cvpr.2018.00577

  58. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: The IEEE International Conference on Computer Vision (ICCV), October 2019

    Google Scholar 

  59. Zhang, S., Song, Z., Cao, X., Zhang, H., Zhou, J.: Task-aware attention model for clothing attribute prediction. IEEE Trans. Circ. Syst. Video Technol. 30, 1051–1064 (2019)

    Article  Google Scholar 

  60. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  61. Zhu, S., Fidler, S., Urtasun, R., Lin, D., Loy, C.C.: Be your own Prada: fashion synthesis with structural coherence. In: 2017 IEEE International Conference on Computer Vision (ICCV), October 2017. https://doi.org/10.1109/iccv.2017.186

  62. Zou, X., Kong, X., Wong, W., Wang, C., Liu, Y., Cao, Y.: FashionAI: a hierarchical dataset for fashion understanding. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Özyeğin University, İstanbul, Turkey

    Furkan Kınlı, Barış Özcan & Furkan Kıraç

Authors
  1. Furkan Kınlı
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barış Özcan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Furkan Kıraç
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Furkan Kınlı .

Editor information

Editors and Affiliations

  1. University of Clermont Auvergne, Clermont Ferrand, France

    Adrien Bartoli

  2. Università degli Studi di Udine, Udine, Italy

    Andrea Fusiello

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kınlı, F., Özcan, B., Kıraç, F. (2020). A Benchmark for Inpainting of Clothing Images with Irregular Holes. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12538. Springer, Cham. https://doi.org/10.1007/978-3-030-66823-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-66823-5_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-66822-8

  • Online ISBN: 978-3-030-66823-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation