Evaluating and Validating Non-photorealistic and Illustrative Rendering

  • Tobias IsenbergEmail author
Part of the Computational Imaging and Vision book series (CIVI, volume 42)


In many areas of non-photorealistic and illustrative rendering, considerable progress has been made toward synthesizing traditional artistic and illustrative techniques. However, evaluation and validation of such images have only been attempted relatively recently. This chapter surveys evaluation approaches that have been applied successfully in non-photorealistic and illustrative rendering. It provides an overview over different evaluation approaches including qualitative and quantitative techniques and gives examples for how to approach evaluation in the NPR context. Collectively, the described techniques do not only answer the question of whether an NPR technique is able to replicate a traditional technique successfully but also what implications the use of NPR techniques has and what opinion people have about different NPR techniques as compared to traditional depictions.


Augmented Reality Line Drawing Virtual Object Shade Image Hand Drawing 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Acevedo, D., Laidlaw, D., Drury, F.: Using visual design expertise to characterize the effectiveness of 2D scientific visualization methods. In: Proceedings Compendium of IEEE InfoVis and Visualization 2005, pp. 111–112 (2005). doi: 10.1109/VIS.2005.109 Google Scholar
  2. 2.
    AlMeraj, Z., Kaplan, C.S., Asente, P., Lank, E.: Towards ground truth in geometric textures. In: Proc. NPAR, pp. 17–26. ACM, New York (2011). doi: 10.1145/2024676.2024679 Google Scholar
  3. 3.
    Bénard, P., Thollot, J., Sillion, F.: Quality assessment of fractalized NPR textures: a perceptual objective metric. In: Proc. APGV, pp. 117–120. ACM, New York (2009). doi: 10.1145/1620993.1621016 CrossRefGoogle Scholar
  4. 4.
    Carpendale, S.: Evaluating information visualizations. In: Information Visualization: Human-Centered Issues and Perspectives. LNCS, vol. 4950, pp. 19–45. Springer, Berlin (2008). doi: 10.1007/978-3-540-70956-5_2 Google Scholar
  5. 5.
    Cohen, J.: The Earth is round (p<0.05). Am. Psychol. 49(12), 997–1003 (1994). doi:  10.1037/0003-066X.49.12.997 CrossRefGoogle Scholar
  6. 6.
    Cole, F.: Line drawings of 3D models. Ph.D. thesis, Princeton University (2009) Google Scholar
  7. 7.
    Cole, F., Golovinskiy, A., Limpaecher, A., Barros, H.S., Finkelstein, A., Funkhouser, T., Rusinkiewic, S.: Where do people draw lines? ACM Trans. Graph. 27(3), 88 (2008). doi: 10.1145/1360612.1360687 CrossRefGoogle Scholar
  8. 8.
    Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., Singh, M.: How well do line drawings depict shape? ACM Trans. Graph. 28(3), 28 (2009). doi: 10.1145/1531326.1531334 CrossRefGoogle Scholar
  9. 9.
    Cole, F., Golovinskiy, A., Limpaecher, A., Barros, H.S., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S.: Where do people draw lines? Commun. ACM 55(1), 107–115 (2012). doi:  10.1145/2063176.2063202 CrossRefGoogle Scholar
  10. 10.
    Duke, D.J., Barnard, P.J., Halper, N., Mellin, M.: Rendering and affect. Comput. Graph. Forum 22(3), 359–368 (2003). doi: 10.1111/1467-8659.00683 CrossRefGoogle Scholar
  11. 11.
    Field, A., Hole, G.: How to Design and Report Experiments. Sage, London (2003) Google Scholar
  12. 12.
    Fischer, J., Cunningham, D., Bartz, D., Wallraven, C., Bülthoff, H., Straßer, W.: Measuring the discernability of virtual objects in conventional and stylized augmented reality. In: Proc. EGVE, pp. 53–61. Eurographics Association, Goslar (2006). doi: 10.2312/EGVE/EGVE06/053-061 Google Scholar
  13. 13.
    Gatzidis, C., Papakonstantinou, S., Brujic-Okretic, V., Baker, S.: Recent advances in the user evaluation methods and studies of non-photorealistic visualisation and rendering techniques. In: Proc. IV, pp. 475–480. IEEE Comput. Soc., Los Alamitos (2008). doi: 10.1109/IV.2008.75 Google Scholar
  14. 14.
    Gerstner, T., DeCarlo, D., Alexa, M., Finkelstein, A., Gingold, Y., Nealen, A.: Pixelated image abstraction. In: Proc. NPAR, pp. 29–36. Eurographics Association, Goslar (2012). doi: 10.2312/PE/NPAR/NPAR12/029-036 Google Scholar
  15. 15.
    Girshick, A., Interrante, V., Haker, S., Lemoine, T.: Line direction matters: an argument for the use of principal directions in 3D line drawings. In: Proc. NPAR, pp. 43–52. ACM, New York (2000). doi: 10.1145/340916.340922 CrossRefGoogle Scholar
  16. 16.
    Gooch, B.: Human facial illustrations: creation and evaluation using behavioral studies and functional magnetic resonance imaging. Ph.D. thesis, University of Utah, USA (2003) Google Scholar
  17. 17.
    Gooch, A.A., Willemsen, P.: Evaluating space perception in NPR immersive environments. In: Proc. NPAR, pp. 105–110. ACM, New York (2002). doi: 10.1145/508530.508549 Google Scholar
  18. 18.
    Gooch, B., Reinhard, E., Gooch, A.A.: Human facial illustrations: creation and psychophysical evaluation. ACM Trans. Graph. 23(1), 27–44 (2004). doi: 10.1145/966131.966133 CrossRefGoogle Scholar
  19. 19.
    Gooch, A.A., Long, J., Ji, L., Estey, A., Gooch, B.S.: Viewing progress in non-photorealistic rendering through Heinlein’s lens. In: Proc. NPAR, pp. 165–171. ACM, New York (2010). doi: 10.1145/1809939.1809959 Google Scholar
  20. 20.
    Greenberg, S., Buxton, B.: Usability evaluation considered harmful (some of the time). In: Proc. CHI, pp. 111–120. ACM, New York (2008). doi: 10.1145/1357054.1357074 Google Scholar
  21. 21.
    Halper, N., Mellin, M., Herrmann, C.S., Linneweber, V., Strothotte, T.: Psychology and non-photorealistic rendering: the beginning of a beautiful relationship. In: Proc. Mensch & Computer, pp. 277–286. Teubner, Stuttgart (2003) CrossRefGoogle Scholar
  22. 22.
    Halper, N., Mellin, M., Herrmann, C.S., Linneweber, V., Strothotte, T.: Towards an understanding of the psychology of non-photorealistic rendering. In: Proc. Workshop Computational Visualistics, Media Informatics and Virtual Communities, pp. 67–78. Deutscher Universitäts, Wiesbaden (2003) CrossRefGoogle Scholar
  23. 23.
    Healey, C.G., Enns, J.T.: Perception and painting: a search for effective, engaging visualizations. IEEE Comput. Graph. Appl. 22(2), 10–15 (2002). doi: 10.1109/38.988741 CrossRefGoogle Scholar
  24. 24.
    Healey, C.G., Tateosian, L., Enns, J.T., Remple, M.: Perceptually-based brush strokes for nonphotorealistic visualization. ACM Trans. Graph. 23(1), 64–96 (2004). doi: 10.1145/966131.966135 CrossRefGoogle Scholar
  25. 25.
    Hertzmann, A.: A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 23(4), 70–81 (2003). doi: 10.1109/MCG.2003.1210867 CrossRefGoogle Scholar
  26. 26.
    Hertzmann, A.: Non-photorealistic rendering and the science of art. In: Proc. NPAR, pp. 147–157. ACM, New York (2010). doi: 10.1145/1809939.1809957 Google Scholar
  27. 27.
    Igarashi, T., Matsuoka, S., Tanaka, H.T.: A sketching interface for 3D freeform design. In: Proc. SIGGRAPH, pp. 409–416. ACM, New York (1999). doi: 10.1145/311535.311602 Google Scholar
  28. 28.
    Inglis, T.C., Kaplany, C.S.: Pixelating vector line art. In: Proc. NPAR, pp. 21–28. Eurographics Association, Goslar (2012). doi: 10.2312/PE/NPAR/NPAR12/021-028 Google Scholar
  29. 29.
    Isenberg, T., Freudenberg, B., Halper, N., Schlechtweg, S., Strothotte, T.: A developer’s guide to silhouette algorithms for polygonal models. IEEE Comput. Graph. Appl. 23(4), 28–37 (2003). doi: 10.1109/MCG.2003.1210862 CrossRefGoogle Scholar
  30. 30.
    Isenberg, T., Neumann, P., Carpendale, S., Sousa, M.C., Jorge, J.A.: Non-photorealistic rendering in context: an observational study. In: Proc. NPAR, pp. 115–126. ACM, New York (2006). doi: 10.1145/1124728.1124747 Google Scholar
  31. 31.
    Jackson, C.D., Acevedo, D., Laidlaw, D.H., Drury, F., Vote, E., Keefe, D.: Designer-critiqued comparison of 2D vector visualization methods: a pilot study. In: ACM SIGGRAPH Sketches & Applications. ACM, New York (2003). doi: 10.1145/965400.965505 Google Scholar
  32. 32.
    Kaptein, M., Robertson, J.: Rethinking statistical analysis methods for CHI. In: Proc. CHI, pp. 1105–1114. ACM, New York (2012). doi: 10.1145/2207676.2208557 Google Scholar
  33. 33.
    Keefe, D.F., Karelitz, D.B., Vote, E.L., Laidlaw, D.H.: Artistic collaboration in designing VR visualizations. IEEE Comput. Graph. Appl. 25(2), 18–23 (2005). doi: 10.1109/MCG.2005.34 CrossRefGoogle Scholar
  34. 34.
    Kerlinger, F.N., Lee, H.B.: Foundations of Behavioral Research, 4th edn. Wadsworth Publishing/Thomson Learning, London (2000) Google Scholar
  35. 35.
    Kim, S., Hagh-Shenas, H., Interrante, V.: Conveying shape with texture: an experimental investigation of the impact of texture type on shape categorization judgments. In: Proc. InfoVis, pp. 163–170. IEEE Comput. Soc., Los Alamitos (2003). doi: 10.1109/INFVIS.2003.1249022 Google Scholar
  36. 36.
    Kim, S., Hagh-Shenas, H., Interrante, V.: Conveying shape with texture: experimental investigation of texture’s effects on shape categorization judgments. IEEE Trans. Vis. Comput. Graph. 10(4), 471–483 (2004). doi: 10.1109/TVCG.2004.5 CrossRefGoogle Scholar
  37. 37.
    Laidlaw, D.H.: Loose, artistic “textures” for visualization. IEEE Comput. Graph. Appl. 21(2), 6–9 (2001). doi: 10.1109/38.909009 CrossRefGoogle Scholar
  38. 38.
    Laidlaw, D., Kirby, R., Jackson, C., Davidson, J., Miller, T., da Silva, M., Warren, W., Tarr, M.: Comparing 2D vector field visualization methods: a user study. IEEE Trans. Vis. Comput. Graph. 11(1), 59–70 (2005). doi: 10.1109/TVCG.2005.4 CrossRefGoogle Scholar
  39. 39.
    Maciejewski, R., Isenberg, T., Andrews, W.M., Ebert, D.S., Sousa, M.C.: Aesthetics of hand-drawn vs. computer-generated stippling. In: Proc. CAe, pp. 53–56. Eurographics Association, Goslar (2007). doi: 10.2312/COMPAESTH/COMPAESTH07/053-056 Google Scholar
  40. 40.
    Maciejewski, R., Isenberg, T., Andrews, W.M., Ebert, D.S., Sousa, M.C., Chen, W.: Measuring stipple aesthetics in hand-drawn and computer-generated images. IEEE Comput. Graph. Appl. 28(2), 62–74 (2008). doi: 10.1109/MCG.2008.35 CrossRefGoogle Scholar
  41. 41.
    Mandryk, R.L., Mould, D., Li, H.: Evaluation of emotional response to non-photorealistic images. In: Proc. NPAR, pp. 7–16. ACM, New York (2011). doi: 10.1145/2024676.2024678 Google Scholar
  42. 42.
    Martín, D., Arroyo, G., Luzón, M.V., Isenberg, T.: Example-based stippling using a scale-dependent grayscale process. In: Proc. NPAR, pp. 51–61. ACM, New York (2010). doi:  10.1145/1809939.1809946 Google Scholar
  43. 43.
    Martín, D., Arroyo, G., Luzón, M.V., Isenberg, T.: Scale-dependent and example-based stippling. Comput. Graph. 35(1), 160–174 (2011). doi: 10.1016/j.cag.2010.11.006 CrossRefGoogle Scholar
  44. 44.
    Mould, D., Mandryk, R.L., Li, H.: Emotional response and visual attention to non-photorealistic images. Comput. Graph. 36(5), 658–672 (2012). doi: 10.1016/j.cag.2012.03.039 CrossRefGoogle Scholar
  45. 45.
    Rautek, P., Bruckner, S., Gröller, E., Viola, I.: Illustrative visualization: new technology or useless tautology? Comput. Graph. 42(3), 4:1–4:8 (2008). doi: 10.1145/1408626.1408633 Google Scholar
  46. 46.
    Saito, T., Takahashi, T.: Comprehensible rendering of 3-D shapes. Comput. Graph. 24(3), 197–206 (1990). doi: 10.1145/97880.97901 Google Scholar
  47. 47.
    Salesin, D.H.: Non-photorealistic animation & rendering: 7 grand challenges. Keynote talk at NPAR (2002) Google Scholar
  48. 48.
    Santella, A., DeCarlo, D.: Visual interest and NPR: an evaluation and manifesto. In: Proc. NPAR, pp. 71–78. ACM, New York (2004). doi: 10.1145/987657.987669 CrossRefGoogle Scholar
  49. 49.
    Schmidt, R., Isenberg, T., Jepp, P., Singh, K., Wyvill, B.: Sketching, scaffolding, and inking: a visual history for interactive 3D modeling. In: Proc. NPAR, pp. 23–32. ACM, New York (2007). doi: 10.1145/1274871.1274875 CrossRefGoogle Scholar
  50. 50.
    Schumann, J., Strothotte, T., Raab, A., Laser, S.: Assessing the effect of non-photorealistic rendered images in CAD. In: Proc. CHI, pp. 35–42. ACM, New York (1996). doi: 10.1145/238386.238398 Google Scholar
  51. 51.
    Seifi, H., DiPaola, S., Enns, J.: Exploring the effect of color palette in painterly rendered character sequences. In: Proc. CAe, pp. 89–97. Eurographics Association, Goslar (2012). doi: 10.2312/COMPAESTH/COMPAESTH12/089-097 Google Scholar
  52. 52.
    Strothotte, T., Preim, B., Raab, A., Schumann, J., Forsey, D.R.: How to render frames and influence people. Comput. Graph. Forum 13(3), 455–466 (1994). doi: 10.1111/1467-8659.1330455 CrossRefGoogle Scholar
  53. 53.
    Tietjen, C., Isenberg, T., Preim, B.: Combining silhouettes, shading, and volume rendering for surgery education and planning. In: Proc. EuroVis, pp. 303–310. Eurographics Association, Goslar (2005). doi: 10.2312/VisSym/EuroVis05/303-310 Google Scholar
  54. 54.
    Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006). doi: 10.1145/1141911.1142018 CrossRefGoogle Scholar
  55. 55.
    Winnemöller, H., Feng, D., Gooch, B., Suzuki, S.: Using NPR to evaluate perceptual shape cues in dynamic environments. In: Proc. NPAR, pp. 85–92. ACM, New York (2007). doi: 10.1145/1274871.1274885 CrossRefGoogle Scholar
  56. 56.
    Zhao, M., Zhu, S.C.: Sisley the abstract painter. In: Proc. NPAR, pp. 99–107. ACM, New York (2010). doi: 10.1145/1809939.1809951 Google Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  1. 1.INRIA SaclayOrsayFrance

Personalised recommendations