Toward the Interactive 3D Modelling Applied to Ponte Rotto in Rome
Abstract
We present the first step of a research aimed at automating a driven interactive 3D modeling of an existing architectural object. The method is based on oriented multiimage spherical panoramas produced by stitching techniques. The photogrammetric process has two steps: the creation of a semiautomatic process to find homolog points in two panoramas; the creation of parametric definitions for an interactive modeling creating points, segments, and surfaces based on the plotted points in the first step. By connecting these two steps, the creation of the model will be automatic, as we indicate the necessary points in just one panoramic photo. The principals of multiview geometry and epipolar geometry were applied to simplify the calculation in the first step in order to create an automatic identification of the correspondent points in the other panorama. The epipolar geometry is described by both analytical and graphical programming, implementing in the first case a C++ application and in the second case a Rhinoceros and Grasshopper application. A case study of the Ponte Rotto in Rome (Italy) is presented.
Keywords
Photogrammetry Imagebased modeling Panoramic photos 3D modelling Epipolar geometry Homolog points Polylines Spherical photogrammetryIntroduction
The interest in panoramic images is growing rapidly. Currently they are widely and mainly used for applications such as 3D virtual tours and Street Views. Furthermore, providing the panorama with accurate information about its location and orientation adds great metric value, allowing the users to acquire metric information about the scene. Regarding documentation, one noteworthy application was the metric documentation of some Syrian monuments in the UNESCO Heritage sites before the war using the spherical photogrammetry technique, presented in (Fangi et al. 2013). Several investigations have been made through the years, as it is a very challenging problem and of interest to a wide range of application fields (Wahbeh 2013).
The calibration and orientation process for panoramic images produced by direct rotating panoramic cameras have been analyzed in depth in (Schneider and HansGerd 2006) and (Parian and Gruen 2010), while Petteri Pëntinen (2004) has investigated the errors which occur in the production of panoramic images from a set of planar images (stitching technique). Luhmann et al. (2004) describe the form of the epipolar line for cylindrical imagery obtained both with stitching and with rotating line scanner camera, demonstrating that the trajectory of the epipolar line on the image is a sinusoid. Fangi and Carla (2013) have analyzed the trajectory of the epipolar line on the projection of spherical images. Investigations about the multiview geometry of panoramic images have been carried out in the field of the computer vision (Torii et al. 2000; Torii and Imiya 2007). In fact, their large field of view (FOV) is very important for many applications which require a synoptic understanding of the scene, such as robotic applications. Fangi (2007) proved that panoramic images produced by the stitching technique may be given a good metric quality by introducing the panoramic spherical photogrammetry tool (PSP), which was mainly conceived and designed for cultural and architectural metric documentation. A considerable number of monuments have been reconstructed using Fangi’s technique (Pisa et al. 2010). The metric quality has been investigated, confirming their capability to reconstruct 3D objects with an accuracy on the order of 10^{−3} to 10^{−4} of the object. Fangi and Carla (2013) provide a survey of the photogrammetric processing of panoramic images and describe in detail the flow chart of PSP algorithms.
This paper presents the first step of a research project that aims to automate a datadriven interactive 3D modelling of an existing architectural object, as this is the most time consuming step of the photogrammetric process. Its starting points are a set of spherical panoramas already oriented by the PSP tool. The graphical programming in 3D modelers, thanks to its practical visualization, helps in the creation of algorithms using a simple interface that can lead directly to the 3D modelling. The present work is organized as following. First, we describe the generation of a spherical panorama using the stitching method, and explain the projection used to map the panosphere image on the plane and the relationship between spherical coordinates and image coordinates. We then illustrate the epipolar geometry of spherical panoramas. Next, a case study of the epipolar geometry of an actual structure is presented; three panoramas of the Ponte Rotto in Rome are used and the trajectory of the epipolar line on the three images is shown. Successively, the visual programming is introduced. An example of interactive 3D modelling based on three virtual panoramic images as well as the interactive 3D modelling of the real application of Ponte Rotto are presented. The conclusions describe the next step toward the automation of the process.
Spherical Panorama
There are several commercial software programs capable of producing accurate panoramic images. In the following application we have used Hugin, an opensource stitching program. The stitching software is able to compensate for both radial distortion and residual horizontal and vertical shift offsets so as to produce a final panoramic image that can be considered virtually free of distortion (Pëntinen 2004).
The sphere can be mapped with different cartographic projection into a plane. Spherical images based on PSP are mapped following an equirectangular projection, that is, a projection in which meridians (or parallels) are represented by vertical (or horizontal) straight lines, and which is neither equivalent nor conformal. The FOV can be 360° and the two poles are represented by straight lines whose lengths are equal to that of the equator.
The xaxis represents the longitude and the yaxis the colatitude; for this reason the projection is also called latitudelongitude projection. This kind of projection is very useful because it has a particularly simple relationship between the position of an image pixel (U, V) on the map and the curvilinear coordinates (colatitude θ and longitude φ) defined on the sphere: U = θ r, V = φ r, where θ is the longitude and φ the complement of the latitude.
Considering a reference system (x, y, z) concentric with the panosphere, the orientation consists in the determination of its position, that is, the coordinates of the center of projection and the angles of the axis with respect to a global reference system. Normally the panosphere is quasivertical and therefore the orientation produces very small angle around the x and yaxes, while the orientation angle around the zaxis can assume values ranging from 0 to 2π according to the relative position of the global reference system and the panosphere reference system.
The PSP tool performs the images’ orientation using a bundle block adjustment. It requires at least five points and allows working both in absolute and relative modes. In the latter case only tie points are used during the orientation and the 3D model is obtained, except for the scale.
Epipolar Geometry of a Spherical Panorama
Epipolar geometry is well known from classical photogrammetry. Given a point P′ in the first image, the corresponding point P″ in the second image lies on a specific line called the epipolar line, which is the intersection of the image with the plane defined by the two centers of the two images O′ and O″ and the point P′ (epipolar plane). Therefore, epipolar geometry provides the line where the corresponding point is constrained, reducing the search space of homolog points from 2D to 1D (from surface to line) thus aiding both automatic and manual detection of the homolog point.
The process can be described in the following steps. Given a point P _{1}′ in the first panorama S_{1} with projection center in O′, it is possible to depict on the second panorama S _{2} the epipolar plane π_{12} that passes through O′ and O″ and the image point P _{1}′. The intersection of the epipolar plane π_{12} and the panosphere S _{2} produces the epipolar line l _{12}. Among all the points which belong to the epipolar line l _{12}, there is the homolog point P″.
The Application: Ponte Rotto
Data Acquisition
Geometrical characteristic of the three spherical images
Image  Width (pixel)  Height (pixel)  360° width (pixel)  FOV (°) 

1  12,507  7,196  61,682  73 
2  13,643  6,597  66,014  74 
3  12,835  7,012  52,816  87 
Orientation of the Panoramic Images
External orientation parameters of the three panoramas
Panorama  X (m)  Y (m)  Z (m)  K (gon)  α_{x} (gon)  α_{y} (gon) 

1  112.894  41.574  7.607  359.0382  1.0802  −0.3498 
2  126.242  51.415  7.342  339.9277  0.0375  −0.5381 
3  143.621  49.14  7.962  332.7084  −0.3430  −0.0254 
Epipolar Geometry of the Images
Imagebased 3D Modelling Using Visual Programming
 1.
The panoramas are inserted in the modeler software as a surface;
 2.
The corresponding points are manually collimated on the surface of every image;
 3.
The (U,V) coordinates of the image surface and the sphere are put in relation.

Inputs: two equirectangular panorama images; images external orientation parameters; the observation point for each panorama (P′ and P″).

Definition of two image surface domains (equirectangular projection image).

Correlation of the global reference system coordinates of a point and the image surface domain to extract the U and V components (image reference system pixel coordinate).

Creation of two spherical surface domains in the projection centers (O _{1} and O _{2}) expressed in the global reference system and rotated according to the orientation parameters.

Connection of the image surface domain with the sphere surface domain for every image in order to obtain the two spatial positions (P′ and P″) of any point P collimated on the image domains.

Creation of the projection rays (R _{1} and R _{2}) respectively from two points O _{1} and P′ for the first sphere and O _{2} and P″ for the second sphere;

Determination of the intersection between the projection rays R _{1} and R _{2} to obtain the 3D position of the surveyed point in the space P.
A 3D Virtual Application
Case Study: Ponte Rotto in Rome
The main differences between this application and the virtual one are the use of real and partial panoramas. In this case, the orientation of a real panorama, which is a very important phase of any photogrammetric restitution process, cannot be considered errorfree. In fact, many factors contribute to the accuracy of the results, such as the errors in the determination of the nodal point, the stitching, the collimation, the distribution and the quantity of points used for the bundle block adjustments.
This is a technique that can be used in many cases, especially in classical architecture, for the mouldings, which are usually horizontal, or to draw the curves of the façade’s vertical elevation using just one projection. Many other constraints could be defined. Therefore, reading and understanding the geometry of architecture allow exploiting its constraints, making it easier to create its digital model using the panoramic image.
Conclusions
The use of panoramas in photogrammetry is becoming more common thanks to digital photography and developments in computer technology. This research aims to achieve a direct connection between the photogrammetry and 3D modelling in order to obtain interactive 3D models. The objective is to be able to choose a single point to be plotted on a single image to create the 3D model in real time. Future developments will regard the improvement of the level of automation to automatically find homolog points in different panoramas and to complete the modelling process using script programming.
Notes
Acknowledgments
The authors thank Gabriele Fangi of the Università Politecnica delle Marche for his collaboration during the orientation phase. All images and photographs in this paper are by the authors.
References
 Fangi, Gabriele. 2007. The multiimage spherical panoramas as a tool for architectural survey. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 36 (5/C53): 311–316.Google Scholar
 Fangi, Gabriele, Livia Piermattei and Wissam Wahbeh. 2013. Beni dell’UNESCO in Siria documentati metricamente prima della guerra con la tecnica della fotogrammetria sferica. Pp. 433442 in Patrimoni e Siti UNESCO. Memoria, Misura e Armonia. Atti del XXXV Congresso Internazionale dei Docenti della Rappresentazione—Decimo Congresso UID, Mario Docci, ed.. Matera: Gangemi Editore.Google Scholar
 Fangi, Gabriele and Nardinocchi Carla. 2013. Photogrammetric processing of spherical panoramas. The Photogrammetric Record 28 (143): 293–311.CrossRefGoogle Scholar
 Luhmann, Thomas and Werner Tecklenburg. 2004. 3D object reconstruction from multiplestation panorama imagery. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 34 (5/W16). (on CDROM). http://www.isprs.org/proceedings/XXXIV/5W16/papers/PanoWS_Dresden2004_Luhmann_b.pdf, accessed 03 January 2015
 Parian, Jafar A. and Armin Gruen. 2010. Sensor modelling, selfcalibration and accuracy testing of panoramic cameras and laser scanners. ISPRS Journal of Remote Sensing and Photogrammetry 65 (1): 60–76.CrossRefGoogle Scholar
 Pisa, Cecilia, Fabiana Zeppa and Gabriele Fangi. 2010. Spherical photogrammetry for cultural heritage. Pp. 3–6 in Proceeding of the Second Workshop on eHeritage and Digital Art Preservation, 25–29 October 2010, Florence, Italy. New York: ACM.Google Scholar
 Pëntinen, Petteri. 2004. On the geometrical quality of panoramic images. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 35 (B5): 82–87.Google Scholar
 Schneider, Danilo and Maas HansGerd. 2006. A geometric model for lineararraybased terrestrial panoramic cameras. Photogrammetric Record 21(115): 198–210.CrossRefGoogle Scholar
 Szeliski, Richard. and HeungYeung Shum. 1997. Creating full view panoramic image mosaics and environment maps. Pp. 251–258 in Proceedings of SIGGRAPH 1997. http://research.microsoft.com/pubs/75673/SzeliskiSG97.pdf, accessed 02 January 2015.
 Szeliski, Richard. 1994. Image mosaicing for telereality applications. Cambridge Research Laboratory, Technical Report Series. http://www.hpl.hp.com/techreports/CompaqDEC/CRL942.pdf, accessed 02 January 2015.
 Torii, Akihiko and Atsushi Imiya. 2007. Computation of epipolar geometry and trifocal tensor from spherical images. Computer Vision Winter Workshop, St Lambrecht, Austria (on CDROM).Google Scholar
 Torii, Akihiko, Atsushi Imiya and Naoya Ohnishi, N., 2000. Two and threeview geometry for spherical cameras. 6th Workshop on Omnidirectional Vision, Camera Network and Nonclassical Cameras, Beijing, China (on CDROM).Google Scholar
 Wahbeh, Wissam. 2012. Architectural Digital Photogrammetry. Pp. 175–180 in Nexus Ph.D. Day: Relationships between Architecture And Mathematics. Milan: McGrawHill.Google Scholar
 Wahbeh, Wissam. 2013. Architectural Photogrammetry panoramic ImageBased Photogrammetry. Pp. 199–201 in Linee di Ricerca nell’area del Disegno, Contributi dalle tesi di dottorato. Laura Carlevaris, ed. Atti del X congresso UID, Matera 2013. Rome: Aracne editrice.Google Scholar