We present the first step of a research aimed at automating a driven interactive 3D modeling of an existing architectural object. The method is based on oriented multi-image spherical panoramas produced by stitching techniques. The photogrammetric process has two steps: the creation of a semi-automatic process to find homolog points in two panoramas; the creation of parametric definitions for an interactive modeling creating points, segments, and surfaces based on the plotted points in the first step. By connecting these two steps, the creation of the model will be automatic, as we indicate the necessary points in just one panoramic photo. The principals of multi-view geometry and epipolar geometry were applied to simplify the calculation in the first step in order to create an automatic identification of the correspondent points in the other panorama. The epipolar geometry is described by both analytical and graphical programming, implementing in the first case a C++ application and in the second case a Rhinoceros and Grasshopper application. A case study of the Ponte Rotto in Rome (Italy) is presented.
The interest in panoramic images is growing rapidly. Currently they are widely and mainly used for applications such as 3D virtual tours and Street Views. Furthermore, providing the panorama with accurate information about its location and orientation adds great metric value, allowing the users to acquire metric information about the scene. Regarding documentation, one noteworthy application was the metric documentation of some Syrian monuments in the UNESCO Heritage sites before the war using the spherical photogrammetry technique, presented in (Fangi et al. 2013). Several investigations have been made through the years, as it is a very challenging problem and of interest to a wide range of application fields (Wahbeh 2013).
The calibration and orientation process for panoramic images produced by direct rotating panoramic cameras have been analyzed in depth in (Schneider and Hans-Gerd 2006) and (Parian and Gruen 2010), while Petteri Pëntinen (2004) has investigated the errors which occur in the production of panoramic images from a set of planar images (stitching technique). Luhmann et al. (2004) describe the form of the epipolar line for cylindrical imagery obtained both with stitching and with rotating line scanner camera, demonstrating that the trajectory of the epipolar line on the image is a sinusoid. Fangi and Carla (2013) have analyzed the trajectory of the epipolar line on the projection of spherical images. Investigations about the multi-view geometry of panoramic images have been carried out in the field of the computer vision (Torii et al. 2000; Torii and Imiya 2007). In fact, their large field of view (FOV) is very important for many applications which require a synoptic understanding of the scene, such as robotic applications. Fangi (2007) proved that panoramic images produced by the stitching technique may be given a good metric quality by introducing the panoramic spherical photogrammetry tool (PSP), which was mainly conceived and designed for cultural and architectural metric documentation. A considerable number of monuments have been reconstructed using Fangi’s technique (Pisa et al. 2010). The metric quality has been investigated, confirming their capability to reconstruct 3D objects with an accuracy on the order of 10−3 to 10−4 of the object. Fangi and Carla (2013) provide a survey of the photogrammetric processing of panoramic images and describe in detail the flow chart of PSP algorithms.
This paper presents the first step of a research project that aims to automate a data-driven interactive 3D modelling of an existing architectural object, as this is the most time consuming step of the photogrammetric process. Its starting points are a set of spherical panoramas already oriented by the PSP tool. The graphical programming in 3D modelers, thanks to its practical visualization, helps in the creation of algorithms using a simple interface that can lead directly to the 3D modelling. The present work is organized as following. First, we describe the generation of a spherical panorama using the stitching method, and explain the projection used to map the pano-sphere image on the plane and the relationship between spherical coordinates and image coordinates. We then illustrate the epipolar geometry of spherical panoramas. Next, a case study of the epipolar geometry of an actual structure is presented; three panoramas of the Ponte Rotto in Rome are used and the trajectory of the epipolar line on the three images is shown. Successively, the visual programming is introduced. An example of interactive 3D modelling based on three virtual panoramic images as well as the interactive 3D modelling of the real application of Ponte Rotto are presented. The conclusions describe the next step toward the automation of the process.
A panoramic image is characterized by a large FOV, which can even extend to a full 360°. Different techniques can be used to produce such images. The possibility of obtaining panoramic imagery simply by merging a set of planar pictures is very interesting. In fact, the stitching technique was invented in the 1990s by Szeliski et al. (1997) for Apple Computer, with the aim of improving the poor resolution of the digital cameras available at that time (Fig. 1).
Images can be taken from a unique station point up to 360° around. They are merged, normally with a scale-invariant feature transform (SIFT) algorithm using the overlapping area between images. Figure 2 shows the tie points extracted by the stitching program to perform the alignment of two images.
More specifically, a spherical panorama image is a planar representation of a sphere (Fig. 3) where images taken from the same point of view with different orientations and partial overlapping have been projected (Szeliski 1994).
There are several commercial software programs capable of producing accurate panoramic images. In the following application we have used Hugin, an open-source stitching program. The stitching software is able to compensate for both radial distortion and residual horizontal and vertical shift offsets so as to produce a final panoramic image that can be considered virtually free of distortion (Pëntinen 2004).
The sphere can be mapped with different cartographic projection into a plane. Spherical images based on PSP are mapped following an equirectangular projection, that is, a projection in which meridians (or parallels) are represented by vertical (or horizontal) straight lines, and which is neither equivalent nor conformal. The FOV can be 360° and the two poles are represented by straight lines whose lengths are equal to that of the equator.
The x-axis represents the longitude and the y-axis the co-latitude; for this reason the projection is also called latitude-longitude projection. This kind of projection is very useful because it has a particularly simple relationship between the position of an image pixel (U, V) on the map and the curvilinear coordinates (co-latitude θ and longitude φ) defined on the sphere: U = θ r, V = φ r, where θ is the longitude and φ the complement of the latitude.
The ray r can be considered equal to 1 without a loss of generality and (U,V) coordinates coincide with the longitude and co-latitude. Figure 4 shows the equirectangular projection of the multi-image panorama. The contribution of each image is also delineated.
Considering a reference system (x, y, z) concentric with the pano-sphere, the orientation consists in the determination of its position, that is, the coordinates of the center of projection and the angles of the axis with respect to a global reference system. Normally the pano-sphere is quasi-vertical and therefore the orientation produces very small angle around the x- and y-axes, while the orientation angle around the z-axis can assume values ranging from 0 to 2π according to the relative position of the global reference system and the pano-sphere reference system.
The PSP tool performs the images’ orientation using a bundle block adjustment. It requires at least five points and allows working both in absolute and relative modes. In the latter case only tie points are used during the orientation and the 3D model is obtained, except for the scale.
Epipolar Geometry of a Spherical Panorama
Epipolar geometry is well known from classical photogrammetry. Given a point P′ in the first image, the corresponding point P″ in the second image lies on a specific line called the epipolar line, which is the intersection of the image with the plane defined by the two centers of the two images O′ and O″ and the point P′ (epipolar plane). Therefore, epipolar geometry provides the line where the corresponding point is constrained, reducing the search space of homolog points from 2D to 1D (from surface to line) thus aiding both automatic and manual detection of the homolog point.
The same consideration can be extended to panoramic images. In this case, the intersection of the epipolar plane π12 with the pano-sphere is a circle l 12 in the 3D space (Fig. 5). The search for the homolog point need only be performed along the epipolar line l 12.
The process can be described in the following steps. Given a point P 1′ in the first panorama S1 with projection center in O′, it is possible to depict on the second panorama S 2 the epipolar plane π12 that passes through O′ and O″ and the image point P 1′. The intersection of the epipolar plane π12 and the pano-sphere S 2 produces the epipolar line l 12. Among all the points which belong to the epipolar line l 12, there is the homolog point P″.
A third panorama improves the determination of the intersection of the rays, as illustrated in Torii et al. (2000) and Torii and Imiya (2007), which present interesting considerations about the geometry of spherical imagery. They define the circle, called epipolar circle C 12, which passes through the following three points: the two projection centers O 1 and O 2, and a point P, where O 1 and O 2 are the projection centers of two pano-sphere and P is a 3D point produced from the intersection of the projective rays r′ and r″ respectively from the first and the second pano-spheres. The circle C 12 is defined on the epipolar plane π 12 (Fig. 6). Subsequently, the planes which have as normal the projective rays r′ and r″ intersect with the epipolar plane π 12 in a point X 12 which also belongs to the epipolar circle (Fig. 7). This point, together with the projection centers, allows defining the epipolar circle.
Considering that the position of P is unknown, then introducing a third pano-sphere, it is possible to define three epipolar circles: C 12, C 13 and C 23. The first is defined by O 1, O 2 and the point X 12, the second by O 1, O 3 and X 13, and the third by O 2, O 3 and X 23. All the three circles pass through the point P (Fig. 8). The intersection of the three epipolar circle can be verified in order to assess the accuracy of the intersection.
The Application: Ponte Rotto
The Pons Aemilius, today called Ponte Rotto, or Broken Bridge, is the oldest Roman stone bridge in Rome. Preceded by a wooden version, it was rebuilt in stone in the second century BC. It once spanned the Tiber, connecting the Forum Boarium with Trastevere; a single arch in mid-river is all that remains today, hence its name (Fig. 9).
The survey of the Ponte Rotto in Rome consisted of nineteen panoramas, of which three have been chosen for the present application (Fig. 10). The camera was a Canon 550D equipped with a 17 mm lens. A Nodal Ninja spherical head was used. In this case the panoramas do not have a 360° FOV. About forty-five images for each pano-sphere have been acquired with a mean overlap of about 35 %. The panoramas were generated by the open-source software Hugin. Table 1 lists the geometrical characteristics of the three spherical images.
Orientation of the Panoramic Images
The three panoramas were oriented by the bundle block adjustment implemented in PSP. Forty natural tie points corresponding to the architectural features were used for the orientation of the three panoramas. Table 2 provides the external orientation parameters.
Epipolar Geometry of the Images
Based on a C++ application the trajectory of the epipolar line in spherical images was implemented and is shown in Fig. 11. More specifically, Fig. 11 shows three images which do not form a 360° panorama, but only a portion of the entire FOV. Figure 11a shows the indication of a point on the first panorama; then the trajectory of the epipolar line of the point indicated in the first panorama on the second and the third panorama, respectively l 12 and l 13 are depicted (Fig. 11b, c). Figure 11d shows the intersection of the epipolar lines l 12 and l 23. The intersection of the two curves indicates two points, of which one is the corresponding point on the second panorama.
Image-based 3D Modelling Using Visual Programming
The use of visual programming in the present process is aimed at directly connecting the collimation of points on the image (input) and the 3D modelling (output) to produce an interactive model. The modelling process in this research starts with a manual collimation of homolog points and has the following steps:
The panoramas are inserted in the modeler software as a surface;
The corresponding points are manually collimated on the surface of every image;
The (U,V) coordinates of the image surface and the sphere are put in relation.
In order to generate an automatic process a parametric modelling definition was created with an order of concepts as follows:
Inputs: two equirectangular panorama images; images external orientation parameters; the observation point for each panorama (P′ and P″).
Definition of two image surface domains (equirectangular projection image).
Correlation of the global reference system coordinates of a point and the image surface domain to extract the U and V components (image reference system pixel coordinate).
Creation of two spherical surface domains in the projection centers (O 1 and O 2) expressed in the global reference system and rotated according to the orientation parameters.
Connection of the image surface domain with the sphere surface domain for every image in order to obtain the two spatial positions (P′ and P″) of any point P collimated on the image domains.
Creation of the projection rays (R 1 and R 2) respectively from two points O 1 and P′ for the first sphere and O 2 and P″ for the second sphere;
Determination of the intersection between the projection rays R 1 and R 2 to obtain the 3D position of the surveyed point in the space P.
Those concepts were translated into components to create the parametric model generation (Fig. 12).
A 3D Virtual Application
To study an interactive modelling definition a virtual space model with known shape and coordinates was created, and in order to have clear and visible features the walls were colored (Fig. 13). Next, two cameras in known positions were positioned and using the spherical rendering, two 360° equirectangular panoramas with a known orientation were generated (Fig. 14).
To simplify the work with many observation points it is useful to use polylines and explode them to use the coordinates of their vertices. The collimation was carried out creating two homolog polylines in the two panoramas in the same order of vertices to verify the definition (Fig. 15). The restitution of a net in the virtual model is performed by intersecting the projection rays of every observation point.
Because there are no orientation errors in this virtual case, the results depend only on the accuracy of the manual collimation. The virtual application has verified the correctness of the definition (Fig. 16).
Case Study: Ponte Rotto in Rome
The main differences between this application and the virtual one are the use of real and partial panoramas. In this case, the orientation of a real panorama, which is a very important phase of any photogrammetric restitution process, cannot be considered error-free. In fact, many factors contribute to the accuracy of the results, such as the errors in the determination of the nodal point, the stitching, the collimation, the distribution and the quantity of points used for the bundle block adjustments.
Since the hardware limits the use to a partial panorama instead of a full 360°, it permits reducing the image dimension or using higher resolution images within the hardware capability. Moreover, it reduces the number of photos that must be captured. Even when only the area of interest is used, the entire panorama (360° × 180°) has to be considered because the partial panorama has to occupy the exact portion of the entire one. For this reason, the image dimensions in pixels and the dimensions of the complete panorama for the same resolution have to be supplied (Fig. 17).
The definition used for the real application is the same presented in the previous paragraph (Fig. 18). Series of collimation points presented by polylines have been used as input to obtain the positions of the survey points in the virtual space by means of the triangulations (Fig. 19).
The dimensional modelling usually starts with the identification of vertices, segments and surfaces. To locate a vertex in the space a triangulation must be done from a minimum of two different panoramas (Fig. 20). The three epipolar circles, explained earlier, are shown in Fig. 21. Many shapes and geometries could be created using techniques which exploit the constraints in the architecture (Wahbeh 2012). For example, drawing planar curves is a common practice in architecture which requires the recognition that the curve is planar. At this point no triangulation is needed, but simply an intersection between straight line and plane (Fig. 22). Therefore, if the plane is identified, just one projection is needed to create the curve points. The plane has to be indicated by three points and the intersection of projected lines, and the plane produces the survey points.
This is a technique that can be used in many cases, especially in classical architecture, for the mouldings, which are usually horizontal, or to draw the curves of the façade’s vertical elevation using just one projection. Many other constraints could be defined. Therefore, reading and understanding the geometry of architecture allow exploiting its constraints, making it easier to create its digital model using the panoramic image.
The use of panoramas in photogrammetry is becoming more common thanks to digital photography and developments in computer technology. This research aims to achieve a direct connection between the photogrammetry and 3D modelling in order to obtain interactive 3D models. The objective is to be able to choose a single point to be plotted on a single image to create the 3D model in real time. Future developments will regard the improvement of the level of automation to automatically find homolog points in different panoramas and to complete the modelling process using script programming.
Fangi, Gabriele. 2007. The multi-image spherical panoramas as a tool for architectural survey. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 36 (5/C53): 311–316.
Fangi, Gabriele, Livia Piermattei and Wissam Wahbeh. 2013. Beni dell’UNESCO in Siria documentati metricamente prima della guerra con la tecnica della fotogrammetria sferica. Pp. 433-442 in Patrimoni e Siti UNESCO. Memoria, Misura e Armonia. Atti del XXXV Congresso Internazionale dei Docenti della Rappresentazione—Decimo Congresso UID, Mario Docci, ed.. Matera: Gangemi Editore.
Fangi, Gabriele and Nardinocchi Carla. 2013. Photogrammetric processing of spherical panoramas. The Photogrammetric Record 28 (143): 293–311.
Luhmann, Thomas and Werner Tecklenburg. 2004. 3D object reconstruction from multiple-station panorama imagery. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 34 (5/W16). (on CD-ROM). http://www.isprs.org/proceedings/XXXIV/5-W16/papers/PanoWS_Dresden2004_Luhmann_b.pdf, accessed 03 January 2015
Parian, Jafar A. and Armin Gruen. 2010. Sensor modelling, self-calibration and accuracy testing of panoramic cameras and laser scanners. ISPRS Journal of Remote Sensing and Photogrammetry 65 (1): 60–76.
Pisa, Cecilia, Fabiana Zeppa and Gabriele Fangi. 2010. Spherical photogrammetry for cultural heritage. Pp. 3–6 in Proceeding of the Second Workshop on eHeritage and Digital Art Preservation, 25–29 October 2010, Florence, Italy. New York: ACM.
Pëntinen, Petteri. 2004. On the geometrical quality of panoramic images. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 35 (B5): 82–87.
Schneider, Danilo and Maas Hans-Gerd. 2006. A geometric model for linear-array-based terrestrial panoramic cameras. Photogrammetric Record 21(115): 198–210.
Szeliski, Richard. and Heung-Yeung Shum. 1997. Creating full view panoramic image mosaics and environment maps. Pp. 251–258 in Proceedings of SIGGRAPH 1997. http://research.microsoft.com/pubs/75673/Szeliski-SG97.pdf, accessed 02 January 2015.
Szeliski, Richard. 1994. Image mosaicing for tele-reality applications. Cambridge Research Laboratory, Technical Report Series. http://www.hpl.hp.com/techreports/Compaq-DEC/CRL-94-2.pdf, accessed 02 January 2015.
Torii, Akihiko and Atsushi Imiya. 2007. Computation of epipolar geometry and trifocal tensor from spherical images. Computer Vision Winter Workshop, St Lambrecht, Austria (on CD-ROM).
Torii, Akihiko, Atsushi Imiya and Naoya Ohnishi, N., 2000. Two- and three-view geometry for spherical cameras. 6th Workshop on Omnidirectional Vision, Camera Network and Non-classical Cameras, Beijing, China (on CD-ROM).
Wahbeh, Wissam. 2012. Architectural Digital Photogrammetry. Pp. 175–180 in Nexus Ph.D. Day: Relationships between Architecture And Mathematics. Milan: McGraw-Hill.
Wahbeh, Wissam. 2013. Architectural Photogrammetry panoramic Image-Based Photogrammetry. Pp. 199–201 in Linee di Ricerca nell’area del Disegno, Contributi dalle tesi di dottorato. Laura Carlevaris, ed. Atti del X congresso UID, Matera 2013. Rome: Aracne editrice.
The authors thank Gabriele Fangi of the Università Politecnica delle Marche for his collaboration during the orientation phase. All images and photographs in this paper are by the authors.
About this article
Cite this article
Wahbeh, W., Nardinocchi, C. Toward the Interactive 3D Modelling Applied to Ponte Rotto in Rome. Nexus Netw J 17, 55–71 (2015). https://doi.org/10.1007/s00004-015-0238-8
- Image-based modeling
- Panoramic photos
- 3D modelling
- Epipolar geometry
- Homolog points
- Spherical photogrammetry