1. Segmentation Based Environment Modeling Using a Single Image
SeungTaek Ryoo
Department of Software, HanShin University, Korea

2. Previous Works 3D Reconstruction Planar Objects
Single View Metrology (Criminisi1999)
Piecewise Planar Objects from Single Images (Sturm1999)
Automatic 3D Modeling of Architecture (Dick2000)
Image Based Modeling and Photo-Editing (Oh2001)
Criminis1999
Sturm1999
Oh2001
Dick2000

3. (Regular Image, OCC Map)
Objective
Objective
Realistic & Real-time View Rendering with Full-View Panorama
Image-Segmentation Based Environment Modeling
Environment Mapping
Desired Image
Input Image
(Regular Image, OCC Map)
Segmented Image
Image Segmentation
Depth Calculation
Depth Image
Environment Model
Environment Modeling
The Structure of Segmentation Based Modeling

4. Image Segmentation Segmentation
indoor scene consists of floor and ceiling.
outdoor scene consists of ground and sky.
the environment map is divided into
a floor (ground), ceiling (sky), and surrounding objects.
In general, An indoor scene consists of a floor and ceiling and the outdoor scene consists of ground and sky.
From such features, the environment map is divided into a floor (ground), ceiling (sky), and surrounding objects.
This figure shows the segmented image using the region based segmentation method.

5. Depth Calculation (1/4) Assumption
View direction of the camera must be parallel to the ground
The floor (ground) and the ceiling must consist of a plane that is parallel to the XZ plane in the world coordinate.
The sky must consist of a hemi-sphere with an infinite radius
The surrounding objects must be perpendicular to the ground
Image plane
From this segmented image, We can acquire the depth image using the following conditions.
View direction of the camera must be parallel to the ground
The floor (ground) and the ceiling must consist of a plane that is parallel to the XZ plane in the world coordinate.
The sky must consist of a hemi-sphere with an infinite radius
The surrounding objects must be perpendicular to the ground o WC m Z R

6. Depth Calculation (2/4) Extracting the depth value from Regular Image
The depth value can be obtained by testing for the intersection points between the floor and the ray, and between the Image plane and the ray
The equation can be obtained with the law of the triangle
Now, We’ll examine the issue of how to extract the depth value from a regular image
This figure shows the world coordinates and image plane, r is the distance between the origin and the image plane
And then setting the floor that parallel to the horizontal planes, d is the height value of the floor
The depth value can be obtained by testing for the intersection points between the floor and the ray, and between the Image plane and the ray.
we can obtain the Z value with the law of the triangle

7. Depth Calculation (3/4) Extracting the depth value from OCC map
depth value of the ceiling (floor) by testing the relationship b/w XZ cylinder and ray, and b/w XY cylinder and ray r b d Z R
Y = d
XY Cylinder
XZ Cylinder
Next, We’ll examine how to extract the depth value using the OCC Map,
We can also obtain depth value of the ceiling (floor) by testing the relationship between XZ cylinder and ray, and between XY cylinder and ray
We can obtain the same equation as before
the depth value of the surrounding objects can be obtained by setting the depth value of the floor on the objects surrounding the floor.

8. Depth Calculation (4/4)
How to get the depth value of the surrounding objects
Set the depth value of the floor(ceiling) close to
the surrounding objects.
Some other assumptions should be made.
the relationship between the surrounding objects.
the relationship between the floor(ceiling) and the surrounding objects.
the relationship between the image plane and the surrounding objects. V U o
Color Image
Segmented Image
Depth Image

9. Environment Modeling Rectangular Grid Triangulation Feature Points
3D Environment model can be made from a rectangular grid modeling method and a triangulation method
To make triangulated model, we need a preprocessing step that extracts feature points
In this study, the feature points can be extracted from color image and segmented image using edge detection methods
From these feature points, we make triangulated model using delaunay triangulation method

10. Texture mapping & Point sample rendering
This figure shows the image of environment model using texture mapping and point sample rendering
As shown in middle one, the stretching effect of the 3D model occurs by the occlusion
Also, holes occurs by the lack of the information when the point sample rendering method is applied
To resolve these problem, we must construct the environment model using a multiple number of images.
Wireframe
Texture Mapping
Point Sample Rendering
Texture mapping : the stretching effect of the 3D model occurs by the occlusion
Point Sample Rendering : holes occurs by the lack of the information

11. Regular Image Regular Image Desired Image Segmentation Environment
Mapping
Now, We’ll talk about the process of the environment mapping and environment modeling from a regular image
This Color figure show the hallway of the building.
we can obtain segmented image using region based segmentation method,
And then, We can get the depth image using the depth calculation equation.
And then, We can build the 3D Environment model using depth image through rectangular grid modeling methods
This figure shows the image of the environment model using Texture mapping methods
Depth Calculation
Environment
Modeling

12. Regular Image
Gyeongbokgung (Royal Palaces)
Seokguram (Stone Cave)

13. OCC Map (Indoor Scene) Segmentation Depth Calculation
Environment Map (Octahedral Map)
Orthogonal Cross Cylinder
Environment Mapping
Segmentation
This Figure shows the process of environment mapping from OCC map and environment modeling from depth image.
Depth Calculation
Environment Modeling

14. OCC Map (Outdoor Scene)
Orthogonal Cross Cylinder
Segmentation Based Environment Model
Segmentation
Desired Image
Depth Calculation

15. Result (Art Gallery) Environment Map (sidestep)
Segmented Based Environment Model (sidestep)

16. Conclusion & Future Works
can be used for environment modeling by extracting the depth value by the image segmentation
navigate through the virtual environment when the viewpoint has been changed
easy to implement on the environment map
Future Works
Multiple Resolution
the problem of the blurring-image effect caused by the fixed resolution
Multiple Images (Registration & Integration)
the problem of the stretching effect of the 3D model caused by the occlusion
Generalization to apply any type of images
only images whose viewpoint and the floor are perpendicular may be used.