WO2015021018A1

WO2015021018A1 - High-speed inspection of entities with respect to reference models

Info

Publication number: WO2015021018A1
Application number: PCT/US2014/049728
Authority: WO
Inventors: Jeff MAHLER
Original assignee: Lynx Laboratories Inc.
Priority date: 2013-08-05
Filing date: 2014-08-05
Publication date: 2015-02-12

Abstract

Implementations are directed to providing a difference model that includes differentiations between a representation of an entity and a reference model. In some implementations, actions include receiving data of the representation that provides a characterization of the entity, receiving data of the reference model, registering the representation and the reference model relative to one another, providing a characterization of the reference model, determining attribute differentiations between the representation and the reference model based on the characterization of the entity and the characterization of the reference model, integrating the attribute differentiations into a difference model, and transmitting instructions to display a graphical representation based on the difference model.

Description

HIGH-SPEED INSPECTION OF ENTITIES

WITH RESPECT TO REFERENCE MODELS

SUMMARY

[0001] Implementations of the present disclosure are generally directed to detecting and assessing differences between an entity and a reference model, the reference model modeling the entity.

[0002] Implementations of the present disclosure include computer-implemented methods for providing a difference model that includes differentiations between a representation of an entity and a reference model. In some implementations, actions include receiving data of the representation that provides a characterization of the entity, receiving data of the reference model, registering the representation and the reference model relative to one another, providing a characterization of the reference model, determining attribute differentiations between the representation and the reference model based on the characterization of the entity and the characterization of the reference model, integrating the attribute differentiations into a difference model, and transmitting instructions to display a graphical representation based on the difference model.

[0003] These and other implementations can each optionally include one or more of the following features: data defining the representation is received in real-time, as images of the entity that is represented by the representation are captured, the graphical representation is displayed in real-time, the representation of the entity is provided as one of a 2.5-dimensional representation, a three-dimensional representation, a volumetric representation, a parametric representation, and an octree representation, the reference model is provided as one of a 2.5-dimensional representation, a three-dimensional representation, a volumetric representation, a parametric representation, and an octree representation, the reference model includes a base representation of the entity that was captured previous to the representation of the entity, the base representation of the entity and the real-world representation of the entity depict the real-world scene, the difference model providing changes in the entity, actions further include scaling at least one of the representation of the entity and the reference model, actions further include converting at least one of the representation of the entity and the reference model to a comparison space, actions further include each attribute differentiation is associated with a respective attribute, attributes comprising one or more of position, depth, color, orientation, and curvature, the data of the representation provides a plurality of points, each point being associated with one or more attributes, and the data of the reference model provides a plurality of points, each point being associated with one or more attributes, determining attribute differentiations between the representation and the reference model includes comparing attributes of point pairs, each point pair including a point of the representation and a point of the reference model, comparing attributes of point pairs is concurrently performed for a plurality of point pairs, the plurality of point pairs includes a sub-set of all point pairs provided for the representation and the reference model, the plurality of point pairs includes all point pairs provided for the representation and the reference model, each attribute differentiation is associated with a weight, the weight reflecting a confidence in a value of the attribute differentiation, the weight is determined based on a difference between a value of the attribute differentiation and a respective value provided in the difference model, the weight is within a range defined from a minimum value and a maximum value, integrating the attribute differentiations into a difference model includes, for each attribute differentiation, determining a new attribute differentiation value based on a value of the attribute differentiation and a value of an existing attribute differentiation already provided in the difference model, actions further include storing the new attribute differentiation value in the difference model as the value of an existing attribute differentiation already provided in the difference model, the new attribute differentiation value is provided as a weighted average of the value of the attribute differentiation and the value of an existing attribute differentiation, the entity includes one or more physical objects, the entity includes an object that is insured, and the graphical representation is displayed to assess an insurance claim, the entity includes a manufactured part, and the graphical representation is displayed to assess defects, the entity includes a crime scene, and the graphical representation is displayed to investigate the crime scene, the entity includes an architectural structure, and the graphical representation is displayed to assess deviations, the entity includes an object that is to be repaired, and the graphical representation is displayed to assess an extent of damage, the entity includes a surveilled entity, and the graphical representation is displayed to assess changes, actions are executed by one or more virtual machines, and actions are executed by a loud computing environment.

[0004] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with implementations of the methods provided herein.

[0005] The present disclosure further provides a system for implementing the methods provided herein. The system includes one or more processors, and a computer- readable storage medium coupled to the one or more processors having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with implementations of the methods provided herein.

[0006] It is appreciated that methods in accordance with the present disclosure can include any combination of the aspects and features described herein. That is, methods in accordance with the present disclosure are not limited to the combinations of aspects and features specifically described herein, but also include any combination of the aspects and features provided.

[0007] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

[0008] FIG. 1 depicts an example high-level architecture in accordance with implementations of the present disclosure.

[0009] FIG. 2 depicts an example inspection system in accordance with

implementations of the present disclosure.

[0010] FIG. 3 depicts an example process that can be executed in accordance with implementations of the present disclosure.

[0011] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0012] Implementations of the present disclosure are generally directed to detecting and assessing differentiations between an entity, e.g., a real-world scene, and a reference model, the reference model modeling the entity. In some examples, the entity includes one or more real-world, physical objects. In some implementations, a real-time comparison is provided between the entity and the reference model. In some examples, the reference model is provided as a computer-readable file that stores data, and that can be processed to display a representation of the entity. In some examples, a difference map can be provided, the difference map including data that indicates differentiations, e.g., mathematical differences, between the entity and the reference model. In some examples, a real-time graphical representation of differentiations between the real-world scene and the reference model can be provided based on the difference map. In some examples, a global difference model can be provided. In some examples, the global difference model can be provided as a multi-dimensional, e.g., three-dimensional, description of the difference between the real-world scene and the reference model.

[0013] Implementations of the present disclosure are discussed in further detail herein with reference to the entity including a real-world scene. It is appreciated that

implementations of the present disclosure are applicable to other entities.

[0014] In accordance with implementations of the present disclosure, input data associated with a real-world scene is received. In some examples, the input data includes a real-world representation and a reference model. The real-world representation can provide an "actual" representation of the physical, real-world scene. In some examples, the real-world representation can be provided as a 2.5-dimensional representation, a mesh representation, or a volumetric representation. The reference model can be provided as an "expected" representation of the real-world scene. In some examples, the reference model can be provided as a 2.5-dimensional representation, a mesh representation, a volumetric representation, or a parametric representation. In some implementations, the reference model can be processed to provide a multi-dimensional, graphical representation of the scene, e.g., a graphical representation of the scene displayed in a display.

[0015] In some implementations, the real-world representation can be received in real-time. For example, the real-world representation can be provided from streaming video of the real-world scene, e.g., a video stream captured by a camera. In this sense, and with respect to generation and receipt of the real-world representation, the term "realtime" can be at least partially based on a video frame rate, e.g., 30 frames per second (fps)). In some implementations, the real-world representation can be provided from a video file that is stored in computer-readable memory. For example, a video of a real- world scene can be captured and can be stored as a computer-readable video file. The video file can be subsequently received from memory to be processed in accordance with implementations of the present disclosure.

[0016] For purposes of illustration and clarity, implementations of the present disclosure will be discussed with reference to an example context. In the example context, the real-world representation is provided as a 2.5 -dimensional representation, and the reference model is provided as a three-dimensional representation. It is appreciated, however, that implementations of the present disclosure can be applicable in other contexts, e.g., the real-world representation being provided as a mesh

representation, or a volumetric representation, and/or the reference model being provided as a 2.5-dimensional representation.

[0017] In accordance with the example context, the real-world representation can be provided as one or more maps. In some examples, as discussed in further detail herein, each map can be provided from a respective image, e.g., a digital image provided from a frame of a video of the real-world scene. In some implementations, a map can include a depth map. Accordingly, a map (depth map) is provided for a respective image. An example depth map can be provided from an image of the real-world scene, and can include data relating to the distance of the surfaces of scene objects from a viewpoint, e.g., of a camera. For example, a depth map can provide a depth value associated with each point, e.g., pixel, within the image. In some examples, the depth value can indicate a distance perpendicular to a plane, e.g., x-y plane, that a focal point of the camera lies in to a surface. In some examples, the depth value can reflect a distance from a focal point of the camera that captured the image and a surface of an object associated with the respective point. In some examples, the depth value can be representative of a

monochromatic color. [0018] In accordance with the example context, the real-world representation can be provided as one or more map pairs. In some examples, each map pair can be provided from a respective image, e.g., a digital image provided from a frame of a video of the real-world scene. In some implementations, each map pair can include a depth map, as discussed above, and a color map. Accordingly, a map pair (depth map and color map) is provided for a respective image. An example color map can be provided from an image of the real-world scene, and can include data relating to the color of the surfaces of scene objects from a viewpoint, e.g., of a camera. For example, a color map can provide a color value associated with each point, e.g., pixel, of objects within the image.

[0019] Implementations of the present disclosure will be discussed in further detail with reference to one or more map pairs. It is appreciated, however, that implementations of the present disclosure can be realized using one or more maps, e.g., depth maps alone, or any other appropriate type of map.

[0020] In some implementations, transformation information can be provided with the real-world representation. In some examples, the transformation information can be specific to a camera that was used to capture the images, e.g., frames of a video, of the real-world scene. In some examples, the transformation information can include a perspective projection matrix (PPM), also referred to as a transformation matrix, that is based on hardware specifications, e.g., of the camera. In some examples, the PPM can be used to transform a point of a two-dimensional image, e.g., a point in a depth map, to a corresponding point in three-dimensional space, and vice-versa. In some examples, the PPM can be referred to as a model (K) of the camera.

[0021] As discussed in further detail herein, the real-world representation and the reference model are processed to determine the differentiation, e.g., mathematical differences, therebetween. In some examples, a difference model can be provided as a multi-dimensional description of the differentiation between the real-world representation and the reference model, and can include information about differentiations in attributes, e.g., position, color, depth, curvature and/or orientation, between points in the real-world representation and respective points in the reference model. In some examples, the difference model can also include a weight associated with each difference, the weight providing a measure of confidence in observed differences. In some examples, the difference model can also include respective directions of observed differences.

[0022] With reference to processing of the real- world representation and the reference model, and in the example context provided above, a map pair is provided for each image of the real-world representation. In the example of a live video of the real-world scene, e.g., real-time observation of the real-world scene, a series of frames can be provided from a camera, e.g., at X fps where X can be provided as a positive integer, such as 30, and each frame can be provided as an image depicting the real-world scene. In some examples, one or more map pairs are provided for comparison to the reference model. In some implementations, each map pair is processed with respect to the reference model. In some examples, each map pair is processed with respect to the reference model as it is received.

[0023] In some implementations, the one or more map pairs are compared to the reference model to determine any differences therebetween. In some examples, maps of the one or more map pairs and/or the reference model are scaled. In this matter, the maps of the one or more map pairs and the reference model are provided in the same scale to facilitate comparison. In some examples, the camera can be associated with a scale

(SCAM) that is inherited by the maps provided from images captured by the camera, and the reference model can include a scale (SRM). In some example, a scaling factor (a) can be applied such that, for example:

SSOURCE = a ' SRM; or

SRM = (l/a)SsouRCE

[0024] In some implementations, and to facilitate comparison between the map pair and the reference model, maps and/or the reference model are converted with respect to space. In some examples, the depth map and/or the reference model can be converted to a comparison space. For example, the depth map can be provided in 2.5-dimensional space and the reference model can be provided in three-dimensional space, as provided in the example context. In order to compare the depth map to the reference model, both the depth map and the reference model are provided in the same space, e.g., comparison space. In some examples, the comparison space can be provided as 2.5-dimensional space. Consequently, the reference model can be converted to 2.5-dimensional space from three-dimensional space. In some examples, the comparison space can be provided as three-dimensional space. Consequently, the depth map can be converted to three- dimensional space from 2.5 -dimensional space. In some examples, the comparison space can be provided as an intermediate space. Consequently, the reference model can be converted to the intermediate space from three-dimensional space, and the depth map can be converted to the intermediate space from 2.5 -dimensional space.

[0025] In some examples, the real-word representation and the reference model can be compared to determine color differences therebetween. Consequently, the color map of a map pair can be converted to an appropriate comparison space, as discussed above with respect to the depth map.

[0026] In accordance with implementations of the present disclosure, each map is registered to the reference model. For example, each depth map can be registered to the reference model. In the case of color comparison, each color map can be registered to the reference model. Registration of the map to the reference model enables a point from the map, e.g., depth map and/or color map, to be compared to a corresponding point of the reference model. In some examples, registration of a map to the reference model can be based on an initial position estimate.

[0027] In further detail, registration can be described as the determination of a geometrical transformation (T) that aligns points in one characterization, e.g., view, of the real-word representation, e.g., as provided in the depth map, and/or the color map, with corresponding points in another characterization, e.g., view, of the real-world scene, e.g., as provided in the reference model.

[0028] Implementations of the present disclosure are discussed in further detail herein with reference to the characterization including a view. It is appreciated that

implementations of the present disclosure are applicable to other characterizations, e.g., structural characterization, mathematical characterization.

[0029] In general, the term "view" can refer to any physical arrangement of a scene, e.g., objects, in space. In some examples, inputs of registration are the two views, e.g., the depth map or color map, and the reference model, to be registered, and the output is the geometrical transformation (T). [0030] In general, the geometrical transformation (T) is a mathematical mapping from points in one view to points in the other view. More specifically, the geometrical transformation (T) maps points from the space X of one view, e.g., the depth map and/or the color map, to the space Y of a second view, e.g., the reference model. For example, the geometrical transformation (T) can be applied to a point in X, e.g., represented by the column vector x, to produce a transformed point x'. This can be provided by the following example representation:

x' = T · x

[0031] If the point y in the space Y corresponds to x, then a successful registration will make x' equal, or approximately equal to y.

[0032] The example discussed above is generally directed to a linear transformation to conduct registration. It is appreciated, however, that implementations of the present disclosure can include non-linear transformation.

[0033] In some examples, a registration error can be provided to determine whether registration is successful. In some example, the registration error can be provided based on the following example relationship:

RERR = |T(x) - y|

In some examples, registration can be repeated until RERR is below a threshold value, e.g., RERRMIN. In some examples, registration can be repeated a threshold number of times, where, if RERR is still not below RERRMIN, an error can be triggered.

[0034] In accordance with implementations of the present disclosure, once a map is registered with respect to the reference model, points in the map(s) can be directly compared to corresponding points in the reference model. More specifically, data describing a point in the real-world scene can be compared with data describing the corresponding point in the reference model. Differences between the points are calculated, and can be included in a difference map and/or a global difference model, discussed in further detail herein. In some examples, differences for a particular point can be used to refine the accuracy of the differences previously detected for the same point. In some examples, the most recent difference data for points can be used for improved registration during subsequent comparisons. [0035] In some implementations, as discussed in further detail below, comparison between a map and a corresponding view of the reference model can be achieved using a multi-core, multi-thread processor that enables multiple comparisons to be executed in parallel. For example, each map can include hundreds of thousands of points that are to be compared to corresponding points in the reference model. In some implementations, multiple points can be compared in parallel. In this manner, processing delay that could attribute to "real-time" can be minimized. In some implementations, all of the points can be compared in parallel. In this manner, any processing delay that could attribute to "realtime" can be further minimized.

[0036] In some implementations, a graphical representation of differences between the real-world scene and the reference model, also referred to as a difference

representation, can be rendered on a user interface, e.g., display. In some examples, the difference representation can be rendered in real-time as comparisons are made and differences are detected between the real-world representation and the reference model. In this context, real-time can at least partially refer to the time between receiving the input data for the real-world representation, e.g., the one or more maps, and rendering the difference representation.

[0037] In accordance with implementations of the present disclosure, real-time can describe actions that can be automatically executed, without requiring human input and without any intentional delay, taking into account processing limitations and any time required to process data. Accordingly, display of the difference representation provides real-time feedback. For example, the real-world representation can include a plurality of map pairs, and the difference representation can be displayed upon processing the first map pair of the plurality of map pairs in view of the reference model. Further, the difference representation can be updated with the processing of each subsequent map pair in view of the reference model. In this manner, the difference representation is updated in real-time as each map pair in the plurality of map pairs is processed in view of the reference model.

[0038] In some implementations, the global difference model can be provided as a computer-readable file that can be processed to display a graphical representation of the scene, also referred to as a scene representation including detected differences. In some examples, the global difference model is provided as a data representation of the differences between the real-world representation and the reference model. For example, the global difference model can be processed to display a scene representation that only includes the differences. In some examples, the global difference model is provided as a data representation of the real- world scene including the differences. For example, the global difference model can be processed to display a scene representation that includes the real-world representation and the differences. In some examples, the global difference model is provided as a data representation of the reference model including the differences. For example, the global difference model can be processed to display a scene representation that includes the reference model and the differences.

[0039] In some examples, the scene representation can be manipulated within the display to enable detailed inspection of the differences. For example, user input can be provided and the scene representation can be zoomed, rotated, panned, annotated, edited and the like based on the user input.

[0040] FIG. 1 depicts an example architecture 100 in accordance with

implementations of the present disclosure. The example architecture 100 includes a real- world scene 102a that includes a real-world object (RWO) 102b. In some examples, the real-world object is a physical object. In the depicted example, the example architecture 100 includes a reference model (RM) 104, an inspection system 106, and real-world scene data (RWSD) 108. In some examples, the inspection system 106 is provided as one or more computer-executable programs that can be executed using one or more computing devices.

[0041] As discussed in further detail herein, the inspection system 106 receives the reference model 104 and the real-world scene data 108, and processes each to provide a global difference model. In some examples, the reference model 104 is selected based on the real-world scene 102a that is to be analyzed. In some implementations, the reference model 104 can be a model of the real-world scene 102a that was created based on previously received images, e.g., video frames, of the real-world scene. For example, a camera can capture video of the real-word scene 102a, which video can be used to provide the reference model 104. In some examples, the reference model 104 can be created based on user input to a computer-executable modeling program. For example, a user, e.g., a designer, an engineer, an architect, can use a computer-aided drawing (CAD) program to generate the reference model.

[0042] In some examples, the real- world scene data 108 is received by the inspection system 106 in real-time. For example, a camera 110, e.g., digital camera, can generate digital video of the real-world scene. In some examples, the video includes one or more frames, each frame being an image, e.g., digital image, of the real-world scene. The video can be provided in real-time, e.g., as it is captured, to a pre-processor 112. The preprocessor 112 processes frames (images) of the video to provide a real-world

representation including respective data sets. In the example context provided above, each data set corresponds to a respective frame (image) and can be provided as a map pair that includes, for example, a depth map (D) and a color map (C), e.g., for q frames, the data sets [Di, Ci], ..., [D_q, C_q] can be provided.

[0043] In some implementations, the inspection system 106 can provide a difference map and/or a global difference model to a visualizer 114. In some examples, the visualizer 112 can process the difference map and/or the global difference model to display respective graphical representations on a user interface, e.g., display, as discussed herein.

[0044] In some implementations, the example architecture 100 is provided in a computing device. In some examples, the computing device is a handheld computing device that can be easily carried by a user. In some implementations, the example architecture 100 is provided in multiple computing devices. In some examples, the multiple computing devices can communicate with each other over a network, e.g., a client communicating with a server over a network. In some examples, the camera 110 can be provided in a first computing device, e.g., a client-side computing device, and the inspection system 106 can be provided by a second computing device, e.g., a server-side computing device.

[0045] FIG. 2 depicts an example inspection system 106 in accordance with implementations of the present disclosure. The example inspection system 106 includes a registration component 200, a conversion component 202, and an inspection component 204. In some examples, each of the registration component 200, the conversion component 202, and the inspection component 204 can be provided as one or more computer-executable programs that can be executed using one or more computing devices. In the depicted example, the inspection component 204 includes a difference sub-component 208, an integration sub-component 208, and a global difference model (GDM) 210.

[0046] In some examples, and as discussed in further detail below, the registration component 200 provides a spatial relationship between the reference model 104 and the real-world scene data 108. This relationship can be used to make a meaningful comparison between the reference model 104 and the real-world scene data 108. In some examples, and as discussed in further detail below, the conversion component 202 converts the reference model 104 to a form (space) that can be used for registration and inspection. For example, the reference model 104 can be converted to a form such as a three-dimensional volume or octree for differencing, or can be projected to a specific camera view for registration. Although the example of FIG. 2 provides for conversion of the reference model 104, it is appreciated that the reference model and/or the real-world scene data 108 can be converted to accommodate registration and inspection. In some examples, and as discussed in further detail below, the inspection component 204 can provide the difference(s) between the real-world scene data and the reference data, e.g., provided from the reference model, and can combine the determined difference information with previous difference measurements. In some examples, the inspection component 204 can execute a plurality of functionality, e.g., algorithms, that are parallelizable. In some examples, the inspection component 204 and/or functionality thereof the can be executed using a multi-core, multi-thread processor, e.g., a graphics processing unit (GPU). In some examples, use of a GPU can enhance real-time speeds by parallelizing execution of functionality discussed herein.

[0047] In some implementations, a visualization component, e.g., the visualizer 114 of FIG. 1, can be provided for rendering and displaying the difference map, and/or global difference model 210 to a user.

[0048] In some implementations, and in order to begin comparing the real-world representation to the reference model and provide the difference map, and/or global difference model, initialization can be executed. In some examples, an estimate of the initial position of the camera with respect to the reference model can be determined. As one example, the camera, e.g., which was used to capture the real-world data, is estimated to be at some known location with respect to the reference model. As another example, the camera can start from any location with respect to the reference model, and user input can be provided to manually estimate the position of the camera with respect to the reference model. For example, the reference model can be rendered using a visualizer, e.g., the visualizer 114 of FIG. 1, and the user can manipulate a camera, e.g., the camera 110, to navigate the viewpoint of the reference model to find the approximate starting point. In some examples, if the reference model has attribute information, e.g., color, in addition to three-dimensional structure, similarity matching can be used to match the color image from the camera, e.g., streaming video of the real-world scene 102a, to some part of the reference model. In some examples, if a match is found, random sample consensus (RANSAC) can be used to estimate the transformation (T) between the viewpoint of the camera (from the real-world data) and the reference model. In some examples, RANSAC can be used in combination with feature matching techniques to provide the transformation. This transformation estimate can be used to initialize registration in the system.

[0049] With further reference to initialization, the global difference model 210 can be initialized. In the example context, the global difference model 210 is stored as a 2.5- dimensional representation with a set of difference maps and weight maps. In some examples, the difference maps store the difference in value for different attributes, e.g., depth, color, orientation, and the weight maps store a confidence of the current difference value for each map. During initialization, the differences and weights can be set to respective initial values, e.g., zero, for a single view in the 2.5-dimensional

representation. In other contexts, e.g., where the global difference model 210 is stored as a volume or an octree, the volume or octree can have weights initialized to zero, and the size of the initial representation will be the same as the size of the reference model.

[0050] With particular reference to the registration component 200, the registration component determines the relationship, e.g., the graphical transformation (T), between the real-world scene data 108 and the reference model 104. For example, the relationship is provided in three-dimensional space. In some examples, the graphical transformation (T) can include a rotation and translation based on n degrees, e.g., 6 degrees, of freedom to constrain between the real-world scene data 108 and the reference model 104.

[0051] In the example context, the input to the registration component 200 can include the real-world scene data 108, e.g., provided as 2.5 -dimensional data, a position estimate, e.g., based on the camera used to record the real-world scene data 108, a representation of the reference model, e.g., a 2.5 -dimensional representation, and an estimate of the difference between the real-world scene and the reference model. In some examples, the representation of the reference model can be referred to as an estimated view of the reference model from the point-of-view of the camera, e.g., if the camera were actually viewing the reference model from the same location as the camera viewed the real-world scene. In some examples, this can be determined based on the position estimate, e.g., based on the model (K). In some examples, the representation of the reference model is a 2.5 -dimensional representation provided from the conversion component 204. In some examples, the representation of the reference model is provided from the point-of-view of the camera used to record the image being processed, as discussed in further detail below. In some examples, the estimate of the difference between the real-world scene and the reference model can include an integrated difference map that is provided as feedback from the integration component 208.

[0052] In some implementations, the registration component 202 processes the integrated difference map and estimated view of the reference model to determine how surfaces would appear from the perspective of the camera, e.g., surface estimate. In some examples, the registration component 200 processes the surface estimate to determine data correspondences between the depth map, in the example context, and the estimated view of the reference model. In some implementations, this can be achieved using dense matching algorithms. This can be accelerated using a GPU, which enables multiple correspondences to be determined in parallel. In some examples, the transformation (T) between the map and the estimated view of the reference model can be determined by solving a point-to-plane objective function using the normal equations for the objective. In some examples, matrices involved in a parallel reduction are computed on the GPU. In this manner, and in some examples, this enables the transformation to be computed on the order of 1 -millisecond, as opposed to multiple seconds, for example, on a CPU. In some examples, this process can repeated for some predetermined number of iterations. In some examples, this process can be repeated until an acceptable transformation

(registration) is provided. The transformation is provided to other components of the inspection system 204 for use in a subsequent registration and in difference

determination, as discussed herein.

[0053] Initial real-word scene data is processed by the inspection system. In some examples, the initial real-word scene data is provided from the first frame of the video that captures the real-world scene. In some examples, the initial real-world data is received, and the estimated view of the reference model and a known difference between the reference model and the real-world scene are combined to estimate the real-world view for registration. In some examples, this combination is used rather than simply using the estimated view of the reference model, because registration could fail in cases of gross differences between the real-world scene and the reference model. Continuing, the initial views are registered as discussed above, and the calculated position is used to project the reference model to the actual view of the camera.

[0054] In some examples, it can be determined whether the estimated view of the reference model is in view of the camera. That is, whether the estimated view of the reference model corresponds to the camera view when the real-world data being analyzed was captured. In some examples, it can be determined whether the currently processed map, e.g., frame from the camera, and the estimated view (frame) of the reference model have some data in common. In some examples, this can be done by some measure of distance between transformations in extended Euclidean three-dimensional space (ES3), for example, or checking whether projecting the reference model to the given viewpoint yields an empty image.

[0055] If it is determined that the reference model is not in view of the camera, the user can manually align the camera with the last position at which the reference model was "visible," as discussed herein. If it is determined that the reference model is in view of the camera, the projection is used to inspect and store the differences between the real- world scene and the reference model, as discussed herein

[0056] In some implementations, the conversion component 202 converts the reference model to one of a plurality of forms, also referred to as spaces, for efficient use in registration and inspection. In some examples, the conversion component 202 projects contents of the reference model to various viewpoints, e.g., based on the position estimate provided from the registration component 200. In some examples, each time a new viewpoint is computed in registration, the reference model is projected to that viewpoint using the model (K) of the camera. In the example context, this results in a 2.5- dimensional representation of the reference model that is how the reference model would be expected to appear, if the camera were viewing the reference model from the given position. As discussed above, this estimated view of the reference model is provided to the registration component 200, and can be used in registration, e.g., a subsequent registration step. Further, this estimated view of the reference model can be provided to the inspection component 204 for inspection, e.g., a current inspection step.

[0057] In some examples, conversion of the reference model to provide the estimated view of the reference model can be omitted when used in conjunction with a visualization system, such as that disclosed in commonly assigned, U.S. Prov. App. No.

XX/XXX,XXX, filed on July X, 2013 (Attorney Docket No. 38277-0003P01 - to be filled in by Amendment), the disclosure of which is expressly incorporated herein by reference in the entirety. In such examples, conversion can be omitted and can be replaced by a query of the viewpoint information from the visualizer, e.g., while the reference model is currently being rendered by the visualization system.

[0058] In some implementations, the inspection component 204 determines the difference between the real-world data and the reference model and combines the measured difference with the difference model. In this manner, the inspection component 204 continuously determined the difference and updates (revises) the difference model as real-world data continues to be available. This can be summarized as difference determination, e.g., performed by the difference component 206, and integration, e.g., performed by the integration component 208.

[0059] In some examples, the difference component 206 processes point data from the map and the reference model to determine differences therebetween. In some examples, differences in depth of every pixel in the real-world data from the reference model are determined. In some examples, if the real-world data and the reference model both have additional attributes, e.g., color, orientation, and/or curvature, differences in these attributes can also be determined. For each point in the real-world data, e.g., the depth map, a position of the point is mapped to a corresponding position in the estimated view (virtual image) of the reference model, e.g., based on the transformation (T). Using the expected measurement of the respective point in the reference image, the difference in position and orientation of the surface between the transformed measurement image and the reference image is determined. Differences for multiple points can be determined in parallel using a GPU.

[0060] Differences can be attribute-specific. For example, a difference in depth indicates a distance between points in physical space and can be provided as a Euclidean distance. As another example, a difference in color indicates a difference in color space, e.g., R, G, B. As another example, a difference in orientation indicates a measure of how much two things, e.g., points, are pointed in same direction, e.g., angular difference. As another example, a difference in curvature can be provided as a difference in magnitude of normal vectors that respectively reflect the degree (or amount) of curvature of a surface.

[0061] In further detail, implementations of the present disclosure can determine the difference between several attributes, e.g., position, color, depth, and/or orientation. For each attribute, there is a distinct notion of "difference." For example, it might not make sense to describe the difference in orientation of the surface between the real-world data and reference model by subtracting the normal vectors. Instead, for example, a measure of cosine similarity can be determined, and can represent the difference in surface orientations. As another example, a kernel function can be used to determine differences in distance, e.g., in a high (or possibly infinite) dimensional space.

[0062] The differences for the multiple points are combined with all previous measurements of the differences during integration, e.g., performed by the integration component 208. In some implementations, currently determined differences can be integrated into the difference map (to provide the integrated difference map), as discussed in further detail below. In some examples, the currently determined differences can be integrated into the global difference model 210 using a "kerne lized stochastic descent" algorithm, which is parallelizable on the GPU. In some examples, kernelized stochastic descent can be described as an algorithm for rapidly finding the maximum likelihood estimate of a noisy depth value from multiple measurements at multiple angles.

[0063] In further detail, after differencing the attributes of the real-world data from the corresponding attributes of the reference model, determined differences can be integrated with the difference map and/or the global difference model. In some implementations, integration includes determining a weighted average of the currently determined difference and the global estimate of the difference, e.g., in parallel on a GPU.

[0064] In some implementations, and as introduced above, each difference value can be associated with a respective weight. In some examples, weights can be provided as fixed values. In some examples, weights can be updated as additional differences are determined. By way of example, a new difference value (ANEW) can be associated with a weight (wNEw) and can be integrated with an existing difference value (ACURRENT) that is associated with an existing weight (WCURRENT). For example, the existing difference value can already be provided in the difference map and/or global difference model. In some examples, WNEW can be determined based on a difference between ANEW and ACURRENT. In some examples, WNEW can range between a minimum value, e.g., 0, and a maximum value, e.g., 1. In some examples, if the difference between ANEW and ACURRENT is equal to or less than a first threshold difference, e.g., 0, WNEW can be set equal to the maximum value, e.g., 1. In some examples, if the difference between ANEW and ACURRENT is equal to or greater than a threshold difference, WNEW can be set equal to a minimum value, e.g., 0. In some examples, if the difference between ANEW and ACURRENT is equal to or less than a second threshold difference, e.g., 0, WNEW can be set equal to the maximum value, e.g., 1. In some examples, WNEW can be set to a value that is somewhere between the minimum value and the maximum value.

[0065] In some implementations, WCURRENT can be updated based on ANEW, ACURRENT, WNEW and WCURRENT. In some examples, WCURRENT can be updated based on the following example relationship:

(^CURRENT) (.^WCURRENT) + ( _NEW) (_w_NEW)

^WCURRENT

^WCURRENT + ^WNEW In some examples, WCURRENT can be capped at a maximum value, e.g., 1. In some examples, WCURRENT can be updated based on the following example relationship:

WCURRENT = MIN (WCURRENT + WNEW; MAX) where MAX is provided as a maximum value, e.g., 1.

[0066] As discussed above, the difference map (integrated difference map), which provides differences in attributes between the real-world data and the reference model from the current viewpoint, is output to the registration component 200, e.g., for subsequent registration (tracking). In some examples, the difference model can be provided to a visualization system, e.g., the visualizer 114 of FIG. 1, for real-time visualization of the difference(s).

[0067] Example Listing 1 and example Listing 2, below, provide respective example pseudo-code for real-time difference (SUBTRACT) and integration (INTEGRATE) in the example context:

SUBTRACT_IMAGES(M, R, ATTR)

Init D to empty image with dimensions of M

for point p in M. points

for attribute attr in ATTR

D.attr[p] = ATTR_DISTANCE(R.attr[p], M.attr[p]) return D

Listing 1 : Difference for 2.5-dimensional Representations

INTEGRATE_2.5D(M, T, K)

load G

if DISTANCE( CLOSEST_VIEW(G, T), T) > EPSILON

EXTEND MODEL(G, T)

for viewpoint D in G

for point p in M.points

meas diff = M.diffs[p]

p hat = TRANSFORM_COORDS(p, D.T)

if IN_IMAGE(p_hat, D)

v = D.diffs[p_hat]

v hat = WEIGHTED_AVG(meas_diff, v, D.weights) D.diffs[p_hat] = v hat

store G

Listing 2: Integration for 2.5-dimensional Representations where M is the map data, p are the points in the map M, ATTR are the attribute(s) of a point p to be compared, R is the estimated view of the reference model to be compared to the map M, EPSILON is a threshold distance, and G is the global difference model. Although a weighted average is implemented in the example of Listing 2, it is contemplated that other techniques for combining values can be implemented.

[0068] In view of example Listings 1 and 2, after the global difference model is loaded (initialized), a distance between the latest camera position and the closest position of a 2.5-dimensional component of the global difference model is determined. In this manner, it can be ensured that the global difference model can incorporate new data when the new measurement data is spatially distinct from past measurements. Each component of the 2.5-dimensional difference model is individually integrated with the new measurement. In some examples, to integrate, the point in the new measurement is converted from the camera coordinates into three-dimensional space, and the camera position from registration is used to transform three-dimensional point into the space of the model component. The point is projected into the difference map at the 2.5- dimensional component and averaged with the measurement at that pixel.

[0069] Listings 3 through 6 below provide example pseudo-code for real-time difference for respective representations (comparison spaces), e.g., octree, volumetric, parametric, discussed in further detail below.

OCTREE DIFF(M, O, DEPTH, ATTR)

Init D to empty image with dimensions of M

for point p in M .points

v = CLOSEST_ELEMENT(p, O, O.root, DEPTH) for attribute attr in ATTR

D.attr[p] = ATTR_DISTANCE(v.attr, M.attr[p])

return D

Listing 3 : Difference for Octree Representations

CLOSEST_ELEMENT(p, O, node, depth)

if node. depth == depth or node. depth == O. depth

elements = {}

for node child in node. children

elements = elements U GET POINT S (child)

elements = SORT_BY_DISTANCE(p, elements)

return elements[0]

for node child in node. children if p in child.bounds

return CLOSEST_POINT(p, O, child, depth+1)

Listing 4: Closest Element Determination for Octree Difference

VOLUMETRIC_DIFF(M, V, K, radius, ATTR)

Init D to empty image with dimensions of M

BEST = NULL

for point p in M.points

vox = CALCULATE_VOXEL(p, V, K)

for d_i in {-radius, radius}

for dj in {-radius, radius}

i = vox.i + d_i

j = vox.j + dj

k = vox.k + d_k

v = V[i, j, k]

if BEST == NULL or SIMIILARITY(v, p) >

SIMILARITY(v, BEST) BEST = v

for attribute attr in ATTR

D.attrtp] = ATTR_DISTANCE(R.attr[p], M.attr[p]) return D

Listing 5 : Difference for Volumetric Representations

PARAMETRIC_DIFF(M, S)

Init D to empty image with dimensions of M

for point p in M.points

x = argmin ( norm2(p - q) s.t q is a point on surface S )

for attribute attr in ATTR

D.attr[p] = ATTR_DISTANCE(R.attr[p], M.attr[p])

return D

Listing 6: Difference for Parametric Representations

[0070] Listings 7 and 8 below provide example pseudo-code for real-time integration for respective representations (comparison spaces), e.g., octree, volumetric, discussed in further detail below.

INTEGRATE OCTREE(M, TAU, K)

load G

for point p in M.points

meas diff = M.diffs[p]

p hat = TRANSFORM COORDS(p, G)

if IN_OCTREE_BOUNDS(p_hat, G)

n = LEAF NODE(G, p hat)

n.diff = WEIGHTED_AVERAGE(meas_diff, n.diff, p hat) store G

Listing 7: Integration for Octree Representations

INTEGRATE_VOLUME(M, TAU, K)

load G

for point p in M .points

meas diff = M.diffs[p]

p hat = TRANSFORM COORDS(p, G)

if IN_VOLUME_BOUNDS(p_hat, G)

v = VOXEL(p_hat, G)

v.diff = WEIGHTED_AVERAGE(v.diff, meas diff, p hat) store G

Listing 8: Integration for Volumetric Representations

[0071] Implementations of the present disclosure provide various visualizations for graphically displaying difference maps and/or models. As introduced above, visualization can be provided during capture, e.g., in real-time as video of the real-world scene is captured, and after the entire difference model has been captured. In some

implementations, the difference model is projected to the current camera position once per iteration, and differences at each point (pixel) are mapped into color space for quick and intuitive visualization. In some examples, differences in a first direction and/or on a first surface of an object can be depicted in a first color, e.g., red, while differences in a second direction and/or on a second surface can be depicted in a second color, e.g., blue. In this manner, the user is able to view and readily discern respective differences in realtime.

[0072] In some implementations, a more sophisticated visualization can be provided based on a complete difference model, e.g., the global difference model. In some examples, using differences provided from every camera view, e.g., in the 2.5- dimensional difference model, differences, e.g., defects, can be represented

(reconstructed) on surfaces of the reference model. In some examples, this can be achieved when the distance metric used to store position differences is invertible. In some examples, the reconstructed differences can be colored and/or shaded based on severity, e.g., magnitude of difference, and can overlay surfaces of the reference model for an intuitive visualization. In some examples, the reference model can be converted to polygons from the depth and color maps.

[0073] Although implementations of the present disclosure are streamlined for realtime evaluation, there are additional implementations that can be considered. Such implementations can include an alternate input source, e.g., a pre-captured model of the real-world scene (as opposed to real-time video capture), and implementations that use a different method of storing the global difference model, which may be useful for different applications.

[0074] With regard to pre-captured models, the inspection system of the present disclosure is able to handle pre-captured models as input, e.g., rather than a stream of depth images, and is able to difference the pre-captured models with respective reference models. In some examples, input types can include a mesh representation or a volumetric representation. In either of these cases, registration and integration are affected. More specifically, and in some examples, registering the pre-captured model with a reference model can be solved using the same point-to-plane objective as discussed above with reference to registering 2.5 -dimensional maps. One difference, however, is finding correspondences. In some examples, correspondences between any of the reference model types and any of the real-world model types can be determined by converting each to a volumetric representation, and finding correspondences by searching a radius around each point. As another example, RANSAC can be used in combination with feature matching techniques. This can be efficiently achieved by the GPU.

[0075] Differencing a mesh representation or a volumetric representation with a reference model has possible alternatives (see Listings 3 through 6, above). For example, differencing using an octree is most efficient with an expected runtime of 0(logs(n)). In such an example (see Listings 3 and 4), an octree representation of the reference model is constructed using known methods for growing and pruning. In some examples, the octree can contain points, polygons, or both. For each point in the measured mesh representation or volumetric representation, the tree is traversed to the node of the tree containing the point at some minimum depth. Each leaf node stemming from this node is visited, and a list of all elements contained in these leaf nodes is provided. The closest element to a query point (the point currently being analyzed) is chosen from the list of points. In some examples, if the octree stores points, the distance measure can be provided as the standard Euclidean distance. In some examples, if the octree stores polygons, the distance measure is point-to-center.

[0076] Continuing, the attributes of the point and octree element can be compared using a standard distance for octree points and point-to-plane distance for octree polygons. In accordance with implementations of the present disclosure, the worst case expected runtime is 0(n²). However, with a fairly constant density of points in the octree, and a minimum depth close to the maximum depth used to construct the tree, the most expensive operation is traversing the tree to the leaf nodes. This method can be used to generate a depth map from a specific viewpoint or a volume containing the measured difference at each reference point.

[0077] In some implementations, it may be useful to difference using a volumetric representation (see Listing 5). This may be the case, for example, when the differences are expected to be small and/or the input models (the real-world representation and the reference model) are both volumetric. In this case, after the models are registered, a radius around each point in the real-world model is searched, and it is determined whether a point in the reference model lies within that radius. This can be accelerated on a GPU and run on the order of milliseconds.

[0078] In some examples, another differencing method can include the use of a

Hausdorff metric, e.g., when the reference model is mesh or parametric. This can run extremely fast when there is a closed form solution for the Hausdorff metric. For example, when the parameterized reference is a plane, the distance can be readily computed as based on the following example relationship:

n^■ p

n

where n is normal, e.g., a vector describing a line that is perpendicular to a tangent line at the point p.

[0079] These computations could be performed in parallel on the GPU, e.g., in closed form cases. When there is no closed form, this method may still be desirable, because the objective can be solved by an iterative optimization method like gradient descent.

[0080] Implementations of the present disclosure also enable the output difference model, e.g., the global difference model, to be stored in various forms. For example, for some applications it may be useful to store the three-dimensional difference between the reference model and the real-world representation in absolute three-dimensional space. That is, without reference to any specific viewpoint as the differences are measured using the 2.5-dimensional representation. This can be useful, for example, in cases where the real-world data is a mesh or volume, because there is no longer a concept of camera view.

[0081] In some examples, the difference model can be stored as a volumetric representation or in an octree. Both storage types can be converted to a more useful form, such as a mesh, e.g., once integration is complete. Listings 7 and 8, above, provide examples for integrating new difference measurements with either of the alternative ways to store the global difference model. For an octree model (Listing 7), each leaf node stores the magnitude and direction of differences averaged over time. After differences between the reference model and the real-world model are determined, the measured differences are propagated to their respective leaf nodes based on their three-dimensional position. In some examples, these new measurements are averaged with the existing measurement estimates at each node. After the entire difference map has been compared, the new differences are sampled from each leaf node to produce a visualization of the difference.

[0082] In a volumetric global difference model (Listing 8), each voxel stores the magnitude and direction of differences averaged over time. In some examples, each voxel is averaged with new measurements as similarly discussed above with reference to the octree, but the computation of the voxel in which differences lie can be computed in constant time. In some examples, a tradeoff is that the volumetric representation requires a cubic amount of memory with respect to the size of the reference model.

[0083] Either of the octree or volumetric representations can be visualized in the same manner as the 2.5-dimensional representation, as discussed above. In some examples, the discussed visualization for 2.5-dimensional models requires some conversion of difference to color and an overlay of this color on a meshed model. Both the octree representation and the volumetric representation can be converted to a mesh format, if necessary, e.g., using Poisson reconstruction and marching cubes.

[0084] FIG. 3 depicts an example process 300 that can be executed in accordance with implementations of the present disclosure. In some examples, the example process 300 can be provided as one or more computer-executable programs executed using one or more computing devices. In some examples, actions of the example process can be executed using a GPU.

[0085] Objects are initialized (302). For example, the inspection system 106 can initialize objects. In some examples, a reference model, a difference map, and a global difference model are initialized. In some examples, a camera that is used to capture video of a real-world scene is initialized. The reference model is scaled (304). In some examples, the reference model is scaled to match a scale of the camera used to capture real- world scene data.

[0086] Data is received (306). In some examples, real-world scene data 108 is received by the registration component 200. In some examples, the data is received in real-time, as the data is generated through observation of a real-world scene, e.g., a camera capturing streaming data. In some examples, the data is received from a previously stored real-world representation (real-world model that had been generated based on observing a real-world scene). The reference model is converted (308). For example, the conversion component 202 can convert the reference model. In some examples, an estimated view of the reference model is provided. In some example, the reference model is converted to a comparison space, e.g., from a three-dimensional space to a 2.5 -dimensional space. In some examples, both the reference model and a real-world representation, e.g., map, provided from the real-world scene data can be converted to the comparison space. In some examples, the real-world representation is converted to the comparison space instead of the reference model.

[0087] The reference model is registered with respect to the real-world representation (310). For example, the registration component 200 can receive the real-world representation, e.g., map, and the estimated view of the reference model, e.g., from the conversion component 202, and can register the real-world representation and the estimated view of the reference model relative to one another, as discussed in detail herein. It is determined whether the registration was successful (312). In some examples, a registration error can be compared to a threshold and, if the registration error is less than the threshold, it can be determined that registration was successful. If the registration was not successful, it can be determined whether a threshold number of attempts has been exceeded (314). In some examples, a number of attempts to register the real-world representation and the estimated view relative to one another can be allowed. If the threshold number of attempts has been exceeded, an error 316 is indicated. If the threshold number of attempts has not been exceeded, an alternative registration can be attempted (318). In some examples, and as discussed above, an alternative registration can include a user providing manual input, e.g., moving a camera, to effect registration.

[0088] If it is determined that registration is successful, differences between the real- world representation and the reference model are determined (320). For example, the difference component 205 determines differences between the real-world representation and the reference model. In some examples, and as discussed herein, one or more attributes of points in the real- world representation are compared to corresponding attributes of corresponding points in the reference model. Listings 1, 3, 5 and 6, above, provide example differencing for respective representations. Differences are integrated (322). For example, the integration component 208 can integrate differences. In some examples, currently determined differences can be incorporated into previously determined differences. For example, a first difference value can be determined for a point at a first time (iteration) based on a first comparison, and a second difference value can be determined for the same point at a second time (iteration) based on a second comparison. The first difference value and the second difference value can be integrated to provide a third difference value, e.g., using weighted averaging.

[0089] In some examples, a difference map (integrated difference map) is rendered (324). For example, the difference map can be rendered as a graphical representation on a display, e.g., using the visualizer 114. In some examples, the difference map is rendered in real-time, as comparisons are made between the real-world representation and the reference model. In this manner, the user is provided with immediate feedback as to the real-world scene, while still capturing (measuring) the real-world scene. Difference data is stored in the global difference model (326). In some examples, the difference data includes the integrated difference data. It is determined whether additional data is available (328). For example, it can be determined whether real-world data is being provided to the inspection system 106. If additional data is available, the example process 300 loops back. In some examples, if additional data is not available, the global difference model can be rendered (330). For example, the visualizer 114 can render the global difference model as a graphical representation in a display.

[0090] In some examples, and as discussed above, actions can be executed in parallel, e.g., using multiple processors in a distributed processing arrangement. In some examples, a plurality of real-world representations can be compared to a plurality of estimated views of the reference model in parallel. To that end, operations such as conversion, registration, difference determination and integration can be performed in parallel, for each real-world representation and estimated view pair.

[0091] Implementations of the present disclosure are applicable in various use cases. As an example, implementations can be provided in an insurance context, where an entity, e.g., physical object, is insured and is damaged. The graphical representation provided based on the difference model can represent the extent of damage to the object, which information can be used as part of an insurance claim process, and/or to determine the extent of repair work required. As another example, the entity can include a manufactured part, and the graphical representation can be displayed to assess defects, e.g., defects occurring during manufacture. As another example, the entity can include a scene, e.g., a crime scene, and the graphical representation can be displayed to investigate the scene, e.g., crime scene investigation. As another example, the entity can include an architectural structure, and the graphical representation can be displayed to assess deviations, e.g., between the actual, physical structure and plans of the structure (such as computer-aided design (CAD) plans). As another example, the entity can include an object that is to be repaired, and the graphical representation can be displayed to assess an extent of damage to the object. As another example, the entity includes an entity that is under surveillance (a surveilled entity), and the graphical representation can be displayed to assess changes to the scene.

[0092] Implementations of the subject matter and the operations described in this specification can be realized in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in any appropriate combinations thereof. Implementations of the subject matter described in this specification can be realized using one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus, e.g., one or more processors. In some examples, program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0093] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0094] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. In some examples, the data processing apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In some examples, the data processing apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0095] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0096] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0097] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. Elements of a computer can include a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non- volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0098] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0099] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a

communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), an inter-network (e.g., the

Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[00100] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any implementation of the present disclosure or of what may be claimed, but rather as descriptions of features specific to example implementations. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a subcombination.

[00101] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[00102] Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for providing a difference model that comprises differentiations between a representation of an entity and a reference model of the entity,

5 the method being executed using one or more processors and comprising:

receiving, by the one or more processors, data of the representation that provides a characterization of the entity;

receiving, by the one or more processors, data of the reference model; registering, by the one or more processors, the representation and the reference o model relative to one another;

providing, by the one or more processors, a characterization of the reference model;

determining, by the one or more processors, attribute differentiations between the representation and the reference model based on the characterization of the entity and the 5 characterization of the reference model;

integrating, by the one or more processors, the attribute differentiations into a difference model; and

transmitting, by the one or more processors, instructions to display a graphical representation based on the difference model.

0

2. The method of claim 1, wherein data defining the representation is received in real-time, as images of the entity that is represented by the representation are captured.

3. The method of any of the foregoing claims, wherein the graphical representation5 is displayed in real-time.

4. The method of any of the foregoing claims, wherein the representation of the entity is provided as one of a 2.5 -dimensional representation, a three-dimensional representation, a volumetric representation, a parametric representation, and an octree0 representation.

5. The method of any of the foregoing claims, wherein the reference model is provided as one of a 2.5 -dimensional representation, a three-dimensional representation, a volumetric representation, a parametric representation, and an octree representation.

6. The method of any of the foregoing claims, wherein the reference model comprises a base representation of the entity that was captured previous to the representation of the entity.

7. The method of claim 6, wherein the base representation of the entity and the real- world representation of the entity depict the real-world scene, the difference model providing changes in the entity.

8. The method of any of the foregoing claims, further comprising scaling at least one of the representation of the entity and the reference model.

9. The method of any of the foregoing claims, further comprising converting at least one of the representation of the entity and the reference model to a comparison space.

10. The method of any of the foregoing claims, wherein each attribute differentiation is associated with a respective attribute, attributes comprising one or more of position, depth, color, orientation, and curvature.

11. The method of any of the foregoing claims, wherein the data of the representation provides a plurality of points, each point being associated with one or more attributes, and the data of the reference model provides a plurality of points, each point being associated with one or more attributes.

12. The method of claim 11, wherein determining attribute differentiations between the representation and the reference model comprises comparing attributes of point pairs, each point pair comprising a point of the representation and a point of the reference model.

13. The method of claim 12, wherein comparing attributes of point pairs is concurrently performed for a plurality of point pairs.

14. The method of claim 13, wherein the plurality of point pairs comprises a sub-set of all point pairs provided for the representation and the reference model.

15. The method of claim 13, wherein the plurality of point pairs comprises all point pairs provided for the representation and the reference model.

16. The method of any of the foregoing claims, wherein each attribute differentiation is associated with a weight, the weight reflecting a confidence in a value of the attribute differentiation.

17. The method of claim 16, wherein the weight is determined based on a difference between a value of the attribute differentiation and a respective value provided in the difference model.

18. The method of claim 17, wherein the weight is within a range defined from a minimum value and a maximum value.

19. The method of any of the foregoing claims, wherein integrating the attribute differentiations into a difference model comprises, for each attribute differentiation, determining a new attribute differentiation value based on a value of the attribute differentiation and a value of an existing attribute differentiation already provided in the difference model.

20. The method of claim 19, further comprising storing the new attribute

differentiation value in the difference model as the value of an existing attribute differentiation already provided in the difference model.

21. The method of claim 19 or claim 20, wherein the new attribute differentiation value is provided as a weighted average of the value of the attribute differentiation and the value of an existing attribute differentiation.

22. The method of claim 1, wherein the entity comprises one or more physical objects.

23. The method of claim 1, wherein the entity comprises an object that is insured, and the graphical representation is displayed to assess an insurance claim.

24. The method of claim 1, wherein the entity comprises a manufactured part, and the graphical representation is displayed to assess defects.

25. The method of claim 1, wherein the entity comprises a crime scene, and the graphical representation is displayed to investigate the crime scene.

26. The method of claim 1, wherein the entity comprises an architectural structure, and the graphical representation is displayed to assess deviations.

27. The method of claim 1 , wherein the entity comprises an object that is to be repaired, and the graphical representation is displayed to assess an extent of damage.

28. The method of claim 1, wherein the entity comprises a surveilled entity, and the graphical representation is displayed to assess changes.

29. The method any of the foregoing claims, wherein the method is executed by one or more virtual machines.

30. The method of any of the foregoing claims, wherein the method is executed by a loud computing environment.

31. A non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with the method of any of claims 1 through 28.

32. A system, comprising:

a computing device; and

a computer-readable storage device coupled to the computing device and having instructions stored thereon which, when executed by the computing device, cause the computing device to perform operations in accordance with the method of any of claims 1 through 28.

33. The system of claim 32, wherein the system is provided in a handheld computing device.

34. The system of claim 32, wherein the system is provided as a client-server system.