Information processing system and method
Field of the invention
The present invention concerns an information processing system comprising an optoelectronic device, associated display means and display control electronics that either is physically a part of said optoelectronic device or physically separate therefrom, but connected therewith via a data link, wherein the optoelectronic device comprises an internal memory, a data processing system for operating the device, including input- and/or output processes, and an optical image recorder for capturing information from the environment surrounding said optoelectronic device. The present invention also concerns a method for operating an information processing system of this kind.
Background of the invention Operator communication with information processing systems such as personal computers and other stationary systems have generally been achieved via keyboard and mouse. There is now a trend towards mobile and hand-held electronic devices such as mobile telephones and personal digital assistants (PDAs) becoming progressively more sophisticated, with growing processing power, memory capacity, functionalities and communication options and bandwidths. This implies that many of the tasks that are presently being performed on personal computers and similar stationary equipment shall be performed on physically much smaller platforms in the future. Before this can become a widespread reality, however, there are two major problems that must be solved, both of which stem from the small physical size of the devices in question.
First, there is not space for a traditional keyboard on which to enter operator commands. The severe restrictions on the number of buttons or other means of input (scroll wheels, four-way buttons, miniature joy sticks) have led to a situation where user control of the device is seriously slowed down, e.g. during text input and menu item selection. Furthermore, extensive use of miniaturized multiple-choice buttons typically leads to increased operator fatigue and higher incidence of erroneous data entries: A common way to implement menu item selection is to employ a four-way button which has to be repeatedly operated in order to step through menu items that are typically arranged in a 1-dimensional list or a 2-dimensional array of symbols. As the
lists and menus tend to increase in size with the increased functionality of mobile devices, the number of button operations required will increase in a highly undesired manner. ■,
Second, the small area available for a display screen implies that displayed features must either be very small and correspondingly difficult to read, or they must be so few as to severely limit the information content in the display.
Several solutions have been tried to solve these problems, but with limited success. One solution is to display only a portion of a page, and scroll or step across it using touch buttons on the mobile device to bring sub-sections of the page into view. Analogously, a cursor or a highlighted area can be moved across the image on the display. Indeed, this is a widely used method to alleviate the problems associated with a small screen, but has a number of drawbacks: It requires dedicated touch buttons that must be operated in a hands-on mode, it is not practical for scrolling and navigation in more than one direction at a time, and it lacks intuitive user features that are desirable in many applications.
Another solution, described in US Pat. No. 6,466,203 is to exhibit the entire page at low magnification on a touch-sensitive display screen initially to provide an overview. The user can recognize the page's general lay-out and presence of hyperlinks. When the user touches a particular location on the touch screen that corresponds to a portion of the page's image, the portion gets displayed so as to fill the display's area. Thus, the user can browse the Web with a display of a limited size. There are several drawbacks associated with this solution: Apart from requiring input via a touch-sensitive screen of small size and high resolution, it requires a particular logic organization and editing of the displayed data to permit extraction of blocks of information, it is not suitable for incremental scanning across a large page, etc.
In US patent application 2002/0024506, a method is disclosed that enables scrolling and zooming in an image displayed on the small screen of a hand¬ held device, using an image capture device that tracks the motion of a reference target, where said target is a part of, or attached to, the users body or clothing. The need for a reference target poses several drawbacks: There will be a delay at start-up while the target is acquired, the movement of the handheld device is limited by the requirement to keep the target in view of
the image capture device, and there may be undesired scrolling or zooming when the user needs to e.g. turn his head.
US patent application 2004/0141085 discloses a system that uses an orientation detecting mechanism (an accelerometer) to determine the orientation of an imaging device, e.g. a digital camera. The acquired orientation information is used to reconfigure the displayed image for increased user viewing convenience. The reconfiguration is explicitly effected on the captured image only. The general problem of using small displays for viewing and performing operations within large and content-rich images, diagrams, etc. is not addressed.
A solution to the problem of providing input to a small portable device with limited space for keyboard is promoted by the firm Memsic Inc., of North Andover, MA in U.S.A. In this case, the portable device is equipped with a small accelerometer which senses tilting of the device. By tilting right/left and/or up/down and clicking on a button on the device, navigation is possible on a screen on the device, i.e. scrolling through lists, menu selection, map scrolling, gaming, etc. An obvious drawback of this approach is the need for an accelerometer in the portable device, where space and cost typically are at an extreme premium. Beyond this, input is sensitive to motion of the device involving changes in speed or direction, and presupposes a given attitude of the portable device relative to the direction of the gravity vector.
A possible solution to the text input problem, described in US Pat. No. 5,818,437 is to assign multiple letters to each of the numerical buttons on the device, and use built-in dictionaries to find words that match the button sequence keyed by the user. Drawbacks of this solution include the need to scroll through several valid alternatives to find the correct word, and having to explicitly spell words, names and abbreviations that are not included in the dictionary.
It is thus a major object of the present invention to provide apparatus and methods for input/output operations involving information processing systems where the available space for keyboard or other mechanical input implements is restricted, and/or where available space for display is restricted.
It is a further major object of the present invention to provide apparatus and methods for input/output operations involving information processing systems where at least the input from the operator takes place on a mobile, e.g. handheld or helmet-mounted component of the system. Summary of the invention
The above objects as well as other features and advantages are realized with an information processing system according to the present invention which is characterized in that the optoelectronic device comprises hardware and software adapted for analyzing information captured by said optical image recorder with respect to temporal variations in the signals from the optical image recorder, said variations being caused by translational and/or rotational motion of said optoelectronic device relative to real or virtual objects, textures or scenery within the field of view of said optical image recorder, that said processing hardware and software are adapted for providing an output signal in scalar or vector form on the basis of the analyzed information, said output signal being indicative of the motion characteristics of said optoelectronic device relative to the real or virtual objects, textures or scenery within the field of view of said optical image recorder, that the display control electronics is connected with said optoelectronic device for receiving said output signal, and that said display control electronics is adapted for modifying or changing a displayed message or image in response to the characteristics of said output signal.
The above objects as well as other features and advantages are also realized with a method according to the invention which is characterized by deriving at least one parameter from translational and/or rotational motion of said optoelectronic device or auxiliary equipment linked to same, relative to real or virtual objects, textures or scenery objects within the field of view of said optical image recorder, and applying said at least one parameter as input to control electronics in said optoelectronic device and to effect a navigation and/or input commands in said display means, partly or completely in response to said at least one parameter.
Further features and advantages shall be apparent from the appended dependent claims.
The invention shall now be explained in more detail by resorting to a discussion of exemplary embodiments thereof and with reference to the accompanying drawing figures, of which figure 1 shows the main components in an information processing system according to the present invention, figures 2a and 2b a first preferred embodiment where the display screen on a mobile phone constitutes part of a virtual display where the viewing field is enhanced by scrolling a window across a larger page, figure 3 a second preferred embodiment involving a virtual keyboard, figure 4 a third preferred embodiment involving menu item selection, figures 5a and 5b an example of a variant of the third preferred embodiment, figure 6 a fourth preferred embodiment where an optoelectronic device according to the invention functions as a computer mouse, and figures 7a and 7b show examples of variants of the fourth embodiment. The main components of an optoelectronic device as used in the information processing system of the invention is shown in fig. 1. The optoelectronic device comprises an optical image recorder, an image motion analyser linked with the former and used for analyzing recorded images and generating a continuously updated image output to a connected display. There shall now be described a set of preferred embodiments where the optoelectronic device is implemented in a mobile telephone or a PDA equipped with a digital camera and a display screen. For convenience the optoelectronic device is termed a "phone" in these preferred embodiments. This is not meant to preclude that the optoelectronic device in question may be another type of mobile platform equipped with a digital camera and a display screen, e.g. a helmet-mounted or head-up device or system.
A first preferred embodiment is shown in figure 2. This embodiment provides a virtual display wherein large pages of text, images, web content, etc. that would otherwise defy normal criteria for readability on the phone display screen, are accessed in selected segments that are scrolled across the larger page simply by moving the phone in three dimensions. This scrolling is
rendered intuitive by correlation between the virtual motion across the page to be viewed with the real motion of the phone. The latter is typically a right/left and up/down motion of the phone above or in front of a surface or object or scene that exhibits adequate structure or texture for the camera to register movement. In cases where the phone camera is equipped with a zoom function with the zooming corresponding to movement in a third dimension, the degree of zoom can be used to control the speed of scrolling for a given motion input, e.g. the more zoom, the slower the scrolling, and vice versa.
Combined with the lateral motion of the phone optionally there may be an in/out motion relative to the surface or object which effects a zoom in/zoom out function on the displayed image of the large page. One may combine scrolling with the zoom function, such that fast scrolling reduces the zoom factor and vice versa. When the scrolling stops, the zoom level then applied may be maintained until the zoom function is activated again, e.g. by the relative in/out motion of the phone. The absolute zoom level shall be user controllable
An alternative variant of phone motion, which is possible in cases where no suitable surface or object is available in reasonably close proximity to the phone, is to rotate the phone right/left and up/down, i.e. panning and tilting the camera to cause the observed scene to move across the camera chip surface. In the virtual display image of the page this can be perceived intuitively as being related to tilting of the observer's head to view different parts of the page. When distant objects are imaged by the camera, motion of the camera in a line of sight to the distant objects will cause the image to change very little, and other means of providing zooming commands must be used. In this case, zooming in/out in the virtual image can be achieved in several ways, namely a) by using the zoom function built into the camera, if such exists; b) by pressing or toggling a mechanical input key, switch or pressure- sensitive field on the phone; c) by moving the phone in a prescribed pattern, e.g. rapid sideways or up/down rotation for zoom in/zoom out; or d) by acoustic input, e.g. voice command.
If the user scrolls to one of the edges of the page to be viewed, further movement/tilting of the phone should not result in a blank screen
(corresponding to virtually looking outside the page). Rather, the scrolling should be stopped at the page edge for as long as the movement/tilting continues, to be resumed again as soon as the user begins scrolling in another direction. The user could be alerted of reaching the edge of the page by e.g. a thumping sound from the phone. Apart from avoiding getting lost in the virtual blank space surrounding the page, this scheme enables the user to virtually drag the page along if he desires to turn around, walk, or otherwise shift position.
It is reasonable to expect that mobile and hand-held optoelectronic devices according to the present invention shall be exposed to mechanical disturbances in the form of unsteady and erratic, or cyclic or continuous motion that is not intended by the operator to provide meaningful motion input to the optical image recorder. To handle such situations, the present invention teaches the introduction of a discrimination threshold below which motion is not register as input activation to the optoelectronic device
("clutching"). This threshold shall typically apply for the recorded speed of motion (translational or rotational), and the magnitude of the threshold may be selected or adjusted according to the type of usage and environment that is encountered in each application. In a variant of this scheme, any predictable motion pattern that shall not serve as input cue to the optoelectronic device may be compensated for in the signal processing, e.g. a constant linear motion due to steady translational motion of the optoelectronic device such as walking. In the latter example, the "dead band" would be centered around a certain speed value corresponding to the constant motion. The algorithm for making the system insensitive to motion that is not intended by the operator to provide meaningful motion input to the optical image recorder may also be set to reject very rapid accelerations or motions, e.g. by setting an upper limit to the legitimate sensitive range. Likewise, the processing can easily be set to pick up a specific motion as an input cue, e.g. cyclic motion within a defined frequency band such as would occur in response to rapid wagging of the optoelectronic device sideways or up and down.
The phone movement can also be used to scroll a virtual keyboard for text input, for instance to message systems like SMS. A single function key is pressed to select a letter, and the range of characters can be extended or various special functions implemented by using movement in the third dimension. This forms the basis of a second preferred embodiment,
illustrated in figure 3. In this embodiment writing via a virtual keyboard is performed without the need for a set of physical keys representing the alphabet or other symbols on the face of the phone. Instead, letters or symbols are selected on a virtual keyboard shown in the display, by a two- step process: First, moving the phone in a lateral or rotational motion causes a visual indicator function to move across the virtual keyboard. When the desired letter or symbol has been reached, the camera is momentarily held still and a "Select" command is given to the phone, by one of the following modalities, namely a) by pressing or toggling a mechanical input key, switch or pressure- sensitive field on the phone; b) by moving the phone in a prescribed pattern, i.e. laterally up/down or rotationally (pan/tilt) or along the optical axis of the phone's camera (in/out); or c) by acoustic input, e.g. voice command.
Several variants of this basic scheme are possible. For instance may the virtual keyboard may be larger than the display, causing only part of it to be visible within the field defined by the phone's display screen. This situation is depicted in figure 3, and in this case the virtual keyboard scrolls across the display, one keyboard letter or symbol being highlighted or framed at a time upon passing a selector field in e.g. the middle of the screen.
In another variant of this embodiment, the whole virtual keyboard is shown inside the display, and a cursor or highlighting field moves across the virtual keyboard when the phone is moved. Selection of upper or lower case characters or special symbols or functions is achieved by entering a command into the phone by one of the modalities a) to c) described earlier in this paragraph. The command may be of the "Caps shift" or "Caps lock" type, depending on the situation.
In a third preferred embodiment illustrated in figure 4, movement of the phone is used for selecting menu items on the phone. It is often annoying and cumbersome to flip through ever-increasing numbers of main menu items, contact list items, or display icons. Instead of laboriously flipping through a large number of menu items by sequentially displaying individual windows on the display, a sweep across a virtual background field where the menu
icons are laid out can be performed quickly and simply. Motion patterns and selection modes can be similar to those described for the virtual keyboard above. Several variants of this basic scheme are possible: A typical phone has many menus. The user could pre-set which menus should be subject to motion controlled selection, and then the camera would automatically be activated when these menus are selected. Menus could be nested hierarchically, and motion control could be used to navigate into and within sub-menus.
A variant of this third embodiment shall now be described, with reference to figures 5a and 5b: Mobile phones often have extensive contact lists, sorted alphabetically. The standard way of accessing the contact list for calling one of the contacts is to scroll through the list until the contact is found. To speed up the process, it is possible to press one of the alphabetic/numeric keys one or several times to select a letter, whereupon the list is directly scrolled to the first contact item with this initial letter. Even so, further scrolling is normally required to find the correct contact item, and on the average, a significant number of key presses have to be performed before a call can be placed. By using the virtual display concept (scrolling of the screen image) in combination with cursor control by phone movement, a more efficient way of contact list access can be implemented:
• Let each contact be represented by a small image (photo of person, company logo, cartoon image, etc).
• Arrange the images in a 2D array
• When the user accesses the contact list, the phone camera is automatically activated to detect phone movement.
• The upper left portion of the 2D image array is presented on the display, with the upper left image highlighted, {see figure 5 a)
• Phone movement causes movement of the highlight between images. If the highlight reaches the display border, the display image is scrolled. • If the highlight rests at the same image for longer than, say, 1 s, an additional pop-up information window is shown next to the image, revealing the contact name, {see figure 5b)
• As soon as the dial-up button is pressed, the phone calls the presently highlighted contact. • If the contact has more than one phone number, the first dial-up or select command produces a pop-up window with all the phone numbers. Phone movement is used to highlight the desired number and the dial-up button is used for placing the call.
This strategy enables fast selection of the desired contact in one sweeping phone movement. It enables more contacts on the screen than the usual text list layout, and is based on human recognition of images/graphical symbols rather than text, which is normally a faster search method. The present invention could also in a general fashion implement a so-called relative pointer device, for instance for controlling the cursor on a computer screen. However, the present invention shall allow the extension of relative pointer technology far beyond that offered by e.g. a conventional computer mouse. This opens for a fourth preferred embodiment, the principle of which is shown in fig.6, which basically renders the use of the optoelectronic device as three-dimensional computer mouse or similar input device to a mobile or stationary information processing system. Cursor movement on the computer screen is controlled by moving the optoelectronic device in one or more of the translational, rotational or combined translational/rotational modes described in the other preferred embodiments above, while mouse clicks can be effected in several ways, e.g. by a) pressing or toggling a mechanical input key, switch or pressure-sensitive field on the optoelectronic device such that right and left mouse buttons could be assigned to user-selectable buttons on the mobile electronic device; b) moving the optoelectronic device in a prescribed pattern, which in principle may be any translational, rotational or combined translational/rotational motion, but in practice a simple lateral up/down or rotational (pan/tilt) motion shall be adequate, although an additional motion mode is along the optical axis of the camera (i.e. in/out); or c) acoustic input, e.g. voice command.
In a first variant of this fourth embodiment, the input from the optoelectronic device to the information processing system is the operator signature, e.g. in legal or economic transactions or in access control. The operator in this case moves the optoelectronic device as he would a pen to create the pattern, which may be his name signature or some other pre-defined identifying pattern. The motion need not take place against any supporting surface, but could be performed in the air in front of the operator ("writing in the air"). In certain instances, it may be advantageous to employ processing of the recorded signature to conform with predefined formats required in e.g. economic transactions. One example of such processing would be to scale the size of the recorded signature to a predefined norm. Further extensions of
this embodiment are obvious to those skilled in the art of image and pattern analysis, e.g. analysis of hand-generated symbols and patterns to generate digital alphanumeric files in the information processing system.
In order to facilitate the use of the phone as a three-dimensional mouse, an alternative to moving the phone in the open air shall be to equip the phone with additional physical hardware, e.g. a rounded object, e.g. ball-like, which may be attached to the phone at the side where the camera is located. By proper positioning of such a ball-type object, the phone may be rested on the desktop or any other object, while the three-dimensional motions are achieved by tilting the phone in the various directions, as allowed by the thickness of the attached, rounded object. Generally the translation movements of the mouse now can be replaced by a rotational movement around any of three axes and over a fairly small angle.
Communication with the computer could be via a wire or by one of several wireless means. As an example, Bluetooth communication would enable using the mobile electronic device for controlling presentations, including pointer control and the possibility for the user to stand up and move around. An important advantage of this embodiment relative to the conventional computer mouse is that contact with or close proximity to a surface is not needed. Furthermore, it is possible to provide input commands linked to a motion in the third dimension, either by increasing or reducing the distance from the camera of the mobile electronic device to an object in front of it or by employing the camera zoom function. This opens up opportunities for 3-D cursor control, e.g. in CAD software or in the display/manipulation of 3 dimensional virtual or real-space objects. Still further, the optoelectronic device can be equipped with a microphone, such as is the case when the optoelectronic device is a mobile telephone with camera. This makes it possible to insert audio files into the computer. Likewise, by means of a speaker or earphone attached to the optoelectronic device, it becomes possible to listen to audio files that have been located in the computer.
In a second variant of the fourth embodiment, relative motion is created by the optoelectronic device being stationary or nearly so. The relative motion is then created by moving an object, e.g. the operator's hand (or finger), in front of the optical image recorder of the optoelectronic device. This variant can be implemented in various ways, as shown in figs 7a and 7b. Figure 7a
shows a person carrying an optoelectronic device according to the invention with a camera viewing out in front of the person. By moving his hands in front of the camera, this causes a cursor to move on the screen of a computer as shown in fig.7a, or on a screen in the optoelectronic device itself. Mouse clicks can be done in several ways, e.g. by closing the hand to a fist, by voice command, by touching the screen at the appropriate spot, etc. In this manner all commands which can be executed by a mouse can be performed by hand (or finger, for a higher degree of precision) motions. Instead of hanging in front of the operator as the optoelectronic device could be mounted in a harness, helmet or the like which is carried by the operator. In fig.7b the optoelectronic device is shown in a mount that is physically separated from the operator, but the basic input mechanisms are the same. Again, the operator may move his hands in free air or resting his hand against the surface of a table, to effect cursor motion. Mouse clicks can be effected in several ways, e.g. via the camera image or by the modalities mentioned in connection with fig. 6. In a variant of this embodiment, the person holds a contrast-enhancing or recognition aiding implement in his hand, e.g. a pen or pencil with a marker spot of high optical contrast on it, optionally in a color selected in an optimized spectral region, to simplify processing of the image. Fingers or hands of the operator can be marked with a spot or symbol of high optical contrast, e.g. to show the positions of the fingertips.
The variants of this fourth embodiment exemplified in figures 7a and 7b can be used to effect handwritten input into the computer, the optoelectronic device or both, including name or other types of signatures for legal and transaction purposes or access control.
Image analysis provides further possibilities for more complex and hence more compact or efficient input coding: For example, two fingers in contact with each other side by side may carry a different message from the same two fingers splayed away from each other. A simple example of how this is when writing to virtual keyboards (cf. below), in which case lower case symbols can be written with a single finger, upper case with two fingers close together.
Likewise, detection of more than one motion vector in the field of view of the optical sensing device of the optoelectronic device can be employed as
input. An example of this is the use of two hands or fingers that are moved independently, but in coordinated fashion.
Moreover, people skilled in the art will recognize the technological relationship of the variant embodiments rendered in figures 7a and 7b with proposals for using nose tip or eyelid movements for controlling a cursor or accessing a keyboard. Approaches of this kind have been developed particularly in order to enable paraplegics to operate computers and the like, and persons skilled in the art will realize that the present invention shall be able to provide similar solutions by employing movements of any body part as practicable. Evidently, in some cases physical impairment may result in a restriction of the operational possibilities, but nevertheless the present invention then can be implemented in order to enable severely handicapped people to undertake normally fairly complicated tasks.
The principle scheme in figure 7b can be used to provide keyboard input via a "virtual keyboard": A keyboard may be printed on the table surface or on a flat sheet lying on the table, and the camera records the position of the operator's fingers on the keyboard. Image analysis software is used to determine which key is indicated, e.g. by analyzing the obscuration of keyboard by the hands and fingers. In cases where fingertips are tagged for enhanced visibility, the tag location relative to the keyboard is analyzed. The actual tapping to indicate selection of an indicated key may be achieved by one of the modalities mentioned above in connection with mouse clicking, or by providing a motion cue to the imaging software. An example of the latter would be to set an activation threshold related to the dwell time on a given key, i.e. the finger must rest a time exceeding a predetermined minimum for a tapping to be registered.
A variant of a virtual keyboard would be to project a keyboard as an image onto the writing surface. Fingertip tags with high contrast could in this case be rendered especially useful by making them fluoresce in response to the light used in projecting the keyboard or another light source.
Another type of virtual keyboard can be established under a wider concept which may be termed a "virtual touch screen": In the latter case, the optoelectronic device is equipped with a camera which is directed towards the screen display of a computer. The operator places a pointer or wand, or
one or more Fingers or hand(s) against the screen, thereby obscuring parts of the displayed image. Image analysis software determines which of the displayed image parts or regions that are indicated, e.g. by analysing the contours of the object held in front of the screen. This principle thus provides the features of a touch-sensitive screen, providing input opportunities for menu selection, etc, but without the need for technically complex physical sensors to record the touching action. When the displayed image is a keyboard, writing may be performed in a manner similar to the case described above where writing was performed on a printed or projected keyboard. The virtual touch screen provides opportunities for more sophisticated input operations than simple two-dimensional coordinate point indication, by making the image analysis software able to discriminate between different motion- and shape patterns for the operator's fingers and hands in front of the screen. An example of this is to use the flat palm of the hand to mark or wipe larger parts of a document or scene, to effect copying, deletion or transfer.
A person skilled in the art will immediately recognize the issues related to possible mismatch in phase and/or frequency between the frame updating in the screen display and in the camera sensor, but shall also be able to provide remedies to problems that would otherwise result from this. These issues shall therefore not be discussed here.
One vital part of any of the indicated embodiments is the processing of camera image sequences to obtain the output signal which indicates movement of the mobile electronic device relative to imaged objects or scenes. This is largely a problem known as motion estimation, which is a well established research field where the present major application is compression of digital video sequences. As an example, the MPEG video compression standards (see e.g.: http://www.chiariglione.org/mpeg/ ) rely heavily on motion estimation. In several of its embodiments, the present invention has an advantage over conventional video compression, in that only the camera motion need be estimated, which means characterization of the general movement of the background of the scene. The motion of smaller objects across the background can be disregarded, which lowers the demand for processing power compared to video compression.
To cope with phenomena such as perspective, camera zoom (or in/out movement), and moving objects, the camera image needs to be split into macroblocks, typically of the size 16x 16 pixels. The motion of each macroblock from one frame to another is then computed, resulting in a motion vector (MV). This is repeated for each macroblock in the image, resulting in a motion vector field (MVF), which then is analyzed to yield the camera motion. Moving objects, mismatched macroblocks due to insufficient texture, etc., will show up as outliers in the MVF and can be disregarded through proper filtering. The implementation of MVF calculation in the present invention will be chosen in each particular embodiment according to the properties of the camera and other phone hardware, available processing power, etc. Block matching is probably the most common way to calculate the MVF (cf., e.g.: A. M. Telkap, Digital Video Processing, Prentice Hall, 1995; J. Watkinson, The Engineer 's Guide to Motion Compensation, Snell & Wilcox, 1994).
Briefly, each macroblock in the new frame is compared with shifted regions of the same size from the previous frame, and the shift which results in the minimum error is selected as the best MV for that macroblock. Other methods of camera motion estimation include phase correlation (cf., e.g.: J. Watkinson, The Engineer 's Guide to Motion Compensation, Snell & Wilcox, 1994; Y. Liang, Phase-Correlation Motion Estimation, Final project, Stanford University) and the utilization of MPEG motion vectors (cf., e.g.: M. PiIu, On Using Raw MPEG Motion Vectors To Determine Global Camera Motion, Hewlett-Packard Company, 1997). The latter is of particular interest for the present invention in case hardware or software MPEG compression is already implemented in the mobile electronic device.
Power consumption is a potential concern in embodiments of the present invention where an imaging camera and/or a display screen are operated on a battery. In general, the camera part of the system draws little power compared to an illuminated display screen. However, when operating the camera as an input tool according to the present invention, several power conservation strategies are possible, since imaging requirements are different from and generally less demanding than those in regular photography. Thus, the total number of available camera pixels need not be active in the primary photo-event or in subsequent signal processing steps. Frame rates may also be reduced, depending on the updating rate and response speed that is
required in applications based on the present invention. Similarly, in embodiments of the present invention where a battery-fed display screen on the optoelectronic device is involved, several strategies are available to conserve power. Among them are for instance a) reducing the frame rate on the display; b) displaying reduced-complexity images, e.g. line or grid drawings and diagrams, on the screen instead of full-pixellated images; c) employing highlight on darker background; d) introducing automatic "sleep mode" with dark or reduced-intensity screen in periods without operator input.