WO2012177237A1 - Gesture based user interface for augmented reality - Google Patents
Gesture based user interface for augmented reality Download PDFInfo
- Publication number
- WO2012177237A1 WO2012177237A1 PCT/US2011/041173 US2011041173W WO2012177237A1 WO 2012177237 A1 WO2012177237 A1 WO 2012177237A1 US 2011041173 W US2011041173 W US 2011041173W WO 2012177237 A1 WO2012177237 A1 WO 2012177237A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- gesture
- keyboard
- command
- key
- resonant frequency
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/044—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by capacitive means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
- G06F2203/04108—Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
Definitions
- the subject disclosure relates generally to a gesture keyboard, e.g., to a gesture based user interface for augmented reality.
- Some computing systems utilize a touch-screen for the entry of commands.
- the touch-screen makes use of a 2-dimensional input space, which cannot make use of the 3-dimensional space above the touch surface.
- the keyboard is separate from the display, such that commands related to typing, e.g., shift lock, cursor movements, and so forth, are not practical to place within the display. This is because the display is too far away from the keypad, and thus, a user's hand(s) move too far to enter such commands.
- a gesture keyboard which is configured to recognize commands, input in the form of gestures and/or as typing, and implement one or more actions based on the commands.
- the one or more actions can be implemented in an augmented reality.
- the gesture keyboard includes keys that type when pressed and can also sense fingers that are near the keyboard or in contact with the keyboard. The fingers can be sensed by their capacitance. The contact with the keyboard can be sensed by electrical contact between the finger and the key. The electrical-contact finger detection can be used to highlight a virtual keyboard that is displayed in virtual-reality spectacles, which can also show the text being typed.
- the capacitive sensing can be used to detect gestures by fingers near the keyboard and the gestures can be used for typing commands.
- a method includes receiving sensory information associated with an object in proximity to, or in contact with, an input device including receiving at least one level of interaction differentiation detected from at least two levels of interaction differentiation, interpreting a command from the sensory information as a function of the at least one level of interaction differentiation, and outputting an action indication based on the command.
- a system in another embodiment, includes an environmental capture component configured to receive at least one gesture within a space relative to a keyboard, an interpretation component configured to identify a command based on the at least one gesture, and an output component configured to render information of the at least one gesture and a result of the command, in which the information is configured to be rendered on a virtual display.
- a computer-readable storage medium having stored thereon computer-executable instructions that, in response to execution, cause a computing device to perform operations, including detecting a gesture that indicates at least one command to be performed, interpreting the gesture as the at least one command selected from a plurality of commands, and initiating a result of the at least one command as a perceivable event within a virtual space.
- a system includes means for receiving an input in a form of a gesture, means for translating the gesture into a command selected from a set of alternative commands as a function of one or more of a proximity level, an electrical continuity level, and an actuation level, and means for outputting a result of the command, in a perceivable format, to a remote display device.
- Still another embodiment is a computing device that includes a keyboard that includes an array of keys, in which at least a subset of keys of the array of keys include a respective displacement actuated switch configured to detect pressure applied to a respective key of at least the subset of keys, and at least one capacitive sensor configured to detect a finger near the keyboard.
- FIG. 1 shows a flow diagram illustrating an example, non-limiting embodiment of a method for recognizing and implementing commands
- FIG. 2 illustrates a specific non-limiting example of a person using a gesture keyboard, according to an aspect
- FIG. 3 illustrates non-limiting, example electrical circuitry that can be utilized to provide a gesture keyboard
- FIG. 4 illustrates a block diagram of an example, non-limiting
- FIG. 5 illustrates a non-limiting, example system for interpreting and implementing commands received in the form of hand gestures or typing;
- FIG. 6 is a flow diagram illustrating an example, non-limiting embodiment of a method for recognizing and implementing commands;
- FIG. 7 is a flow diagram illustrating an example, non-limiting embodiment of a method for recognizing and implementing commands
- FIG. 8 is a flow diagram illustrating an example, non-limiting embodiment of a method for recognizing and implementing commands
- FIG. 9 is a flow diagram illustrating an example, non-limiting embodiment of a method for recognizing and implementing commands
- FIG. 10 illustrates a flow diagram of an example, non-limiting embodiment of a set of computer readable instructions for a gesture based keyboard in accordance with at least some aspects of the subject disclosure
- FIG. 11 is a block diagram of an example, non-limiting embodiment of a gesture keyboard computing device in accordance with at least some aspects of the subject disclosure.
- FIG. 12 is a block diagram illustrating an example computing device that is arranged for a gesture based keyboard in accordance with at least some embodiments of the subject disclosure.
- the keyboard is separate from the display, such that commands related to typing (e.g., shift lock, cursor movements, and so forth) are not practical to place within the display. This is because the display is too far away from the user's physical input, e.g., farther than the keys that are currently being used by the user.
- the mouse pad which is typically located near the user's thumbs in a laptop, can be used for similar commands because the hands do not need to move far.
- the new commands that have been evolving in the context of touch- screens in 2-dimensional screen space (e.g., sweeping motion, rotation, radial motion) are not suited to typing motions on a mouse pad or tablet.
- these new touch-screen commands cannot make use of the space above the touch surface, so that motions toward or away from the touch screen do not have any effect. Additionally, these new touch-screen commands are based on hardware that may soon become obsolete, namely, the display. For instance, it is anticipated that displays will be replaced or supplemented by virtual-reality headgear (discussed below). Thus, these commands, in their present form, may also become obsolete. [0029] Various non-limiting embodiments are directed to the use of various gestures in an environment in which the various gestures can be applied at substantially the same time as typing, and thereby can improve the working conditions of people who type. Typing is an activity that consumes many hours of people's time, but the present typing paradigm is inefficient and stressful.
- a QWERTY keyboard may include 26 letter keys and 10 digit keys, plus 48 other keys: the ratio of other keys to essential number/letter keys is 4:3. Of these 48 keys, some are typographic symbol keys (e.g., the key for colon and semicolon); subtracting these typographic keys reduces the number of "other" keys to 37, still outnumbering the number/letter/symbol keys.
- typographic symbol keys e.g., the key for colon and semicolon
- a naive person would be unable to decipher most of the keys.
- Many are cryptic (e.g., "F6") and unusable without experience or a lookup table.
- the ratio of cryptic to patent keys is roughly 2: 1. There is little logic to the arrangement and relations of the roughly 27 cryptic mystery keys.
- Mouse/click commands can also be an integral part of a computer system. However, these are often worse than the keys themselves. Precise movements are required by the small dimensions of a mouse pad, which causes users to waste their energy; also, the mouse pad is not able to carry out any action by itself, but requires an auxiliary left click or right click.
- a virtual-reality device can superimpose text and a keyboard image onto the real world from the user's perspective.
- a method includes receiving sensory information associated with an object in proximity to, or in contact with, an input device including receiving at least one level of interaction differentiation detected from at least two levels of interaction differentiation, interpreting a command from the sensory information as a function of the at least one level of interaction differentiation, and outputting an action indication based on the command.
- the at least two levels of interaction are two levels of interaction
- receiving the sensory information further includes at least one of:
- the method includes increasing a number of command gestures based on interpreting the proximity level and sensing the electrical continuity level.
- interpreting the proximity level includes ascertaining a height of the object above the input device based on an output from at least one resonant circuit and locating a lateral position of the object with respect to the input device. Further, ascertaining the height includes comparing a first resonant frequency of a first component of the input device to a second resonant frequency of a second component of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency and measuring a distance between the object and the input device based on the lowest resonant frequency including determining that the object is closer to the first component or the second component based on the lowest resonant frequency. In another example, ascertaining the height includes comparing responses of components of the input device and applying a trigonometric function to the responses to ascertain the height of the object indirectly.
- locating the lateral position includes calculating a first resonant frequency of a first component of the input device and a second resonant frequency of a second component of the input device and determining the lateral position as a function of a local minima between the first resonant frequency and the second resonant frequency.
- locating the lateral position includes detecting a difference between a first resonant frequency of a first component and a second resonant frequency of a second component adjacent to the first component and determining that the object is closer to the first component or the second component as a function of the difference.
- outputting the action indication includes highlighting an item on a virtual display that includes a first portion and a second portion, in which the first portion includes the command associated with the action indication and the second portion includes a virtual representation of the input device. Further to this example, highlighting the item is performed in response to the object being in physical contact with the input device.
- receiving sensory information includes detecting two or more objects in proximity to, or in contact with, the input device at a same time and identifying the two or more objects as a single object for purposes of a gesture command, in which receiving the sensory information includes receiving the sensory information associated with the single object.
- outputting the action indication includes transmitting, to a display device, a signal including the action indication and an interpretation of the command for output by the display device.
- interpreting the command includes interpolating motion of the object. Further, in some examples, interpolating includes receiving data from a plurality of components of the input device and based on the data, constructing a smooth line to locate the object.
- a system in another embodiment, includes an environmental capture component configured to receive at least one gesture within a space relative to a keyboard, an interpretation component configured to identify a command based on the at least one gesture, and an output component configured to render information of the at least one gesture and a result of the command, in which the information is configured to be rendered on a virtual display.
- the interpretation component is further configured to ascertain a proximity or an electrical continuity associated with the at least one gesture. Further to this example, the interpretation component is configured to ascertain the proximity by detecting cursor control and is configured to ascertain that a mechanical actuation indicates manual commands, and in which the electrical continuity facilitates a highlighting of a key in the virtual display.
- the environmental capture component is configured to detect a movement in the space relative to and near the keyboard and detect pressure applied to the keyboard. Further to this example, the output component is configured to highlight at least a portion of the virtual display in response to detection of the pressure applied to the keyboard.
- the system also includes a sensor configured to measure a capacitance as input to a determination of a height and a lateral location of a fingertip above the keyboard.
- the fingertip and a key of the keyboard form a capacitor of an resonant circuit, and in which the capacitance between the fingertip and the key is, by physics, proportional to l/d, where J is a separation in units of one-half a size of the fingertip and the key, and in which a frequency of the resonant circuit is a measure of distance.
- the system includes an evaluation component configured to measure a beat frequency that represents an extent to which the fingertip is closer to a first key than to a second key in a key pair. The beat frequency of the first key and the second key disappears in response to the fingertip being between the first key and the second key.
- the virtual display includes augmented reality spectacles.
- the keyboard includes two portions having respective thumb-actuated space bars, and the two portions are configured to be folded together to cover keys associated with the two portions.
- a computer-readable storage medium having stored thereon computer-executable instructions that, in response to execution, cause a computing device to perform operations, including detecting a gesture that indicates at least one command to be performed, interpreting the gesture as the at least one command selected from a plurality of commands, and initiating a result of the at least one command as a perceivable event within a virtual space.
- detecting the gesture can include distinguishing the gesture that indicates the at least one command from a set of common proximity- level gestures, in which the set of common proximity-level gestures are ignored.
- detecting the gesture can include receiving an actuation that indicates an electrical contact between an external actor and a conductive actuation key on a keyboard.
- detecting the gesture can include comparing a first resonant frequency of a first conductive actuation key of an input device with a second resonant frequency of a second conductive actuation key of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency and measuring a distance between an external actor and the input device based on the lowest resonant frequency including determining that the external actor is closer to the first conductive actuation key or the second conductive actuation key based on the lowest resonant frequency.
- detecting the gesture can include calculating a first resonant frequency of a first conductive actuation key of an input device and a second resonant frequency of a second conductive actuation key of the input device and determining a lateral position of an external actor as a function of a local minima between the first resonant frequency and the second resonant frequency.
- interpreting the gesture includes
- initiating the result includes highlighting an item on a virtual display that includes a first portion and a second portion, illustrating a command associated with the gesture by the first portion, and presenting a representation of an input device by the second portion.
- highlighting the item is in response to physical contact between an external actor and the input device.
- a system in yet another embodiment, includes means for receiving an input in a form of a gesture, means for translating the gesture into a command selected from a set of alternative commands as a function of one or more of a proximity level, an electrical continuity level, and an actuation level.
- the system also includes means for outputting a result of the command, in a perceivable format, to a remote display device.
- the system can also include means for interpreting the proximity level as a function of capacitance, means for sensing the electrical continuity level based on a grounding of a surface of a component of an input device by an object, and means for detecting the actuation level from movement of the component by an external force.
- Still another embodiment described herein is a computing device that includes a keyboard that includes an array of keys, in which at least a subset of keys of the array of keys include a respective displacement actuated switch configured to detect pressure applied to a respective key of at least the subset of keys, and at least one capacitive sensor configured to detect a finger near the keyboard.
- the computing device also includes a translation module configured to translate a gesture near the keyboard into a command and a processor configured to change a display as a function of the command.
- the keyboard is configured to detect an electrical contact between the finger and at least the subset of keys to receive interaction information from a capacitive interaction, a conductive interaction, or a mechanical interaction.
- the display is a remote virtual display and the processor is further configured to generate signals and transmit the signals to the remote virtual display.
- the translation module is further configured to access a data store that includes a set of command gestures
- the data store includes a set of common proximity-level gestures that are distinguishable from the set of command gestures.
- a gesture keyboard which is configured to recognize commands, input in the form of gestures, and implement one or more actions based on the commands.
- one or more of the commands can be input in the form of typing.
- the commands can be input as a combination of gestures and typing.
- typing refers to traditional typing actions (e.g., contact with a key on a keyboard) and “gesture” refers to motions above or near the keyboard (e.g., actions other than typing).
- the one or more commands can be implemented in an augmented reality.
- FIG. 1 shows a flow diagram illustrating an example, non- limiting embodiment of a method for recognizing and implementing commands. The flow diagram in FIG. 1 could be implemented using, for example, any of the systems, such as the system 400 (of FIG. 4), described herein.
- the keyboard can include capacitive keyboards.
- the spectacles can include head- mounted gear.
- the sensory information is associated with an object in proximity to, or in contact with, an input device.
- the input device can be a keyboard and the object can be a fingertip or another item used by an external actor (e.g., a person) to interact with the keyboard.
- the person can interact with the keyboard to enter commands in the form of hand gestures or by typing. Such commands can be received as the sensory information.
- the receiving, at 100 can include receiving at least one level of interaction differentiation detected from at least two levels of interaction
- the interaction differentiation can distinguish between commands entered by typing and commands entered as gestures.
- the at least two levels of interaction differentiation include a proximity level and an electrical continuity level.
- the proximity level can be sensed by capacitance and the electrical continuity level can be sensed by grounding of the surface of a key by the user's finger.
- another level of interaction differentiation includes an actuation level. The actuation level can be sensed by actuation of the key by force or depression.
- a command is interpreted from the sensory information as a function of the at least one level of interaction differentiation (e.g., one of a proximity level, an electrical continuity level, or, in some aspects, an actuation level) that was detected.
- the interpreted command can be selected from a plurality of commands.
- the interpreted command, received in the form of a gesture can also be distinguished from a set of common proximity-level gestures, which are not intended to function as a command. If it is determined that the gesture is a common proximity-level gesture, the gesture is ignored (e.g., not implemented as a command). Further information related to various types of gestures is provided below.
- an action indication is output based on the command.
- the output indication can include highlighting a key in a virtual display that represents a virtual keyboard.
- Other examples include cursor movements, page movements, highlighting for cutting, copying, and pasting, and other actions to be implemented within a document that can be represented in the virtual display.
- the output indication can be presented as a perceivable event within a virtual space.
- an output indication, as the actuation level can include adding a symbol to a text file.
- the virtual display can be implemented with the use of augmented reality spectacles.
- FIG. 2 illustrated is a specific non-limiting example of a person using a gesture keyboard, according to an aspect.
- a left hand 202 and a right hand 204 positioned over a keyboard 206.
- a finger of the right hand 204 is located a distance hi above the right side of the keyboard 206.
- the outspread left hand 202 is a distance h2 above the left side of the keyboard 206.
- both a height and a lateral location of a fingertip above the keyboard can be detected. Additional information related to detection of the height and lateral location will be provided in more detail below.
- the user may be grounded in some circumstances, and the keyboard might include a raised rim or a lightly-biased grounding bar, against which the wrists can rest, to assure that the user and the user's hands are at ground potential.
- a grounding wrist strap may be provided as an option.
- the hands 202, 204 are poised over the keyboard 206 in a position similar to the position that is used to input the one or more gestures.
- the one or more gestures can be capacitive gestures.
- keys are actuated by "hand capacitance" due to proximity of a finger.
- keyboard keys are connected to circuits that are used to detect the proximity of a fingertip. The detection of the fingertip proximity to the keyboard is utilized to detect finger motions in space relative to the keyboard, and, thus, hand gestures can be detected.
- the hand gestures function as commands, and can also function as typing commands, according to some aspects.
- the disclosed aspects can utilize the capacitive hand gestures to replace key-actuated and mouse-pad commands, such as the commands used for moving the screen, cursor movements, return function, control function, clicking, and so forth.
- key-actuated and mouse-pad commands such as the commands used for moving the screen, cursor movements, return function, control function, clicking, and so forth.
- commands are implemented with precision motions and often caused errors.
- the commands can be input with less precise motion, resulting in fewer errors.
- a sensor is configured to measure a capacitance as input to a determination of a height and a lateral location of a fingertip above the keyboard 206.
- hand capacitance can be utilized by the sensor to determine the height and lateral location.
- a variable capacitor is used for radio tuning, and the antenna can be de-tuned by the presence of a hand or other body part near the antenna, especially when the affecting person is grounded.
- Hand capacitance was used intentionally as the basis of the theramin, the first musical instrument played without direct contact.
- the theramin produces sounds of arbitrary pitch, which can sweep rapidly in frequency as the player's hand moves.
- the theramin employs an antenna, which is coupled to an audio-frequency oscillator including a resonant LC (inductor-capacitor) circuit.
- LC capacitor-capacitor
- the antenna's capacitance is varied by the player's hand moving closer to, or farther from, the antenna, the frequency of the LC oscillations vary.
- the oscillations are amplified electronically and then sent to a loudspeaker, producing a sound of pitch that depends on the position of the person's hand.
- Any conductive or partly-conductive object has capacitance, and its capacitance is affected by another object brought close (as the capacitance of the other object is also affected).
- a person' s fingertip for example, has a certain capacitance; another small conductive object, which might be a key on a keyboard, also has capacitance, and the two have a mutual capacitance. Their mutual capacitance is determined by geometry.
- the capacitance is large because the plates are close. If a first plate is grounded, and a voltage is impressed on a second plate by a battery, a charge appears on the second plate (but not the first plate, because the first plate is grounded).
- the capacitance of an object is defined as the amount of electrical charge that will accumulate on that object when it is raised to a potential of one volt.
- a fingertip can be approximately modeled as a sphere, for the purpose of investigating its capacitance.
- the resonant frequency will vary roughly as the square root of d, because the frequency is proportional to the inverse square root of the product of L and C. Because d is in units comparable to the size of either the finger or the key, the frequency of the key-fingertip circuit will be a sensitive measure of the distance, especially when the fingertip is near to the key.
- each key has its own resonant circuit formed with the key as part of a capacitor and a fixed inductance, then the key that has the lowest resonant frequency is the key closest to the fingertip, and that frequency is a measure of the distance from the fingertip to the nearest key (e.g., the height).
- the height of a finger can also be indirectly detected by comparing responses of keys, by trigonometry.
- the ratio of the distance from a hovering fingertip to a key directly beneath, to the distance of that fingertip to the adjacent key increases as the fingertip moves toward the key beneath the fingertip.
- the distance to a key next to it is 1.41 d, because the two keys form a 45-45-90 right triangle.
- a 30-60-90 right triangle is formed and the distance ratio is 1.15 instead of 1.41.
- the distance to the nearest finger can also be calculated in a similar manner, by ratios.
- a sensor can be configured to detect the location by finding the key that, in a local area, has the lowest frequency in its LC circuit. If the fingertip causes two keys to have the same resonant frequency, then the finger is in between the two keys.
- the resonant frequency varies as the square root of the distance, the ratio of change in frequency to change in distance is greatest at shortest distances, and is much less at larger distances; that is, the device is more sensitive at closer distances.
- This physical fact increases the utility of the gesture-based system disclosed herein, because a gesture, such as holding the hand flat and parallel to the keyboard (discussed below), which can be performed at a greater height, is less likely to cause confusion in the gesture-detecting circuitry.
- a beat frequency which represents an extent to which the fingertip is closer to a first key than to a second key in a key pair is provided.
- the resonant frequency of each key can be utilized directly, finding local minima of frequency that indicates a finger above.
- the resonant frequency of each key can be utilized indirectly, by finding the beat frequency, which is the difference of the frequencies of two adjoining keys
- the beat frequency represents the extent to which the fingertip is closer to one key than the other key. When the beat frequency between two keys disappears, then the finger is halfway between those keys. Similarly, when the user's hand is far away, then the beat frequency will be essentially zero between any pair of keys, since each key will have almost the same frequency (due to the variation as the square root of d, discussed above). In general, a single fingertip near the keyboard will be encircled by rings or loops of approximately zero beat frequency, but the keys in each ring or loop will have a non-zero beat frequency with respect to keys that are farther from or closer to the point under the key.
- second-order beat frequencies which are beat frequencies between the first-order beat frequencies.
- the second-order beat frequencies can provide a computational advantage in some circumstances.
- Third and higher orders can also be found. There may be a limit to this process, however, because the number of results shrinks with each iteration as an outer layer of keys is removed.
- the first level is proximity, which is sensed by capacitance; the second is electrical continuity, which is sensed by grounding of the surface of a key by the user's finger; and the third is actuation of the key by force or depression, as in a standard keyboard.
- force can be sensed by keys with no moving parts or minimally-moving parts, such as piezoelectric devices, capacitive devices similar to a capacitor microphone, or other devices.
- the capacitive sensing may be used for cursor control.
- the electrical continuity may be used for highlighting a key in the virtual display (e.g., when the "H" key is lightly touched, it can be highlighted in the virtual keyboard display).
- the mechanical key actuation may be used for typing a symbol or for commands such as "enter” and "shift” that are ingrained in users and perhaps should not be changed for that reason.
- one way to mitigate confusion between capacitive sweeps and conductive sweeps is to only register conductive sweeps that extend over a certain minimum distance or last for a certain minimum time. Further information related to gestures and associated commands will be provided below.
- the sensing of hand capacitance can be coupled with a virtual display 208, shown as including a virtual keyboard 210 and two documents 212 (e.g., text documents or other types of documents).
- the virtual display 208 can be provided by augmented-reality spectacles 214.
- the use of spectacles 214 can reduce the unit to pocket-size and can also provide a 3-D display with additional display and control aspects.
- the spectacles 214 can be configured to allow the user to see both the text, images, or other information that is being worked and also the virtual keyboard 210, the keys of which can be highlighted 216 in response to a light touch to the keyboard 206 by the user.
- Spectacles such as those discussed below can replace the current flat displays because the spectacles are smaller, lighter, less likely to be stolen (because they are more likely to be carried on the person), with smaller environmental impact, and can be less expensive.
- the spectacles are an augmented-reality device that can be used with many applications. Further, the wearing of spectacles can become common, such as it is common to wear ear- mounted telephones and music devices.
- the spectacles 214 provide a virtual-reality device that superimposes text and a keyboard image onto the real world. This can permit typing notes during a meeting without having to divide attention between the person talking and the textual notes being taken. Further, the notes can remain confidential since the notes will not be on a screen, only on the spectacles.
- the disclosed aspects are not limited to spectacles and are compatible with laptop computers and tablets.
- a split display, with a virtual keyboard on the lower half of the display and text on the upper half can be utilized.
- One type of spectacles that can be utilized with the various aspects are Cinemizer spectacles from Carl Zeiss, which allow private movie watching.
- the Cinemizer physically resembles wrap-around sunglasses.
- Using two high resolution liquid crystal display (LCD) screens and two focusable lens systems the image is similar to that of a real screen with 1/20-inch pixels and a 45 diagonal (aspect ratio 3:4) placed 6 feet away. In a non-limiting test, the power lasted 4 hours, and the weight was 100 g.
- LCD liquid crystal display
- spectacles utilized with the disclosed aspects can use partially-silvered, zero-diopter lenses that reflect light rays projected from the temple portions into the eyes of the user.
- the spectacles can employ low-power laser diodes (similar to the type used in laser pointers), which can be modulated at megahertz speeds simply by turning the power on and off.
- the modulated laser beams can be steered by a mirror or mirrors in a raster pattern, to produce images. Rather than scan the laser beams onto a surface, the modulated beams can be scanned directly into the eye (a "virtual retinal display"), using the partially-silvered inside surface of the eyeglasses lens as a converging mirror.
- a spectacle that can be utilized is a spectacle that has the lasers and beam- steering mirrors mounted in the eyeglass temples.
- the steering mirrors reflect the beams off the partially-metalized inside of the lens, as mentioned above, into the eye.
- the eyeglass lens can include a fully- silvered surface on the outer portion, while the portion through which the user looks could be shaped as a paraboloid, ellipsoid of revolution, or other suitable shape, to act as a converging (focusing) element.
- the effect can be similar to the effect as that of a converging lens placed directly in front of the user's eye, with the scanning mirror located on the lens axis to shine the beam through the lens and thence into the user's eye; the lens can form the virtual image.
- the apparent size would depend on the focal length of the lens.
- Two laser diodes and two beam-steering mirrors can provide a 3-D image that would allow different text files (or other images) to be pushed into the background and pulled forward, using a push/pull gesture.
- Ordinary red laser light, as from a laser pointer, can also be suitable with the various aspects discussed herein.
- the term “spectacles” is intended to cover any device developed in the future, such as contact lenses that form images, implants, and so forth.
- the term “spectacles” is also intended to include a monocle or other single- eye device.
- the virtual display 208 might have two portions, an upper portion showing text (and/or figures, etc.) and a lower portion showing the keyboard (analogous to a laptop).
- the upper portion showing text and the lower portion showing the keyboard can be similar to the two documents 212.
- a key that is grounded can be highlighted in the virtual display, and this can make typing much easier.
- the less-expert touch typist, who now glances away from the display to find a number key, or even the "z" key, will not need to avert her eyes because the touched key can be highlighted (as shown on the keyboard 210).
- the touched key can appear bright, flashing, in different color, or otherwise highlighted, and therefore the desired key can easily be found with a gliding finger. Once located, the key can be depressed to type the symbol without any chance (or minimized chance) for error.
- Fig. 3 illustrates non-limiting, example electrical circuitry 300 that can be utilized to provide a gesture keyboard.
- a standard keyboard can be fitted with metal foils 302 or other conductive parts on the key surfaces.
- the foils 302 can function as both capacitor plates and grounding electrodes, and these electrodes can be coupled to circuits below.
- the frequency of the circuit changes.
- a lead 310 from the foil 302 can provide an input to an electrical contact sensing module 312, which can interact with a processor 314.
- the electrical contact sensing module 312 is configured to detect the electrical continuity.
- the foil 302 on key 306 acts as a capacitor plate in opposition to the grounded finger 304, and connects though the lead 310 with an inductor 318 and thereby forms a resonant tank circuit (other types of resonant circuits can also be used).
- the circuit also includes, for example, a magnetically-coupled pickup coil 316 that is coupled to a frequency detection module 320 to determine whether the finger 304 is near the key 306, far from the key 306, or even in contact with the key 306, by picking up signals from the resonance in the coil 318. Physical contact of the finger 304 with the foil 302 can also be sensed by a dedicated contact-sensing circuit 312.
- the pickup coil 316 can drive the oscillations of the tank circuit formed by the inductor 318 and capacitor 302/304 with, for example, positive feedback.
- the frequency detection module 320 is utilized as a sensor that detects the proximity.
- a pressure switch 322 located under the key 306 can be detected by a debounce module 324, which can communicate with the processor 314.
- the pressure switch 322 and/or debounce module 324 facilitate sensing the mechanical actuation.
- the processor 314 is configured to monitor the keys and can ascertain the position of the finger by the frequency, or, in an alternate aspect, the processor 314 can monitor the keys by a parallel circuit that detects only continuity.
- the processor 314 can cause an output (e.g., a result of a command, information, an action indication, and so forth) to be rendered on a display 326, which can be presented to the user in a perceivable format.
- Doubled fingers or multiple pursed fingers can be interpreted and treated by the circuitry in a similar manner as an isolated, or a single finger.
- a single finger or group can be differentiated from open-handed gestures with fingers separated by, for example, about two inches as illustrated by the left hand 202 in Fig. 2.
- the open- hand gesture shown in Fig. 2 can be differentiated on the basis of its decreased capacitance gradient, especially when farther from the keyboard (e.g., when hi is less than hi in Fig. 2). Gestures that require precision are to be minimized because these gestures will, similar to the fine motions required for hitting a particular key, be difficult to perform accurately.
- gestures specifically intended to replace the worst of the current mystery-key/point-click commands can be selected. These include cursor movements, page movements, and highlighting for cutting, copying, and pasting, which typically utilize not one but two fingers to simultaneously be located within a quarter-inch at respective target keys (e.g., "ctrl V" for pasting).
- a type of gesture that can be exploited is the sweep, a lateral motion of the hand or fingers across the keyboard, for example, by moving the hand or one or more fingers parallel to the keyboard surface (which can be detected by capacitance sensing).
- a sweep can be performed repeatedly and quickly by circling the hand above the keyboard around a horizontal axis, and can easily be varied from brisk bold motions for large displacements to small or slower motions for fine displacements of a cursor, or of the display, which is an intuitive motion.
- sweep control can be finer and more intuitive if the motion parallel to the keyboard surface is multiplied by the inverse of the height, for example, by multiplying the speed by the lowest resonant frequency. This can cause a greater response when the finger is closer to the keyboard, which is an intuitive variation.
- a sweep can also be performed conductively, for example, by lightly brushing the fingers over the keyboard, making electrical contact but not actuating the keys for typing. This is discussed in further detail below.
- touch-screen contact sweep is used in some devices to input a command for lateral motion of the displayed image.
- the touch-screen contact sweep can be performed capacitively or conductively. Virtual momentum and friction can be used also. To mitigate confusion, a contact sweep can be ignored unless it is of a certain distance and/or duration, for example.
- the sweep is only horizontal, then its vertical counterpart can be called a roll.
- the touch-screen contact sweep and roll can be distinct gestures or subsumed in a single lateral movement at an arbitrary angle, whereby display elements could be moved diagonally.
- a variation on the sweep is the dither, a back-and-forth motion that includes consecutive alternating opposite motions parallel to the keyboard.
- gestures are pushing or pulling.
- the hand is spread flat, fingers extended, forming a capacitor ground plate parallel to the keyboard, and moved toward or away from the keyboard.
- This gesture can be performed independently on the two sides of the keyboard, allowing two commands each with two senses.
- a stationary hover of the open hand can also constitute a command analogous to "ctrl" or "alt,” for example, a mode changer that would modify the gesture being made by the other hand.
- the push-pull gesture can be used to move virtual text (or other display) toward or away from the user. This might be useful for switching documents: one could be brought forward, whereupon another would recede out of eye convergence range (accomplished with the 3-D feature of the spectacles) and could also be made fainter so as not to distract. Open documents could also be moved laterally by horizontal sweeping and vertical rolling.
- circling Another example type of gesture is circling, which can be performed either capacitively or conductively. Since there are two sides to the keyboard and two senses of rotation, circling can convey four different commands (more if the diameter of the circle is detected and made a factor in the gesture).
- a further example of a gesture is holding the fingers in one area of the keyboard, where the distance and direction from the center have a similar effect as deflection of a joystick. (In polar coordinates, the distance and direction from the center would be r and ⁇ .)
- This can be used to control the cursor, in the manner of certain laptops that have a nubbin, which moves the cursor in the direction the nubbin is deflected.
- This type of control can benefit from virtual physics. If the deflection were taken as an accelerating virtual force on a virtually-massive cursor (possibly with virtual friction), then the interface can be more natural.
- a reversal of the fingers might be interpreted differently from an advance of the fingers away from the center, the first leading to a different acceleration as a function of position. Since the height of a finger is already detected, this gesture could be generalized to include the height as a controlling factor. In cylindrical coordinates, r, ⁇ , and z would all be input variables.
- gestures are expansion/contraction, which can be accomplished by moving two fingers or hands closer and farther to one another.
- This gesture can be used in touch-screen devices to command expansion or contraction of the displayed image. In the keyboard, this gesture could have a different meaning.
- a further example gesture is rotating with two fingers, which can be used to turn an image.
- gestures discussed herein include gestures than can be mined for use as typing commands, and therefore can allow for a new interface that utilizes fewer keys and less precision. It also offers programmers an opportunity to write new code that obeys, rather than frustrates, the user. The best
- gestures that occur normally such as moving both hands onto position over the keyboard simultaneously when commencing to type, should not be construed as commands and may be excluded (or ignored).
- gestures might also be used for menu navigation, perhaps after switching modes via a gesture or key.
- the disclosed aspects can be incorporated into a larger gesture- controlled virtual world that is extraneous to typing.
- FIG. 4 illustrates a block diagram of an example, non-limiting embodiment of a system that is configured to recognize and implement commands.
- a gesture-based system 400 is depicted that includes an environmental capture component 402 configured to receive at least one gesture 404 within a space relative to a keyboard 406.
- the gesture 404 is intended to be a command 408, such as, for example, cursor control, scrolling of a document, and so forth.
- the command 408 is received in the form of typing 410.
- the space relative to the keyboard 406 can be a location on (e.g., physical contact with) the keyboard 406 or a location above (e.g., no physical contact with) the keyboard 406.
- the environmental capture component 402 is configured to detect the presence of fingers of either or both hands, even though the fingers might not be in physical contact with the keyboard 406.
- the keyboard 406 can be a standard keyboard retrofitted with metal foils or other conductive parts on the key surfaces, which will be discussed in further detail below.
- the keyboard 406 can be a two-part folding keyboard, in which the two halves are divided in the manner of "ergonomic" keyboards, but with a thumb-actuated space bar on each half. If connected by a stiff but yielding universal joint located in the middle of the upper edge, the keyboard halves can be aligned on a tabletop in the usual orientation, spread slightly into the ergonomic configuration, or draped over the lap. When not in use, the two halves can be folded together to cover the keys, resulting in a package measuring about 4.5 by 6 inches, the size of a paperback novel.
- One-piece keyboards can also be used, according to an aspect.
- a processor can be located in the keyboard (as in present laptops) or in a separate unit.
- the two sides of the keyboard can be used independently, and, thus, the mechanical design can also provide for some separation mechanism.
- Fig. 5 illustrates a non-limiting, example system for interpreting and implementing commands received in the form of hand gestures or typing. As shown, Fig.
- FIG. 5 depicts a gesture -based system 500 that includes an environmental capture component 502 configured to receive at least one command 504 within a space relative to a keyboard 506.
- the keyboard 506 is similar to keyboard 406 in Fig. 4.
- the environmental capture component 502 is configured to detect a movement in the space relative to and near the keyboard 506 (e.g., gestures) and to detect pressure applied to the keyboard 506 (e.g., typing).
- one or more sensors can be configured to detect a proximity 508, an electrical continuity 510, or a mechanical actuation 512.
- a sensor can be configured to detect a height and a lateral location of a fingertip above a keyboard by capacitance.
- gesture-based system 500 Also included in gesture-based system 500 is an interpretation component 514 configured to identify the command 516 based on at least one gesture and/or the typing.
- the interpretation component 514 is configured to identify at least one command as a function of the proximity 508 by capacitance, for example.
- the interpretation component 514 is configured to identify the at least one command based on the mechanical actuation 512, which can indicate manual commands. Further, the interpretation component 514 can identify a command and can interpret that the electrical continuity 510 should produce highlighting of a key in a virtual display.
- the gesture-based system 500 also includes an evaluation component 516 configured to measure a beat frequency that represents an extent to which the fingertip is closer to a first key than to a second key in a key pair.
- the evaluation component 516 is configured to use the resonant frequency of each key directly, finding local minima of frequency that indicate a finger above.
- the evaluation component 516 is configured to use the resonant frequency of each key indirectly, by finding the beat frequency, which is the difference of the frequencies of two adjoining keys (modulation).
- evaluation component 516 is configured to find second-order beat frequencies or higher order beat frequencies.
- the keyboard 506 includes a circuit that detects when frequency of an oscillator goes below a fixed or predetermined frequency that correlates with a certain distance above the keyboard 506. This can create a virtual barrier, penetration of which by a fingertip can be an event that causes that key to register. There could be two or more such barriers, which can be utilized to differentiate gestures performed at different heights. There are many possible detection methodologies.
- the disclosed aspects include any program or circuitry that detects fingers or other objects near the keyboard 506, and tracks the fingers or objects in such a way that gestures near the keyboard 506 can be discriminated and determined.
- an output component 518 configured to render information 520 of the command 504 and a result 522 of the command.
- the information 520 and/or the result 522 is configured to be rendered on a virtual display 524.
- the information 520 and/or the result 522 is configured to be rendered in a virtual space.
- the output component 518 is configured to highlight at least a portion of the virtual display 524 in response to detection of pressure applied to the keyboard 506.
- the keyboard 506 does not use capacitance to actuate a key; instead, the keyboard 506 uses mechanical key actuation, which provides beneficial tactile and auditory feedback to the user.
- the keyboard 506 thus allows the fingers to touch the keys without causing an actuation, and therefore permits the user to make hand gestures in the space above the keyboard 506 without worrying about lightly hitting the keyboard 506. This freedom from error can make performance of the gestures easier and less stressful.
- FIG. 6 is a flow diagram illustrating an example, non-limiting
- method 600 for recognizing and implementing commands.
- the method 600 in FIG. 6 could be implemented using, for example, any of the systems, such as the system 400, described herein and could be used to recognize and implement commands.
- method 600 can include receiving sensory
- an input device including receiving at least one level of interaction differentiation detected from at least two levels of interaction differentiation.
- the at least two levels of interaction differentiation can include a proximity level and an electrical continuity level.
- a third level of interaction differentiation includes an actuation level.
- the proximity level is interpreted as a function of capacitance.
- the electrical continuity level is sensed based on a grounding of a surface of at least one component of the input device by the object.
- the actuation level is detected, at 608, from movement of the at least one component by an external force. It should be noted that at least one of interpreting, at 604, sensing, at 606, or detecting, at 608 can be implemented. For example, the electrical continuity level might be sensed, but the proximity level might not be interpreted and the actuation level might not be detected, according to an aspect.
- a command is interpreted from the sensory information as a function of the at least one level of interaction differentiation.
- an action indication is output based on the command.
- the action indication can highlight a key on a virtual keyboard and/or perform an action within a virtual display.
- the virtual keyboard and/or virtual display can be realized through the use of spectacles or through the use of other devices that can provide the virtual reality.
- FIG. 7 is a flow diagram illustrating an example, non-limiting
- method 700 for recognizing and implementing commands.
- the method 700 in FIG. 7 could be implemented using, for example, any of the systems, such as the system 500, described herein and could be used to recognize and implement commands.
- method 700 can include receiving sensory
- the information associated with an object in proximity to, or in contact with, an input device including receiving at least one level of interaction differentiation detected from at least two levels of interaction differentiation.
- the at least two levels of interaction differentiation can include a proximity level and an electrical continuity level.
- a command is interpreted from the sensory information as a function of the at least one level of interaction differentiation.
- Interpreting the command can include ascertaining a height of the object above the input device based on an output from at least one resonant circuit, at 706, and locating a lateral position of the object with respect to the input device, at 708.
- ascertaining the height of the object includes comparing a first resonant frequency of a first component of the input device to a second resonant frequency of a second component of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency. Further to this implementation, ascertaining the height of the object, at 706, includes measuring a distance between the object and the input device based on the lowest resonant frequency including determining that the object is closer to the first component or the second component based on the lowest resonant frequency, at 710.
- ascertaining the height of the object includes comparing responses of components of the input device and applying a trigonometric function to the responses to ascertain the height of the object indirectly, at 712.
- locating the lateral position, at 708, includes calculating a first resonant frequency of a first component of the input device and a second resonant frequency of a second component of the input device; and determining the lateral position as a function of a local minima between the first resonant frequency and the second resonant frequency, at 714.
- locating the lateral position, at 708, includes detecting a difference between a first resonant frequency of a first component and a second resonant frequency of a second component adjacent to the first component, and determining that the object is closer to the first component or the second component as a function of the difference, at 716.
- An action indication is output, at 718, based on the command.
- the action indication can be output to a virtual display and/or a virtual space that cannot be viewed by others and, therefore, remains confidential to the user.
- FIG. 8 is a flow diagram illustrating an example, non-limiting
- a method 800 for recognizing and implementing commands could be implemented using, for example, any of the systems, such as the system 400, described herein.
- sensory information is received.
- the sensory information is associated with an object in proximity to, or in contact with, an input device.
- the receiving can include receiving at least one level of interaction differentiation detected from at least two levels of interaction
- the at least two levels of interaction differentiation include a proximity level and an electrical continuity level.
- a third level of interaction differentiation includes an actuation level.
- a command is interpreted from the sensory information as a function of the at least one level of interaction differentiation.
- the interpretation of the command can include detecting two or more fingers and interpreting the two or more fingers as a single, isolated finger for purposes of interpreting the sensory information.
- an action indication is output based on the command.
- outputting the action indication includes, at 808, highlighting an item on a virtual display that includes a first portion and a second portion. The first portion includes the command associated with the action indication and the second portion includes a virtual representation of the input device. In an example, the highlighting the item is performed in response to the object being in physical contact with the input device.
- outputting the action indication includes transmitting a signal including the action indication and an interpretation of the command, at 810. The signal can be transmitted to a display device that is configured to output the action indication and the interpretation of the command.
- the display device can be implemented through the use of spectacles or another device that can be configured to present a virtual display to the user.
- FIG. 9 is a flow diagram illustrating an example, non-limiting
- a method 900 for recognizing and implementing commands could be implemented using, for example, any of the systems, such as the system 500, described herein and could be used to recognize and implement commands.
- sensory information is received.
- the sensory information is associated with an object in proximity to, or in contact with, an input device.
- the receiving, at 902 also includes, at 904, detecting two or more objects in proximity to, or in contact with, the input device at a same time, or at substantially the same time, and identifying the two or more objects as a single object for purposes of a command.
- the receiving the sensory information includes receiving the sensory information associated with the single object.
- a command is interpreted from the sensory information as a function of the at least one level of interaction differentiation.
- the interpretation, at 906, includes interpolating motion of the object, at 908.
- interpolating motion of the object includes receiving data from a plurality of components of the input device and, based on the data, constructing a smooth line to locate the object, at 910.
- resonant frequencies of adjacent keys can provide an indication of relative distance of the fingertip from the two keys, and therefore can locate the fingertip to a precision greater than that of the key array spacing.
- method 900 can include increasing a number of command gestures based on interpreting the proximity level and sensing the electrical continuity level, at 912.
- An action indication is output, at 914, based on the command.
- FIG. 10 illustrates a flow diagram of an example, non-limiting
- Computer-readable storage medium 1000 can include computer executable instructions. At 1002, these instructions can operate to detect a gesture or typing that indicates at least one command to be performed. In an example, to detect the gesture, the instructions can operate to distinguish the gesture that indicates the at least one command from a set of common proximity-level gestures, and the set of common proximity-level gestures are ignored.
- the instructions can operate to receive an actuation that indicates an electrical contact between an external actor and a conductive actuation key on a keyboard.
- the instructions can operate to compare a first resonant frequency of a first conductive actuation key of an input device with a second resonant frequency of a second conductive actuation key of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency. Further to this example, the instructions can operate to measure a distance between an external actor and the input device based on the lowest resonant frequency including determining that the external actor is closer to the first conductive actuation key or the second conductive actuation key based on the lowest resonant frequency.
- the instructions can operate to calculate a first resonant frequency of a first conductive actuation key of an input device and a second resonant frequency of a second conductive actuation key of the input device. Further to this example, the instructions can operate to determine a lateral position of an external actor as a function of a local minima between the first resonant frequency and the second resonant frequency.
- the instructions can operate to interpret the gesture as the at least one command selected from a plurality of commands.
- the instructions can operate to ascertain the gesture is at least one of a sweep gesture, a push-pull gesture, a circle gesture, a deflection gesture, an expansion- contraction gesture, or a rotating gesture.
- the instructions can operate to initiate a result of the at least one command as a perceivable event within a virtual space.
- the instructions can operate to highlight an item on a virtual display that includes a first portion and a second portion, illustrating a command associated with the gesture by the first portion, and presenting a representation of an input device by the second portion. Highlighting the item can be in response to physical contact between an external actor and the input device.
- FIG. 11 is a block diagram of an example, non-limiting embodiment of a gesture keyboard computing device in accordance with at least some aspects of the subject disclosure.
- a computing device 1100 can include a keyboard 1102.
- the keyboard 1102 is configured to detect an electrical contact between the finger and at least the subset of keys to receive interaction information from a capacitive interaction, a conductive interaction, or a mechanical interaction.
- the keyboard 1102 can include an array of keys, in which at least a subset of keys of the array of keys include a respective displacement actuated switch configured to detect pressure applied to a respective key of at least the subset of keys.
- the keyboard 1102 can also include at least one capacitive sensor 1106. The at least one capacitive sensor 1106 is configured to detect a finger near the keyboard 1102.
- the computing device 1100 also include a translation module 1108 configured to translate a gesture near the keyboard 1102 into a command.
- the translation module 1108 is further configured to access a data store 1110 that includes a set of command gestures corresponding to different commands.
- the data store 1110 includes a set of common proximity- level gestures that are distinguishable from the set of command gestures.
- Computing device 1100 also includes a processor 1112 configured to change a display 1114 as a function of the command.
- the display 1114 As a function of the command.
- the processor 1112 is further configured to generate signals and transmit the signals to the remote virtual display.
- various aspects related to a gesture keyboard can be implemented without a physical display and can be utilized with a folding or roll-up keyboard that is easy to carry.
- the compact system be can easily be used in awkward locations, such as on airplanes.
- the various aspects are also compatible with interim laptop computers having real (e.g., physical) displays.
- the various aspects, with the use of spectacles can provide a virtual-reality device that
- the keyboard can be smaller than conventional keyboards, since many keys are not utilized and/or not included on the keyboard.
- large gestures can be utilized with the disclosed aspects, which can mitigate an amount of mistakes, extra work, and aggravation. For example, the user does not have to remove his hands from the typing position in order to make the commands, because the gestures are right above the keyboard.
- the disclosed aspects provide for fast hand alignment for touch-typing, with highlighted keys triggered by light contact. The user does not need to glance at the keyboard to orient her hands, because the hand position is evident on the display.
- the disclosed aspects provide a large number of gestures, which are scalable, meaning that the size of the gesture can be translated into part of the command.
- the commands can be used for all ordinary commands of degree (e.g., cursor control, display motions) and also for discrete commands that do not incorporate any magnitude (e.g., shift lock, etc.).
- FIG. 12 is a block diagram illustrating an example computing device 1200 that is arranged for a gesture based keyboard in accordance with at least some embodiments of the subject disclosure.
- a very basic configuration 1202 In a very basic configuration 1202,
- computing device 1200 typically includes one or more processors 1204 and a system memory 1206.
- a memory bus 1208 may be used for communicating between processor 1204 and system memory 1206.
- processor 1204 may be of any type including but not limited to a microprocessor ( ⁇ ), a microcontroller ( ⁇ ), a digital signal processor (DSP), or any combination thereof.
- Processor 1204 may include one more levels of caching, such as a level one cache 1210 and a level two cache 1212, a processor core 1214, and registers 1216.
- An example processor core 1214 may include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof.
- An example memory controller 1218 may also be used with processor 1204, or in some implementations memory controller 1218 may be an internal part of processor 1204.
- system memory 1206 may be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof.
- System memory 1206 may include an operating system 1220, one or more applications 1222, and program data 1224.
- Application 1222 may include a gesture detection and interpretation algorithm 1226 that is arranged to perform the functions as described herein including those described with respect to gesture-based system 400 of FIG. 4.
- Program data 1224 may include gesture commands and common proximity-level gesture information 1228 that may be useful for operation with gesture detection and interpretation algorithm 1226 as is described herein.
- application 1222 may be arranged to operate with program data 1224 on operating system 1220 such that a gesture based keyboard and an augmented virtual reality experience may be provided.
- This described basic configuration 1202 is illustrated in FIG. 12 by those components within the inner dashed line.
- Computing device 1200 may have additional features or functionality, and additional interfaces to facilitate communications between basic configuration 1202 and any required devices and interfaces.
- a bus/interface controller 1230 may be used to facilitate communications between basic configuration 1202 and one or more data storage devices 1232 via a storage interface bus 1234.
- Data storage devices 1232 may be removable storage devices 1236, non-removable storage devices 1238, or a combination thereof. Examples of removable storage and non-removable storage devices include magnetic disk devices such as flexible disk drives and hard- disk drives (HDD), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSD), and tape drives to name a few.
- HDD hard- disk drives
- optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives
- SSD solid state drives
- Example computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.
- System memory 1206, removable storage devices 1236, and nonremovable storage devices 1238 are examples of computer storage media.
- Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information and which may be accessed by computing device 1200. Any such computer storage media may be part of computing device 1200.
- Computing device 1200 may also include an interface bus 1240 for facilitating communication from various interface devices (e.g., output devices 1242, peripheral interfaces 1244, and communication devices 1246) to basic configuration 1202 via bus/interface controller 1230.
- Example output devices 1242 include a graphics processing unit 1248 and an audio processing unit 1250, which may be configured to communicate to various external devices such as a display or speakers via one or more A/V ports 1252.
- Example peripheral interfaces 1244 include a serial interface controller 1254 or a parallel interface controller 1256, which may be configured to communicate with external devices such as input devices (e.g., mouse, pen, voice input device, etc.) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I/O ports 1258.
- An example communication device 1246 includes a network controller 1260, which may be arranged to facilitate communications with one or more other computing devices 1262 over a network communication link via one or more communication ports 1264.
- the network communication link may be one example of a
- Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media.
- a "modulated data signal” may be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
- communication media may include wired media such as a wired network or direct- wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media.
- RF radio frequency
- IR infrared
- computer readable media may include both storage media and communication media.
- the computer-readable instructions described herein can be implemented as computer-readable instructions stored on a computer-readable medium.
- the computer-readable instructions can be executed by a processor of a mobile unit, a network element, and/or any other computing device.
- the implementer may opt for a mainly hardware and/or firmware vehicle; if flexibility is paramount, the implementer may opt for a mainly software implementation; or, yet again alternatively, the implementer may opt for some combination of hardware, software, and/or firmware.
- a signal bearing medium examples include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a CD, a DVD, a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).
- a typical data processing system generally includes one or more of a system unit housing, a video display device, a memory such as volatile and non-volatile memory, processors such as microprocessors and digital signal processors, computational entities such as operating systems, drivers, graphical user interfaces, and applications programs, one or more interaction devices, such as a touch pad or screen, and/or control systems including feedback loops and control motors (e.g., feedback for sensing position and/or velocity; control motors for moving and/or adjusting components and/or quantities).
- a typical data processing system may be implemented utilizing any suitable commercially available components, such as those typically found in data computing/communication and/or network computing/communication systems.
- any two components so associated can also be viewed as being “operably connected”, or “operably coupled”, to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable”, to each other to achieve the desired functionality.
- operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.
- a system having at least one of A, B, and C would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.).
- a convention analogous to "at least one of A, B, or C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., " a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.).
- a range includes each individual member.
- a group having 1-3 cells refers to groups having 1, 2, or 3 cells.
- a group having 1-5 cells refers to groups having 1, 2, 3, 4, or 5 cells, and so forth.
Abstract
Description
Claims
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/320,904 US9547438B2 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
PCT/US2011/041173 WO2012177237A1 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
JP2014508330A JP2014515147A (en) | 2011-06-21 | 2011-06-21 | Gesture-based user interface for augmented reality |
KR1020137022939A KR101516513B1 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
CN201180071787.5A CN103635869B (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
US15/361,638 US9823752B2 (en) | 2011-06-21 | 2016-11-28 | Gesture based user interface for augmented reality |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2011/041173 WO2012177237A1 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/320,904 A-371-Of-International US9547438B2 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
US15/361,638 Continuation US9823752B2 (en) | 2011-06-21 | 2016-11-28 | Gesture based user interface for augmented reality |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012177237A1 true WO2012177237A1 (en) | 2012-12-27 |
Family
ID=47361356
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2011/041173 WO2012177237A1 (en) | 2011-06-21 | 2011-06-21 | Gesture based user interface for augmented reality |
Country Status (5)
Country | Link |
---|---|
US (2) | US9547438B2 (en) |
JP (1) | JP2014515147A (en) |
KR (1) | KR101516513B1 (en) |
CN (1) | CN103635869B (en) |
WO (1) | WO2012177237A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2016519819A (en) * | 2013-04-02 | 2016-07-07 | クイックステップ テクノロジーズ リミテッド ライアビリティ カンパニー | A device that interacts with an electronic device and / or computer device in a non-contact manner, and a device including the device |
CN108519855A (en) * | 2018-04-17 | 2018-09-11 | 北京小米移动软件有限公司 | Characters input method and device |
US10474287B2 (en) | 2007-01-03 | 2019-11-12 | Apple Inc. | Double-sided touch-sensitive panel with shield and drive combined layer |
Families Citing this family (51)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9430145B2 (en) * | 2011-04-06 | 2016-08-30 | Samsung Electronics Co., Ltd. | Dynamic text input using on and above surface sensing of hands and fingers |
US20150169059A1 (en) * | 2012-04-18 | 2015-06-18 | Nokia Corporation | Display apparatus with haptic feedback |
WO2013166261A1 (en) * | 2012-05-03 | 2013-11-07 | Georgia Tech Research Corporation | Methods, controllers and computer program products for accessibility to computing devices |
US9696547B2 (en) * | 2012-06-25 | 2017-07-04 | Microsoft Technology Licensing, Llc | Mixed reality system learned input and functions |
US9300645B1 (en) * | 2013-03-14 | 2016-03-29 | Ip Holdings, Inc. | Mobile IO input and output for smartphones, tablet, and wireless devices including touch screen, voice, pen, and gestures |
US10940311B2 (en) * | 2013-03-29 | 2021-03-09 | Neurometrix, Inc. | Apparatus and method for button-free control of a wearable transcutaneous electrical nerve stimulator using interactive gestures and other means |
US9829992B2 (en) | 2013-05-02 | 2017-11-28 | Synaptics Incorporated | Multi-function keys providing additional functions and previews of functions |
WO2015016894A1 (en) * | 2013-07-31 | 2015-02-05 | Hewlett-Packard Development Company, L.P. | Gesture detection |
US9804712B2 (en) | 2013-08-23 | 2017-10-31 | Blackberry Limited | Contact-free interaction with an electronic device |
US20150185858A1 (en) * | 2013-12-26 | 2015-07-02 | Wes A. Nagara | System and method of plane field activation for a gesture-based control system |
US10001645B2 (en) * | 2014-01-17 | 2018-06-19 | Sony Interactive Entertainment America Llc | Using a second screen as a private tracking heads-up display |
FR3017723B1 (en) * | 2014-02-19 | 2017-07-21 | Fogale Nanotech | METHOD OF MAN-MACHINE INTERACTION BY COMBINING TOUCH-FREE AND CONTACTLESS CONTROLS |
JP6307627B2 (en) * | 2014-03-14 | 2018-04-04 | 株式会社ソニー・インタラクティブエンタテインメント | Game console with space sensing |
US9746984B2 (en) * | 2014-08-19 | 2017-08-29 | Sony Interactive Entertainment Inc. | Systems and methods for providing feedback to a user while interacting with content |
US9696549B2 (en) | 2014-12-22 | 2017-07-04 | International Business Machines Corporation | Selectively pairing an application presented in virtual space with a physical display |
CN104656893B (en) * | 2015-02-06 | 2017-10-13 | 西北工业大学 | The long-distance interactive control system and method in a kind of information physical space |
JP6596883B2 (en) | 2015-03-31 | 2019-10-30 | ソニー株式会社 | Head mounted display, head mounted display control method, and computer program |
US9939908B2 (en) * | 2015-09-28 | 2018-04-10 | Paypal, Inc. | Multi-device authentication |
EP3384367A4 (en) * | 2015-12-01 | 2019-07-31 | Quantum Interface, LLC | Motion based interface systems and apparatuses and methods for making and using same using directionally activatable attributes or attribute control objects |
WO2017096096A1 (en) * | 2015-12-01 | 2017-06-08 | Quantum Interface, Llc. | Motion based systems, apparatuses and methods for establishing 3 axis coordinate systems for mobile devices and writing with virtual keyboards |
US10452227B1 (en) | 2016-03-31 | 2019-10-22 | United Services Automobile Association (Usaa) | System and method for data visualization and modification in an immersive three dimensional (3-D) environment |
CN105975067A (en) * | 2016-04-28 | 2016-09-28 | 上海创米科技有限公司 | Key input device and method applied to virtual reality product |
EP3460633A4 (en) | 2016-06-22 | 2019-04-17 | Huawei Technologies Co., Ltd. | Head-mounted display apparatus and processing method thereof |
CN106249882B (en) | 2016-07-26 | 2022-07-12 | 华为技术有限公司 | Gesture control method and device applied to VR equipment |
WO2018047791A1 (en) * | 2016-09-09 | 2018-03-15 | パナソニックIpマネジメント株式会社 | Input device and electronic device comprising same |
TWI609316B (en) * | 2016-09-13 | 2017-12-21 | 精元電腦股份有限公司 | Devices to overlay an virtual keyboard on head mount display |
CN109952551A (en) * | 2016-09-16 | 2019-06-28 | 触觉实验室股份有限公司 | Touch sensitive keyboard |
US10817126B2 (en) * | 2016-09-20 | 2020-10-27 | Apple Inc. | 3D document editing system |
CN106527916A (en) * | 2016-09-22 | 2017-03-22 | 乐视控股(北京)有限公司 | Operating method and device based on virtual reality equipment, and operating equipment |
US10845891B2 (en) * | 2016-09-30 | 2020-11-24 | Disney Enterprises, Inc. | Displaying proximate keys in a virtual reality environment |
CN106502417A (en) * | 2016-11-08 | 2017-03-15 | 北京暴风魔镜科技有限公司 | Mobile virtual real world devices are adapted to the methods, devices and systems of computer peripheral equipment |
EP3635521A4 (en) * | 2017-06-08 | 2021-05-19 | Honeywell International Inc. | Apparatus and method for visual-assisted training, collaboration, and monitoring in augmented/virtual reality in industrial automation systems and other systems |
US10394342B2 (en) * | 2017-09-27 | 2019-08-27 | Facebook Technologies, Llc | Apparatuses, systems, and methods for representing user interactions with real-world input devices in a virtual space |
US10732774B1 (en) * | 2018-03-21 | 2020-08-04 | Rockwell Collins, Inc. | Increasing cursor control device sensitivity |
US10719173B2 (en) * | 2018-04-04 | 2020-07-21 | Facebook Technologies, Llc | Transcribing augmented reality keyboard input based on hand poses for improved typing accuracy |
JP7245631B2 (en) | 2018-10-26 | 2023-03-24 | Dynabook株式会社 | Electronics |
KR102174039B1 (en) * | 2018-12-12 | 2020-11-04 | 한국광기술원 | Apparatus and Method for Editing Files Anywhere Using Augmented Reality |
CN110275610B (en) * | 2019-05-27 | 2022-09-30 | 山东科技大学 | Cooperative gesture control coal mining simulation control method based on LeapMotion somatosensory controller |
KR102137092B1 (en) * | 2019-08-06 | 2020-07-23 | 삼성전기주식회사 | Switching operatiion sensing device implemented touch sensing and force sensing using a dual sensing structure, and electronic device |
US11009969B1 (en) | 2019-12-03 | 2021-05-18 | International Business Machines Corporation | Interactive data input |
KR102333090B1 (en) * | 2019-12-17 | 2021-12-01 | 삼성전기주식회사 | Sensing coil device, and switching operation sensing device |
TWI778347B (en) * | 2020-01-06 | 2022-09-21 | 美商矽成積體電路股份有限公司 | Proximity detection method and proximity detection keyboard |
KR20230144042A (en) | 2021-02-08 | 2023-10-13 | 사이트풀 컴퓨터스 리미티드 | Extended Reality for Productivity |
JP2024507749A (en) | 2021-02-08 | 2024-02-21 | サイトフル コンピューターズ リミテッド | Content sharing in extended reality |
EP4288950A1 (en) | 2021-02-08 | 2023-12-13 | Sightful Computers Ltd | User interactions in extended reality |
BR112023017828A2 (en) | 2021-03-03 | 2023-10-03 | Guardian Glass Llc | SYSTEMS AND/OR METHODS FOR CREATING AND DETECTING CHANGES IN ELECTRIC FIELDS |
US11757951B2 (en) | 2021-05-28 | 2023-09-12 | Vizio, Inc. | System and method for configuring video watch parties with gesture-specific telemojis |
WO2023009580A2 (en) | 2021-07-28 | 2023-02-02 | Multinarity Ltd | Using an extended reality appliance for productivity |
US20230334795A1 (en) | 2022-01-25 | 2023-10-19 | Multinarity Ltd | Dual mode presentation of user interface elements |
US20230324997A1 (en) * | 2022-04-11 | 2023-10-12 | Meta Platforms Technologies, Llc | Virtual Keyboard Selections Using Multiple Input Modalities |
US11948263B1 (en) | 2023-03-14 | 2024-04-02 | Sightful Computers Ltd | Recording the complete physical and extended reality environments of a user |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5790107A (en) * | 1995-06-07 | 1998-08-04 | Logitech, Inc. | Touch sensing method and apparatus |
US20020101546A1 (en) * | 1995-05-23 | 2002-08-01 | Colorlink, Inc. | Color filters, sequencers and displays using color selective light modulators |
US20050002158A1 (en) * | 2002-02-25 | 2005-01-06 | Robert Olodort | Mobile computer with foldable keyboard |
US20080042989A1 (en) * | 1998-01-26 | 2008-02-21 | Apple Inc. | Typing with a touch sensor |
US20080048997A1 (en) * | 1992-06-08 | 2008-02-28 | Synaptics Incorporated | Object position detector with edge motion feature and gesture recognition |
US20090009491A1 (en) * | 2007-07-04 | 2009-01-08 | Grivna Edward L | Capacitive sensing control knob |
US20090243997A1 (en) * | 2008-03-27 | 2009-10-01 | Immersion Corporation | Systems and Methods For Resonance Detection |
US20100295559A1 (en) * | 2009-05-22 | 2010-11-25 | Freescale Semiconductor, Inc. | Device with proximity detection capability |
US20110095997A1 (en) * | 2009-10-27 | 2011-04-28 | Qrg Limited | Touchscreen electrode arrangement |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4524348A (en) | 1983-09-26 | 1985-06-18 | Lefkowitz Leonard R | Control interface |
JP3484355B2 (en) | 1998-09-28 | 2004-01-06 | オムロンヘルスケア株式会社 | Biological detector |
JPH11194882A (en) | 1998-01-06 | 1999-07-21 | Poseidon Technical Systems:Kk | Keyboard and input device |
US7821425B2 (en) | 2002-07-12 | 2010-10-26 | Atmel Corporation | Capacitive keyboard with non-locking reduced keying ambiguity |
WO2004093005A1 (en) * | 2003-04-15 | 2004-10-28 | Fujitsu Limited | Information processor |
WO2006133593A1 (en) | 2005-04-30 | 2006-12-21 | Apexone Microelectronics Inc. | Capacitance touchpad data input device and keyboard |
JP4631676B2 (en) | 2005-11-30 | 2011-02-16 | トヨタ自動車株式会社 | Two-dimensional coordinate detection device |
US7903092B2 (en) | 2006-05-25 | 2011-03-08 | Atmel Corporation | Capacitive keyboard with position dependent reduced keying ambiguity |
US8786554B2 (en) | 2006-07-10 | 2014-07-22 | Atmel Corporation | Priority and combination suppression techniques (PST/CST) for a capacitive keyboard |
JP4766340B2 (en) * | 2006-10-13 | 2011-09-07 | ソニー株式会社 | Proximity detection type information display device and information display method using the same |
US8219936B2 (en) * | 2007-08-30 | 2012-07-10 | Lg Electronics Inc. | User interface for a mobile device using a user's gesture in the proximity of an electronic device |
US8179375B2 (en) * | 2008-01-04 | 2012-05-15 | Tactus Technology | User interface system and method |
US20090251422A1 (en) | 2008-04-08 | 2009-10-08 | Honeywell International Inc. | Method and system for enhancing interaction of a virtual keyboard provided through a small touch screen |
TW200949632A (en) | 2008-05-22 | 2009-12-01 | Prospect Technology Corp | A trigger-threshold adaptable and auto-adjustable capacitive induction-type keying device |
TWI392876B (en) | 2008-08-08 | 2013-04-11 | Sony Corp | Capacitance change measuring circuit of capacitive sensor device, capacitive sensor module, method of measuring capacitance change of capacitive sensor device, and electronic device |
KR20100031204A (en) * | 2008-09-12 | 2010-03-22 | 삼성전자주식회사 | Input device based on a proximity sensor and operation method using the same |
JP4775669B2 (en) | 2008-10-10 | 2011-09-21 | ソニー株式会社 | Information processing apparatus, information processing method, information processing system, and information processing program |
US8497786B2 (en) | 2009-03-26 | 2013-07-30 | Freescale Semiconductor, Inc. | Capacitive keyboard with enhanced electrode areas |
US20100253629A1 (en) | 2009-04-03 | 2010-10-07 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Combined Mutual Capacitance and Switch-Actuated Keyboard for Enhanced Texting in an Electronic Device |
FR2949007B1 (en) * | 2009-08-07 | 2012-06-08 | Nanotec Solution | DEVICE AND METHOD FOR CONTROL INTERFACE SENSITIVE TO A MOVEMENT OF A BODY OR OBJECT AND CONTROL EQUIPMENT INCORPORATING THIS DEVICE. |
US8816964B2 (en) * | 2010-11-26 | 2014-08-26 | Mckesson Financial Holdings | Sensor-augmented, gesture-enabled keyboard and associated apparatus and computer-readable storage medium |
-
2011
- 2011-06-21 CN CN201180071787.5A patent/CN103635869B/en not_active Expired - Fee Related
- 2011-06-21 JP JP2014508330A patent/JP2014515147A/en active Pending
- 2011-06-21 KR KR1020137022939A patent/KR101516513B1/en not_active IP Right Cessation
- 2011-06-21 WO PCT/US2011/041173 patent/WO2012177237A1/en active Application Filing
- 2011-06-21 US US13/320,904 patent/US9547438B2/en not_active Expired - Fee Related
-
2016
- 2016-11-28 US US15/361,638 patent/US9823752B2/en not_active Expired - Fee Related
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080048997A1 (en) * | 1992-06-08 | 2008-02-28 | Synaptics Incorporated | Object position detector with edge motion feature and gesture recognition |
US20020101546A1 (en) * | 1995-05-23 | 2002-08-01 | Colorlink, Inc. | Color filters, sequencers and displays using color selective light modulators |
US5790107A (en) * | 1995-06-07 | 1998-08-04 | Logitech, Inc. | Touch sensing method and apparatus |
US20080042989A1 (en) * | 1998-01-26 | 2008-02-21 | Apple Inc. | Typing with a touch sensor |
US20050002158A1 (en) * | 2002-02-25 | 2005-01-06 | Robert Olodort | Mobile computer with foldable keyboard |
US20090009491A1 (en) * | 2007-07-04 | 2009-01-08 | Grivna Edward L | Capacitive sensing control knob |
US20090243997A1 (en) * | 2008-03-27 | 2009-10-01 | Immersion Corporation | Systems and Methods For Resonance Detection |
US20100295559A1 (en) * | 2009-05-22 | 2010-11-25 | Freescale Semiconductor, Inc. | Device with proximity detection capability |
US20110095997A1 (en) * | 2009-10-27 | 2011-04-28 | Qrg Limited | Touchscreen electrode arrangement |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10474287B2 (en) | 2007-01-03 | 2019-11-12 | Apple Inc. | Double-sided touch-sensitive panel with shield and drive combined layer |
US11112904B2 (en) | 2007-01-03 | 2021-09-07 | Apple Inc. | Double-sided touch-sensitive panel with shield and drive combined layer |
JP2016519819A (en) * | 2013-04-02 | 2016-07-07 | クイックステップ テクノロジーズ リミテッド ライアビリティ カンパニー | A device that interacts with an electronic device and / or computer device in a non-contact manner, and a device including the device |
US10222913B2 (en) | 2013-04-02 | 2019-03-05 | Quickstep Technologies Llc | Device for contactless interaction with an electronic and/or computer apparatus, and apparatus equipped with such a device |
CN108519855A (en) * | 2018-04-17 | 2018-09-11 | 北京小米移动软件有限公司 | Characters input method and device |
Also Published As
Publication number | Publication date |
---|---|
KR20130129271A (en) | 2013-11-27 |
JP2014515147A (en) | 2014-06-26 |
CN103635869B (en) | 2017-02-15 |
US9823752B2 (en) | 2017-11-21 |
KR101516513B1 (en) | 2015-05-04 |
US20120326961A1 (en) | 2012-12-27 |
CN103635869A (en) | 2014-03-12 |
US20170075429A1 (en) | 2017-03-16 |
US9547438B2 (en) | 2017-01-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9823752B2 (en) | Gesture based user interface for augmented reality | |
US11720176B2 (en) | Device having integrated interface system | |
US11029843B2 (en) | Touch sensitive keyboard | |
EP2717120B1 (en) | Apparatus, methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications | |
EP2235638B1 (en) | Hand-held device with touchscreen and digital tactile pixels and operating method therefor | |
US8907897B2 (en) | Optical capacitive thumb control with pressure sensor | |
EP2433203B1 (en) | Hand-held device with two-finger touch triggered selection and transformation of active elements | |
US20230359278A1 (en) | Tactile Feedback |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 13320904 Country of ref document: US |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11868049 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20137022939 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2014508330 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 11868049 Country of ref document: EP Kind code of ref document: A1 |