WO2009130573A1

WO2009130573A1 - Method and device for determining gestures made by a user with his hand

Info

Publication number: WO2009130573A1
Application number: PCT/IB2009/005323
Authority: WO
Inventors: Sergei Startchik
Original assignee: Sergei Startchik
Priority date: 2008-04-22
Filing date: 2009-04-22
Publication date: 2009-10-29

Abstract

Method for determining gestures made by a user with his palm and/or with his fingers comprising: measuring electric signals generated by a neural activity at a Plurality of locations at a wrist of the user; processing the measured electric signals in order to determine which nerve is activated and to which extent; determining a gesture made by the user on the basis of the determined neural activity; and wrist device (1) for determining gestures made by a user with his palm and/or with his fingers comprising: a bracelet for securing the wrist device (1) around a wrist of the user; a plurality of electrodes (2) and/or electromagnetic coils (6) placed on and/or within the bracelet for measuring electric signals generated by a neural activity within the wrist.

Description

Method and Device for Determining Gestures Made by a User with His Hand

Field of the invention

The p resent invention relates to a method and a device for determining gestures mi de by a user with his hand. The present invention relates in particular tc a method and a device for determining hand gestures by measuring n ϊural activity in the wrist.

Background of the invention Measμrement of neural activity was mainly done within medical monitoring amework and often several other parameters are measured at the same time, In US20070276213, a device including a glove with electrodes measures h ;art activity. In EP 1815784 EMGs are measured, processed and wirelessly tr insmitted to a stand-alone processing device. Measurement points are located all over the human body. In WO2004090661 , US2001044588, US2007019 881 and GB2442330, portable systems for processing various physiologies parameters are described. A specially designed headset in

US2007093 06 allows a better measurement of EEG signals.

In W D2007075703, a system to recognize continuous 3D motion of a hand and handwriting in particular is suggested. Electrodes combined with a glove are p aced over the palm to measure muscle activity. Processing of the signals is dc ne on a computer.

In US2007276280, US2007270918 and EP 1888598, EMG measureme its are done to be transferred to a computer for further analysis by a doctor.

Processing neural signals to obtain clearer information about neural activity is d ϊscribed in WO2007021645 where various filtering techniques are applied to c ne measured signal. In WO2007076566, a density vector field for

EEG is computed in a special way. An algorithm to integrate and combine measurements done at various points is described in WO2007140536. A method for c earer separation of signals is presented in WO2008019407.

Operε ting devices using hand gestures that are captured electronically was done b ' different methods. In US2008084385, a device fixed to a palm uses gyrosα pes to detect hand gestures. In US2008046425, a tactile screen is required. Se feral methods based on optical or video capture of gestures were implemented like in US2008040692. In US2008009310 and US20080014917, gestures witf a mobile phone in the hand are detected when the camera of the mobile phon ; is viewing an image containing hidden information. Performing gestures tha produce an electric contact is described in US 7,012,097, where two fingers n sed to be touched to activate a device. The whole hand is covered with a glove that includes electrodes. Operation of devices using neural activity is describe J in WO2008025048, but no gestures are involved. In WO2007149 >53, a general aptitude to execute different tasks by an individual is estimated bajsed on global measurement of neural activity.

Summary of the invention

An ob ective of the present invention is to provide a method and a device for reliably id sntifying gestures executed by the hand of a user.

Anoth ;r aim of the present invention is to provide a device for identifying gestures exe Duted by the hand of a user, which can be worn by the user during extended peqiods of time without affecting its mobility.

These; aims are achieved by a method and a device comprising the features of thje corresponding independent claim.

These aims are achieved in particular by a method for determining gestures ma _e by a user with his palm and/or with his fingers comprising: measuring € ectric signals generated by a neural activity at a plurality of locations at a wrist of the user; processing the measured electric signals in order to dete mnine which nerve is activated and to which extent; determining a gesture mad< by the user on the basis of the determined neural activity. These aims are also achieved in particular by a wrist device for determining gestures made by a user with his palm and/or with his fingers comprising: a bracelet for securing the wrist device around a wrist of the user; a plurality of e ectrodes and/or electromagnetic coils placed on and/or within the bracelet for measuring electric signals generated by a neural activity within the wrist.

Accor ling to an embodiment of the invention, the electric signals comprise ele ctric potentials and/or electric currents measured by a plurality of electrodes in| contact with the skin surface of said wrist.

In a V ariant embodiment of the invention, the electric signals comprise electromagn ϊtic fields measured by a plurality of electromagnetic coils placed in close vicinity to said wrist. The electromagnetic fields are preferably measured at numerous points around the wrist so as to disambiguate signal capturing from nerves.

The determination of the gesture made by the user is preferably performed b ! using information about the position of the measurement points around the wrist in order to allow signal processing algorithms obtaining disambiguatφd conclusion about which nerve is active and to which extent.

Prefei aoly, an alphabet of gestures is determined, each gesture correspondir g for example to a specific command for a specific electronic device. The letermination of gestures is then preferably calibrated with the help of one or mo e gestures of the alphabet.

According to the invention, an electronic device is operated with gestures made by a i ser with his palm and/or with his fingers by determining a gesture made by th s user with his palm and/or with his fingers with the method described al ove and generating and sending to the electronic device at least one commar d signal corresponding to the determined hand gesture.

The i frist device of the invention preferably comprises a control unit electrically connected to the electrodes and/or electromagnetic coils for collecting a id interpreting the measured signals, where the control unit comprises f >r example a controller for generating one or more command signals base i on a determined gesture, and an interface for sending the one or more comm nd signal to an external electronic device.

The r ϊsult is thus a hand gesture operated device comprising a wrist device acc< rding to the invention and an electronic device electrically connected t the wrist device, wherein the electronic device is operated by command si jnals generated by the wrist device, the commands corresponding to gestures letermined by the wrist device. The electronic device operated by the commar ds generated by the wrist device of the invention is for example a watch, a me bile phone, a smartphone, a PDA, a wireless virtual keyboard for connection t a a computer, a remote control, or a voice generation device that pronounces iand signs executed by the user.

The device of the invention measures the activity of the wrist nerves during exec Jtion of various gestures of the palm and fingers of the user and reliably anc rapidly identifies the gestures being executed. Automatically identified ge stures are then for example used to operate one or more various devices cor nected or combined with the device of the invention, like for example a v atch, a mobile phone, a PDA, a remote control and/or a keyboard, These devic ;s operated by gestures are operated with high flexibility since they are operatei by using only one hand and without depending on the buttons' size. Device s do not need to be taken, held and touched during operation thus offering mot lity. Free hands offer security as well.

Brief description of the drawings

The invention will be better understood by reading the description below, illustrated by the figures, where:

Fig. 1 shows an embodiment of the device of the invention for use with a mobile phor e;

Fig. : is a view of a transversal cut of a human wrist with the position of the nerves; Fig. : illustrates the construction of the device of the invention according to a preferre d embodiment;

Fig. illustrates the construction of the device of the invention according to a preferre d embodiment;

Fig. J shows the form of signals during nerve activation;

Fig. ( » shows example of gestures corresponding to different possible commands;

Fig- 7 illustrates an example of classification of distinguishable and possible har id gestures;

Fig. £ illustrates an algorithm for identifying hand gestures; Fig. S is a scheme of a possible realization of the device of the invention.

Detailed description of the invention

The activity of nerves controlling human fingers and palm produces electric curi snts and electromagnetic fields that despite their weakness and according i the invention are measured in a non-invasive way. The human palm is con oiled by two sets of nerves as shown on Fig. 2. The median nerve 20 controls the first four fingers and the ulnar nerve 21 controls the fifth and some muse ΪS of the fourth finger. On the transversal cut of Fig. 2 it is illustrated that the me< ian nerve 20 is located in the center of the wrist and the ulnar nerve 21 is locate I on the periphery of the wrist section. When these nerves activate muscles of the palm and fingers, they produce electric currents and electromagnetic fit Ids. According to the invention, the neural activity in the wrist is determined >y measuring these electric currents and/or electromagnetic fields.

The evice of the invention is thus preferably a wrist device measuring the electric field produced by the neural activity in the wrist of a user. The device corn irises a base unit which includes for example a wrist-watch like housing, a nodule for generating power disposed within the housing and a strap for s jcuring the housing to the wrist. The device of the invention furthermore comprises electrodes in close contact with the skin for capturing direct elect icai signals available on the skin surface and/or electromagnetic sensors like for example coils with central measuring element for capturing weak magnetic field emitted by the nerves and thus don't need to have a conductive < iontact with the skin. According to the invention, the hand gestures are determined on the basis of a combination of electric and/or magnetic signals from numen >us measurement points.

Acco d ing to an embodiment illustrated in Fig. 3, the device of the invention of erates with electric currents. The device 1 comprises electrodes 2 measuring < slectric potentials in contact with a user's skin. The wrist device 1 comprises bracelet with a housing made of non-conductive material, such as plastic for e ample. Numerous electrodes 2 are in-built into the housing on its inside face, i.e. the face in contact with the wrist. The contact between the skin and the eleqtrodes 2 is increased when the wrist device 1 is secured to the wrist of a user.

The ej lectrical signals measured by each electrode 2 are conducted to a control unit 9 through electrically conductive wires 5. The control unit 9 for example cόi nprises a power supply, an AD converter, a DSP₁ RAM memory and/or a flash memory.

The number and placement of the electrodes 2 on the inner face of the bracelet is regulated by three criteria. First, their density is preferably high in order to ma <e redundant measurements and account for loose contacts of a certain num er of them with the skin.

Seco d, they are preferably placed within the housing of the device 1 in such a way Io reduce cross-talk between wires 5 and electrodes 2. Technique similar to lo\ f voltage differential is used to obtain a clearer measurement of a signal.

Third, the type of electrodes 2 preferably varies to better capture differences i i the sensed current and locate its origin. Measurements are done in a synchπ nous manner so that the distances covered by the signals are reflected in the impulses measured and allow locating which nerve was activated t etween the set of median nerve 20. The disambiguation of measureme nts of impulses is done in a similar way to deconvolution. In order to obtain as π uch information as possible about electric currents going in nerves and disamb-iguate the impulses coming from different nerves.

An example of neural activity measured at one point is shown in Fig. 5

housing is ( one such that their positions belong to planes that have the axis of the wrist, a id thus the axis of the median nerves 20, in common. Coils radial distance, ar d thus the distance from the coils to the inner surface of the wrist device 1 wl ich is in contact with the user's skin, preferably varies, as shown in the upper j art of Fig. 4. The reason behind such placement is to sample the magnetic fi Id at different distances from the nerve that emits it. Variations of distances e re preferably similar to those between the nerves in the median nerve 20 in order to capture the same resolution of differences in magnetic field that than of ocation of nerves.

To sSample magnetic field, the coils 6 preferably each have a central element the t measures the strength of the field. This element is connected to the control jnit 3 that collects and processes information. To avoid interference between rr easurements and communications, they are preferably done at different oments in time (one cycle for measurement, one cycle for communication). In a variant embodiment, the coils 6 have different sizes allowing sa npling at different resolutions.

An eftnbodiment of the circuitry of the wrist device 1 is illustrated in Fig. 9 by way of example for the case where electromagnetic fields are measured.

The measu ement points are each equipped with a coil 6, a measuring element 7 and a wi e 5 connected to the control unit 3. The control unit 3 contains an analog-to-c igital converter and a DSP for processing the signals. The program is stored n a non-volatile memory and RAM memory is available for computatio is. The control unit 3 is for example used to synchronize the measurements. An < mbodiment of the general algorithm for determining the gestures is illustrated i i Fig. 8. The network of measurement points is reset and scanned, The measi red values are converted to digital form for further processing. The values are weighted with respect to the type of coils and their positions. The obtained v< lues are processed by applying models of magnetic fields in order to infer most •robable positions of field sources. Redundant conclusions along the wrist axis illow confirming the presence of magnetic field sources inside the wrist. The Ic cation and strength of each source is registered. The pattern of the magnetic field sources and areas that do not produce magnetic fields are aligned with a model of the wrist section. Sources positions aligned with nerves in the medic n nerve allow concluding about the activity of particular nerves and their strengt l. The combination of these pieces of information allows obtaining a vector of eε ch nerve's activity. This vector is then translated to the gestures' alphabet wi h sufficient discriminative power. The identified gesture from the alphabet is ( ommunicated to the device for operation.

The i osition of the ulnar nerve 21 and of the median nerve 20 in the hand differs from one person to another. To account for this position, a calibration o the device is necessary. In this calibration step, a user wearing the wrist device 1 follows a procedure where he executes gestures suggested by the device 1. At each executed gesture, the device 1 registers a set of measurements corresponding to basic gestures and stores the signatures of each nerve's behaviour for subsequent detection.

Analysis of neural activity measured at numerous points and based on that informa ion robustly inferring which gesture is being executed.

The circuitry of the wrist device 1 and the general algorithm for determining the gestures are described above in the case of a wrist device 1 measuring t ie magnetic fields emitted by the nerves' activity. Similar circuitry and algorith n however apply to the wrist device 1 of the invention measuring the electric c urrents.

Furth irmore, in a variant embodiment the wrist device 1 of the invention comprises b )th electrodes and coils and the determination of the hand gesture is made on t ie basis of the measurement of both electric currents and magnetic fields produc ed by the nerves' activity.

The measurement of the neural activity with the wrist device 1 and deducing th hand gestures from that activity is of direct benefit for example for operating va ϊous electronic equipments. Gestures where fixed hand position is maintained o mark the gesture or sequence of such positions marked by a brief stop is of pε rticular interest.

Selet ting appropriate gestures is critical to the acceptance of the technology by people and several criteria are important. First, the gestures should be d sychologically significant, easy to memorize and coherent with the task they co rrespond to. Second, the gestures should be easy to execute with one hand b! / most of people and there should not be ambiguity between two gestures as to how they are executed by two persons. In other words, gestures should be distinguishable enough. Identifiable and physically possible and comfortable gestures are suggested for use with the device of the invention.

The nuscles controlled by the wrist nerves move the proximal phalanx 16, the inte mediary phalanx 17 and the distal phalanx 18 as shown in Fig. 7.

Only a limit* id number of positions of the fingers can be easily distinguished by humans. Th sse are schematically represented in the lower part of Fig. 7.

The j roximal phalanx 16 is easily placed by a human in roughly three positions: A - u wards from the plane of the palm, B a gned with the plane of the palm and C - p inting downwards from the plane of the palm.

The intermediary phalanx 17 that is shown with dotted line in Fig. 7 can also have three positions:

A - aligned with the previous proximal phalanx 16, B - at 45 degrees with respect to the previous proximal phalanx 16 and

C - a approximately 120 degrees with respect to the previous proximal phalanx 16 or pushed inwards at its maximum).

The I; ist distal phalanx 18 can only have two distinguishable positions:

A - al gned with the previous intermediary phalanx 17 and B - at approximately 45 degrees with respect to the previous phalanx 17

(or contract* d). All fit ger positions defined with the above phalanx positions are listed in the table be ow and are indicated by codes of type "CBB".

Types Positions

Usable AAA, ABB, BAA, BBB, BCB, CAA, CCB

Difficult AAB, ABA, BAB, BBA, BCA, CAB, CBA, CCA

Relaxed poj ition CBB

Table 1 : Finder positions codes for index finger

It shi mid be noted that position CBB is the one corresponding to a relaxed pos ion of the hand and thus do not correspond to any neural activity in the wrist. P.< sitions AAB₁ ABA, BAB, BBA, BCA, CAB, CBA, CCA are actually difficult to e ecute with one hand and thus have limited usage. The remaining positions m< rked as "usable" in Table 1 can be applied for operating devices.

Usab e positions are not applied blindly to all the fingers and some constraints xist. The thumb can move independently of the other fingers, but constraints nainly exist between index, middle, oricular and small finger. In Table 2, some extreme position combinations between index and middle fingers are marked is difficult to execute.

This mits the number of gestures that can be used by a system made of a wrist devii e according to the invention and another device that is operated by the signals received from the wrist device. The total number of usable combinatior s is slightly higher than one hundred, which is more than a necessary phabet of gestures that can be retained by a person.

Acco ding to an embodiment of the invention, the outer face of the wrist device iinco porates another device that can be of various types, such as for example a ψobile phone, and/or a PDA, etc. According to this embodiment, the incorporatec device is partly or entirely operated by command signals emitted by the wrist device of the invention, each command signal corresponding to a particular ha nd gesture detected as explained above.

The i corporated device is for example a watch. In this particular case, the wrist de vice of the invention is in-built into the watch bracelet belt. The electrodes nd/or coils are built into the non-conductive bracelet or electrically insulated fn ( m the metal bracelet, for example by plastic rings. The power supply of th wrist device is for example coupled with that of the watch, which can be one of existing mechanical components for self-winding or a battery being rechai jed during motion.

Seve al simple operations requested by gestures made with one hand can be very iseful for watch operation. One of simple operations is for example the activatioh of a stopwatch by using the gesture 12 shown in Fig. 6 and correspondir g to the natural gesture of stopwatch activation.

In an ther example, the wrist device of the invention is combined with a mobile phon i designed as a wrist device. Several functions of mobile phones require mult pie steps and imply the use of two hands, which is often not practical in oday's moving environment. Several functionalities of a mobile phone becon ie more accessible if operated by the gesture of one hand, without having to ke eepp the telephone in the hand. Examples include the gesture of answering a ;all 11 and the gesture of checking the mailbox 13 as shown in Fig. 6. To avoid unintentionally executing operations on such device, a special gesture tna is not common to execute can be required to "unlock" the device and enter in o gesture-command mode.

Com >uter keyboards often take too much space, especially in the case of laptops md SMS messaging devices. In still another example, two wrist devices of t ie invention are worn by a user, one on each hand, and function as replacemen for a keyboard. An independent power supply and wireless communica ion module send information to a computer about the gestures of typing 15 as shown in Fig. 8.

In another example, the wrist device of the invention is combined with a remote con rol. An additional infrared module beams to a television commands correspondi ig to detected hand gestures for switching to next/previous channel, setting high jr/lower volume, muting, etc.

In a ast example, the wrist device of the invention is combined with a translator fr >m hand signs language. According to this example, the wrist device of the inver tion, worn by a user, recognizes the hand signs made by the user and sends corresponding command signals to the translator. The translator then for ex. mple synthesizes the words corresponding to the hand signs, which are sent tc an integrated speaker. Such application of the device of the invention th JS allows for a much better integration of the user into the society. Indeed, pe< pie that don't understand hand signs can still understand the user because th( words expressed by his hands are spoken by the translator.

Othe examples of implementation of the wrist device of the invention are possible. In particular, the wrist device of the invention can be used to operate any other h andheld device with an appropriate gesture alphabet corresponding to commani Is for the handheld device.

According to the method of invention, fixed positions are detected to generate e ectric signals emitted by nerves in the wrist to maintain fixed positions of palm and fingers. Those signals are measured and interpreted to generate the commands to the equipments. Acco ding to the description above, the wrist device of the invention determines the hand gestures, in particular the finger movements, made by a user by me< suring the neural activity in the user's wrist, using electrodes and/or magnetic α ils placed around the wrist. The wrist device of the invention can however cc mprise further elements for detecting the hand gestures and/or movements such as for example accelerometers, gyroscopes, etc., thus allowing an extension of the signs and/or gestures alphabet that can be used with the dev ce of the invention.

Claims

1. Method or determining gestures made by a user with his palm and/or with his finge s comprising: measuring electric signals generated by a neural activity at a plurality )f locations at a wrist of the user; processing the measured electric signals in order to determine which ne rve is activated and to which extent; determining a gesture made by the user on the basis of the determrrfed neural activity.

2. Method Recording to the preceding claim, wherein said electric signals comprise electric potentials and/or electric currents measured by a plurality of ^: electπ >des in contact with the skin surface of said wrist.

3. Method ccording to one of the preceding claims, wherein said electric signals c omprise electromagnetic fields measured by a plurality of electrom agnetic coils placed in close vicinity to said wrist.

4. Method according to the preceding claim, wherein said electromagnetic fields are measured at numerous points around said wrist so as to disambk uate signal capturing from nerves.

5. Method i iccording to one of the preceding claims, wherein determining a gesture i nade by the user is performed by using information about the position )f measurement points around said wrist and signal processing algorithn s allowing to obtain disambiguated conclusion about which nerve is active ar d to which extent.

6. Method « iccording to one of the preceding claims, comprising the step of determin ng an alphabet of gestures.

7. Method jccording to the preceding claim, comprising the step of calibrating the dete mination of gestures with the help of one or more gestures of said alphabe

electrom agnetic coils (6) for collecting and interpreting said measured signals.

11. Wrist de rice (1) according to the preceding claim, wherein said control unit (9, 3) co nprises a controller for generating one or more command signals based on a determined gesture, and an interface for sending said one or more coi nmand signal to an external electronic device.

12. A hand ( esture operated device comprising: a wrist device (1) according to one of claims 9 to 11 ; an electronic device electrically connected to said wrist device

(1). wherein said electronic device is operated by command signals generated by said wrist device (1), said command signals corresponding to gestures] determined by said wrist device (1).

13. Hand gφture operated device according to the preceding claim, wherein said ele tronic device is a watch.

14. Hand gejsture operated device according to claim 12, wherein said electronic device is a mobile phone and/or a smartphone and/or a PDA.

15. Hand ge sture operated device according to claim 12, wherein said electronic device is a wireless virtual keyboard for connection to a computer.

16. Hand ge sture operated device according to claim 12, wherein said electronic device is a remote control.

17. Hand gefeture operated device according to claim 12, wherein said electronic device is a voice generation device that pronounces hand signs executed by said use