EP2490459A1

EP2490459A1 - Method for voice signal blending

Info

Publication number: EP2490459A1
Application number: EP11155021A
Authority: EP
Inventors: Bernd Iser; Arthur Wolf; Patrick Hannon; Mohamed Krini
Original assignee: SVOX AG
Current assignee: SVOX AG
Priority date: 2011-02-18
Filing date: 2011-02-18
Publication date: 2012-08-22
Anticipated expiration: 2031-02-18
Also published as: EP2490459B1

Abstract

In a method for voice signal blending in an indoor communication system, particularly in a vehicle, or in a hands-free telephony system or an automatic speech recognition system, there are at least two microphones, the voice signals of which should be blended to be delivered to at least one loudspeaker. Moreover, feedback suppression or compensation is provided. This feedback suppression or compensation is effected before blending the signals of the at least two microphones.

Description

Field of the invention

The invention refers to a method and system installed in a car for the communication of people sitting in remote locations. Therefore at least one loudspeaker is installed and at least two microphones, one assigned to each person. Alternatively, the invention relates to method applied to a hands-free telephony system or to an automatic speech recognition system, where conditions and requirements are quite similar. More specifically, the invention relates to a method according to the introductory clause of claim 1 as well as to a software product according to claim 8 and a system according to claim 9.

Background of the invention

EP1850640B1 shows and describes a typical setup for a basic intercom system with two microphones and one loudspeaker. The mixer in this setup decides depending on a criterion which of the microphone signals shall be switched to the next module or the output. This criterion can depend on the detected occupation of a seat. However, it has been also proposed in US006549629B2 such decision criterion can depend on the Signal plus Noise to Noise Ratio (SNNR).
The fundamental disadvantage of such a system is that the feedback is still contained in the signal and therefore SNNR is not a good approach to a Signal to Noise Ratio (SNR). Using SNNR as SNR causes a misinterpretation of the signal power when estimating the SNNR for each microphone. The result of such a misinterpretation is the preferred usage of the microphone containing the highest amount of feedback.
A further disadvantage of these systems is a possible change in background-noise level when switching from one speaking person to another (e.g. driver speaks to a passenger in the back while having his window open, and then the co-driver speaks to the passenger in the back having his window closed and driver as well as co-driver are each equipped with a microphone). This change in background-noise level of the signal played back over the loudspeaker might be experienced as unpleasant by a listener.
Another disadvantage that occurs is, when the play-back level between two speaking persons differs. This can occur if for example the driver is quite tall and therefore the distance to the microphone is little, which means significant speech level in the microphone signal. If for example the co-driver is a small person and therefore the distance to the microphone is large, which means low speech level within the microphone signal this basic setup results in a playback signal for the loudspeaker that differs in speech level for the driver and co-driver in an unpleasant manner concerning the listener impression.
US-A-2005/0265560 discloses a system, where a beamformer is used for microphone signal blending. The, thus, blended output signal is then subjected to feedback suppression. This has the disadvantage that the power of the signals to be blended is relative high and, moreover, the output signal of the beamformer is adulterated by the feedback component of the signal. The suppression, however, is merely the suppression of frequencies, which are just developing resonance oscillations, which means that feedback components remain, and only resonance oscillations are prevented.

Summary of the invention

It is an object of the present invention to improve the quality of the blended signal and to find a method that is more robust in relation to feedback components.
This object is achieved by the measure according to the characterizing clause of claim 1. The feedback suppression or compensation is effected before blending the signals of the at least two microphones. If in this connection the term "feedback suppression or compensation" is used, it is in any case a minimization of feedback components of a signal. A feedback compensation is for example described in EP 1 679 874 B1 . Feedback suppression can be applied by filtering, for example with a notch-filter. Another known compensation method uses frequency shifting. These suppression methods reduce the development of feedback, but existing feedback components remain.
The method for voice signal blending is applied in a communication system, such as an indoor communication system, particularly in a vehicle, or in a hands-free telephony system or an automatic speech recognition system, comprising at least two microphones and at least one loudspeaker. The microphone signals are blended with respective weights to be delivered to the at least one loudspeaker. The feedback suppression or compensation is effected before blending the signals of the at least two microphones. This improves the quality of the blended signal and is more robust in relation to feedback components.
In preferred embodiments the feedback components are minimized by estimating the feedback signal or its energy level, preferably its power spectral density, and applying a Wiener filter which eliminates the estimated feedback signal from the signals to be blended.
According to a preferred embodiment, for at least a sub-band of the microphone signals the energies of the microphone signals are determined and at least the energies of the noise components are estimated, wherein for blending, a higher weight is given to at least the sub-band with the highest ratio of signal energy to noise energy. Sub-dividing the entire frequency band into sub-bands enables it to give individual sub-bands a different weight. Blending will be made for each sub-band with the respective weights.
It is favourable, if a signal level adjustment of the at least two microphone signals to a predetermined value is carried out particularly immediately before or immediately after blending, wherein the adjustment is depending on the energies of the noise components and minimizes the perceivable difference in the noise level when blending from one microphone to the other. In this way changes of noise levels are better avoided.
There can be a perceivable difference in the speech level when blending from one microphone to the other. To solve this problem, the signal levels of the microphone signals are adjusted to a predetermined value so that there is no perceivable difference in the speech level when blending from one microphone to the other.
The noise level of at least two microphones can be unpleasantly different. An adjustment can be achieved, if the characteristics of claim 5 are fulfilled.
The microphone signals comprise different components, each with a certain level. If a person close to a microphone is speaking, then the signal of this microphone comprises this voice with a corresponding voice level. In a preferred embodiment the voice levels of the at least two microphone signals are adjusted substantially to the same level by adjusting the microphone signals. Then the difference of the noise levels of the at least two adjusted microphone signals is determined. The noise suppressions in the microphone signals are controlled by adapting the parameters of the noise suppression characteristic in such a way, that the noise suppressed signals have substantially the same level of residual noise and the same level of speech signal in it for each microphone signal.
In a preferred embodiment the respective speaking and/or listening, non-speaking party is detected by one of the following measures:

a) by analyzing the signal of the at least two microphones during blending;
b) by sensing by means of vehicle sensors, e.g. for the seat occupancy.

In further preferred embodiments at least one of the following measures is taken:

a) the signal weight of the microphone signal where no speaking party is detected is reduced, when blending;
b) predetermined amplifier characteristics of the gain control, e.g. noise dependent gain control, for listeners and/or speakers are provided and that amplifier characteristic is chosen, which corresponds to the position of the respective listener and/or speaker;
c) equalizing is effected by means of an equalizer after voice signal blending and that the settings of the equalizer are chosen depending on the detection so as to enhance the perceived quality of the output for the listening and/or speaking party.

Brief description of the drawings

Further details and advantages will become apparent from the following description of embodiments with reference to the drawings, in which

Fig. 1: shows a block diagram of a typical setup for a basic intercom system with two microphones and one loudspeaker;
Fig. 2a: is a block diagram of a first embodiment according to the invention, while
Fig. 2b: shows a second embodiment according to the invention, and
Fig. 3: depicts a third embodiment according to the invention;
Fig. 4a: illustrates a circuit for voice blending according to a first structure according to the invention;
Fig. 4b: is a circuit for voice blending according to a second structure according to the invention; and
Fig. 4c: shows a circuit for voice blending according to a third structure according to the invention;
Fig. 5: illustrates setting an equalizer after voice blending; and
Fig. 6: depicts a sample of a characteristic used for chosing a gain factor depending on a noise estimate..

Detailed description of the invention

Fig. 1 shows a typical setup for a basic intercom system with two microphones 1 and 2 and one loudspeaker 3. Between a pre-processing stage 4, directly connected to the microphones 1, 2 and a post-processing stage 6, leading to the loudspeaker 3 is a voice mixer 5 which receives the pre-processed signals of the microphones 1, 2 and decides depending on a criterion which of the microphone signals shall be switched to its output and to the next module 6.
In Fig. 2a, one embodiment of the invention is illustrated, where the signals of the microphones 1, 2, after an optional pre-processing stage 4a, are subjected to feedback suppression or compensation in a stage 7. Thus, the first component after some optional pre-processing is a feedback suppression or feedback compensation at 7. This module 7 suppresses or compensates for the portion of the loudspeaker signal that is coupling back into the microphone and therefore being an undesired signal component.
Before reaching the voice blender 5a (Fig. 2a) or after the blender 5a (Fig. 2b), noise suppression in a noise suppression module 8 can be effected. This module 8 suppresses the background noise components of the microphone signals resulting from the background noise being present in the car cabin and being picked up by the microphones. A practical embodiment with a noise suppression postponed to the voice blender 5a is shown in Fig. 4b.
The signal m_i of microphone i is made up of the signal from the speaker in the cabin s_i, the feedback of the system from the loudspeaker into the microphone f_i and the background noise of the cabin recorded by the microphone b_i.
This is suitably depicted by the following $m_{i} (k μ) = s_{i} (k μ) + f_{i} (k μ) + b_{i} (k μ)$
Where i corresponds to the microphone index, k to the time interval and µ to the frequency band.
In the following the noise suppression is located before the voice blender 5a which is not necessarily required (without limitation of generality, see e.g. Fig. 4b).
In any case (vide Figs. 2a, 2b), the blended and noise suppressed signal goes suitably to a NDGC or noise dependent gain control 9.

Voice decision

The voice blender module blends the at least two feedback suppressed or compensated and noise suppressed signals according to Fig. 2a together to one output signal following below described criterion. According to Fig. 2b the blended signals are only feedback suppressed or compensated.
The blending is made by giving weights to the microphones. The blending or weighting criterion can either be evaluated in a non-frequency selective manner or in a frequency selective manner resulting in a non-frequency selective weighting or in a frequency selective weighting of the microphone signals ^mi . The criterion used in this invention for the weighting according to Fig. 4a, 4b, 4c is a function of the ratio of the energy of every microphone signal S_mimi subtracted by the feedback ƒ_i (respectively S_ƒiƒi for the energy of the feedback component at every microphone) due to the coupling of the loudspeaker signal back into the microphone signal and further subtracted by the noise b_i (respectively S_bibi for the energy of the noise component at every microphone) within the microphone signal resulting from the noise present in the car cabin and being picked up by the microphone (for the setup according to Fig. 2b each microphone signal is only subtracted by the feedback) to the estimated noise present in each microphone signal. Double indices are used for the energy of the signal, and for power spectral densities. $w_{i} (k μ) = f (S_{mm} (k μ), S_{ff} (k μ), S_{bb} (k μ))$
For ^k being the time block index and ^µ being the sub-band index. The output of the voice blender 5a is denoted with: $v (k μ) = \sum_{i = 0}^{N - 1} w_{i} (k μ) (m_{i} (k μ) - f_{i} (k μ) + b_{i} (k μ))$
With N being the number of microphones.
For estimating the power of the microphone signal a first order IIR filter for smoothing can be applied according to $S_{m_{i} m_{i}} (k μ) = β_{m} {|m_{i} (k μ)|}^{2} + (1 - β_{m}) S_{m_{i} m_{i}} (k - 1, μ)$
Where ß_m is a smoothing constant that has to be chosen between zero and one. This means the current short-term power is weighted with ß_m and the estimate of the previous time frame is weighted with (1-ß_m).
The feedback component S_ƒiƒi (k,µ) is estimated in the feedback compensation or feedback suppression module and is therefore given.
The noise component can be estimated using a minimum tracker according to $S_{b_{i} b_{i}} (k μ) = \min \{S_{m_{i} m_{i}} (k μ), S_{b_{i} b_{i}} (k - 1, μ)\} (1 + ε)$
Where ^ε is a small number depending on the sampling rate. A good choice for a sampling rate of 44.1 kHz is e.g. epsilon=0.00001. This means that the minimum of the current time frame and the previous one is weighted by (1+epsilon) resulting in a signal that tracks local minima of the microphone signal power for estimating the noise signal power.
One possible implementation of w_i (k, µ) is: $w_{i} (k μ) = \frac{S_{m_{i} m_{i}} (k μ) - S_{f_{i} f_{i}} (k μ) - S_{b_{i} b_{i}} (k μ)}{S_{b_{i} b_{i}} (k μ) \sum_{i} \frac{S_{m_{i} m_{i}} (k μ) - S_{f_{i} f_{i}} (k μ) - S_{b_{i} b_{i}} (k μ)}{S_{b_{i} b_{i}} (k μ)}}$
This means that the modified SNR, after subtracting the power of the feedback signal and of the background noise from the microphone signal power, is set into relation with the sum over all microphones of the modified SNR. The resulting weight is more robust against misinterpretations of feedback power as desired speech power as would be the case using an SNNR as described in US006549629B2 .

Perception enhancement

Further criteria that can be included in the above weights are:

Adjusting the signal before or after the blending in terms of level to a common background noise level so that there is no perceivable difference for the listening party considering background-noise level when switching from one microphone to the other, e.g. using $w_{i, noise} (k μ) = w_{i} (k μ) \frac{S_{bb, desired} (k μ)}{\frac{1}{M} \sum_{μ = 0}^{M - 1} S_{b_{i} b_{i}} (k μ)}$
Where M is the amount of sub-bands µ and S_bb,desired(k,µ) is the desired noise level. This means applying a gain to the signal with lower noise level than the desired one to reach the desired noise level.
Adjusting the signal before or after the blending in terms of level to a common speech level so that there is no perceivable difference for the listening party considering speech level when switching from one microphone to the other. $S_{s_{i} s_{i}} (k μ) = S_{m_{i} m_{i}} (k μ) - S_{f_{i} f_{i}} (k μ) - S_{b_{i} b_{i}} (k μ)$
$w_{i, speech} (k μ) = w_{i} (k μ) \frac{S_{ss, desired} (k μ)}{\frac{1}{M} \sum_{μ = 0}^{M - 1} S_{s_{i} s_{i}} (k μ)}$
Where S_ss,desired(k,µ) is the desired speech level.

A combination of the above mentioned criteria with some trade-offs is permitted as well. A possibility of combining both criteria at the same time without having to trade off is described in the following.

Speaker individual noise suppression parameterization

For obtaining the same noise level for each speaker, the voice blender module 5a determines the difference in noise levels between the different signals and causes noise suppression to adapt the parameters of the noise suppression characteristic to result in a noise suppressed signal having the same level of residual noise in it for each speaker.
When having in mind that we as well want to have a balanced speech level between the different speakers we have to evaluate the differences in noise level after having balanced the signals for their speech level and use the resulting differences in noise level to adjust the parameters of the noise suppression module for each speaker.
This allows compensating for different background-noise levels and different speech levels at the same time. No trade off between the both optimization criteria is necessary. This means applying Equation 8 and adjusting the filter coefficients of the noise suppression (e.g. a Wiener filter) to (see Fig. 4b): $H (k μ) = \min \{β, 1 - \sum_{i} w_{i, speech} (k μ) \frac{S_{b_{i} b_{i}} (k μ)}{S_{m_{i} m_{i}} (k μ)}\}$
With $β = β_{NR} \frac{\frac{1}{M} \sum_{μ = 0}^{M - 1} \sum_{i} w_{i, speech} (k μ) S_{s_{i} s_{i}} (k μ)}{S_{ss, desired} (k μ)}$
And β_NR being the regular spectral floor or maximum attenuation of the Wiener filter (typically β_NR = ―10dB ).

Joint information usage

Fig. 4a shows, how a voice blender for carrying out the method according to the invention may be structured. It is clearly visible that feedback suppression or feedback compensation is done for each of the signals stemming from the microphones 1 and 2 in modules 7a and 7b. The output of these modules 7a, 7b is delivered to the voice blender 5a, i.e. to noise estimation modules 10, 10' (see equations 1-4), to power estimation modules 11, 11' (see equation 8) and to multiplicators 12 and 12' for weighting the incoming signals.
To this end, the multiplicators 12, 12' are each controlled by a module Wi which determines the blender weights for each signal of the microphones 1, 2. At the output of blender 5a is a summing point 13.
It is clear, that the representation as distinct blocks 10, 11, 12 does not mean that these blocks have to be realized as distinctive devices. In practice, all these activities or at least part of them will be handled by software.
As has been mentioned above, the output signal v(µ,k) of the voice blender 5a can be used to influence noise suppression in module 8. This is shown in Fig. 4b, where the weighting signals W_i,speech(k,µ) are supplied to module 8. Noise suppression down to zero gives mostly a bad feeling to the listener, for which reason it is preferred to adjust noise suppression to a predetermined level to a predetermined minimum level. This level typically equals 0.316 meaning a maximum suppression of -10 dB.
Alternatively or in combination with the above-mentioned measures (see also Fig. 3 which shows the information exchange between the modules in dotted lines), a configuration according to Fig. 4c is possible within the scope of the present invention taking equation 8 in mind. According to this figure, the weighting signals W_i,speech(k,µ) are supplied to module 9, the noise dependent gain control, which, to have an information of noise present in the signal, receives also the output signal S_bibi(k,µ) of the noise estimation modules 10 and 10'.
In a further modification according to Fig. 5, the weighting signal W_i of the voice blender 5a could be used to control an equalizer 6a which forms part of the post-processor 6 (Fig. 1 to Fig. 2b).
To figures 3, 4c and 5, the following details are now given:

(i) Noise dependent gain control

The decision of the voice blender module is used for the noise dependent gain control module. The noise dependent gain control module is responsible for increasing the level of the output signal by applying a gain g_ℓ,NDGC (S_bℓbℓ(k,µ)),depending on the level of noise perceived by the listening party. This increase of the output signal level depends on a characteristic which maps the level of background noise in the car cabin to a gain applied to the output signal. The characteristic chosen for this mapping depends on which speaker is active. This information is delivered by the voice blender module in the form of the microphone index of the most active microphone ℓ(k). $ℓ (k) = \arg \{\max_{i} \{\sum_{μ = 0}^{M - 1} w_{i, speech} (k μ)\}\}$
$g_{ℓ, NDGC} (S_{b_{ℓ} b_{ℓ}} (k μ)) = {\begin{matrix} 0, for S_{b_{ℓ} b_{ℓ}} (k μ) \leq S_{bb, ℓ, low} (k μ) \\ a_{ℓ} S_{b_{ℓ} b_{ℓ}} (k μ), for S_{bb, ℓ, low} (k μ) < S_{b_{ℓ} b_{ℓ}} (k μ) < S_{bb, ℓ, high} (k μ) \\ g_{ℓ, NDGC, \max} (S_{b_{ℓ} b_{ℓ}} (k μ)), for S_{b_{ℓ} b_{ℓ}} (k μ) \geq S_{bb, ℓ, high} (k μ) \end{matrix}$
$a_{ℓ} = \frac{g_{ℓ, NDGC, \max} (S_{b_{ℓ} b_{ℓ}} (k μ))}{S_{bb, ℓ, high} (k μ) - S_{bb, ℓ, low} (k μ)}$
Where g _ℓ,NDGC, _max (S _bℓbℓ(k,µ)) represents the maximum gain of the NDGC (e.g. 10dB). So all properties of the different NDGC characteristics are described by g_ℓ,NDGC,max (S_bℓbℓ(k,µ)), S_bb,ℓ,high(k,µ) and S_bb,ℓ,low(k,µ) .
In Equation 12, S_bℓbℓ(k,µ) represents the noise component that is present in the microphone signal of the active speaker. Some sample characteristics are depicted in Fig. 6.

(ii) Equalizer

As has been mentioned above, part of post processing 6 of an intercom system is the equalizer 6a. This equalizer 6a can as well be implemented as a multi-channel equalizer individual for each channel. The settings of the equalizer 6a are chosen depending on the voice blender decision of which person is speaking to enhance the perceived quality of the output for the listening party (see Fig.3 and Fig. 5).

(iii) Further optional modifications

A multi-microphone array can be used for beamforming. This allows for localization of the speaker in the car cabin (similar to the decision ^ℓ(k) of the voice blender). This information is used by the voice blender to decide which speaker is active.
Detection of position of listening party (e.g. by weight sensors in the seats) to further enhance joint information usage.
- o Voice blender knows where no speaker is and can set weights accordingly.

Furthermore by knowing who speaks and who listens, NDGC mapping characteristic and equalizer setting can be chosen in an even more specific manner.
Numerous modifications can be made within the scope of the invention; for example instead of voice blending, beamforming could be performed, as has been indicated above, so that the expression "blending" has to be understood in a broader sense.

Claims

Method for voice signal blending in a communication system, such as an indoor communication system, particularly in a vehicle, or in a hands-free telephony system or an automatic speech recognition system, comprising at least two microphones and at least one loudspeaker, wherein the microphone signals are blended with respective weights to be delivered to the at least one loudspeaker and feedback suppression or compensation is provided, characterized in that feedback suppression or compensation is effected before blending the signals of the at least two microphones.
Method according to claim 1, characterized in that for at least a sub-band of the microphone signals the energies of the microphone signals are determined and at least the energies of the noise components are estimated,, wherein for blending, a higher weight is given to at least the sub-band with the highest ratio of signal energy to noise energy.
Method according to claim 1 or 2, characterized in that a signal level adjustment of the at least two microphone signals to a predetermined value is carried out particularly immediately before or immediately after blending, wherein the adjustment is depending on the energies of the noise components and minimizes the perceivable difference in the noise level when blending from one microphone to the other.
Method according to any of the preceding claims, characterized in that signal levels of the microphone signals are adjusted to a predetermined value so that there is no perceivable difference in the signal level when blending from one microphone to the other.
Method according to claim 4, characterized in that voice levels of the at least two microphone signals are adjusted substantially to the same level, then the difference of the noise levels of the at least two adjusted microphone signals is determined and the noise suppression is controlled by adapting the parameters of the noise suppression characteristic, to generate a noise suppressed signal having substantially the same level of residual noise and the same level of speech signal in it for each microphone signal.
Method according to any of the preceding claims, characterized in that the respective speaking and/or listening, non-speaking party is detected by one of the following measures:
a) by analyzing the signal of the at least two microphones during blending;

b) by sensing by means of vehicle sensors, e.g. for the seat occupancy.
Method according to claim 6, characterized in that at least one of the following measures is taken after detection:
a) the signal weight of the microphone signal where no speaking party is detected is reduced, when blending;

b) predetermined amplifier characteristics of the gain control, e.g. noise dependent gain control, for listeners and/or speakers are provided and that amplifier characteristic is chosen, which corresponds to the position of the respective listener and/or speaker;

c) equalizing is effected by means of an equalizer after voice signal blending and that the settings of the equalizer are chosen depending on the detection so as to enhance the perceived quality of the output for the listening and/or speaking party.
Software product which executes the method according to any of the preceding claims.
Communication system, such as an in-door communication system, a hands-free telephony system or an automatic speech recognition system, comprising at least one loudspeaker and at least two microphones, as well as a signal treatment device, which carries out a method according to any of claims 1 to 7.