KotJohansson/palemoon27
1
0
Fork 0
mirror of https://github.com/roytam1/palemoon27.git synced 2026-05-29 18:40:41 +00:00
Code Issues Projects Releases Wiki Activity

import changes from `dev' branch of rmottola/Arctic-Fox:

Browse Source 
- bug 1179662 rename AudioNode::Stream() to GetStream() as it may return null r=padenot (d2d7e5f90)
-  bug 1197043 remove unnecessary aSampleRate parameter for AudioNodeStream creation r=padenot (3733ceb56)
- bug 1197043 rename Add/RemoveStream to Add/RemoveStreamGraphThread r=padenot (f648b8251)
- bug 1197043 introduce MediaStreamGraph::AddStream() r=padenot (ac021d4b2)
- bug 1197043 move AudioNodeStream creation to stream class r=padenot (a90a05910)
- bug 1197043 use flags to distinguish between external streams and events r=padenot (024dd96f1)
- Bug 1176300 - Move libsoundtouch to lgpllibs; r=glandium (99a546adf)
- Bug 1176300 - Add lgpllibs library to build system; r=glandium (bb4d07670)
- Bug 1176300 - Update libsoundtouch to patched r222; r=padenot (hg rev 8c32c900cb48)
- Bug 1176300 - Add soundtouch factory functions for DLL memory handling on windows; r=padenot (hg rev 84a1ffbc2db8)
- Bug 901633 - Part 1 - Implement a generic audio packetizer. r=jesup (a38c2d70b)
- Bug 901633 - Part 2 - Make AudioChannelFormat and AudioSegment more generic. r=roc (556b7349f)
- Bug 901633 - Part 3 - Fix TrackEncoder to use the new AudioChunk methods. r=jesup (56e018f83)
- Bug 901633 - Part 4 - Update AudioNodeStream to use new chunk methods. r=roc (9df19b894)
- Bug 901633 - Part 6 - Update DelayBuffer to use the new AudioChunk methods. r=karlt (6d5684334)
- Bug 901633 - Part 7 - Update AudioNodeExternalInputStream to use the new AudioChunk methods. r=karlt (caa3afa01)
- Bug 1155089: Fix hazard analysis bustage on a CLOSED TREE. r=bustage (8d23ccf39)
- Bug 1166183 - Back out the direct listener removal landed by mistake in bug 1141781. r=jesup (745d683d4)
- Bug 1166183 - Reset PipelineListener's flag after ReplaceTrack(). r=bwc (2fb38ca01)
- Bug 1170059 - Fix -Wunreachable-code clang warnings in webrtc/signaling. r=jesup (0d99b30ca)
- Bug 1139144 - Remove unused empty() definition from databuffer.h. r=mt (2fef64e3c)
- Bug 822129: don't alloc/free on every packet send in MediaPipeline r=bwc (ddfdc9455)
- Bug 1172397 - Check for Conduit/Type mismatch on every frame. r=jesup, r=bwc (a399a3336)
- Bug 1137169 - Uninitialised value uses related to mozilla::dom::WebAdioUtils::SpeexResamplerProcess. r=rjesup. (ce14ac278)
- Bug 901633 - Part 5 - Make MediaPipeline downmix and properly convert audio for webrtc.org code. r=jesup (89138b5d5)
- Bug 901633 - Part 8 - Use our new generic packetizer in the MediaPipeline so that we can packetize stereo easily. r=jesup (fb5d075b6)
- Bug 901633 - Part 9 - Make the necessary changes to VoEExternalMediaImpl::ExternalRecordingInsertData so that it the number of channels is forwarded down the webrtc.org code. r=jesup (d5d7dd4ca)
- Bug 901633 - Part 10 - Change the receiving side of the MediaPipeline so that it can detect and handle stereo. r=jesup (a73c1520f)
- Bug 901633 - Part 11 - Add an API in webrtc.org's output mixer to get the output channel count. r=jesup (4b396b85e)
- Bug 901633 - Part 12 - Add a function to deinterleave and convert an audio buffer. r=jesup (47ce3c7a5)
- Bug 901633 - Part 13 - Teach the resampler at the input of the MSG to dynamically change its channel count if needed. r=jesup (120c8d037)
- Bug 901633 - Part 14 - Add testing for our audio processing functions. r=jesup (5aa95b82e)
- Bug 901633 - Part 15 - Remove an allocation on the sending side, out of the packetizer. r=jesup (df8aed252)
- Bug 901633 - Part 16 - Remove another allocation in the sending side r=jesup (1e2fc8bca)
- Bug 1196408 - Make sure we only report a corrupt/slow video frame once. r=cpearce (5bae2f17a)
- bug 1162364 report telemetry on WMFMediaDataDecoder errors r=cpearce,f=vladan,bsmedberg (e217618ef)
- Bug 1193864 - Fixed dom/media/platforms/wmf/ compilation on mingw. r=cpearce (4e8c0ecd7)
- Bug 1141139: Enable low latency decoding on Windows. r=cpearce (9e0a36e27)
- Bug 1193547 - Fallback to software decoding explicitly if the GPU doesn't support decoding the current resolution in hardware. r=cpearce,jya (7fbab8784)
- Bug 1196417 - Make video software fallback only affect the current video instead of the entire browser. r=cpearce (3e83f0677)
This commit is contained in:
roytam1
2021-10-12 10:01:23 +08:00
parent a51ed11b22
commit c1485b21e1
102 changed files with 1910 additions and 905 deletions
Download Patch File Download Diff File Expand all files Collapse all files
b2g/installer/package-manifest.in
+1
View File
@@ -129,6 +129,7 @@
#ifndef MOZ_FOLD_LIBS
@BINPATH@/@DLL_PREFIX@mozsqlite3@DLL_SUFFIX@
#endif
@BINPATH@/@DLL_PREFIX@lgpllibs@DLL_SUFFIX@
@RESPATH@/blocklist.xml
@RESPATH@/ua-update.json
#ifdef XP_UNIX
browser/installer/package-manifest.in
+1
View File
@@ -162,6 +162,7 @@
@BINPATH@/@DLL_PREFIX@mozsqlite3@DLL_SUFFIX@
#endif
#endif
@BINPATH@/@DLL_PREFIX@lgpllibs@DLL_SUFFIX@
@RESPATH@/browser/blocklist.xml
#ifdef XP_UNIX
#ifndef XP_MACOSX
config/external/lgpllibs/moz.build
Vendored
+14
View File
@@ -0,0 +1,14 @@
# -*- Mode: python; c-basic-offset: 4; indent-tabs-mode: nil; tab-width: 40 -*-
# vim: set filetype=python:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# The lgpllibs library stores symbols from third-party LGPL licensed libraries,
# such as libav and libsoundtouch. It fulfills the requirement of dynamically
# linking these symbols into gecko.
#
# Any library added here should also be reflected in the about:license page.
SharedLibrary('lgpllibs')
SHARED_LIBRARY_NAME = 'lgpllibs'
config/external/moz.build
Vendored
+1
View File
@@ -7,6 +7,7 @@
external_dirs = []
DIRS += [
'lgpllibs',
'sqlite',
]
if not CONFIG['MOZ_NATIVE_JPEG']:
config/system-headers
+1
View File
@@ -1268,6 +1268,7 @@ X11/Xos.h
X11/Xutil.h
zmouse.h
soundtouch/SoundTouch.h
soundtouch/SoundTouchFactory.h
#if MOZ_NATIVE_PNG==1
png.h
#endif
dom/media/AudioChannelFormat.cpp
-75
View File
@@ -15,79 +15,4 @@ GetAudioChannelsSuperset(uint32_t aChannels1, uint32_t aChannels2)
return std::max(aChannels1, aChannels2);
}
/**
* UpMixMatrix represents a conversion matrix by exploiting the fact that
* each output channel comes from at most one input channel.
*/
struct UpMixMatrix {
uint8_t mInputDestination[CUSTOM_CHANNEL_LAYOUTS];
};
static const UpMixMatrix
gUpMixMatrices[CUSTOM_CHANNEL_LAYOUTS*(CUSTOM_CHANNEL_LAYOUTS - 1)/2] =
{
// Upmixes from mono
{ { 0, 0 } },
{ { 0, IGNORE, IGNORE } },
{ { 0, 0, IGNORE, IGNORE } },
{ { 0, IGNORE, IGNORE, IGNORE, IGNORE } },
{ { IGNORE, IGNORE, 0, IGNORE, IGNORE, IGNORE } },
// Upmixes from stereo
{ { 0, 1, IGNORE } },
{ { 0, 1, IGNORE, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, IGNORE, IGNORE } },
// Upmixes from 3-channel
{ { 0, 1, 2, IGNORE } },
{ { 0, 1, 2, IGNORE, IGNORE } },
{ { 0, 1, 2, IGNORE, IGNORE, IGNORE } },
// Upmixes from quad
{ { 0, 1, 2, 3, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, 2, 3 } },
// Upmixes from 5-channel
{ { 0, 1, 2, 3, 4, IGNORE } }
};
void
AudioChannelsUpMix(nsTArray<const void*>* aChannelArray,
uint32_t aOutputChannelCount,
const void* aZeroChannel)
{
uint32_t inputChannelCount = aChannelArray->Length();
uint32_t outputChannelCount =
GetAudioChannelsSuperset(aOutputChannelCount, inputChannelCount);
NS_ASSERTION(outputChannelCount > inputChannelCount,
"No up-mix needed");
MOZ_ASSERT(inputChannelCount > 0, "Bad number of channels");
MOZ_ASSERT(outputChannelCount > 0, "Bad number of channels");
aChannelArray->SetLength(outputChannelCount);
if (inputChannelCount < CUSTOM_CHANNEL_LAYOUTS &&
outputChannelCount <= CUSTOM_CHANNEL_LAYOUTS) {
const UpMixMatrix& m = gUpMixMatrices[
gMixingMatrixIndexByChannels[inputChannelCount - 1] +
outputChannelCount - inputChannelCount - 1];
const void* outputChannels[CUSTOM_CHANNEL_LAYOUTS];
for (uint32_t i = 0; i < outputChannelCount; ++i) {
uint8_t channelIndex = m.mInputDestination[i];
if (channelIndex == IGNORE) {
outputChannels[i] = aZeroChannel;
} else {
outputChannels[i] = aChannelArray->ElementAt(channelIndex);
}
}
for (uint32_t i = 0; i < outputChannelCount; ++i) {
aChannelArray->ElementAt(i) = outputChannels[i];
}
return;
}
for (uint32_t i = inputChannelCount; i < outputChannelCount; ++i) {
aChannelArray->ElementAt(i) = aZeroChannel;
}
}
} // namespace mozilla
dom/media/AudioChannelFormat.h
+95 -26
View File
@@ -58,24 +58,6 @@ const int gMixingMatrixIndexByChannels[CUSTOM_CHANNEL_LAYOUTS - 1] =
uint32_t
GetAudioChannelsSuperset(uint32_t aChannels1, uint32_t aChannels2);
/**
* Given an array of input channel data, and an output channel count,
* replaces the array with an array of upmixed channels.
* This shuffles the array and may set some channel buffers to aZeroChannel.
* Don't call this with input count >= output count.
* This may return *more* channels than requested. In that case, downmixing
* is required to to get to aOutputChannelCount. (This is how we handle
* odd cases like 3 -> 4 upmixing.)
* If aChannelArray.Length() was the input to one of a series of
* GetAudioChannelsSuperset calls resulting in aOutputChannelCount,
* no downmixing will be required.
*/
void
AudioChannelsUpMix(nsTArray<const void*>* aChannelArray,
uint32_t aOutputChannelCount,
const void* aZeroChannel);
/**
* DownMixMatrix represents a conversion matrix efficiently by exploiting the
* fact that each input channel contributes to at most one output channel,
@@ -124,19 +106,19 @@ gDownMixMatrices[CUSTOM_CHANNEL_LAYOUTS*(CUSTOM_CHANNEL_LAYOUTS - 1)/2] =
* input count <= output count.
*/
template<typename T>
void AudioChannelsDownMix(const nsTArray<const void*>& aChannelArray,
T** aOutputChannels,
uint32_t aOutputChannelCount,
uint32_t aDuration)
void AudioChannelsDownMix(const nsTArray<const T*>& aChannelArray,
T** aOutputChannels,
uint32_t aOutputChannelCount,
uint32_t aDuration)
{
uint32_t inputChannelCount = aChannelArray.Length();
const void* const* inputChannels = aChannelArray.Elements();
const T* const* inputChannels = aChannelArray.Elements();
NS_ASSERTION(inputChannelCount > aOutputChannelCount, "Nothing to do");
if (inputChannelCount > 6) {
// Just drop the unknown channels.
for (uint32_t o = 0; o < aOutputChannelCount; ++o) {
memcpy(aOutputChannels[o], inputChannels[o], aDuration*sizeof(T));
PodCopy(aOutputChannels[o], inputChannels[o], aDuration);
}
return;
}
@@ -153,8 +135,7 @@ void AudioChannelsDownMix(const nsTArray<const void*>& aChannelArray,
for (uint32_t s = 0; s < aDuration; ++s) {
// Reserve an extra junk channel at the end for the cases where we
// want an input channel to contribute to nothing
T outputChannels[CUSTOM_CHANNEL_LAYOUTS + 1];
memset(outputChannels, 0, sizeof(T)*(CUSTOM_CHANNEL_LAYOUTS));
T outputChannels[CUSTOM_CHANNEL_LAYOUTS + 1] = {0};
for (uint32_t c = 0; c < inputChannelCount; ++c) {
outputChannels[m.mInputDestination[c]] +=
m.mInputCoefficient[c]*(static_cast<const T*>(inputChannels[c]))[s];
@@ -171,6 +152,94 @@ void AudioChannelsDownMix(const nsTArray<const void*>& aChannelArray,
}
}
/**
* UpMixMatrix represents a conversion matrix by exploiting the fact that
* each output channel comes from at most one input channel.
*/
struct UpMixMatrix {
uint8_t mInputDestination[CUSTOM_CHANNEL_LAYOUTS];
};
static const UpMixMatrix
gUpMixMatrices[CUSTOM_CHANNEL_LAYOUTS*(CUSTOM_CHANNEL_LAYOUTS - 1)/2] =
{
// Upmixes from mono
{ { 0, 0 } },
{ { 0, IGNORE, IGNORE } },
{ { 0, 0, IGNORE, IGNORE } },
{ { 0, IGNORE, IGNORE, IGNORE, IGNORE } },
{ { IGNORE, IGNORE, 0, IGNORE, IGNORE, IGNORE } },
// Upmixes from stereo
{ { 0, 1, IGNORE } },
{ { 0, 1, IGNORE, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, IGNORE, IGNORE } },
// Upmixes from 3-channel
{ { 0, 1, 2, IGNORE } },
{ { 0, 1, 2, IGNORE, IGNORE } },
{ { 0, 1, 2, IGNORE, IGNORE, IGNORE } },
// Upmixes from quad
{ { 0, 1, 2, 3, IGNORE } },
{ { 0, 1, IGNORE, IGNORE, 2, 3 } },
// Upmixes from 5-channel
{ { 0, 1, 2, 3, 4, IGNORE } }
};
/**
* Given an array of input channel data, and an output channel count,
* replaces the array with an array of upmixed channels.
* This shuffles the array and may set some channel buffers to aZeroChannel.
* Don't call this with input count >= output count.
* This may return *more* channels than requested. In that case, downmixing
* is required to to get to aOutputChannelCount. (This is how we handle
* odd cases like 3 -> 4 upmixing.)
* If aChannelArray.Length() was the input to one of a series of
* GetAudioChannelsSuperset calls resulting in aOutputChannelCount,
* no downmixing will be required.
*/
template<typename T>
void
AudioChannelsUpMix(nsTArray<const T*>* aChannelArray,
uint32_t aOutputChannelCount,
const T* aZeroChannel)
{
uint32_t inputChannelCount = aChannelArray->Length();
uint32_t outputChannelCount =
GetAudioChannelsSuperset(aOutputChannelCount, inputChannelCount);
NS_ASSERTION(outputChannelCount > inputChannelCount,
"No up-mix needed");
MOZ_ASSERT(inputChannelCount > 0, "Bad number of channels");
MOZ_ASSERT(outputChannelCount > 0, "Bad number of channels");
aChannelArray->SetLength(outputChannelCount);
if (inputChannelCount < CUSTOM_CHANNEL_LAYOUTS &&
outputChannelCount <= CUSTOM_CHANNEL_LAYOUTS) {
const UpMixMatrix& m = gUpMixMatrices[
gMixingMatrixIndexByChannels[inputChannelCount - 1] +
outputChannelCount - inputChannelCount - 1];
const T* outputChannels[CUSTOM_CHANNEL_LAYOUTS];
for (uint32_t i = 0; i < outputChannelCount; ++i) {
uint8_t channelIndex = m.mInputDestination[i];
if (channelIndex == IGNORE) {
outputChannels[i] = aZeroChannel;
} else {
outputChannels[i] = aChannelArray->ElementAt(channelIndex);
}
}
for (uint32_t i = 0; i < outputChannelCount; ++i) {
aChannelArray->ElementAt(i) = outputChannels[i];
}
return;
}
for (uint32_t i = inputChannelCount; i < outputChannelCount; ++i) {
aChannelArray->ElementAt(i) = aZeroChannel;
}
}
} // namespace mozilla
dom/media/AudioPacketizer.h
+196
View File
@@ -0,0 +1,196 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*-*/
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this file,
* You can obtain one at http://mozilla.org/MPL/2.0/. */
#ifndef AudioPacketizer_h_
#define AudioPacketizer_h_
#include <mozilla/PodOperations.h>
#include <mozilla/Assertions.h>
#include <nsAutoPtr.h>
#include <AudioSampleFormat.h>
// Enable this to warn when `Output` has been called but not enough data was
// buffered.
// #define LOG_PACKETIZER_UNDERRUN
namespace mozilla {
/**
* This class takes arbitrary input data, and returns packets of a specific
* size. In the process, it can convert audio samples from 16bit integers to
* float (or vice-versa).
*
* Input and output, as well as length units in the public interface are
* interleaved frames.
*
* Allocations of output buffer can be performed by this class. Buffers can
* simply be delete-d. This is because packets are intended to be sent off to
* non-gecko code using normal pointers/length pairs
*
* Alternatively, consumers can pass in a buffer in which the output is copied.
* The buffer needs to be large enough to store a packet worth of audio.
*
* The implementation uses a circular buffer using absolute virtual indices.
*/
template <typename InputType, typename OutputType>
class AudioPacketizer
{
public:
AudioPacketizer(uint32_t aPacketSize, uint32_t aChannels)
: mPacketSize(aPacketSize)
, mChannels(aChannels)
, mReadIndex(0)
, mWriteIndex(0)
// Start off with a single packet
, mStorage(new InputType[aPacketSize * aChannels])
, mLength(aPacketSize * aChannels)
{
MOZ_ASSERT(aPacketSize > 0 && aChannels > 0,
"The packet size and the number of channel should be strictly positive");
}
void Input(const InputType* aFrames, uint32_t aFrameCount)
{
uint32_t inputSamples = aFrameCount * mChannels;
// Need to grow the storage. This should rarely happen, if at all, once the
// array has the right size.
if (inputSamples > EmptySlots()) {
// Calls to Input and Output are roughtly interleaved
// (Input,Output,Input,Output, etc.), or balanced
// (Input,Input,Input,Output,Output,Output), so we update the buffer to
// the exact right size in order to not waste space.
uint32_t newLength = AvailableSamples() + inputSamples;
uint32_t toCopy = AvailableSamples();
nsAutoPtr<InputType> oldStorage = mStorage;
mStorage = new InputType[newLength];
// Copy the old data at the beginning of the new storage.
if (WriteIndex() >= ReadIndex()) {
PodCopy(mStorage.get(),
oldStorage.get() + ReadIndex(),
AvailableSamples());
} else {
uint32_t firstPartLength = mLength - ReadIndex();
uint32_t secondPartLength = AvailableSamples() - firstPartLength;
PodCopy(mStorage.get(),
oldStorage.get() + ReadIndex(),
firstPartLength);
PodCopy(mStorage.get() + firstPartLength,
oldStorage.get(),
secondPartLength);
}
mWriteIndex = toCopy;
mReadIndex = 0;
mLength = newLength;
}
if (WriteIndex() + inputSamples <= mLength) {
PodCopy(mStorage.get() + WriteIndex(), aFrames, aFrameCount * mChannels);
} else {
uint32_t firstPartLength = mLength - WriteIndex();
uint32_t secondPartLength = inputSamples - firstPartLength;
PodCopy(mStorage.get() + WriteIndex(), aFrames, firstPartLength);
PodCopy(mStorage.get(), aFrames + firstPartLength, secondPartLength);
}
mWriteIndex += inputSamples;
}
OutputType* Output()
{
uint32_t samplesNeeded = mPacketSize * mChannels;
OutputType* out = new OutputType[samplesNeeded];
Output(out);
return out;
}
void Output(OutputType* aOutputBuffer)
{
uint32_t samplesNeeded = mPacketSize * mChannels;
// Under-run. Pad the end of the buffer with silence.
if (AvailableSamples() < samplesNeeded) {
#ifdef LOG_PACKETIZER_UNDERRUN
char buf[256];
snprintf(buf, 256,
"AudioPacketizer %p underrun: available: %u, needed: %u\n",
this, AvailableSamples(), samplesNeeded);
NS_WARNING(buf);
#endif
uint32_t zeros = samplesNeeded - AvailableSamples();
PodZero(aOutputBuffer + AvailableSamples(), zeros);
samplesNeeded -= zeros;
}
if (ReadIndex() + samplesNeeded <= mLength) {
ConvertAudioSamples<InputType,OutputType>(mStorage.get() + ReadIndex(),
aOutputBuffer,
samplesNeeded);
} else {
uint32_t firstPartLength = mLength - ReadIndex();
uint32_t secondPartLength = samplesNeeded - firstPartLength;
ConvertAudioSamples<InputType, OutputType>(mStorage.get() + ReadIndex(),
aOutputBuffer,
firstPartLength);
ConvertAudioSamples<InputType, OutputType>(mStorage.get(),
aOutputBuffer + firstPartLength,
secondPartLength);
}
mReadIndex += samplesNeeded;
}
uint32_t PacketsAvailable() const {
return AvailableSamples() / mChannels / mPacketSize;
}
bool Empty() const {
return mWriteIndex == mReadIndex;
}
bool Full() const {
return mWriteIndex - mReadIndex == mLength;
}
uint32_t PacketSize() const {
return mPacketSize;
}
uint32_t Channels() const {
return mChannels;
}
private:
uint32_t ReadIndex() const {
return mReadIndex % mLength;
}
uint32_t WriteIndex() const {
return mWriteIndex % mLength;
}
uint32_t AvailableSamples() const {
return mWriteIndex - mReadIndex;
}
uint32_t EmptySlots() const {
return mLength - AvailableSamples();
}
// Size of one packet of audio, in frames
uint32_t mPacketSize;
// Number of channels of the stream flowing through this packetizer
uint32_t mChannels;
// Two virtual index into the buffer: the read position and the write
// position.
uint64_t mReadIndex;
uint64_t mWriteIndex;
// Storage for the samples
nsAutoPtr<InputType> mStorage;
// Length of the buffer, in samples
uint32_t mLength;
};
} // mozilla
#endif // AudioPacketizer_h_
dom/media/AudioSampleFormat.h
+45
View File
@@ -96,6 +96,51 @@ FloatToAudioSample<int16_t>(float aValue)
return int16_t(clamped);
}
template <typename T> T IntegerToAudioSample(int16_t aValue);
template <> inline float
IntegerToAudioSample<float>(int16_t aValue)
{
return aValue / 32768.0f;
}
template <> inline int16_t
IntegerToAudioSample<int16_t>(int16_t aValue)
{
return aValue;
}
template<typename SrcT, typename DstT>
inline void
ConvertAudioSample(SrcT aIn, DstT& aOut);
template<>
inline void
ConvertAudioSample(int16_t aIn, int16_t & aOut)
{
aOut = aIn;
}
template<>
inline void
ConvertAudioSample(int16_t aIn, float& aOut)
{
aOut = AudioSampleToFloat(aIn);
}
template<>
inline void
ConvertAudioSample(float aIn, float& aOut)
{
aOut = aIn;
}
template<>
inline void
ConvertAudioSample(float aIn, int16_t& aOut)
{
aOut = FloatToAudioSample<int16_t>(aIn);
}
// Sample buffer conversion
template <typename From, typename To> inline void
dom/media/AudioSegment.cpp
+28 -125
View File
@@ -5,7 +5,6 @@
#include "AudioSegment.h"
#include "AudioStream.h"
#include "AudioMixer.h"
#include "AudioChannelFormat.h"
#include "Latency.h"
@@ -13,49 +12,18 @@
namespace mozilla {
template <class SrcT, class DestT>
static void
InterleaveAndConvertBuffer(const SrcT** aSourceChannels,
int32_t aLength, float aVolume,
int32_t aChannels,
DestT* aOutput)
const uint8_t SilentChannel::gZeroChannel[MAX_AUDIO_SAMPLE_SIZE*SilentChannel::AUDIO_PROCESSING_FRAMES] = {0};
template<>
const float* SilentChannel::ZeroChannel<float>()
{
DestT* output = aOutput;
for (int32_t i = 0; i < aLength; ++i) {
for (int32_t channel = 0; channel < aChannels; ++channel) {
float v = AudioSampleToFloat(aSourceChannels[channel][i])*aVolume;
*output = FloatToAudioSample<DestT>(v);
++output;
}
}
return reinterpret_cast<const float*>(SilentChannel::gZeroChannel);
}
void
InterleaveAndConvertBuffer(const void** aSourceChannels,
AudioSampleFormat aSourceFormat,
int32_t aLength, float aVolume,
int32_t aChannels,
AudioDataValue* aOutput)
template<>
const int16_t* SilentChannel::ZeroChannel<int16_t>()
{
switch (aSourceFormat) {
case AUDIO_FORMAT_FLOAT32:
InterleaveAndConvertBuffer(reinterpret_cast<const float**>(aSourceChannels),
aLength,
aVolume,
aChannels,
aOutput);
break;
case AUDIO_FORMAT_S16:
InterleaveAndConvertBuffer(reinterpret_cast<const int16_t**>(aSourceChannels),
aLength,
aVolume,
aChannels,
aOutput);
break;
case AUDIO_FORMAT_SILENCE:
// nothing to do here.
break;
}
return reinterpret_cast<const int16_t*>(SilentChannel::gZeroChannel);
}
void
@@ -66,54 +34,6 @@ AudioSegment::ApplyVolume(float aVolume)
}
}
static const int AUDIO_PROCESSING_FRAMES = 640; /* > 10ms of 48KHz audio */
static const uint8_t gZeroChannel[MAX_AUDIO_SAMPLE_SIZE*AUDIO_PROCESSING_FRAMES] = {0};
void
DownmixAndInterleave(const nsTArray<const void*>& aChannelData,
AudioSampleFormat aSourceFormat, int32_t aDuration,
float aVolume, uint32_t aOutputChannels,
AudioDataValue* aOutput)
{
nsAutoTArray<const void*,GUESS_AUDIO_CHANNELS> channelData;
nsAutoTArray<float,AUDIO_PROCESSING_FRAMES*GUESS_AUDIO_CHANNELS> downmixConversionBuffer;
nsAutoTArray<float,AUDIO_PROCESSING_FRAMES*GUESS_AUDIO_CHANNELS> downmixOutputBuffer;
channelData.SetLength(aChannelData.Length());
if (aSourceFormat != AUDIO_FORMAT_FLOAT32) {
NS_ASSERTION(aSourceFormat == AUDIO_FORMAT_S16, "unknown format");
downmixConversionBuffer.SetLength(aDuration*aChannelData.Length());
for (uint32_t i = 0; i < aChannelData.Length(); ++i) {
float* conversionBuf = downmixConversionBuffer.Elements() + (i*aDuration);
const int16_t* sourceBuf = static_cast<const int16_t*>(aChannelData[i]);
for (uint32_t j = 0; j < (uint32_t)aDuration; ++j) {
conversionBuf[j] = AudioSampleToFloat(sourceBuf[j]);
}
channelData[i] = conversionBuf;
}
} else {
for (uint32_t i = 0; i < aChannelData.Length(); ++i) {
channelData[i] = aChannelData[i];
}
}
downmixOutputBuffer.SetLength(aDuration*aOutputChannels);
nsAutoTArray<float*,GUESS_AUDIO_CHANNELS> outputChannelBuffers;
nsAutoTArray<const void*,GUESS_AUDIO_CHANNELS> outputChannelData;
outputChannelBuffers.SetLength(aOutputChannels);
outputChannelData.SetLength(aOutputChannels);
for (uint32_t i = 0; i < (uint32_t)aOutputChannels; ++i) {
outputChannelData[i] = outputChannelBuffers[i] =
downmixOutputBuffer.Elements() + aDuration*i;
}
if (channelData.Length() > aOutputChannels) {
AudioChannelsDownMix(channelData, outputChannelBuffers.Elements(),
aOutputChannels, aDuration);
}
InterleaveAndConvertBuffer(outputChannelData.Elements(), AUDIO_FORMAT_FLOAT32,
aDuration, aVolume, aOutputChannels, aOutput);
}
void AudioSegment::ResampleChunks(SpeexResamplerState* aResampler, uint32_t aInRate, uint32_t aOutRate)
{
if (mChunks.IsEmpty()) {
@@ -165,9 +85,9 @@ void
AudioSegment::Mix(AudioMixer& aMixer, uint32_t aOutputChannels,
uint32_t aSampleRate)
{
nsAutoTArray<AudioDataValue, AUDIO_PROCESSING_FRAMES* GUESS_AUDIO_CHANNELS>
nsAutoTArray<AudioDataValue, SilentChannel::AUDIO_PROCESSING_FRAMES* GUESS_AUDIO_CHANNELS>
buf;
nsAutoTArray<const void*, GUESS_AUDIO_CHANNELS> channelData;
nsAutoTArray<const AudioDataValue*, GUESS_AUDIO_CHANNELS> channelData;
uint32_t offsetSamples = 0;
uint32_t duration = GetDuration();
@@ -197,11 +117,11 @@ AudioSegment::Mix(AudioMixer& aMixer, uint32_t aOutputChannels,
// desired input and output channels.
channelData.SetLength(c.mChannelData.Length());
for (uint32_t i = 0; i < channelData.Length(); ++i) {
channelData[i] = c.mChannelData[i];
channelData[i] = static_cast<const AudioDataValue*>(c.mChannelData[i]);
}
if (channelData.Length() < aOutputChannels) {
// Up-mix.
AudioChannelsUpMix(&channelData, aOutputChannels, gZeroChannel);
AudioChannelsUpMix(&channelData, aOutputChannels, SilentChannel::ZeroChannel<AudioDataValue>());
for (uint32_t channel = 0; channel < aOutputChannels; channel++) {
AudioDataValue* ptr =
PointerForOffsetInChannel(buf.Elements(), outBufferLength,
@@ -246,9 +166,8 @@ AudioSegment::Mix(AudioMixer& aMixer, uint32_t aOutputChannels,
void
AudioSegment::WriteTo(uint64_t aID, AudioMixer& aMixer, uint32_t aOutputChannels, uint32_t aSampleRate)
{
nsAutoTArray<AudioDataValue,AUDIO_PROCESSING_FRAMES*GUESS_AUDIO_CHANNELS> buf;
nsAutoTArray<const void*,GUESS_AUDIO_CHANNELS> channelData;
// Offset in the buffer that will end up sent to the AudioStream, in samples.
nsAutoTArray<AudioDataValue,SilentChannel::AUDIO_PROCESSING_FRAMES*GUESS_AUDIO_CHANNELS> buf;
// Offset in the buffer that will be written to the mixer, in samples.
uint32_t offset = 0;
if (GetDuration() <= 0) {
@@ -262,39 +181,23 @@ AudioSegment::WriteTo(uint64_t aID, AudioMixer& aMixer, uint32_t aOutputChannels
for (ChunkIterator ci(*this); !ci.IsEnded(); ci.Next()) {
AudioChunk& c = *ci;
uint32_t frames = c.mDuration;
// If we have written data in the past, or we have real (non-silent) data
// to write, we can proceed. Otherwise, it means we just started the
// AudioStream, and we don't have real data to write to it (just silence).
// To avoid overbuffering in the AudioStream, we simply drop the silence,
// here. The stream will underrun and output silence anyways.
if (c.mBuffer && c.mBufferFormat != AUDIO_FORMAT_SILENCE) {
channelData.SetLength(c.mChannelData.Length());
for (uint32_t i = 0; i < channelData.Length(); ++i) {
channelData[i] = c.mChannelData[i];
}
if (channelData.Length() < aOutputChannels) {
// Up-mix. Note that this might actually make channelData have more
// than aOutputChannels temporarily.
AudioChannelsUpMix(&channelData, aOutputChannels, gZeroChannel);
}
if (channelData.Length() > aOutputChannels) {
// Down-mix.
DownmixAndInterleave(channelData, c.mBufferFormat, frames,
c.mVolume, aOutputChannels, buf.Elements() + offset);
} else {
InterleaveAndConvertBuffer(channelData.Elements(), c.mBufferFormat,
frames, c.mVolume,
aOutputChannels,
buf.Elements() + offset);
}
} else {
// Assumes that a bit pattern of zeroes == 0.0f
memset(buf.Elements() + offset, 0, aOutputChannels * frames * sizeof(AudioDataValue));
switch (c.mBufferFormat) {
case AUDIO_FORMAT_S16:
WriteChunk<int16_t>(c, aOutputChannels, buf.Elements() + offset);
break;
case AUDIO_FORMAT_FLOAT32:
WriteChunk<float>(c, aOutputChannels, buf.Elements() + offset);
break;
case AUDIO_FORMAT_SILENCE:
// The mixer is expecting interleaved data, so this is ok.
PodZero(buf.Elements() + offset, c.mDuration * aOutputChannels);
break;
default:
MOZ_ASSERT(false, "Not handled");
}
offset += frames * aOutputChannels;
offset += c.mDuration * aOutputChannels;
#if !defined(MOZILLA_XPCOMRT_API)
if (!c.mTimeStamp.IsNull()) {
dom/media/AudioSegment.h
+106 -9
View File
@@ -8,6 +8,7 @@
#include "MediaSegment.h"
#include "AudioSampleFormat.h"
#include "AudioChannelFormat.h"
#include "SharedBuffer.h"
#include "WebAudioUtils.h"
#ifdef MOZILLA_INTERNAL_API
@@ -56,21 +57,82 @@ const int GUESS_AUDIO_CHANNELS = 2;
const uint32_t WEBAUDIO_BLOCK_SIZE_BITS = 7;
const uint32_t WEBAUDIO_BLOCK_SIZE = 1 << WEBAUDIO_BLOCK_SIZE_BITS;
void InterleaveAndConvertBuffer(const void** aSourceChannels,
AudioSampleFormat aSourceFormat,
int32_t aLength, float aVolume,
int32_t aChannels,
AudioDataValue* aOutput);
template <typename SrcT, typename DestT>
static void
InterleaveAndConvertBuffer(const SrcT* const* aSourceChannels,
uint32_t aLength, float aVolume,
uint32_t aChannels,
DestT* aOutput)
{
DestT* output = aOutput;
for (size_t i = 0; i < aLength; ++i) {
for (size_t channel = 0; channel < aChannels; ++channel) {
float v = AudioSampleToFloat(aSourceChannels[channel][i])*aVolume;
*output = FloatToAudioSample<DestT>(v);
++output;
}
}
}
template <typename SrcT, typename DestT>
static void
DeinterleaveAndConvertBuffer(const SrcT* aSourceBuffer,
uint32_t aFrames, uint32_t aChannels,
DestT** aOutput)
{
for (size_t i = 0; i < aChannels; i++) {
size_t interleavedIndex = i;
for (size_t j = 0; j < aFrames; j++) {
ConvertAudioSample(aSourceBuffer[interleavedIndex],
aOutput[i][j]);
interleavedIndex += aChannels;
}
}
}
class SilentChannel
{
public:
static const int AUDIO_PROCESSING_FRAMES = 640; /* > 10ms of 48KHz audio */
static const uint8_t gZeroChannel[MAX_AUDIO_SAMPLE_SIZE*AUDIO_PROCESSING_FRAMES];
// We take advantage of the fact that zero in float and zero in int have the
// same all-zeros bit layout.
template<typename T>
static const T* ZeroChannel();
};
/**
* Given an array of input channels (aChannelData), downmix to aOutputChannels,
* interleave the channel data. A total of aOutputChannels*aDuration
* interleaved samples will be copied to a channel buffer in aOutput.
*/
void DownmixAndInterleave(const nsTArray<const void*>& aChannelData,
AudioSampleFormat aSourceFormat, int32_t aDuration,
float aVolume, uint32_t aOutputChannels,
AudioDataValue* aOutput);
template <typename SrcT, typename DestT>
void
DownmixAndInterleave(const nsTArray<const SrcT*>& aChannelData,
int32_t aDuration, float aVolume, uint32_t aOutputChannels,
DestT* aOutput)
{
if (aChannelData.Length() == aOutputChannels) {
InterleaveAndConvertBuffer(aChannelData.Elements(),
aDuration, aVolume, aOutputChannels, aOutput);
} else {
nsAutoTArray<SrcT*,GUESS_AUDIO_CHANNELS> outputChannelData;
nsAutoTArray<SrcT, SilentChannel::AUDIO_PROCESSING_FRAMES * GUESS_AUDIO_CHANNELS> outputBuffers;
outputChannelData.SetLength(aOutputChannels);
outputBuffers.SetLength(aDuration * aOutputChannels);
for (uint32_t i = 0; i < aOutputChannels; i++) {
outputChannelData[i] = outputBuffers.Elements() + aDuration * i;
}
AudioChannelsDownMix(aChannelData,
outputChannelData.Elements(),
aOutputChannels,
aDuration);
InterleaveAndConvertBuffer(outputChannelData.Elements(),
aDuration, aVolume, aOutputChannels, aOutput);
}
}
/**
* An AudioChunk represents a multi-channel buffer of audio samples.
@@ -190,6 +252,13 @@ struct AudioChunk {
return amount;
}
template<typename T>
const nsTArray<const T*>& ChannelData()
{
MOZ_ASSERT(AudioSampleTypeToFormat<T>::Format == mBufferFormat);
return *reinterpret_cast<nsTArray<const T*>*>(&mChannelData);
}
StreamTime mDuration; // in frames within the buffer
nsRefPtr<ThreadSharedObject> mBuffer; // the buffer object whose lifetime is managed; null means data is all zeroes
nsTArray<const void*> mChannelData; // one pointer per channel; empty if and only if mBuffer is null
@@ -356,6 +425,34 @@ public:
}
};
template<typename SrcT>
void WriteChunk(AudioChunk& aChunk,
uint32_t aOutputChannels,
AudioDataValue* aOutputBuffer)
{
nsAutoTArray<const SrcT*,GUESS_AUDIO_CHANNELS> channelData;
channelData = aChunk.ChannelData<SrcT>();
if (channelData.Length() < aOutputChannels) {
// Up-mix. Note that this might actually make channelData have more
// than aOutputChannels temporarily.
AudioChannelsUpMix(&channelData, aOutputChannels, SilentChannel::ZeroChannel<SrcT>());
}
if (channelData.Length() > aOutputChannels) {
// Down-mix.
DownmixAndInterleave(channelData, aChunk.mDuration,
aChunk.mVolume, aOutputChannels, aOutputBuffer);
} else {
InterleaveAndConvertBuffer(channelData.Elements(),
aChunk.mDuration, aChunk.mVolume,
aOutputChannels,
aOutputBuffer);
}
}
} // namespace mozilla
#endif /* MOZILLA_AUDIOSEGMENT_H_ */
dom/media/AudioStream.cpp
+5 -2
View File
@@ -14,7 +14,6 @@
#include "mozilla/Mutex.h"
#include "mozilla/Snprintf.h"
#include <algorithm>
#include "soundtouch/SoundTouch.h"
#include "Latency.h"
#include "CubebUtils.h"
#include "nsPrintfCString.h"
@@ -129,6 +128,7 @@ AudioStream::AudioStream()
, mOutChannels(0)
, mWritten(0)
, mAudioClock(this)
, mTimeStretcher(nullptr)
, mLatencyRequest(HighLatency)
, mReadPoint(0)
, mDumpFile(nullptr)
@@ -151,6 +151,9 @@ AudioStream::~AudioStream()
if (mDumpFile) {
fclose(mDumpFile);
}
if (mTimeStretcher) {
soundtouch::destroySoundTouchObj(mTimeStretcher);
}
}
size_t
@@ -173,7 +176,7 @@ nsresult AudioStream::EnsureTimeStretcherInitializedUnlocked()
{
mMonitor.AssertCurrentThreadOwns();
if (!mTimeStretcher) {
mTimeStretcher = new soundtouch::SoundTouch();
mTimeStretcher = soundtouch::createSoundTouchObj();
mTimeStretcher->setSampleRate(mInRate);
mTimeStretcher->setChannels(mOutChannels);
mTimeStretcher->setPitch(1.0);
dom/media/AudioStream.h
+2 -5
View File
@@ -15,10 +15,7 @@
#include "mozilla/RefPtr.h"
#include "mozilla/UniquePtr.h"
#include "CubebUtils.h"
namespace soundtouch {
class SoundTouch;
}
#include "soundtouch/SoundTouchFactory.h"
namespace mozilla {
@@ -332,7 +329,7 @@ private:
// Number of frames written to the buffers.
int64_t mWritten;
AudioClock mAudioClock;
nsAutoPtr<soundtouch::SoundTouch> mTimeStretcher;
soundtouch::SoundTouch* mTimeStretcher;
nsRefPtr<AsyncLatencyLogger> mLatencyLog;
// copy of Latency logger's starting time for offset calculations
dom/media/MediaDecoderStateMachine.cpp
+18 -6
View File
@@ -217,7 +217,6 @@ MediaDecoderStateMachine::MediaDecoderStateMachine(MediaDecoder* aDecoder,
mCurrentTimeBeforeSeek(0),
mCorruptFrames(30),
mDecodingFirstFrame(true),
mDisabledHardwareAcceleration(false),
mSentLoadedMetadataEvent(false),
mSentFirstFrameLoadedEvent(false),
mSentPlaybackEndedEvent(false),
@@ -2453,11 +2452,17 @@ MediaDecoderStateMachine::Reset()
DecodeTaskQueue()->Dispatch(resetTask.forget());
}
void MediaDecoderStateMachine::CheckTurningOffHardwareDecoder(VideoData* aData)
bool MediaDecoderStateMachine::CheckFrameValidity(VideoData* aData)
{
MOZ_ASSERT(OnTaskQueue());
AssertCurrentThreadInMonitor();
// If we've sent this frame before then only return the valid state,
// don't update the statistics.
if (aData->mSentToCompositor) {
return !aData->mImage || aData->mImage->IsValid();
}
// Update corrupt-frames statistics
if (aData->mImage && !aData->mImage->IsValid()) {
MediaDecoder::FrameStatistics& frameStats = mDecoder->GetFrameStatistics();
@@ -2466,18 +2471,19 @@ void MediaDecoderStateMachine::CheckTurningOffHardwareDecoder(VideoData* aData)
// hardware acceleration. We use 10 as the corrupt value because RollingMean<>
// only supports integer types.
mCorruptFrames.insert(10);
if (!mDisabledHardwareAcceleration &&
mReader->VideoIsHardwareAccelerated() &&
if (mReader->VideoIsHardwareAccelerated() &&
frameStats.GetPresentedFrames() > 30 &&
mCorruptFrames.mean() >= 1 /* 10% */) {
nsCOMPtr<nsIRunnable> task =
NS_NewRunnableMethod(mReader, &MediaDecoderReader::DisableHardwareAcceleration);
DecodeTaskQueue()->Dispatch(task.forget());
mDisabledHardwareAcceleration = true;
mCorruptFrames.clear();
gfxCriticalNote << "Too many dropped/corrupted frames, disabling DXVA";
}
return false;
} else {
mCorruptFrames.insert(0);
return true;
}
}
@@ -2499,14 +2505,21 @@ void MediaDecoderStateMachine::RenderVideoFrames(int32_t aMaxFrames,
TimeStamp lastFrameTime;
for (uint32_t i = 0; i < frames.Length(); ++i) {
VideoData* frame = frames[i]->As<VideoData>();
bool valid = CheckFrameValidity(frame);
frame->mSentToCompositor = true;
if (!valid) {
continue;
}
int64_t frameTime = frame->mTime;
if (frameTime < 0) {
// Frame times before the start time are invalid; drop such frames
continue;
}
TimeStamp t;
if (aMaxFrames > 1) {
MOZ_ASSERT(!aClockTimeStamp.IsNull());
@@ -2609,7 +2622,6 @@ void MediaDecoderStateMachine::UpdateRenderedVideoFrames()
VERBOSE_LOG("discarding video frame mTime=%lld clock_time=%lld",
currentFrame->mTime, clockTime);
}
CheckTurningOffHardwareDecoder(currentFrame->As<VideoData>());
currentFrame = VideoQueue().PopFront();
}
dom/media/MediaDecoderStateMachine.h
+1 -3
View File
@@ -478,7 +478,7 @@ protected:
void UpdatePlaybackPositionInternal(int64_t aTime);
// Decode monitor must be held.
void CheckTurningOffHardwareDecoder(VideoData* aData);
bool CheckFrameValidity(VideoData* aData);
// Sets VideoQueue images into the VideoFrameContainer. Called on the shared
// state machine thread. Decode monitor must be held. The first aMaxFrames
@@ -1255,8 +1255,6 @@ private:
// successeeding.
bool mDecodingFirstFrame;
bool mDisabledHardwareAcceleration;
// True if we are back from DECODER_STATE_DORMANT state and
// LoadedMetadataEvent was already sent.
bool mSentLoadedMetadataEvent;
dom/media/MediaFormatReader.cpp
+11 -4
View File
@@ -76,6 +76,7 @@ MediaFormatReader::MediaFormatReader(AbstractMediaDecoder* aDecoder,
, mSeekable(false)
, mIsEncrypted(false)
, mTrackDemuxersMayBlock(false)
, mHardwareAccelerationDisabled(false)
{
MOZ_ASSERT(aDemuxer);
MOZ_COUNT_CTOR(MediaFormatReader);
@@ -350,8 +351,11 @@ MediaFormatReader::EnsureDecodersCreated()
false);
mVideo.mDecoderInitialized = false;
// If we've disabled hardware acceleration for this reader, then we can't use
// the shared decoder.
if (mSharedDecoderManager &&
mPlatform->SupportsSharedDecoders(mInfo.mVideo)) {
mPlatform->SupportsSharedDecoders(mInfo.mVideo) &&
!mHardwareAccelerationDisabled) {
mVideo.mDecoder =
mSharedDecoderManager->CreateVideoDecoder(mPlatform,
mVideo.mInfo ?
@@ -362,13 +366,16 @@ MediaFormatReader::EnsureDecodersCreated()
mVideo.mTaskQueue,
mVideo.mCallback);
} else {
// Decoders use the layers backend to decide if they can use hardware decoding,
// so specify LAYERS_NONE if we want to forcibly disable it.
mVideo.mDecoder =
mPlatform->CreateDecoder(mVideo.mInfo ?
*mVideo.mInfo->GetAsVideoInfo() :
mInfo.mVideo,
mVideo.mTaskQueue,
mVideo.mCallback,
mLayersBackendType,
mHardwareAccelerationDisabled ? LayersBackend::LAYERS_NONE :
mLayersBackendType,
mDecoder->GetImageContainer());
}
NS_ENSURE_TRUE(mVideo.mDecoder != nullptr, false);
@@ -473,8 +480,8 @@ void
MediaFormatReader::DisableHardwareAcceleration()
{
MOZ_ASSERT(OnTaskQueue());
if (HasVideo()) {
mPlatform->DisableHardwareAcceleration();
if (HasVideo() && !mHardwareAccelerationDisabled) {
mHardwareAccelerationDisabled = true;
Flush(TrackInfo::kVideoTrack);
mVideo.mDecoder->Shutdown();
mVideo.mDecoder = nullptr;
dom/media/MediaFormatReader.h
+2
View File
@@ -394,6 +394,8 @@ private:
// Set to true if any of our track buffers may be blocking.
bool mTrackDemuxersMayBlock;
bool mHardwareAccelerationDisabled;
// Seeking objects.
bool IsSeeking() const { return mPendingSeekTime.isSome(); }
void AttemptSeek();
dom/media/MediaRecorder.cpp
+7 -6
View File
@@ -770,12 +770,13 @@ MediaRecorder::MediaRecorder(AudioNode& aSrcAudioNode,
if (aSrcAudioNode.NumberOfOutputs() > 0) {
AudioContext* ctx = aSrcAudioNode.Context();
AudioNodeEngine* engine = new AudioNodeEngine(nullptr);
mPipeStream = ctx->Graph()->CreateAudioNodeStream(engine,
MediaStreamGraph::EXTERNAL_STREAM,
ctx->SampleRate());
AudioNodeStream* ns = aSrcAudioNode.Stream();
AudioNodeStream::Flags flags =
AudioNodeStream::EXTERNAL_OUTPUT |
AudioNodeStream::NEED_MAIN_THREAD_FINISHED;
mPipeStream = AudioNodeStream::Create(ctx->Graph(), engine, flags);
AudioNodeStream* ns = aSrcAudioNode.GetStream();
if (ns) {
mInputPort = mPipeStream->AllocateInputPort(aSrcAudioNode.Stream(),
mInputPort = mPipeStream->AllocateInputPort(aSrcAudioNode.GetStream(),
MediaInputPort::FLAG_BLOCK_INPUT,
0,
aSrcOutput);
@@ -1156,7 +1157,7 @@ MediaRecorder::GetSourceMediaStream()
return mDOMStream->GetStream();
}
MOZ_ASSERT(mAudioNode != nullptr);
return mPipeStream != nullptr ? mPipeStream.get() : mAudioNode->Stream();
return mPipeStream ? mPipeStream.get() : mAudioNode->GetStream();
}
nsIPrincipal*
dom/media/MediaStreamGraph.cpp
+25 -74
View File
@@ -20,7 +20,6 @@
#include "ImageContainer.h"
#include "AudioCaptureStream.h"
#include "AudioChannelService.h"
#include "AudioNodeEngine.h"
#include "AudioNodeStream.h"
#include "AudioNodeExternalInputStream.h"
#include "mozilla/dom/AudioContextBinding.h"
@@ -106,7 +105,7 @@ MediaStreamGraphImpl::FinishStream(MediaStream* aStream)
static const GraphTime START_TIME_DELAYED = -1;
void
MediaStreamGraphImpl::AddStream(MediaStream* aStream)
MediaStreamGraphImpl::AddStreamGraphThread(MediaStream* aStream)
{
// Check if we're adding a stream to a suspended context, in which case, we
// add it to mSuspendedStreams, and delay setting mBufferStartTime
@@ -133,7 +132,7 @@ MediaStreamGraphImpl::AddStream(MediaStream* aStream)
}
void
MediaStreamGraphImpl::RemoveStream(MediaStream* aStream)
MediaStreamGraphImpl::RemoveStreamGraphThread(MediaStream* aStream)
{
// Remove references in mStreamUpdates before we allow aStream to die.
// Pending updates are not needed (since the main thread has already given
@@ -1667,7 +1666,7 @@ public:
explicit CreateMessage(MediaStream* aStream) : ControlMessage(aStream) {}
virtual void Run() override
{
mStream->GraphImpl()->AddStream(mStream);
mStream->GraphImpl()->AddStreamGraphThread(mStream);
}
virtual void RunDuringShutdown() override
{
@@ -2086,7 +2085,7 @@ MediaStream::Destroy()
mStream->RemoveAllListenersImpl();
auto graph = mStream->GraphImpl();
mStream->DestroyImpl();
graph->RemoveStream(mStream);
graph->RemoveStreamGraphThread(mStream);
}
virtual void RunDuringShutdown()
{ Run(); }
@@ -2506,24 +2505,19 @@ SourceMediaStream::ResampleAudioToGraphSampleRate(TrackData* aTrackData, MediaSe
AudioSegment* segment = static_cast<AudioSegment*>(aSegment);
int channels = segment->ChannelCount();
// If this segment is just silence, we delay instanciating the resampler.
if (channels) {
if (aTrackData->mResampler) {
MOZ_ASSERT(aTrackData->mResamplerChannelCount == segment->ChannelCount());
} else {
SpeexResamplerState* state = speex_resampler_init(channels,
aTrackData->mInputRate,
GraphImpl()->GraphRate(),
SPEEX_RESAMPLER_QUALITY_DEFAULT,
nullptr);
if (!state) {
return;
}
aTrackData->mResampler.own(state);
#ifdef DEBUG
aTrackData->mResamplerChannelCount = channels;
#endif
// If this segment is just silence, we delay instanciating the resampler. We
// also need to recreate the resampler if the channel count changes.
if (channels && aTrackData->mResamplerChannelCount != channels) {
SpeexResamplerState* state = speex_resampler_init(channels,
aTrackData->mInputRate,
GraphImpl()->GraphRate(),
SPEEX_RESAMPLER_QUALITY_MIN,
nullptr);
if (!state) {
return;
}
aTrackData->mResampler.own(state);
aTrackData->mResamplerChannelCount = channels;
}
segment->ResampleChunks(aTrackData->mResampler, aTrackData->mInputRate, GraphImpl()->GraphRate());
}
@@ -2885,7 +2879,7 @@ ProcessedMediaStream::DestroyImpl()
MediaStream::DestroyImpl();
// The stream order is only important if there are connections, in which
// case MediaInputPort::Disconnect() called SetStreamOrderDirty().
// MediaStreamGraphImpl::RemoveStream() will also call
// MediaStreamGraphImpl::RemoveStreamGraphThread() will also call
// SetStreamOrderDirty(), for other reasons.
}
@@ -3138,10 +3132,7 @@ SourceMediaStream*
MediaStreamGraph::CreateSourceStream(DOMMediaStream* aWrapper)
{
SourceMediaStream* stream = new SourceMediaStream(aWrapper);
NS_ADDREF(stream);
MediaStreamGraphImpl* graph = static_cast<MediaStreamGraphImpl*>(this);
stream->SetGraphImpl(graph);
graph->AppendMessage(new CreateMessage(stream));
AddStream(stream);
return stream;
}
@@ -3149,10 +3140,7 @@ ProcessedMediaStream*
MediaStreamGraph::CreateTrackUnionStream(DOMMediaStream* aWrapper)
{
TrackUnionStream* stream = new TrackUnionStream(aWrapper);
NS_ADDREF(stream);
MediaStreamGraphImpl* graph = static_cast<MediaStreamGraphImpl*>(this);
stream->SetGraphImpl(graph);
graph->AppendMessage(new CreateMessage(stream));
AddStream(stream);
return stream;
}
@@ -3160,54 +3148,17 @@ ProcessedMediaStream*
MediaStreamGraph::CreateAudioCaptureStream(DOMMediaStream* aWrapper)
{
AudioCaptureStream* stream = new AudioCaptureStream(aWrapper);
NS_ADDREF(stream);
MediaStreamGraphImpl* graph = static_cast<MediaStreamGraphImpl*>(this);
stream->SetGraphImpl(graph);
graph->AppendMessage(new CreateMessage(stream));
AddStream(stream);
return stream;
}
AudioNodeExternalInputStream*
MediaStreamGraph::CreateAudioNodeExternalInputStream(AudioNodeEngine* aEngine, TrackRate aSampleRate)
void
MediaStreamGraph::AddStream(MediaStream* aStream)
{
MOZ_ASSERT(NS_IsMainThread());
if (!aSampleRate) {
aSampleRate = aEngine->NodeMainThread()->Context()->SampleRate();
}
AudioNodeExternalInputStream* stream = new AudioNodeExternalInputStream(
aEngine, aSampleRate, aEngine->NodeMainThread()->Context()->Id());
NS_ADDREF(stream);
NS_ADDREF(aStream);
MediaStreamGraphImpl* graph = static_cast<MediaStreamGraphImpl*>(this);
stream->SetGraphImpl(graph);
graph->AppendMessage(new CreateMessage(stream));
return stream;
}
AudioNodeStream*
MediaStreamGraph::CreateAudioNodeStream(AudioNodeEngine* aEngine,
AudioNodeStreamKind aKind,
TrackRate aSampleRate)
{
MOZ_ASSERT(NS_IsMainThread());
if (!aSampleRate) {
aSampleRate = aEngine->NodeMainThread()->Context()->SampleRate();
}
// MediaRecorders use an AudioNodeStream, but no AudioNode
AudioNode* node = aEngine->NodeMainThread();
dom::AudioContext::AudioContextId contextIdForStream = node ? node->Context()->Id() :
NO_AUDIO_CONTEXT;
AudioNodeStream* stream = new AudioNodeStream(aEngine, aKind, aSampleRate,
contextIdForStream);
NS_ADDREF(stream);
MediaStreamGraphImpl* graph = static_cast<MediaStreamGraphImpl*>(this);
stream->SetGraphImpl(graph);
if (aEngine->HasNode()) {
stream->SetChannelMixingParametersImpl(aEngine->NodeMainThread()->ChannelCount(),
aEngine->NodeMainThread()->ChannelCountModeValue(),
aEngine->NodeMainThread()->ChannelInterpretationValue());
}
graph->AppendMessage(new CreateMessage(stream));
return stream;
aStream->SetGraphImpl(graph);
graph->AppendMessage(new CreateMessage(aStream));
}
class GraphStartedRunnable final : public nsRunnable
dom/media/MediaStreamGraph.h
+4 -18
View File
@@ -900,9 +900,7 @@ protected:
// Resampler if the rate of the input track does not match the
// MediaStreamGraph's.
nsAutoRef<SpeexResamplerState> mResampler;
#ifdef DEBUG
int mResamplerChannelCount;
#endif
StreamTime mStart;
// End-time of data already flushed to the track (excluding mData)
StreamTime mEndOfFlushedData;
@@ -1254,23 +1252,11 @@ public:
* Create a stream that will mix all its audio input.
*/
ProcessedMediaStream* CreateAudioCaptureStream(DOMMediaStream* aWrapper);
// Internal AudioNodeStreams can only pass their output to another
// AudioNode, whereas external AudioNodeStreams can pass their output
// to an nsAudioStream for playback.
enum AudioNodeStreamKind { SOURCE_STREAM, INTERNAL_STREAM, EXTERNAL_STREAM };
/**
* Create a stream that will process audio for an AudioNode.
* Takes ownership of aEngine. aSampleRate is the sampling rate used
* for the stream. If 0 is passed, the sampling rate of the engine's
* node will get used.
*/
AudioNodeStream* CreateAudioNodeStream(AudioNodeEngine* aEngine,
AudioNodeStreamKind aKind,
TrackRate aSampleRate = 0);
AudioNodeExternalInputStream*
CreateAudioNodeExternalInputStream(AudioNodeEngine* aEngine,
TrackRate aSampleRate = 0);
/**
* Add a new stream to the graph. Main thread.
*/
void AddStream(MediaStream* aStream);
/* From the main thread, ask the MSG to send back an event when the graph
* thread is running, and audio is being processed. */
dom/media/MediaStreamGraphImpl.h
+2 -2
View File
@@ -423,12 +423,12 @@ public:
/**
* Add aStream to the graph and initializes its graph-specific state.
*/
void AddStream(MediaStream* aStream);
void AddStreamGraphThread(MediaStream* aStream);
/**
* Remove aStream from the graph. Ensures that pending messages about the
* stream back to the main thread are flushed.
*/
void RemoveStream(MediaStream* aStream);
void RemoveStreamGraphThread(MediaStream* aStream);
/**
* Remove aPort from the graph and release it.
*/
dom/media/compiledtest/TestAudioPacketizer.cpp
+174
View File
@@ -0,0 +1,174 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this file,
* You can obtain one at http://mozilla.org/MPL/2.0/. */
#include <stdint.h>
#include <assert.h>
#define _USE_MATH_DEFINES
#include <math.h>
#include "../AudioPacketizer.h"
using namespace mozilla;
template<typename T>
class AutoBuffer
{
public:
explicit AutoBuffer(size_t aLength)
{
mStorage = new T[aLength];
}
~AutoBuffer() {
delete [] mStorage;
}
T* Get() {
return mStorage;
}
private:
T* mStorage;
};
int16_t Sequence(int16_t* aBuffer, uint32_t aSize, uint32_t aStart = 0)
{
uint32_t i;
for (i = 0; i < aSize; i++) {
aBuffer[i] = aStart + i;
}
return aStart + i;
}
void IsSequence(int16_t* aBuffer, uint32_t aSize, uint32_t aStart = 0)
{
for (uint32_t i = 0; i < aSize; i++) {
if (aBuffer[i] != static_cast<int64_t>(aStart + i)) {
fprintf(stderr, "Buffer is not a sequence at offset %u\n", i);
assert(false);
}
}
assert("Buffer is a sequence.");
}
void Zero(int16_t* aBuffer, uint32_t aSize)
{
for (uint32_t i = 0; i < aSize; i++) {
if (aBuffer[i] != 0) {
fprintf(stderr, "Buffer is not null at offset %u\n", i);
assert(false);
}
}
}
double sine(uint32_t aPhase) {
return sin(aPhase * 2 * M_PI * 440 / 44100);
}
int main() {
for (int16_t channels = 1; channels < 2; channels++) {
// Test that the packetizer returns zero on underrun
{
AudioPacketizer<int16_t, int16_t> ap(441, channels);
for (int16_t i = 0; i < 10; i++) {
int16_t* out = ap.Output();
Zero(out, 441);
delete out;
}
}
// Simple test, with input/output buffer size aligned on the packet size,
// alternating Input and Output calls.
{
AudioPacketizer<int16_t, int16_t> ap(441, channels);
int16_t seqEnd = 0;
for (int16_t i = 0; i < 10; i++) {
AutoBuffer<int16_t> b(441 * channels);
int16_t prevEnd = seqEnd;
seqEnd = Sequence(b.Get(), channels * 441, prevEnd);
ap.Input(b.Get(), 441);
int16_t* out = ap.Output();
IsSequence(out, 441 * channels, prevEnd);
delete out;
}
}
// Simple test, with input/output buffer size aligned on the packet size,
// alternating two Input and Output calls.
{
AudioPacketizer<int16_t, int16_t> ap(441, channels);
int16_t seqEnd = 0;
for (int16_t i = 0; i < 10; i++) {
AutoBuffer<int16_t> b(441 * channels);
AutoBuffer<int16_t> b1(441 * channels);
int16_t prevEnd0 = seqEnd;
seqEnd = Sequence(b.Get(), 441 * channels, prevEnd0);
int16_t prevEnd1 = seqEnd;
seqEnd = Sequence(b1.Get(), 441 * channels, seqEnd);
ap.Input(b.Get(), 441);
ap.Input(b1.Get(), 441);
int16_t* out = ap.Output();
int16_t* out2 = ap.Output();
IsSequence(out, 441 * channels, prevEnd0);
IsSequence(out2, 441 * channels, prevEnd1);
delete out;
delete out2;
}
}
// Input/output buffer size not aligned on the packet size,
// alternating two Input and Output calls.
{
AudioPacketizer<int16_t, int16_t> ap(441, channels);
int16_t prevEnd = 0;
int16_t prevSeq = 0;
for (int16_t i = 0; i < 10; i++) {
AutoBuffer<int16_t> b(480 * channels);
AutoBuffer<int16_t> b1(480 * channels);
prevSeq = Sequence(b.Get(), 480 * channels, prevSeq);
prevSeq = Sequence(b1.Get(), 480 * channels, prevSeq);
ap.Input(b.Get(), 480);
ap.Input(b1.Get(), 480);
int16_t* out = ap.Output();
int16_t* out2 = ap.Output();
IsSequence(out, 441 * channels, prevEnd);
prevEnd += 441 * channels;
IsSequence(out2, 441 * channels, prevEnd);
prevEnd += 441 * channels;
delete out;
delete out2;
}
printf("Available: %d\n", ap.PacketsAvailable());
}
// "Real-life" test case: streaming a sine wave through a packetizer, and
// checking that we have the right output.
// 128 is, for example, the size of a Web Audio API block, and 441 is the
// size of a webrtc.org packet when the sample rate is 44100 (10ms)
{
AudioPacketizer<int16_t, int16_t> ap(441, channels);
AutoBuffer<int16_t> b(128 * channels);
uint32_t phase = 0;
uint32_t outPhase = 0;
for (int16_t i = 0; i < 1000; i++) {
for (int32_t j = 0; j < 128; j++) {
for (int32_t c = 0; c < channels; c++) {
// int16_t sinewave at 440Hz/44100Hz sample rate
b.Get()[j * channels + c] = (2 << 14) * sine(phase);
}
phase++;
}
ap.Input(b.Get(), 128);
while (ap.PacketsAvailable()) {
int16_t* packet = ap.Output();
for (uint32_t k = 0; k < ap.PacketSize(); k++) {
for (int32_t c = 0; c < channels; c++) {
assert(packet[k * channels + c] ==
static_cast<int16_t>(((2 << 14) * sine(outPhase))));
}
outPhase++;
}
delete [] packet;
}
}
}
}
printf("OK\n");
return 0;
}
dom/media/compiledtest/TestAudioSegment.cpp
+268
View File
@@ -0,0 +1,268 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this file,
* You can obtain one at http://mozilla.org/MPL/2.0/. */
#include "AudioSegment.h"
#include <assert.h>
#include <iostream>
using namespace mozilla;
namespace mozilla {
uint32_t
GetAudioChannelsSuperset(uint32_t aChannels1, uint32_t aChannels2)
{
return std::max(aChannels1, aChannels2);
}
}
/* Helper function to give us the maximum and minimum value that don't clip,
* for a given sample format (integer or floating-point). */
template<typename T>
T GetLowValue();
template<typename T>
T GetHighValue();
template<typename T>
T GetSilentValue();
template<>
float GetLowValue<float>() {
return -1.0;
}
template<>
int16_t GetLowValue<short>() {
return -INT16_MAX;
}
template<>
float GetHighValue<float>() {
return 1.0;
}
template<>
int16_t GetHighValue<short>() {
return INT16_MAX;
}
template<>
float GetSilentValue() {
return 0.0;
}
template<>
int16_t GetSilentValue() {
return 0;
}
// Get an array of planar audio buffers that has the inverse of the index of the
// channel (1-indexed) as samples.
template<typename T>
const T* const* GetPlanarChannelArray(size_t aChannels, size_t aSize)
{
T** channels = new T*[aChannels];
for (size_t c = 0; c < aChannels; c++) {
channels[c] = new T[aSize];
for (size_t i = 0; i < aSize; i++) {
channels[c][i] = FloatToAudioSample<T>(1. / (c + 1));
}
}
return channels;
}
template<typename T>
void DeletePlanarChannelsArray(const T* const* aArrays, size_t aChannels)
{
for (size_t channel = 0; channel < aChannels; channel++) {
delete [] aArrays[channel];
}
delete [] aArrays;
}
template<typename T>
T** GetPlanarArray(size_t aChannels, size_t aSize)
{
T** channels = new T*[aChannels];
for (size_t c = 0; c < aChannels; c++) {
channels[c] = new T[aSize];
for (size_t i = 0; i < aSize; i++) {
channels[c][i] = 0.0f;
}
}
return channels;
}
template<typename T>
void DeletePlanarArray(T** aArrays, size_t aChannels)
{
for (size_t channel = 0; channel < aChannels; channel++) {
delete [] aArrays[channel];
}
delete [] aArrays;
}
// Get an array of audio samples that have the inverse of the index of the
// channel (1-indexed) as samples.
template<typename T>
const T* GetInterleavedChannelArray(size_t aChannels, size_t aSize)
{
size_t sampleCount = aChannels * aSize;
T* samples = new T[sampleCount];
for (size_t i = 0; i < sampleCount; i++) {
uint32_t channel = (i % aChannels) + 1;
samples[i] = FloatToAudioSample<T>(1. / channel);
}
return samples;
}
template<typename T>
void DeleteInterleavedChannelArray(const T* aArray)
{
delete [] aArray;
}
bool FuzzyEqual(float aLhs, float aRhs) {
return std::abs(aLhs - aRhs) < 0.01;
}
template<typename SrcT, typename DstT>
void TestInterleaveAndConvert()
{
size_t arraySize = 1024;
size_t maxChannels = 8; // 7.1
for (uint32_t channels = 1; channels < maxChannels; channels++) {
const SrcT* const* src = GetPlanarChannelArray<SrcT>(channels, arraySize);
DstT* dst = new DstT[channels * arraySize];
InterleaveAndConvertBuffer(src, arraySize, 1.0, channels, dst);
uint32_t channelIndex = 0;
for (size_t i = 0; i < arraySize * channels; i++) {
assert(FuzzyEqual(dst[i],
FloatToAudioSample<DstT>(1. / (channelIndex + 1))));
channelIndex++;
channelIndex %= channels;
}
DeletePlanarChannelsArray(src, channels);
delete [] dst;
}
}
template<typename SrcT, typename DstT>
void TestDeinterleaveAndConvert()
{
size_t arraySize = 1024;
size_t maxChannels = 8; // 7.1
for (uint32_t channels = 1; channels < maxChannels; channels++) {
const SrcT* src = GetInterleavedChannelArray<SrcT>(channels, arraySize);
DstT** dst = GetPlanarArray<DstT>(channels, arraySize);
DeinterleaveAndConvertBuffer(src, arraySize, channels, dst);
for (size_t channel = 0; channel < channels; channel++) {
for (size_t i = 0; i < arraySize; i++) {
assert(FuzzyEqual(dst[channel][i],
FloatToAudioSample<DstT>(1. / (channel + 1))));
}
}
DeleteInterleavedChannelArray(src);
DeletePlanarArray(dst, channels);
}
}
uint8_t gSilence[4096] = {0};
template<typename T>
T* SilentChannel()
{
return reinterpret_cast<T*>(gSilence);
}
template<typename T>
void TestUpmixStereo()
{
size_t arraySize = 1024;
nsTArray<T*> channels;
nsTArray<const T*> channelsptr;
channels.SetLength(1);
channelsptr.SetLength(1);
channels[0] = new T[arraySize];
for (size_t i = 0; i < arraySize; i++) {
channels[0][i] = GetHighValue<T>();
}
channelsptr[0] = channels[0];
AudioChannelsUpMix(&channelsptr, 2, ::SilentChannel<T>());
for (size_t channel = 0; channel < 2; channel++) {
for (size_t i = 0; i < arraySize; i++) {
if (channelsptr[channel][i] != GetHighValue<T>()) {
assert(false);
}
}
}
assert(true);
delete channels[0];
}
template<typename T>
void TestDownmixStereo()
{
const size_t arraySize = 1024;
nsTArray<const T*> inputptr;
nsTArray<T*> input;
T** output;
output = new T*[1];
output[0] = new T[arraySize];
input.SetLength(2);
inputptr.SetLength(2);
for (size_t channel = 0; channel < input.Length(); channel++) {
input[channel] = new T[arraySize];
for (size_t i = 0; i < arraySize; i++) {
input[channel][i] = channel == 0 ? GetLowValue<T>() : GetHighValue<T>();
}
inputptr[channel] = input[channel];
}
AudioChannelsDownMix(inputptr, output, 1, arraySize);
for (size_t i = 0; i < arraySize; i++) {
if (output[0][i] != GetSilentValue<T>()) {
assert(false);
}
}
assert(true);
delete output[0];
delete output;
}
int main(int argc, char* argv[]) {
TestInterleaveAndConvert<float, float>();
TestInterleaveAndConvert<float, int16_t>();
TestInterleaveAndConvert<int16_t, float>();
TestInterleaveAndConvert<int16_t, int16_t>();
TestDeinterleaveAndConvert<float, float>();
TestDeinterleaveAndConvert<float, int16_t>();
TestDeinterleaveAndConvert<int16_t, float>();
TestDeinterleaveAndConvert<int16_t, int16_t>();
TestUpmixStereo<float>();
TestUpmixStereo<int16_t>();
TestDownmixStereo<float>();
TestDownmixStereo<int16_t>();
return 0;
}
dom/media/compiledtest/moz.build
+7 -3
View File
@@ -6,11 +6,15 @@
GeckoCppUnitTests([
'TestAudioBuffers',
'TestAudioMixer'
'TestAudioMixer',
'TestAudioPacketizer',
'TestAudioSegment'
])
FAIL_ON_WARNINGS = True
LOCAL_INCLUDES += [
'..',
]
USE_LIBS += [
'lgpllibs',
]
dom/media/encoder/TrackEncoder.cpp
+23 -16
View File
@@ -123,9 +123,6 @@ AudioTrackEncoder::AppendAudioSegment(const AudioSegment& aSegment)
return NS_OK;
}
static const int AUDIO_PROCESSING_FRAMES = 640; /* > 10ms of 48KHz audio */
static const uint8_t gZeroChannel[MAX_AUDIO_SAMPLE_SIZE*AUDIO_PROCESSING_FRAMES] = {0};
/*static*/
void
AudioTrackEncoder::InterleaveTrackData(AudioChunk& aChunk,
@@ -133,19 +130,29 @@ AudioTrackEncoder::InterleaveTrackData(AudioChunk& aChunk,
uint32_t aOutputChannels,
AudioDataValue* aOutput)
{
if (aChunk.mChannelData.Length() < aOutputChannels) {
// Up-mix. This might make the mChannelData have more than aChannels.
AudioChannelsUpMix(&aChunk.mChannelData, aOutputChannels, gZeroChannel);
}
if (aChunk.mChannelData.Length() > aOutputChannels) {
DownmixAndInterleave(aChunk.mChannelData, aChunk.mBufferFormat, aDuration,
aChunk.mVolume, aOutputChannels, aOutput);
} else {
InterleaveAndConvertBuffer(aChunk.mChannelData.Elements(),
aChunk.mBufferFormat, aDuration, aChunk.mVolume,
aOutputChannels, aOutput);
}
switch(aChunk.mBufferFormat) {
case AUDIO_FORMAT_S16: {
nsAutoTArray<const int16_t*, 2> array;
array.SetLength(aOutputChannels);
for (uint32_t i = 0; i < array.Length(); i++) {
array[i] = static_cast<const int16_t*>(aChunk.mChannelData[i]);
}
InterleaveTrackData(array, aDuration, aOutputChannels, aOutput, aChunk.mVolume);
break;
}
case AUDIO_FORMAT_FLOAT32: {
nsAutoTArray<const float*, 2> array;
array.SetLength(aOutputChannels);
for (uint32_t i = 0; i < array.Length(); i++) {
array[i] = static_cast<const float*>(aChunk.mChannelData[i]);
}
InterleaveTrackData(array, aDuration, aOutputChannels, aOutput, aChunk.mVolume);
break;
}
case AUDIO_FORMAT_SILENCE: {
MOZ_ASSERT(false, "To implement.");
}
};
}
/*static*/
dom/media/encoder/TrackEncoder.h
+22
View File
@@ -151,6 +151,28 @@ public:
uint32_t aTrackEvents,
const MediaSegment& aQueuedMedia) override;
template<typename T>
static
void InterleaveTrackData(nsTArray<const T*>& aInput,
int32_t aDuration,
uint32_t aOutputChannels,
AudioDataValue* aOutput,
float aVolume)
{
if (aInput.Length() < aOutputChannels) {
// Up-mix. This might make the mChannelData have more than aChannels.
AudioChannelsUpMix(&aInput, aOutputChannels, SilentChannel::ZeroChannel<T>());
}
if (aInput.Length() > aOutputChannels) {
DownmixAndInterleave(aInput, aDuration,
aVolume, aOutputChannels, aOutput);
} else {
InterleaveAndConvertBuffer(aInput.Elements(), aDuration, aVolume,
aOutputChannels, aOutput);
}
}
/**
* Interleaves the track data and stores the result into aOutput. Might need
* to up-mix or down-mix the channel data if the channels number of this chunk
dom/media/moz.build
+1
View File
@@ -96,6 +96,7 @@ EXPORTS += [
'AudioChannelFormat.h',
'AudioCompactor.h',
'AudioMixer.h',
'AudioPacketizer.h',
'AudioSampleFormat.h',
'AudioSegment.h',
'AudioStream.h',
dom/media/platforms/PlatformDecoderModule.h
-2
View File
@@ -100,8 +100,6 @@ public:
// feeding it to MediaDataDecoder::Input.
virtual ConversionRequired DecoderNeedsConversion(const TrackInfo& aConfig) const = 0;
virtual void DisableHardwareAcceleration() {}
virtual bool SupportsSharedDecoders(const VideoInfo& aConfig) const {
return !AgnosticMimeType(aConfig.mMimeType);
}
dom/media/platforms/SharedDecoderManager.cpp
-7
View File
@@ -101,13 +101,6 @@ SharedDecoderManager::CreateVideoDecoder(
return proxy.forget();
}
void
SharedDecoderManager::DisableHardwareAcceleration()
{
MOZ_ASSERT(mPDM);
mPDM->DisableHardwareAcceleration();
}
bool
SharedDecoderManager::Recreate(const VideoInfo& aConfig)
{
dom/media/platforms/SharedDecoderManager.h
-1
View File
@@ -41,7 +41,6 @@ public:
friend class SharedDecoderProxy;
friend class SharedDecoderCallback;
void DisableHardwareAcceleration();
bool Recreate(const VideoInfo& aConfig);
private:
dom/media/platforms/wmf/DXVA2Manager.cpp
+135
View File
@@ -62,14 +62,120 @@ public:
ImageContainer* aContainer,
Image** aOutImage) override;
virtual bool SupportsConfig(IMFMediaType* aType) override;
private:
nsRefPtr<IDirect3D9Ex> mD3D9;
nsRefPtr<IDirect3DDevice9Ex> mDevice;
nsRefPtr<IDirect3DDeviceManager9> mDeviceManager;
RefPtr<D3D9RecycleAllocator> mTextureClientAllocator;
nsRefPtr<IDirectXVideoDecoderService> mDecoderService;
UINT32 mResetToken;
};
void GetDXVA2ExtendedFormatFromMFMediaType(IMFMediaType *pType,
DXVA2_ExtendedFormat *pFormat)
{
// Get the interlace mode.
MFVideoInterlaceMode interlace =
(MFVideoInterlaceMode)MFGetAttributeUINT32(pType, MF_MT_INTERLACE_MODE, MFVideoInterlace_Unknown);
if (interlace == MFVideoInterlace_MixedInterlaceOrProgressive) {
pFormat->SampleFormat = DXVA2_SampleFieldInterleavedEvenFirst;
} else {
pFormat->SampleFormat = (UINT)interlace;
}
pFormat->VideoChromaSubsampling =
MFGetAttributeUINT32(pType, MF_MT_VIDEO_CHROMA_SITING, MFVideoChromaSubsampling_Unknown);
pFormat->NominalRange =
MFGetAttributeUINT32(pType, MF_MT_VIDEO_NOMINAL_RANGE, MFNominalRange_Unknown);
pFormat->VideoTransferMatrix =
MFGetAttributeUINT32(pType, MF_MT_YUV_MATRIX, MFVideoTransferMatrix_Unknown);
pFormat->VideoLighting =
MFGetAttributeUINT32(pType, MF_MT_VIDEO_LIGHTING, MFVideoLighting_Unknown);
pFormat->VideoPrimaries =
MFGetAttributeUINT32(pType, MF_MT_VIDEO_PRIMARIES, MFVideoPrimaries_Unknown);
pFormat->VideoTransferFunction =
MFGetAttributeUINT32(pType, MF_MT_TRANSFER_FUNCTION, MFVideoTransFunc_Unknown);
}
HRESULT ConvertMFTypeToDXVAType(IMFMediaType *pType, DXVA2_VideoDesc *pDesc)
{
ZeroMemory(pDesc, sizeof(*pDesc));
// The D3D format is the first DWORD of the subtype GUID.
GUID subtype = GUID_NULL;
HRESULT hr = pType->GetGUID(MF_MT_SUBTYPE, &subtype);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
pDesc->Format = (D3DFORMAT)subtype.Data1;
UINT32 width = 0;
UINT32 height = 0;
hr = MFGetAttributeSize(pType, MF_MT_FRAME_SIZE, &width, &height);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
pDesc->SampleWidth = width;
pDesc->SampleHeight = height;
UINT32 fpsNumerator = 0;
UINT32 fpsDenominator = 0;
hr = MFGetAttributeRatio(pType, MF_MT_FRAME_RATE, &fpsNumerator, &fpsDenominator);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
pDesc->InputSampleFreq.Numerator = fpsNumerator;
pDesc->InputSampleFreq.Denominator = fpsDenominator;
GetDXVA2ExtendedFormatFromMFMediaType(pType, &pDesc->SampleFormat);
pDesc->OutputFrameFreq = pDesc->InputSampleFreq;
if ((pDesc->SampleFormat.SampleFormat == DXVA2_SampleFieldInterleavedEvenFirst) ||
(pDesc->SampleFormat.SampleFormat == DXVA2_SampleFieldInterleavedOddFirst)) {
pDesc->OutputFrameFreq.Numerator *= 2;
}
return S_OK;
}
static const GUID DXVA2_ModeH264_E = {
0x1b81be68, 0xa0c7, 0x11d3, { 0xb9, 0x84, 0x00, 0xc0, 0x4f, 0x2e, 0x73, 0xc5 }
};
// This tests if a DXVA video decoder can be created for the given media type/resolution.
// It uses the same decoder device (DXVA2_ModeH264_E - DXVA2_ModeH264_VLD_NoFGT) as the H264
// decoder MFT provided by windows (CLSID_CMSH264DecoderMFT) uses, so we can use it to determine
// if the MFT will use software fallback or not.
bool
D3D9DXVA2Manager::SupportsConfig(IMFMediaType* aType)
{
DXVA2_VideoDesc desc;
HRESULT hr = ConvertMFTypeToDXVAType(aType, &desc);
NS_ENSURE_TRUE(SUCCEEDED(hr), false);
UINT configCount;
DXVA2_ConfigPictureDecode* configs = nullptr;
hr = mDecoderService->GetDecoderConfigurations(DXVA2_ModeH264_E, &desc, nullptr, &configCount, &configs);
NS_ENSURE_TRUE(SUCCEEDED(hr), false);
nsRefPtr<IDirect3DSurface9> surface;
hr = mDecoderService->CreateSurface(desc.SampleWidth, desc.SampleHeight, 0, (D3DFORMAT)MAKEFOURCC('N', 'V', '1', '2'),
D3DPOOL_DEFAULT, 0, DXVA2_VideoDecoderRenderTarget,
surface.StartAssignment(), NULL);
if (!SUCCEEDED(hr)) {
CoTaskMemFree(configs);
return false;
}
for (UINT i = 0; i < configCount; i++) {
nsRefPtr<IDirectXVideoDecoder> decoder;
IDirect3DSurface9* surfaces = surface;
hr = mDecoderService->CreateVideoDecoder(DXVA2_ModeH264_E, &desc, &configs[i], &surfaces, 1, decoder.StartAssignment());
if (SUCCEEDED(hr) && decoder) {
CoTaskMemFree(configs);
return true;
}
}
CoTaskMemFree(configs);
return false;
}
D3D9DXVA2Manager::D3D9DXVA2Manager()
: mResetToken(0)
{
@@ -180,6 +286,35 @@ D3D9DXVA2Manager::Init(nsACString& aFailureReason)
return hr;
}
HANDLE deviceHandle;
nsRefPtr<IDirectXVideoDecoderService> decoderService;
hr = deviceManager->OpenDeviceHandle(&deviceHandle);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
hr = deviceManager->GetVideoService(deviceHandle, IID_PPV_ARGS(decoderService.StartAssignment()));
deviceManager->CloseDeviceHandle(deviceHandle);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
UINT deviceCount;
GUID* decoderDevices = nullptr;
hr = decoderService->GetDecoderDeviceGuids(&deviceCount, &decoderDevices);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
bool found = false;
for (UINT i = 0; i < deviceCount; i++) {
if (decoderDevices[i] == DXVA2_ModeH264_E) {
found = true;
break;
}
}
CoTaskMemFree(decoderDevices);
if (!found) {
return E_FAIL;
}
mDecoderService = decoderService;
mResetToken = resetToken;
mD3D9 = d3d9Ex;
mDevice = device;
dom/media/platforms/wmf/DXVA2Manager.h
+2
View File
@@ -44,6 +44,8 @@ public:
virtual ~DXVA2Manager();
virtual bool SupportsConfig(IMFMediaType* aType) { return true; }
protected:
Mutex mLock;
DXVA2Manager();
dom/media/platforms/wmf/MFTDecoder.cpp
-4
View File
@@ -180,10 +180,6 @@ MFTDecoder::CreateOutputSample(RefPtr<IMFSample>* aOutSample)
hr = wmf::MFCreateAlignedMemoryBuffer(bufferSize, alignment, byRef(buffer));
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
DWORD maxLength = 0;
DWORD currentLength = 0;
BYTE* dst = nullptr;
hr = sample->AddBuffer(buffer);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
dom/media/platforms/wmf/MFTDecoder.h
+6 -5
View File
@@ -59,6 +59,11 @@ public:
int64_t aTimestampUsecs);
HRESULT Input(IMFSample* aSample);
HRESULT CreateInputSample(const uint8_t* aData,
uint32_t aDataSize,
int64_t aTimestampUsecs,
RefPtr<IMFSample>* aOutSample);
// Retrieves output from the MFT. Call this once Input() returns
// MF_E_NOTACCEPTING. Some MFTs with hardware acceleration (the H.264
// decoder MFT in particular) can't handle it if clients hold onto
@@ -80,14 +85,10 @@ public:
// Sends a message to the MFT.
HRESULT SendMFTMessage(MFT_MESSAGE_TYPE aMsg, ULONG_PTR aData);
private:
HRESULT SetDecoderOutputType(ConfigureOutputCallback aCallback, void* aData);
private:
HRESULT CreateInputSample(const uint8_t* aData,
uint32_t aDataSize,
int64_t aTimestampUsecs,
RefPtr<IMFSample>* aOutSample);
HRESULT CreateOutputSample(RefPtr<IMFSample>* aOutSample);
dom/media/platforms/wmf/WMFDecoderModule.cpp
+3 -39
View File
@@ -42,13 +42,6 @@ WMFDecoderModule::~WMFDecoderModule()
}
}
void
WMFDecoderModule::DisableHardwareAcceleration()
{
sDXVAEnabled = false;
sIsIntelDecoderEnabled = false;
}
static void
SetNumOfDecoderThreads()
{
@@ -99,11 +92,11 @@ WMFDecoderModule::CreateVideoDecoder(const VideoInfo& aConfig,
FlushableTaskQueue* aVideoTaskQueue,
MediaDataDecoderCallback* aCallback)
{
nsAutoPtr<WMFVideoMFTManager> manager =
nsAutoPtr<WMFVideoMFTManager> manager(
new WMFVideoMFTManager(aConfig,
aLayersBackend,
aImageContainer,
sDXVAEnabled && ShouldUseDXVA(aConfig));
sDXVAEnabled));
nsRefPtr<MFTDecoder> mft = manager->Init();
@@ -122,7 +115,7 @@ WMFDecoderModule::CreateAudioDecoder(const AudioInfo& aConfig,
FlushableTaskQueue* aAudioTaskQueue,
MediaDataDecoderCallback* aCallback)
{
nsAutoPtr<WMFAudioMFTManager> manager = new WMFAudioMFTManager(aConfig);
nsAutoPtr<WMFAudioMFTManager> manager(new WMFAudioMFTManager(aConfig));
nsRefPtr<MFTDecoder> mft = manager->Init();
if (!mft) {
@@ -134,35 +127,6 @@ WMFDecoderModule::CreateAudioDecoder(const AudioInfo& aConfig,
return decoder.forget();
}
bool
WMFDecoderModule::ShouldUseDXVA(const VideoInfo& aConfig) const
{
static bool isAMD = false;
static bool initialized = false;
if (!initialized) {
nsCOMPtr<nsIGfxInfo> gfxInfo = services::GetGfxInfo();
nsAutoString vendor;
gfxInfo->GetAdapterVendorID(vendor);
isAMD = vendor.Equals(widget::GfxDriverInfo::GetDeviceVendor(widget::VendorAMD), nsCaseInsensitiveStringComparator()) ||
vendor.Equals(widget::GfxDriverInfo::GetDeviceVendor(widget::VendorATI), nsCaseInsensitiveStringComparator());
initialized = true;
}
if (!isAMD) {
return true;
}
// Don't use DXVA for 4k videos or above, since it seems to perform poorly.
return aConfig.mDisplay.width <= 1920 && aConfig.mDisplay.height <= 1200;
}
bool
WMFDecoderModule::SupportsSharedDecoders(const VideoInfo& aConfig) const
{
// If DXVA is enabled, but we're not going to use it for this specific config, then
// we can't use the shared decoder.
return !AgnosticMimeType(aConfig.mMimeType) &&
(!sDXVAEnabled || ShouldUseDXVA(aConfig));
}
bool
WMFDecoderModule::SupportsMimeType(const nsACString& aMimeType)
{
dom/media/platforms/wmf/WMFDecoderModule.h
-4
View File
@@ -33,9 +33,6 @@ public:
bool SupportsMimeType(const nsACString& aMimeType) override;
virtual void DisableHardwareAcceleration() override;
virtual bool SupportsSharedDecoders(const VideoInfo& aConfig) const override;
virtual ConversionRequired
DecoderNeedsConversion(const TrackInfo& aConfig) const override;
@@ -52,7 +49,6 @@ public:
// Called from any thread, must call init first
static int GetNumDecoderThreads();
private:
bool ShouldUseDXVA(const VideoInfo& aConfig) const;
bool mWMFInitialized;
};
dom/media/platforms/wmf/WMFMediaDataDecoder.cpp
+12
View File
@@ -66,6 +66,9 @@ WMFMediaDataDecoder::ProcessShutdown()
if (mMFTManager) {
mMFTManager->Shutdown();
mMFTManager = nullptr;
if (!mRecordedError && mHasSuccessfulOutput) {
//SendTelemetry(S_OK);
}
}
mDecoder = nullptr;
}
@@ -101,6 +104,10 @@ WMFMediaDataDecoder::ProcessDecode(MediaRawData* aSample)
if (FAILED(hr)) {
NS_WARNING("MFTManager rejected sample");
mCallback->Error();
if (!mRecordedError) {
//SendTelemetry(hr);
mRecordedError = true;
}
return;
}
@@ -116,6 +123,7 @@ WMFMediaDataDecoder::ProcessOutput()
HRESULT hr = S_OK;
while (SUCCEEDED(hr = mMFTManager->Output(mLastStreamOffset, output)) &&
output) {
mHasSuccessfulOutput = true;
mCallback->Output(output);
}
if (hr == MF_E_TRANSFORM_NEED_MORE_INPUT) {
@@ -125,6 +133,10 @@ WMFMediaDataDecoder::ProcessOutput()
} else if (FAILED(hr)) {
NS_WARNING("WMFMediaDataDecoder failed to output data");
mCallback->Error();
if (!mRecordedError) {
//SendTelemetry(hr);
mRecordedError = true;
}
}
}
dom/media/platforms/wmf/WMFMediaDataDecoder.h
+4
View File
@@ -112,6 +112,10 @@ private:
bool mIsFlushing;
bool mIsShutDown;
// For telemetry
bool mHasSuccessfulOutput = false;
bool mRecordedError = false;
};
} // namespace mozilla
dom/media/platforms/wmf/WMFVideoMFTManager.cpp
+110 -36
View File
@@ -211,9 +211,16 @@ WMFVideoMFTManager::InitInternal(bool aForceD3D9)
RefPtr<IMFAttributes> attr(decoder->GetAttributes());
UINT32 aware = 0;
if (attr) {
attr->GetUINT32(MF_SA_D3D_AWARE, &aware);
attr->SetUINT32(CODECAPI_AVDecNumWorkerThreads,
WMFDecoderModule::GetNumDecoderThreads());
attr->GetUINT32(MF_SA_D3D_AWARE, &aware);
attr->SetUINT32(CODECAPI_AVDecNumWorkerThreads,
WMFDecoderModule::GetNumDecoderThreads());
hr = attr->SetUINT32(CODECAPI_AVLowLatencyMode, TRUE);
if (SUCCEEDED(hr)) {
LOG("Enabling Low Latency Mode");
}
else {
LOG("Couldn't enable Low Latency Mode");
}
}
if (useDxva) {
@@ -227,42 +234,19 @@ WMFVideoMFTManager::InitInternal(bool aForceD3D9)
if (SUCCEEDED(hr)) {
mUseHwAccel = true;
} else {
mDXVA2Manager = nullptr;
mDXVAFailureReason = nsPrintfCString("MFT_MESSAGE_SET_D3D_MANAGER failed with code %X", hr);
}
} else {
}
else {
mDXVAFailureReason.AssignLiteral("Decoder returned false for MF_SA_D3D_AWARE");
}
}
// Setup the input/output media types.
RefPtr<IMFMediaType> inputType;
hr = wmf::MFCreateMediaType(byRef(inputType));
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
hr = inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
hr = inputType->SetGUID(MF_MT_SUBTYPE, GetMediaSubtypeGUID());
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
hr = inputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_MixedInterlaceOrProgressive);
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
RefPtr<IMFMediaType> outputType;
hr = wmf::MFCreateMediaType(byRef(outputType));
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
hr = outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
GUID outputSubType = mUseHwAccel ? MFVideoFormat_NV12 : MFVideoFormat_YV12;
hr = outputType->SetGUID(MF_MT_SUBTYPE, outputSubType);
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
hr = decoder->SetMediaTypes(inputType, outputType);
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
mDecoder = decoder;
hr = SetDecoderMediaTypes();
NS_ENSURE_TRUE(SUCCEEDED(hr), nullptr);
LOG("Video Decoder initialized, Using DXVA: %s", (mUseHwAccel ? "Yes" : "No"));
// Just in case ConfigureVideoFrameGeometry() does not set these
@@ -275,6 +259,37 @@ WMFVideoMFTManager::InitInternal(bool aForceD3D9)
return decoder.forget();
}
HRESULT
WMFVideoMFTManager::SetDecoderMediaTypes()
{
// Setup the input/output media types.
RefPtr<IMFMediaType> inputType;
HRESULT hr = wmf::MFCreateMediaType(byRef(inputType));
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
hr = inputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
hr = inputType->SetGUID(MF_MT_SUBTYPE, GetMediaSubtypeGUID());
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
hr = inputType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_MixedInterlaceOrProgressive);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
RefPtr<IMFMediaType> outputType;
hr = wmf::MFCreateMediaType(byRef(outputType));
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
hr = outputType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
GUID outputSubType = mUseHwAccel ? MFVideoFormat_NV12 : MFVideoFormat_YV12;
hr = outputType->SetGUID(MF_MT_SUBTYPE, outputSubType);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
return mDecoder->SetMediaTypes(inputType, outputType);
}
HRESULT
WMFVideoMFTManager::Input(MediaRawData* aSample)
{
@@ -282,10 +297,55 @@ WMFVideoMFTManager::Input(MediaRawData* aSample)
// This can happen during shutdown.
return E_FAIL;
}
HRESULT hr = mDecoder->CreateInputSample(aSample->Data(),
uint32_t(aSample->Size()),
aSample->mTime,
&mLastInput);
NS_ENSURE_TRUE(SUCCEEDED(hr) && mLastInput != nullptr, hr);
// Forward sample data to the decoder.
return mDecoder->Input(aSample->Data(),
uint32_t(aSample->Size()),
aSample->mTime);
return mDecoder->Input(mLastInput);
}
// The MFTransform we use for decoding h264 video will silently fall
// back to software decoding (even if we've negotiated DXVA) if the GPU
// doesn't support decoding the given resolution. It will then upload
// the software decoded frames into d3d textures to preserve behaviour.
//
// Unfortunately this seems to cause corruption (see bug 1193547) and is
// slow because the upload is done into a non-shareable texture and requires
// us to copy it.
//
// This code tests if the given resolution can be supported directly on the GPU,
// and makes sure we only ask the MFT for DXVA if it can be supported properly.
bool
WMFVideoMFTManager::MaybeToggleDXVA(IMFMediaType* aType)
{
// SupportsConfig only checks for valid h264 decoders currently.
if (!mDXVA2Manager || mStreamType != H264) {
return false;
}
if (mDXVA2Manager->SupportsConfig(aType)) {
if (!mUseHwAccel) {
// DXVA disabled, but supported for this resolution
ULONG_PTR manager = ULONG_PTR(mDXVA2Manager->GetDXVADeviceManager());
HRESULT hr = mDecoder->SendMFTMessage(MFT_MESSAGE_SET_D3D_MANAGER, manager);
if (SUCCEEDED(hr)) {
mUseHwAccel = true;
return true;
}
}
} else if (mUseHwAccel) {
// DXVA enabled, and not supported for this resolution
HRESULT hr = mDecoder->SendMFTMessage(MFT_MESSAGE_SET_D3D_MANAGER, 0);
MOZ_ASSERT(SUCCEEDED(hr), "Attempting to fall back to software failed?");
mUseHwAccel = false;
return true;
}
return false;
}
HRESULT
@@ -295,6 +355,20 @@ WMFVideoMFTManager::ConfigureVideoFrameGeometry()
HRESULT hr = mDecoder->GetOutputMediaType(mediaType);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
// If we enabled/disabled DXVA in response to a resolution
// change then we need to renegotiate our media types,
// and resubmit our previous frame (since the MFT appears
// to lose it otherwise).
if (MaybeToggleDXVA(mediaType)) {
hr = SetDecoderMediaTypes();
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
HRESULT hr = mDecoder->GetOutputMediaType(mediaType);
NS_ENSURE_TRUE(SUCCEEDED(hr), hr);
mDecoder->Input(mLastInput);
}
// Verify that the video subtype is what we expect it to be.
// When using hardware acceleration/DXVA2 the video format should
// be NV12, which is DXVA2's preferred format. For software decoding
@@ -527,7 +601,7 @@ WMFVideoMFTManager::Output(int64_t aStreamOffset,
}
// Else unexpected error, assert, and bail.
NS_WARNING("WMFVideoMFTManager::Output() unexpected error");
return E_FAIL;
return hr;
}
nsRefPtr<VideoData> frame;
dom/media/platforms/wmf/WMFVideoMFTManager.h
+6
View File
@@ -56,6 +56,10 @@ private:
int64_t aStreamOffset,
VideoData** aOutVideoData);
HRESULT SetDecoderMediaTypes();
bool MaybeToggleDXVA(IMFMediaType* aType);
// Video frame geometry.
VideoInfo mVideoInfo;
uint32_t mVideoStride;
@@ -67,6 +71,8 @@ private:
RefPtr<layers::ImageContainer> mImageContainer;
nsAutoPtr<DXVA2Manager> mDXVA2Manager;
RefPtr<IMFSample> mLastInput;
const bool mDXVAEnabled;
const layers::LayersBackend mLayersBackend;
bool mUseHwAccel;
dom/media/webaudio/AnalyserNode.cpp
+3 -2
View File
@@ -85,8 +85,9 @@ AnalyserNode::AnalyserNode(AudioContext* aContext)
, mMaxDecibels(-30.)
, mSmoothingTimeConstant(.8)
{
mStream = aContext->Graph()->CreateAudioNodeStream(new AnalyserNodeEngine(this),
MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(),
new AnalyserNodeEngine(this),
AudioNodeStream::NO_STREAM_FLAGS);
// Enough chunks must be recorded to handle the case of fftSize being
// increased to maximum immediately before getFloatTimeDomainData() is
dom/media/webaudio/AudioBufferSourceNode.cpp
+2 -1
View File
@@ -557,7 +557,8 @@ AudioBufferSourceNode::AudioBufferSourceNode(AudioContext* aContext)
, mStartCalled(false)
{
AudioBufferSourceNodeEngine* engine = new AudioBufferSourceNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::SOURCE_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NEED_MAIN_THREAD_FINISHED);
engine->SetSourceStream(mStream);
mStream->AddMainThreadListener(this);
}
dom/media/webaudio/AudioDestinationNode.cpp
+5 -1
View File
@@ -333,7 +333,11 @@ AudioDestinationNode::AudioDestinationNode(AudioContext* aContext,
aLength, aSampleRate) :
static_cast<AudioNodeEngine*>(new DestinationNodeEngine(this));
mStream = graph->CreateAudioNodeStream(engine, MediaStreamGraph::EXTERNAL_STREAM);
AudioNodeStream::Flags flags =
AudioNodeStream::NEED_MAIN_THREAD_CURRENT_TIME |
AudioNodeStream::NEED_MAIN_THREAD_FINISHED |
AudioNodeStream::EXTERNAL_OUTPUT;
mStream = AudioNodeStream::Create(graph, engine, flags);
mStream->AddMainThreadListener(this);
mStream->AddAudioOutput(&gWebAudioOutputKey);
dom/media/webaudio/AudioDestinationNode.h
+3
View File
@@ -47,6 +47,9 @@ public:
virtual void SetChannelCount(uint32_t aChannelCount,
ErrorResult& aRv) override;
// Returns the stream or null after unlink.
AudioNodeStream* Stream() { return mStream; }
void Mute();
void Unmute();
dom/media/webaudio/AudioNode.h
+2 -1
View File
@@ -167,7 +167,8 @@ public:
uint32_t mOutputPort;
};
AudioNodeStream* Stream() { return mStream; }
// Returns the stream, if any.
AudioNodeStream* GetStream() { return mStream; }
const nsTArray<InputNode>& InputNodes() const
{
dom/media/webaudio/AudioNodeExternalInputStream.cpp
+36 -23
View File
@@ -13,7 +13,7 @@ using namespace mozilla::dom;
namespace mozilla {
AudioNodeExternalInputStream::AudioNodeExternalInputStream(AudioNodeEngine* aEngine, TrackRate aSampleRate, uint32_t aContextId)
: AudioNodeStream(aEngine, MediaStreamGraph::INTERNAL_STREAM, aSampleRate, aContextId)
: AudioNodeStream(aEngine, NO_STREAM_FLAGS, aSampleRate, aContextId)
{
MOZ_COUNT_CTOR(AudioNodeExternalInputStream);
}
@@ -23,50 +23,52 @@ AudioNodeExternalInputStream::~AudioNodeExternalInputStream()
MOZ_COUNT_DTOR(AudioNodeExternalInputStream);
}
/* static */ already_AddRefed<AudioNodeExternalInputStream>
AudioNodeExternalInputStream::Create(MediaStreamGraph* aGraph,
AudioNodeEngine* aEngine)
{
MOZ_ASSERT(NS_IsMainThread());
MOZ_ASSERT(aGraph->GraphRate() == aEngine->NodeMainThread()->Context()->SampleRate());
nsRefPtr<AudioNodeExternalInputStream> stream =
new AudioNodeExternalInputStream(aEngine, aGraph->GraphRate(),
aEngine->NodeMainThread()->Context()->Id());
aGraph->AddStream(stream);
return stream.forget();
}
/**
* Copies the data in aInput to aOffsetInBlock within aBlock.
* aBlock must have been allocated with AllocateInputBlock and have a channel
* count that's a superset of the channels in aInput.
*/
template <typename T>
static void
CopyChunkToBlock(const AudioChunk& aInput, AudioChunk *aBlock,
CopyChunkToBlock(AudioChunk& aInput, AudioChunk *aBlock,
uint32_t aOffsetInBlock)
{
uint32_t blockChannels = aBlock->ChannelCount();
nsAutoTArray<const void*,2> channels;
nsAutoTArray<const T*,2> channels;
if (aInput.IsNull()) {
channels.SetLength(blockChannels);
PodZero(channels.Elements(), blockChannels);
} else {
channels.SetLength(aInput.ChannelCount());
PodCopy(channels.Elements(), aInput.mChannelData.Elements(), channels.Length());
const nsTArray<const T*>& inputChannels = aInput.ChannelData<T>();
channels.SetLength(inputChannels.Length());
PodCopy(channels.Elements(), inputChannels.Elements(), channels.Length());
if (channels.Length() != blockChannels) {
// We only need to upmix here because aBlock's channel count has been
// chosen to be a superset of the channel count of every chunk.
AudioChannelsUpMix(&channels, blockChannels, nullptr);
AudioChannelsUpMix(&channels, blockChannels, static_cast<T*>(nullptr));
}
}
uint32_t duration = aInput.GetDuration();
for (uint32_t c = 0; c < blockChannels; ++c) {
float* outputData = aBlock->ChannelFloatsForWrite(c) + aOffsetInBlock;
if (channels[c]) {
switch (aInput.mBufferFormat) {
case AUDIO_FORMAT_FLOAT32:
ConvertAudioSamplesWithScale(
static_cast<const float*>(channels[c]), outputData, duration,
aInput.mVolume);
break;
case AUDIO_FORMAT_S16:
ConvertAudioSamplesWithScale(
static_cast<const int16_t*>(channels[c]), outputData, duration,
aInput.mVolume);
break;
default:
NS_ERROR("Unhandled format");
}
ConvertAudioSamplesWithScale(channels[c], outputData, aInput.GetDuration(), aInput.mVolume);
} else {
PodZero(outputData, duration);
PodZero(outputData, aInput.GetDuration());
}
}
}
@@ -97,7 +99,18 @@ static void ConvertSegmentToAudioBlock(AudioSegment* aSegment,
uint32_t duration = 0;
for (AudioSegment::ChunkIterator ci(*aSegment); !ci.IsEnded(); ci.Next()) {
CopyChunkToBlock(*ci, aBlock, duration);
switch (ci->mBufferFormat) {
case AUDIO_FORMAT_S16: {
CopyChunkToBlock<int16_t>(*ci, aBlock, duration);
break;
}
case AUDIO_FORMAT_FLOAT32: {
CopyChunkToBlock<float>(*ci, aBlock, duration);
break;
}
case AUDIO_FORMAT_SILENCE:
break;
}
duration += ci->GetDuration();
}
}
dom/media/webaudio/AudioNodeExternalInputStream.h
+4 -1
View File
@@ -21,9 +21,12 @@ namespace mozilla {
class AudioNodeExternalInputStream final : public AudioNodeStream
{
public:
static already_AddRefed<AudioNodeExternalInputStream>
Create(MediaStreamGraph* aGraph, AudioNodeEngine* aEngine);
protected:
AudioNodeExternalInputStream(AudioNodeEngine* aEngine, TrackRate aSampleRate,
uint32_t aContextId);
protected:
~AudioNodeExternalInputStream();
public:
dom/media/webaudio/AudioNodeStream.cpp
+33 -7
View File
@@ -26,14 +26,14 @@ namespace mozilla {
*/
AudioNodeStream::AudioNodeStream(AudioNodeEngine* aEngine,
MediaStreamGraph::AudioNodeStreamKind aKind,
Flags aFlags,
TrackRate aSampleRate,
AudioContext::AudioContextId aContextId)
: ProcessedMediaStream(nullptr),
mEngine(aEngine),
mSampleRate(aSampleRate),
mAudioContextId(aContextId),
mKind(aKind),
mFlags(aFlags),
mNumberOfInputChannels(2),
mMarkAsFinishedAfterThisBlock(false),
mAudioParamStream(false),
@@ -53,6 +53,30 @@ AudioNodeStream::~AudioNodeStream()
MOZ_COUNT_DTOR(AudioNodeStream);
}
/* static */ already_AddRefed<AudioNodeStream>
AudioNodeStream::Create(MediaStreamGraph* aGraph, AudioNodeEngine* aEngine,
Flags aFlags)
{
MOZ_ASSERT(NS_IsMainThread());
// MediaRecorders use an AudioNodeStream, but no AudioNode
AudioNode* node = aEngine->NodeMainThread();
MOZ_ASSERT(!node || aGraph->GraphRate() == node->Context()->SampleRate());
dom::AudioContext::AudioContextId contextIdForStream = node ? node->Context()->Id() :
NO_AUDIO_CONTEXT;
nsRefPtr<AudioNodeStream> stream =
new AudioNodeStream(aEngine, aFlags, aGraph->GraphRate(),
contextIdForStream);
if (aEngine->HasNode()) {
stream->SetChannelMixingParametersImpl(aEngine->NodeMainThread()->ChannelCount(),
aEngine->NodeMainThread()->ChannelCountModeValue(),
aEngine->NodeMainThread()->ChannelInterpretationValue());
}
aGraph->AddStream(stream);
return stream.forget();
}
size_t
AudioNodeStream::SizeOfExcludingThis(MallocSizeOf aMallocSizeOf) const
{
@@ -406,7 +430,7 @@ AudioNodeStream::AccumulateInputChunk(uint32_t aInputIndex, const AudioChunk& aC
AudioChunk* aBlock,
nsTArray<float>* aDownmixBuffer)
{
nsAutoTArray<const void*,GUESS_AUDIO_CHANNELS> channels;
nsAutoTArray<const float*,GUESS_AUDIO_CHANNELS> channels;
UpMixDownMixChunk(&aChunk, aBlock->mChannelData.Length(), channels, *aDownmixBuffer);
for (uint32_t c = 0; c < channels.Length(); ++c) {
@@ -429,15 +453,17 @@ AudioNodeStream::AccumulateInputChunk(uint32_t aInputIndex, const AudioChunk& aC
void
AudioNodeStream::UpMixDownMixChunk(const AudioChunk* aChunk,
uint32_t aOutputChannelCount,
nsTArray<const void*>& aOutputChannels,
nsTArray<const float*>& aOutputChannels,
nsTArray<float>& aDownmixBuffer)
{
static const float silenceChannel[WEBAUDIO_BLOCK_SIZE] = {0.f};
aOutputChannels.AppendElements(aChunk->mChannelData);
for (uint32_t i = 0; i < aChunk->mChannelData.Length(); i++) {
aOutputChannels.AppendElement(static_cast<const float*>(aChunk->mChannelData[i]));
}
if (aOutputChannels.Length() < aOutputChannelCount) {
if (mChannelInterpretation == ChannelInterpretation::Speakers) {
AudioChannelsUpMix(&aOutputChannels, aOutputChannelCount, nullptr);
AudioChannelsUpMix(&aOutputChannels, aOutputChannelCount, SilentChannel::ZeroChannel<float>());
NS_ASSERTION(aOutputChannelCount == aOutputChannels.Length(),
"We called GetAudioChannelsSuperset to avoid this");
} else {
@@ -571,7 +597,7 @@ AudioNodeStream::AdvanceOutputSegment()
StreamBuffer::Track* track = EnsureTrack(AUDIO_TRACK);
AudioSegment* segment = track->Get<AudioSegment>();
if (mKind == MediaStreamGraph::EXTERNAL_STREAM) {
if (mFlags & EXTERNAL_OUTPUT) {
segment->AppendAndConsumeChunk(&mLastChunks[0]);
} else {
segment->AppendNullData(mLastChunks[0].GetDuration());
dom/media/webaudio/AudioNodeStream.h
+24 -7
View File
@@ -43,15 +43,33 @@ public:
typedef nsAutoTArray<AudioChunk, 1> OutputChunks;
// Flags re main thread updates and stream output.
typedef unsigned Flags;
enum : Flags {
NO_STREAM_FLAGS = 0U,
NEED_MAIN_THREAD_FINISHED = 1U << 0,
NEED_MAIN_THREAD_CURRENT_TIME = 1U << 1,
// Internal AudioNodeStreams can only pass their output to another
// AudioNode, whereas external AudioNodeStreams can pass their output
// to other ProcessedMediaStreams or hardware audio output.
EXTERNAL_OUTPUT = 1U << 2,
};
/**
* Create a stream that will process audio for an AudioNode.
* Takes ownership of aEngine.
*/
static already_AddRefed<AudioNodeStream>
Create(MediaStreamGraph* aGraph, AudioNodeEngine* aEngine, Flags aKind);
protected:
/**
* Transfers ownership of aEngine to the new AudioNodeStream.
*/
AudioNodeStream(AudioNodeEngine* aEngine,
MediaStreamGraph::AudioNodeStreamKind aKind,
Flags aFlags,
TrackRate aSampleRate,
AudioContext::AudioContextId aContextId);
protected:
~AudioNodeStream();
public:
@@ -119,9 +137,8 @@ public:
}
virtual bool MainThreadNeedsUpdates() const override
{
// Only source and external streams need updates on the main thread.
return (mKind == MediaStreamGraph::SOURCE_STREAM && mFinished) ||
mKind == MediaStreamGraph::EXTERNAL_STREAM;
return ((mFlags & NEED_MAIN_THREAD_FINISHED) && mFinished) ||
(mFlags & NEED_MAIN_THREAD_CURRENT_TIME);
}
virtual bool IsIntrinsicallyConsumed() const override
{
@@ -166,7 +183,7 @@ protected:
AudioChunk* aBlock,
nsTArray<float>* aDownmixBuffer);
void UpMixDownMixChunk(const AudioChunk* aChunk, uint32_t aOutputChannelCount,
nsTArray<const void*>& aOutputChannels,
nsTArray<const float*>& aOutputChannels,
nsTArray<float>& aDownmixBuffer);
uint32_t ComputedNumberOfChannels(uint32_t aInputChannelCount);
@@ -185,7 +202,7 @@ protected:
// AudioContext. It is set on the main thread, in the constructor.
const AudioContext::AudioContextId mAudioContextId;
// Whether this is an internal or external stream
const MediaStreamGraph::AudioNodeStreamKind mKind;
const Flags mFlags;
// The number of input channels that this stream requires. 0 means don't care.
uint32_t mNumberOfInputChannels;
// The mixing modes
dom/media/webaudio/AudioParam.cpp
+3 -4
View File
@@ -100,9 +100,8 @@ AudioParam::Stream()
AudioNodeEngine* engine = new AudioNodeEngine(nullptr);
nsRefPtr<AudioNodeStream> stream =
mNode->Context()->Graph()->CreateAudioNodeStream(engine,
MediaStreamGraph::INTERNAL_STREAM,
Node()->Context()->SampleRate());
AudioNodeStream::Create(mNode->Context()->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
// Force the input to have only one channel, and make it down-mix using
// the speaker rules if needed.
@@ -113,7 +112,7 @@ AudioParam::Stream()
mStream = stream.forget();
// Setup the AudioParam's stream as an input to the owner AudioNode's stream
AudioNodeStream* nodeStream = mNode->Stream();
AudioNodeStream* nodeStream = mNode->GetStream();
if (nodeStream) {
mNodeStreamPort =
nodeStream->AllocateInputPort(mStream, MediaInputPort::FLAG_BLOCK_INPUT);
dom/media/webaudio/BiquadFilterNode.cpp
+2 -1
View File
@@ -250,7 +250,8 @@ BiquadFilterNode::BiquadFilterNode(AudioContext* aContext)
, mGain(new AudioParam(this, SendGainToStream, 0.f, "gain"))
{
BiquadFilterNodeEngine* engine = new BiquadFilterNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/ChannelMergerNode.cpp
+3 -2
View File
@@ -73,8 +73,9 @@ ChannelMergerNode::ChannelMergerNode(AudioContext* aContext,
ChannelInterpretation::Speakers)
, mInputCount(aInputCount)
{
mStream = aContext->Graph()->CreateAudioNodeStream(new ChannelMergerNodeEngine(this),
MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(),
new ChannelMergerNodeEngine(this),
AudioNodeStream::NO_STREAM_FLAGS);
}
ChannelMergerNode::~ChannelMergerNode()
dom/media/webaudio/ChannelSplitterNode.cpp
+3 -2
View File
@@ -60,8 +60,9 @@ ChannelSplitterNode::ChannelSplitterNode(AudioContext* aContext,
ChannelInterpretation::Speakers)
, mOutputCount(aOutputCount)
{
mStream = aContext->Graph()->CreateAudioNodeStream(new ChannelSplitterNodeEngine(this),
MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(),
new ChannelSplitterNodeEngine(this),
AudioNodeStream::NO_STREAM_FLAGS);
}
ChannelSplitterNode::~ChannelSplitterNode()
dom/media/webaudio/ConvolverNode.cpp
+2 -1
View File
@@ -191,7 +191,8 @@ ConvolverNode::ConvolverNode(AudioContext* aContext)
, mNormalize(true)
{
ConvolverNodeEngine* engine = new ConvolverNodeEngine(this, mNormalize);
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
}
ConvolverNode::~ConvolverNode()
dom/media/webaudio/DelayBuffer.cpp
+5 -6
View File
@@ -156,7 +156,7 @@ DelayBuffer::ReadChannels(const double aPerFrameDelays[WEBAUDIO_BLOCK_SIZE],
for (uint32_t channel = aFirstChannel;
channel < readChannelsEnd; ++channel) {
aOutputChunk->ChannelFloatsForWrite(channel)[i] += multiplier *
static_cast<const float*>(mUpmixChannels[channel])[readOffset];
mUpmixChannels[channel][readOffset];
}
}
@@ -238,24 +238,23 @@ DelayBuffer::UpdateUpmixChannels(int aNewReadChunk, uint32_t aChannelCount,
return;
}
static const float silenceChannel[WEBAUDIO_BLOCK_SIZE] = {};
NS_WARN_IF_FALSE(mHaveWrittenBlock || aNewReadChunk != mCurrentChunk,
"Smoothing is making feedback delay too small.");
mLastReadChunk = aNewReadChunk;
mUpmixChannels = mChunks[aNewReadChunk].mChannelData;
mUpmixChannels = mChunks[aNewReadChunk].ChannelData<float>();
MOZ_ASSERT(mUpmixChannels.Length() <= aChannelCount);
if (mUpmixChannels.Length() < aChannelCount) {
if (aChannelInterpretation == ChannelInterpretation::Speakers) {
AudioChannelsUpMix(&mUpmixChannels, aChannelCount, silenceChannel);
AudioChannelsUpMix(&mUpmixChannels,
aChannelCount, SilentChannel::ZeroChannel<float>());
MOZ_ASSERT(mUpmixChannels.Length() == aChannelCount,
"We called GetAudioChannelsSuperset to avoid this");
} else {
// Fill up the remaining channels with zeros
for (uint32_t channel = mUpmixChannels.Length();
channel < aChannelCount; ++channel) {
mUpmixChannels.AppendElement(silenceChannel);
mUpmixChannels.AppendElement(SilentChannel::ZeroChannel<float>());
}
}
}
dom/media/webaudio/DelayBuffer.h
+1 -1
View File
@@ -94,7 +94,7 @@ private:
// Circular buffer for capturing delayed samples.
FallibleTArray<AudioChunk> mChunks;
// Cache upmixed channel arrays.
nsAutoTArray<const void*,GUESS_AUDIO_CHANNELS> mUpmixChannels;
nsAutoTArray<const float*,GUESS_AUDIO_CHANNELS> mUpmixChannels;
double mSmoothingRate;
// Current delay, in fractional ticks
double mCurrentDelay;
dom/media/webaudio/DelayNode.cpp
+2 -1
View File
@@ -195,7 +195,8 @@ DelayNode::DelayNode(AudioContext* aContext, double aMaxDelay)
DelayNodeEngine* engine =
new DelayNodeEngine(this, aContext->Destination(),
aContext->SampleRate() * aMaxDelay);
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/DynamicsCompressorNode.cpp
+2 -1
View File
@@ -203,7 +203,8 @@ DynamicsCompressorNode::DynamicsCompressorNode(AudioContext* aContext)
, mRelease(new AudioParam(this, SendReleaseToStream, 0.25f, "release"))
{
DynamicsCompressorNodeEngine* engine = new DynamicsCompressorNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/GainNode.cpp
+2 -1
View File
@@ -127,7 +127,8 @@ GainNode::GainNode(AudioContext* aContext)
, mGain(new AudioParam(this, SendGainToStream, 1.0f, "gain"))
{
GainNodeEngine* engine = new GainNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/MediaStreamAudioDestinationNode.cpp
+2 -1
View File
@@ -39,7 +39,8 @@ MediaStreamAudioDestinationNode::MediaStreamAudioDestinationNode(AudioContext* a
ProcessedMediaStream* outputStream = mDOMStream->GetStream()->AsProcessedStream();
MOZ_ASSERT(!!outputStream);
AudioNodeEngine* engine = new AudioNodeEngine(this);
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::EXTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::EXTERNAL_OUTPUT);
mPort = outputStream->AllocateInputPort(mStream);
nsIDocument* doc = aContext->GetParentObject()->GetExtantDoc();
dom/media/webaudio/MediaStreamAudioSourceNode.cpp
+1 -1
View File
@@ -66,7 +66,7 @@ MediaStreamAudioSourceNode::Init(DOMMediaStream* aMediaStream, ErrorResult& aRv)
mInputStream = aMediaStream;
AudioNodeEngine* engine = new MediaStreamAudioSourceNodeEngine(this);
mStream = graph->CreateAudioNodeExternalInputStream(engine);
mStream = AudioNodeExternalInputStream::Create(graph, engine);
ProcessedMediaStream* outputStream = static_cast<ProcessedMediaStream*>(mStream.get());
mInputPort = outputStream->AllocateInputPort(inputStream,
MediaInputPort::FLAG_BLOCK_INPUT);
dom/media/webaudio/OscillatorNode.cpp
+2 -1
View File
@@ -384,7 +384,8 @@ OscillatorNode::OscillatorNode(AudioContext* aContext)
, mStartCalled(false)
{
OscillatorNodeEngine* engine = new OscillatorNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::SOURCE_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NEED_MAIN_THREAD_FINISHED);
engine->SetSourceStream(mStream);
mStream->AddMainThreadListener(this);
}
dom/media/webaudio/PannerNode.cpp
+4 -3
View File
@@ -240,8 +240,9 @@ PannerNode::PannerNode(AudioContext* aContext)
, mConeOuterAngle(360.)
, mConeOuterGain(0.)
{
mStream = aContext->Graph()->CreateAudioNodeStream(new PannerNodeEngine(this),
MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(),
new PannerNodeEngine(this),
AudioNodeStream::NO_STREAM_FLAGS);
// We should register once we have set up our stream and engine.
Context()->Listener()->RegisterPannerNode(this);
}
@@ -543,7 +544,7 @@ PannerNode::FindConnectedSources(AudioNode* aNode,
// Check if this node is an AudioBufferSourceNode that still have a stream,
// which means it has not finished playing.
AudioBufferSourceNode* node = inputNodes[i].mInputNode->AsAudioBufferSourceNode();
if (node && node->Stream()) {
if (node && node->GetStream()) {
aSources.AppendElement(node);
}
}
dom/media/webaudio/ScriptProcessorNode.cpp
+2 -1
View File
@@ -481,7 +481,8 @@ ScriptProcessorNode::ScriptProcessorNode(AudioContext* aContext,
aContext->Destination(),
BufferSize(),
aNumberOfInputChannels);
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/StereoPannerNode.cpp
+2 -2
View File
@@ -179,8 +179,8 @@ StereoPannerNode::StereoPannerNode(AudioContext* aContext)
, mPan(new AudioParam(this, SendPanToStream, 0.f, "pan"))
{
StereoPannerNodeEngine* engine = new StereoPannerNodeEngine(this, aContext->Destination());
mStream = aContext->Graph()->CreateAudioNodeStream(engine,
MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
engine->SetSourceStream(mStream);
}
dom/media/webaudio/WaveShaperNode.cpp
+2 -1
View File
@@ -288,7 +288,8 @@ WaveShaperNode::WaveShaperNode(AudioContext* aContext)
mozilla::HoldJSObjects(this);
WaveShaperNodeEngine* engine = new WaveShaperNodeEngine(this);
mStream = aContext->Graph()->CreateAudioNodeStream(engine, MediaStreamGraph::INTERNAL_STREAM);
mStream = AudioNodeStream::Create(aContext->Graph(), engine,
AudioNodeStream::NO_STREAM_FLAGS);
}
WaveShaperNode::~WaveShaperNode()
media/libsoundtouch/README_MOZILLA
+1 -1
View File
@@ -1,5 +1,5 @@
These files are from the SoundTouch library (http://www.surina.net/soundtouch/),
and are extracted from the revision r198 of the svn repository at
and are extracted from the revision r222 of the svn repository at
https://soundtouch.svn.sourceforge.net/svnroot/soundtouch/trunk.
The whole library is not used, only the relevant files are imported in the tree,
media/libsoundtouch/moz-libsoundtouch.patch
+2 -41
View File
@@ -56,7 +56,7 @@ diff -u /src/cpu_detect_x86.cpp /src/cpu_detect_x86.cpp
diff -u /src/STTypes.h /src/STTypes.h
--- /src/STTypes.h
+++ /src/STTypes.h
@@ -54,12 +54,17 @@
@@ -54,12 +54,13 @@
#define SOUNDTOUCH_ALIGN_POINTER_16(x) ( ( (ulongptr)(x) + 15 ) & ~(ulongptr)15 )
@@ -67,27 +67,14 @@ diff -u /src/STTypes.h /src/STTypes.h
-#endif
+#include "soundtouch_config.h"
+#if defined(WIN32) && defined(GKMEDIAS_SHARED_LIBRARY)
+#ifdef BUILDING_SOUNDTOUCH
+#if defined(WIN32)
+#define EXPORT __declspec(dllexport)
+#else
+#define EXPORT __declspec(dllimport)
+#endif
+#else
+#define EXPORT
+#endif
namespace soundtouch
{
@@ -164,7 +169,7 @@
};
// define ST_NO_EXCEPTION_HANDLING switch to disable throwing std exceptions:
-// #define ST_NO_EXCEPTION_HANDLING 1
+#define ST_NO_EXCEPTION_HANDLING 1
#ifdef ST_NO_EXCEPTION_HANDLING
// Exceptions disabled. Throw asserts instead if enabled.
#include <assert.h>
diff -u /src/SoundTouch.h /src/SoundTouch.h
--- /src/SoundTouch.h
+++ /src/SoundTouch.h
@@ -103,18 +90,6 @@ diff -u /src/SoundTouch.h /src/SoundTouch.h
diff -u /src/FIRFilter.cpp /src/FIRFilter.cpp
--- /src/FIRFilter.cpp
+++ /src/FIRFilter.cpp
@@ -46,6 +46,11 @@
#include "FIRFilter.h"
#include "cpu_detect.h"
+#ifdef _MSC_VER
+#include <malloc.h>
+#define alloca _alloca
+#endif
+
using namespace soundtouch;
/*****************************************************************************
@@ -291,9 +296,11 @@
FIRFilter * FIRFilter::newInstance()
@@ -143,17 +118,3 @@ diff -u /src/TDStretch.cpp /src/TDStretch.cpp
// Check if MMX/SSE instruction set extensions supported by CPU
diff -u /src/SoundTouch.cpp /src/SoundTouch.cpp
--- /src/SoundTouch.cpp
+++ /src/SoundTouch.cpp
@@ -80,6 +80,11 @@
#include "RateTransposer.h"
#include "cpu_detect.h"
+#ifdef _MSC_VER
+#include <malloc.h>
+#define alloca _alloca
+#endif
+
using namespace soundtouch;
/// test if two floating point numbers are equal
media/libsoundtouch/src/AAFilter.cpp
+1 -1
View File
@@ -12,7 +12,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-05 15:40:22 -0600 (Sun, 05 Jan 2014) $
// Last changed : $Date: 2014-01-05 21:40:22 +0000 (Sun, 05 Jan 2014) $
// File revision : $Revision: 4 $
//
// $Id: AAFilter.cpp 177 2014-01-05 21:40:22Z oparviai $
media/libsoundtouch/src/AAFilter.h
+1 -1
View File
@@ -13,7 +13,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-07 13:41:23 -0600 (Tue, 07 Jan 2014) $
// Last changed : $Date: 2014-01-07 19:41:23 +0000 (Tue, 07 Jan 2014) $
// File revision : $Revision: 4 $
//
// $Id: AAFilter.h 187 2014-01-07 19:41:23Z oparviai $
media/libsoundtouch/src/FIFOSampleBuffer.cpp
+1 -1
View File
@@ -15,7 +15,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2012-11-08 12:53:01 -0600 (Thu, 08 Nov 2012) $
// Last changed : $Date: 2012-11-08 18:53:01 +0000 (Thu, 08 Nov 2012) $
// File revision : $Revision: 4 $
//
// $Id: FIFOSampleBuffer.cpp 160 2012-11-08 18:53:01Z oparviai $
media/libsoundtouch/src/FIFOSampleBuffer.h
+1 -1
View File
@@ -15,7 +15,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-05 15:40:22 -0600 (Sun, 05 Jan 2014) $
// Last changed : $Date: 2014-01-05 21:40:22 +0000 (Sun, 05 Jan 2014) $
// File revision : $Revision: 4 $
//
// $Id: FIFOSampleBuffer.h 177 2014-01-05 21:40:22Z oparviai $
media/libsoundtouch/src/FIFOSamplePipe.h
+1 -1
View File
@@ -17,7 +17,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2012-06-13 14:29:53 -0500 (Wed, 13 Jun 2012) $
// Last changed : $Date: 2012-06-13 19:29:53 +0000 (Wed, 13 Jun 2012) $
// File revision : $Revision: 4 $
//
// $Id: FIFOSamplePipe.h 143 2012-06-13 19:29:53Z oparviai $
media/libsoundtouch/src/FIRFilter.cpp
+33 -32
View File
@@ -11,10 +11,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2013-06-12 10:24:44 -0500 (Wed, 12 Jun 2013) $
// Last changed : $Date: 2015-02-21 21:24:29 +0000 (Sat, 21 Feb 2015) $
// File revision : $Revision: 4 $
//
// $Id: FIRFilter.cpp 171 2013-06-12 15:24:44Z oparviai $
// $Id: FIRFilter.cpp 202 2015-02-21 21:24:29Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -46,11 +46,6 @@
#include "FIRFilter.h"
#include "cpu_detect.h"
#ifdef _MSC_VER
#include <malloc.h>
#define alloca _alloca
#endif
using namespace soundtouch;
/*****************************************************************************
@@ -77,8 +72,7 @@ FIRFilter::~FIRFilter()
// Usual C-version of the filter routine for stereo sound
uint FIRFilter::evaluateFilterStereo(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples) const
{
uint i, j, end;
LONG_SAMPLETYPE suml, sumr;
int j, end;
#ifdef SOUNDTOUCH_FLOAT_SAMPLES
// when using floating point samples, use a scaler instead of a divider
// because division is much slower operation than multiplying.
@@ -92,9 +86,12 @@ uint FIRFilter::evaluateFilterStereo(SAMPLETYPE *dest, const SAMPLETYPE *src, ui
end = 2 * (numSamples - length);
#pragma omp parallel for
for (j = 0; j < end; j += 2)
{
const SAMPLETYPE *ptr;
LONG_SAMPLETYPE suml, sumr;
uint i;
suml = sumr = 0;
ptr = src + j;
@@ -135,28 +132,31 @@ uint FIRFilter::evaluateFilterStereo(SAMPLETYPE *dest, const SAMPLETYPE *src, ui
// Usual C-version of the filter routine for mono sound
uint FIRFilter::evaluateFilterMono(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples) const
{
uint i, j, end;
LONG_SAMPLETYPE sum;
int j, end;
#ifdef SOUNDTOUCH_FLOAT_SAMPLES
// when using floating point samples, use a scaler instead of a divider
// because division is much slower operation than multiplying.
double dScaler = 1.0 / (double)resultDivider;
#endif
assert(length != 0);
end = numSamples - length;
#pragma omp parallel for
for (j = 0; j < end; j ++)
{
const SAMPLETYPE *pSrc = src + j;
LONG_SAMPLETYPE sum;
uint i;
sum = 0;
for (i = 0; i < length; i += 4)
{
// loop is unrolled by factor of 4 here for efficiency
sum += src[i + 0] * filterCoeffs[i + 0] +
src[i + 1] * filterCoeffs[i + 1] +
src[i + 2] * filterCoeffs[i + 2] +
src[i + 3] * filterCoeffs[i + 3];
sum += pSrc[i + 0] * filterCoeffs[i + 0] +
pSrc[i + 1] * filterCoeffs[i + 1] +
pSrc[i + 2] * filterCoeffs[i + 2] +
pSrc[i + 3] * filterCoeffs[i + 3];
}
#ifdef SOUNDTOUCH_INTEGER_SAMPLES
sum >>= resultDivFactor;
@@ -166,16 +166,15 @@ uint FIRFilter::evaluateFilterMono(SAMPLETYPE *dest, const SAMPLETYPE *src, uint
sum *= dScaler;
#endif // SOUNDTOUCH_INTEGER_SAMPLES
dest[j] = (SAMPLETYPE)sum;
src ++;
}
return end;
}
uint FIRFilter::evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels) const
uint FIRFilter::evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels)
{
uint i, j, end, c;
LONG_SAMPLETYPE *sum=(LONG_SAMPLETYPE*)alloca(numChannels*sizeof(*sum));
int j, end;
#ifdef SOUNDTOUCH_FLOAT_SAMPLES
// when using floating point samples, use a scaler instead of a divider
// because division is much slower operation than multiplying.
@@ -186,17 +185,21 @@ uint FIRFilter::evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uin
assert(src != NULL);
assert(dest != NULL);
assert(filterCoeffs != NULL);
assert(numChannels < 16);
end = numChannels * (numSamples - length);
for (c = 0; c < numChannels; c ++)
{
sum[c] = 0;
}
#pragma omp parallel for
for (j = 0; j < end; j += numChannels)
{
const SAMPLETYPE *ptr;
LONG_SAMPLETYPE sums[16];
uint c, i;
for (c = 0; c < numChannels; c ++)
{
sums[c] = 0;
}
ptr = src + j;
@@ -205,7 +208,7 @@ uint FIRFilter::evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uin
SAMPLETYPE coef=filterCoeffs[i];
for (c = 0; c < numChannels; c ++)
{
sum[c] += ptr[0] * coef;
sums[c] += ptr[0] * coef;
ptr ++;
}
}
@@ -213,13 +216,11 @@ uint FIRFilter::evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uin
for (c = 0; c < numChannels; c ++)
{
#ifdef SOUNDTOUCH_INTEGER_SAMPLES
sum[c] >>= resultDivFactor;
sums[c] >>= resultDivFactor;
#else
sum[c] *= dScaler;
sums[c] *= dScaler;
#endif // SOUNDTOUCH_INTEGER_SAMPLES
*dest = (SAMPLETYPE)sum[c];
dest++;
sum[c] = 0;
dest[j+c] = (SAMPLETYPE)sums[c];
}
}
return numSamples - length;
@@ -258,7 +259,7 @@ uint FIRFilter::getLength() const
//
// Note : The amount of outputted samples is by value of 'filter_length'
// smaller than the amount of input samples.
uint FIRFilter::evaluate(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels) const
uint FIRFilter::evaluate(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels)
{
assert(length > 0);
assert(lengthDiv8 * 8 == length);
media/libsoundtouch/src/FIRFilter.h
+4 -4
View File
@@ -11,10 +11,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2013-06-12 10:24:44 -0500 (Wed, 12 Jun 2013) $
// Last changed : $Date: 2015-02-21 21:24:29 +0000 (Sat, 21 Feb 2015) $
// File revision : $Revision: 4 $
//
// $Id: FIRFilter.h 171 2013-06-12 15:24:44Z oparviai $
// $Id: FIRFilter.h 202 2015-02-21 21:24:29Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -71,7 +71,7 @@ protected:
virtual uint evaluateFilterMono(SAMPLETYPE *dest,
const SAMPLETYPE *src,
uint numSamples) const;
virtual uint evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels) const;
virtual uint evaluateFilterMulti(SAMPLETYPE *dest, const SAMPLETYPE *src, uint numSamples, uint numChannels);
public:
FIRFilter();
@@ -91,7 +91,7 @@ public:
uint evaluate(SAMPLETYPE *dest,
const SAMPLETYPE *src,
uint numSamples,
uint numChannels) const;
uint numChannels);
uint getLength() const;
media/libsoundtouch/src/RateTransposer.cpp
+1 -1
View File
@@ -10,7 +10,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2014-04-06 15:57:21 +0000 (Sun, 06 Apr 2014) $
// File revision : $Revision: 4 $
//
// $Id: RateTransposer.cpp 195 2014-04-06 15:57:21Z oparviai $
media/libsoundtouch/src/RateTransposer.h
+1 -1
View File
@@ -14,7 +14,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2014-04-06 15:57:21 +0000 (Sun, 06 Apr 2014) $
// File revision : $Revision: 4 $
//
// $Id: RateTransposer.h 195 2014-04-06 15:57:21Z oparviai $
media/libsoundtouch/src/STTypes.h
+6 -9
View File
@@ -8,10 +8,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2015-05-18 15:25:07 +0000 (Mon, 18 May 2015) $
// File revision : $Revision: 3 $
//
// $Id: STTypes.h 195 2014-04-06 15:57:21Z oparviai $
// $Id: STTypes.h 215 2015-05-18 15:25:07Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -56,13 +56,9 @@ typedef unsigned long ulong;
#include "soundtouch_config.h"
#if defined(WIN32) && defined(GKMEDIAS_SHARED_LIBRARY)
#ifdef BUILDING_SOUNDTOUCH
#if defined(WIN32)
#define EXPORT __declspec(dllexport)
#else
#define EXPORT __declspec(dllimport)
#endif
#else
#define EXPORT
#endif
@@ -80,7 +76,7 @@ namespace soundtouch
/// runtime performance so recommendation is to keep this off.
// #define USE_MULTICH_ALWAYS
#if (defined(__SOFTFP__))
#if (defined(__SOFTFP__) && defined(ANDROID))
// For Android compilation: Force use of Integer samples in case that
// compilation uses soft-floating point emulation - soft-fp is way too slow
#undef SOUNDTOUCH_FLOAT_SAMPLES
@@ -169,7 +165,7 @@ namespace soundtouch
};
// define ST_NO_EXCEPTION_HANDLING switch to disable throwing std exceptions:
#define ST_NO_EXCEPTION_HANDLING 1
// #define ST_NO_EXCEPTION_HANDLING 1
#ifdef ST_NO_EXCEPTION_HANDLING
// Exceptions disabled. Throw asserts instead if enabled.
#include <assert.h>
@@ -177,6 +173,7 @@ namespace soundtouch
#else
// use c++ standard exceptions
#include <stdexcept>
#include <string>
#define ST_THROW_RT_ERROR(x) {throw std::runtime_error(x);}
#endif
media/libsoundtouch/src/SoundTouch.cpp
+6 -9
View File
@@ -41,10 +41,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2014-10-08 15:26:57 +0000 (Wed, 08 Oct 2014) $
// File revision : $Revision: 4 $
//
// $Id: SoundTouch.cpp 195 2014-04-06 15:57:21Z oparviai $
// $Id: SoundTouch.cpp 201 2014-10-08 15:26:57Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -80,11 +80,6 @@
#include "RateTransposer.h"
#include "cpu_detect.h"
#ifdef _MSC_VER
#include <malloc.h>
#define alloca _alloca
#endif
using namespace soundtouch;
/// test if two floating point numbers are equal
@@ -353,8 +348,8 @@ void SoundTouch::flush()
int i;
int nUnprocessed;
int nOut;
SAMPLETYPE *buff=(SAMPLETYPE*)alloca(64*channels*sizeof(SAMPLETYPE));
SAMPLETYPE *buff = new SAMPLETYPE[64 * channels];
// check how many samples still await processing, and scale
// that by tempo & rate to get expected output sample count
nUnprocessed = numUnprocessedSamples();
@@ -383,6 +378,8 @@ void SoundTouch::flush()
}
}
delete[] buff;
// Clear working buffers
pRateTransposer->clear();
pTDStretch->clearInput();
media/libsoundtouch/src/SoundTouch.h
+4 -4
View File
@@ -41,10 +41,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2015-05-18 15:28:41 +0000 (Mon, 18 May 2015) $
// File revision : $Revision: 4 $
//
// $Id: SoundTouch.h 195 2014-04-06 15:57:21Z oparviai $
// $Id: SoundTouch.h 216 2015-05-18 15:28:41Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -79,10 +79,10 @@ namespace soundtouch
{
/// Soundtouch library version string
#define SOUNDTOUCH_VERSION "1.8.0"
#define SOUNDTOUCH_VERSION "1.9.0"
/// SoundTouch library version id
#define SOUNDTOUCH_VERSION_ID (10800)
#define SOUNDTOUCH_VERSION_ID (10900)
//
// Available setting IDs for the 'setSetting' & 'get_setting' functions:
media/libsoundtouch/src/SoundTouchFactory.cpp
+31
View File
@@ -0,0 +1,31 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim:set ts=2 sw=2 sts=2 et cindent: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#include <soundtouch/SoundTouch.h>
namespace soundtouch
{
EXPORT
soundtouch::SoundTouch*
createSoundTouchObj()
{
return new soundtouch::SoundTouch();
}
EXPORT
void
destroySoundTouchObj(soundtouch::SoundTouch* aObj)
{
// SoundTouch runs deletes in its destructor, meaning they need to be run in
// the DLL context. Gecko should send its SoundTouch obj pointers here to be
// cleaned up.
if (aObj) {
delete aObj;
}
}
}
media/libsoundtouch/src/SoundTouchFactory.h
+22
View File
@@ -0,0 +1,22 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim:set ts=2 sw=2 sts=2 et cindent: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
// Code for dealing with creating/deleting SoundTouch objects across DLL
// boundaries.
#include <soundtouch/STTypes.h>
#include <soundtouch/SoundTouch.h>
namespace soundtouch
{
EXPORT
soundtouch::SoundTouch*
createSoundTouchObj();
EXPORT
void
destroySoundTouchObj(soundtouch::SoundTouch* aObj);
}
media/libsoundtouch/src/TDStretch.cpp
+28 -11
View File
@@ -13,10 +13,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2015-02-22 15:07:12 +0000 (Sun, 22 Feb 2015) $
// File revision : $Revision: 1.12 $
//
// $Id: TDStretch.cpp 195 2014-04-06 15:57:21Z oparviai $
// $Id: TDStretch.cpp 205 2015-02-22 15:07:12Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -292,9 +292,9 @@ inline void TDStretch::overlap(SAMPLETYPE *pOutput, const SAMPLETYPE *pInput, ui
int TDStretch::seekBestOverlapPositionFull(const SAMPLETYPE *refPos)
{
int bestOffs;
double bestCorr, corr;
double norm;
double bestCorr;
int i;
double norm;
bestCorr = FLT_MIN;
bestOffs = 0;
@@ -302,14 +302,22 @@ int TDStretch::seekBestOverlapPositionFull(const SAMPLETYPE *refPos)
// Scans for the best correlation value by testing each possible position
// over the permitted range.
bestCorr = calcCrossCorr(refPos, pMidBuffer, norm);
#pragma omp parallel for
for (i = 1; i < seekLength; i ++)
{
// Calculates correlation value for the mixing position corresponding
// to 'i'. Now call "calcCrossCorrAccumulate" that is otherwise same as
// "calcCrossCorr", but saves time by reusing & updating previously stored
double corr;
// Calculates correlation value for the mixing position corresponding to 'i'
#ifdef _OPENMP
// in parallel OpenMP mode, can't use norm accumulator version as parallel executor won't
// iterate the loop in sequential order
corr = calcCrossCorr(refPos + channels * i, pMidBuffer, norm);
#else
// In non-parallel version call "calcCrossCorrAccumulate" that is otherwise same
// as "calcCrossCorr", but saves time by reusing & updating previously stored
// "norm" value
corr = calcCrossCorrAccumulate(refPos + channels * i, pMidBuffer, norm);
#endif
// heuristic rule to slightly favour values close to mid of the range
double tmp = (double)(2 * i - seekLength) / (double)seekLength;
corr = ((corr + 0.1) * (1.0 - 0.25 * tmp * tmp));
@@ -317,8 +325,15 @@ int TDStretch::seekBestOverlapPositionFull(const SAMPLETYPE *refPos)
// Checks for the highest correlation value
if (corr > bestCorr)
{
bestCorr = corr;
bestOffs = i;
// For optimal performance, enter critical section only in case that best value found.
// in such case repeat 'if' condition as it's possible that parallel execution may have
// updated the bestCorr value in the mean time
#pragma omp critical
if (corr > bestCorr)
{
bestCorr = corr;
bestOffs = i;
}
}
}
// clear cross correlation routine state if necessary (is so e.g. in MMX routines).
@@ -883,9 +898,10 @@ void TDStretch::calculateOverlapLength(int overlapInMsec)
/// Calculate cross-correlation
double TDStretch::calcCrossCorr(const float *mixingPos, const float *compare, double &norm) const
double TDStretch::calcCrossCorr(const float *mixingPos, const float *compare, double &anorm) const
{
double corr;
double norm;
int i;
corr = norm = 0;
@@ -907,6 +923,7 @@ double TDStretch::calcCrossCorr(const float *mixingPos, const float *compare, do
mixingPos[i + 3] * mixingPos[i + 3];
}
anorm = norm;
return corr / sqrt((norm < 1e-9 ? 1.0 : norm));
}
media/libsoundtouch/src/TDStretch.h
+1 -1
View File
@@ -13,7 +13,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-04-06 10:57:21 -0500 (Sun, 06 Apr 2014) $
// Last changed : $Date: 2014-04-06 15:57:21 +0000 (Sun, 06 Apr 2014) $
// File revision : $Revision: 4 $
//
// $Id: TDStretch.h 195 2014-04-06 15:57:21Z oparviai $
media/libsoundtouch/src/cpu_detect.h
+1 -1
View File
@@ -12,7 +12,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2008-02-10 10:26:55 -0600 (Sun, 10 Feb 2008) $
// Last changed : $Date: 2008-02-10 16:26:55 +0000 (Sun, 10 Feb 2008) $
// File revision : $Revision: 4 $
//
// $Id: cpu_detect.h 11 2008-02-10 16:26:55Z oparviai $
media/libsoundtouch/src/cpu_detect_x86.cpp
+1 -1
View File
@@ -11,7 +11,7 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-07 12:24:28 -0600 (Tue, 07 Jan 2014) $
// Last changed : $Date: 2014-01-07 18:24:28 +0000 (Tue, 07 Jan 2014) $
// File revision : $Revision: 4 $
//
// $Id: cpu_detect_x86.cpp 183 2014-01-07 18:24:28Z oparviai $
media/libsoundtouch/src/mmx_optimized.cpp
+3 -2
View File
@@ -20,10 +20,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-07 12:25:40 -0600 (Tue, 07 Jan 2014) $
// Last changed : $Date: 2015-02-22 15:10:38 +0000 (Sun, 22 Feb 2015) $
// File revision : $Revision: 4 $
//
// $Id: mmx_optimized.cpp 184 2014-01-07 18:25:40Z oparviai $
// $Id: mmx_optimized.cpp 206 2015-02-22 15:10:38Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -287,6 +287,7 @@ void TDStretchMMX::overlapStereo(short *output, const short *input) const
FIRFilterMMX::FIRFilterMMX() : FIRFilter()
{
filterCoeffsAlign = NULL;
filterCoeffsUnalign = NULL;
}
media/libsoundtouch/src/moz.build
+10 -5
View File
@@ -8,6 +8,7 @@ EXPORTS.soundtouch += [
'FIFOSamplePipe.h',
'SoundTouch.h',
'soundtouch_config.h',
'SoundTouchFactory.h',
'STTypes.h',
]
@@ -21,6 +22,7 @@ UNIFIED_SOURCES += [
'InterpolateShannon.cpp',
'RateTransposer.cpp',
'SoundTouch.cpp',
'SoundTouchFactory.cpp',
'TDStretch.cpp',
]
@@ -32,12 +34,15 @@ if CONFIG['INTEL_ARCHITECTURE']:
SOURCES += ['mmx_optimized.cpp']
SOURCES['mmx_optimized.cpp'].flags += CONFIG['MMX_FLAGS']
if CONFIG['GKMEDIAS_SHARED_LIBRARY']:
NO_VISIBILITY_FLAGS = True
if CONFIG['OS_ARCH'] != 'WINNT':
# GCC/Clang require permissions to be explicitly set for the soundtouch
# header.
CXXFLAGS += ['-include', 'soundtouch_perms.h']
else:
# Windows need alloca renamed to _alloca
DEFINES['alloca'] = '_alloca'
FINAL_LIBRARY = 'gkmedias'
FINAL_LIBRARY = 'lgpllibs'
# Use abort() instead of exception in SoundTouch.
DEFINES['ST_NO_EXCEPTION_HANDLING'] = 1
DEFINES['BUILDING_SOUNDTOUCH'] = True
media/libsoundtouch/src/soundtouch_perms.h
+18
View File
@@ -0,0 +1,18 @@
/* -*- Mode: C++; tab-width: 2; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim:set ts=2 sw=2 sts=2 et cindent: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
// Include file for fixing symbol visibility on non-windows platforms, until
// system headers wrappers work uniformly across all of them.
#ifndef MOZILLA_SOUNDTOUCH_PERMS_H
#define MOZILLA_SOUNDTOUCH_PERMS_H
#pragma GCC visibility push(default)
#include "SoundTouch.h"
#include "SoundTouchFactory.h"
#pragma GCC visibility pop
#endif // MOZILLA_SOUNDTOUCH_PERMS_H
media/libsoundtouch/src/sse_optimized.cpp
+10 -8
View File
@@ -23,10 +23,10 @@
///
////////////////////////////////////////////////////////////////////////////////
//
// Last changed : $Date: 2014-01-07 12:25:40 -0600 (Tue, 07 Jan 2014) $
// Last changed : $Date: 2015-02-21 21:24:29 +0000 (Sat, 21 Feb 2015) $
// File revision : $Revision: 4 $
//
// $Id: sse_optimized.cpp 184 2014-01-07 18:25:40Z oparviai $
// $Id: sse_optimized.cpp 202 2015-02-21 21:24:29Z oparviai $
//
////////////////////////////////////////////////////////////////////////////////
//
@@ -71,7 +71,7 @@ using namespace soundtouch;
#include <math.h>
// Calculates cross correlation of two buffers
double TDStretchSSE::calcCrossCorr(const float *pV1, const float *pV2, double &norm) const
double TDStretchSSE::calcCrossCorr(const float *pV1, const float *pV2, double &anorm) const
{
int i;
const float *pVec1;
@@ -141,7 +141,8 @@ double TDStretchSSE::calcCrossCorr(const float *pV1, const float *pV2, double &n
// return value = vSum[0] + vSum[1] + vSum[2] + vSum[3]
float *pvNorm = (float*)&vNorm;
norm = (pvNorm[0] + pvNorm[1] + pvNorm[2] + pvNorm[3]);
float norm = (pvNorm[0] + pvNorm[1] + pvNorm[2] + pvNorm[3]);
anorm = norm;
float *pvSum = (float*)&vSum;
return (double)(pvSum[0] + pvSum[1] + pvSum[2] + pvSum[3]) / sqrt(norm < 1e-9 ? 1.0 : norm);
@@ -258,14 +259,17 @@ uint FIRFilterSSE::evaluateFilterStereo(float *dest, const float *source, uint n
assert(((ulongptr)filterCoeffsAlign) % 16 == 0);
// filter is evaluated for two stereo samples with each iteration, thus use of 'j += 2'
#pragma omp parallel for
for (j = 0; j < count; j += 2)
{
const float *pSrc;
float *pDest;
const __m128 *pFil;
__m128 sum1, sum2;
uint i;
pSrc = (const float*)source; // source audio data
pSrc = (const float*)source + j * 2; // source audio data
pDest = dest + j * 2; // destination audio data
pFil = (const __m128*)filterCoeffsAlign; // filter coefficients. NOTE: Assumes coefficients
// are aligned to 16-byte boundary
sum1 = sum2 = _mm_setzero_ps();
@@ -298,12 +302,10 @@ uint FIRFilterSSE::evaluateFilterStereo(float *dest, const float *source, uint n
// to sum the two hi- and lo-floats of these registers together.
// post-shuffle & add the filtered values and store to dest.
_mm_storeu_ps(dest, _mm_add_ps(
_mm_storeu_ps(pDest, _mm_add_ps(
_mm_shuffle_ps(sum1, sum2, _MM_SHUFFLE(1,0,3,2)), // s2_1 s2_0 s1_3 s1_2
_mm_shuffle_ps(sum1, sum2, _MM_SHUFFLE(3,2,1,0)) // s2_3 s2_2 s1_1 s1_0
));
source += 4;
dest += 4;
}
// Ideas for further improvement:
media/mtransport/databuffer.h
+31 -7
View File
@@ -17,30 +17,54 @@ namespace mozilla {
class DataBuffer {
public:
DataBuffer() : data_(nullptr), len_(0) {}
DataBuffer() : data_(nullptr), len_(0), capacity_(0) {}
DataBuffer(const uint8_t *data, size_t len) {
Assign(data, len);
Assign(data, len, len);
}
DataBuffer(const uint8_t *data, size_t len, size_t capacity) {
Assign(data, len, capacity);
}
void Assign(const uint8_t *data, size_t len) {
Allocate(len);
// to ensure extra space for expansion
void Assign(const uint8_t *data, size_t len, size_t capacity) {
MOZ_RELEASE_ASSERT(len <= capacity);
Allocate(capacity); // sets len_ = capacity
memcpy(static_cast<void *>(data_.get()),
static_cast<const void *>(data), len);
len_ = len;
}
void Allocate(size_t len) {
data_.reset(new uint8_t[len ? len : 1]); // Don't depend on new [0].
void Allocate(size_t capacity) {
data_.reset(new uint8_t[capacity ? capacity : 1]); // Don't depend on new [0].
len_ = capacity_ = capacity;
}
void EnsureCapacity(size_t capacity) {
if (capacity_ < capacity) {
uint8_t *new_data = new uint8_t[ capacity ? capacity : 1];
memcpy(static_cast<void *>(new_data),
static_cast<const void *>(data_.get()), len_);
data_.reset(new_data); // after copying! Deletes old data
capacity_ = capacity;
}
}
// used when something writes to the buffer (having checked
// capacity() or used EnsureCapacity()) and increased the length.
void SetLength(size_t len) {
MOZ_RELEASE_ASSERT(len <= capacity_);
len_ = len;
}
const uint8_t *data() const { return data_.get(); }
uint8_t *data() { return data_.get(); }
size_t len() const { return len_; }
const bool empty() const { return len_ != 0; }
size_t capacity() const { return capacity_; }
private:
UniquePtr<uint8_t[]> data_;
size_t len_;
size_t capacity_;
DISALLOW_COPY_ASSIGN(DataBuffer);
};
media/webrtc/signaling/src/mediapipeline/MediaPipeline.cpp
+163 -184
View File
@@ -633,12 +633,10 @@ void MediaPipelineTransmit::AttachToTrack(const std::string& track_id) {
stream_->AddListener(listener_);
// // Is this a gUM mediastream? If so, also register the Listener directly with
// // the SourceMediaStream that's attached to the TrackUnion so we can get direct
// // unqueued (and not resampled) data
// if (domstream_->AddDirectListener(listener_)) {
// listener_->direct_connect_ = true;
// }
// Is this a gUM mediastream? If so, also register the Listener directly with
// the SourceMediaStream that's attached to the TrackUnion so we can get direct
// unqueued (and not resampled) data
listener_->direct_connect_ = domstream_->AddDirectListener(listener_);
#ifndef MOZILLA_INTERNAL_API
// this enables the unit tests that can't fiddle with principals and the like
@@ -755,115 +753,91 @@ nsresult MediaPipeline::PipelineTransport::SendRtpPacket(
const void *data, int len) {
nsAutoPtr<DataBuffer> buf(new DataBuffer(static_cast<const uint8_t *>(data),
len));
len, len + SRTP_MAX_EXPANSION));
RUN_ON_THREAD(sts_thread_,
WrapRunnable(
RefPtr<MediaPipeline::PipelineTransport>(this),
&MediaPipeline::PipelineTransport::SendRtpPacket_s,
buf),
&MediaPipeline::PipelineTransport::SendRtpRtcpPacket_s,
buf, true),
NS_DISPATCH_NORMAL);
return NS_OK;
}
nsresult MediaPipeline::PipelineTransport::SendRtpPacket_s(
nsAutoPtr<DataBuffer> data) {
ASSERT_ON_THREAD(sts_thread_);
if (!pipeline_)
return NS_OK; // Detached
nsresult MediaPipeline::PipelineTransport::SendRtpRtcpPacket_s(
nsAutoPtr<DataBuffer> data,
bool is_rtp) {
if (!pipeline_->rtp_.send_srtp_) {
MOZ_MTLOG(ML_DEBUG, "Couldn't write RTP packet; SRTP not set up yet");
ASSERT_ON_THREAD(sts_thread_);
if (!pipeline_) {
return NS_OK; // Detached
}
TransportInfo& transport = is_rtp ? pipeline_->rtp_ : pipeline_->rtcp_;
if (!transport.send_srtp_) {
MOZ_MTLOG(ML_DEBUG, "Couldn't write RTP/RTCP packet; SRTP not set up yet");
return NS_OK;
}
MOZ_ASSERT(pipeline_->rtp_.transport_);
NS_ENSURE_TRUE(pipeline_->rtp_.transport_, NS_ERROR_NULL_POINTER);
MOZ_ASSERT(transport.transport_);
NS_ENSURE_TRUE(transport.transport_, NS_ERROR_NULL_POINTER);
// libsrtp enciphers in place, so we need a new, big enough
// buffer.
// XXX. allocates and deletes one buffer per packet sent.
// Bug 822129
int max_len = data->len() + SRTP_MAX_EXPANSION;
ScopedDeletePtr<unsigned char> inner_data(
new unsigned char[max_len]);
memcpy(inner_data, data->data(), data->len());
// libsrtp enciphers in place, so we need a big enough buffer.
MOZ_ASSERT(data->capacity() >= data->len() + SRTP_MAX_EXPANSION);
int out_len;
nsresult res = pipeline_->rtp_.send_srtp_->ProtectRtp(inner_data,
data->len(),
max_len,
&out_len);
if (!NS_SUCCEEDED(res))
nsresult res;
if (is_rtp) {
res = transport.send_srtp_->ProtectRtp(data->data(),
data->len(),
data->capacity(),
&out_len);
} else {
res = transport.send_srtp_->ProtectRtcp(data->data(),
data->len(),
data->capacity(),
&out_len);
}
if (!NS_SUCCEEDED(res)) {
return res;
}
MOZ_MTLOG(ML_DEBUG, pipeline_->description_ << " sending RTP packet.");
pipeline_->increment_rtp_packets_sent(out_len);
return pipeline_->SendPacket(pipeline_->rtp_.transport_, inner_data,
out_len);
// paranoia; don't have uninitialized bytes included in data->len()
data->SetLength(out_len);
MOZ_MTLOG(ML_DEBUG, pipeline_->description_ << " sending " <<
(is_rtp ? "RTP" : "RTCP") << " packet");
if (is_rtp) {
pipeline_->increment_rtp_packets_sent(out_len);
} else {
pipeline_->increment_rtcp_packets_sent();
}
return pipeline_->SendPacket(transport.transport_, data->data(), out_len);
}
nsresult MediaPipeline::PipelineTransport::SendRtcpPacket(
const void *data, int len) {
nsAutoPtr<DataBuffer> buf(new DataBuffer(static_cast<const uint8_t *>(data),
len));
len, len + SRTP_MAX_EXPANSION));
RUN_ON_THREAD(sts_thread_,
WrapRunnable(
RefPtr<MediaPipeline::PipelineTransport>(this),
&MediaPipeline::PipelineTransport::SendRtcpPacket_s,
buf),
&MediaPipeline::PipelineTransport::SendRtpRtcpPacket_s,
buf, false),
NS_DISPATCH_NORMAL);
return NS_OK;
}
nsresult MediaPipeline::PipelineTransport::SendRtcpPacket_s(
nsAutoPtr<DataBuffer> data) {
ASSERT_ON_THREAD(sts_thread_);
if (!pipeline_)
return NS_OK; // Detached
if (!pipeline_->rtcp_.send_srtp_) {
MOZ_MTLOG(ML_DEBUG, "Couldn't write RTCP packet; SRTCP not set up yet");
return NS_OK;
}
MOZ_ASSERT(pipeline_->rtcp_.transport_);
NS_ENSURE_TRUE(pipeline_->rtcp_.transport_, NS_ERROR_NULL_POINTER);
// libsrtp enciphers in place, so we need a new, big enough
// buffer.
// XXX. allocates and deletes one buffer per packet sent.
// Bug 822129.
int max_len = data->len() + SRTP_MAX_EXPANSION;
ScopedDeletePtr<unsigned char> inner_data(
new unsigned char[max_len]);
memcpy(inner_data, data->data(), data->len());
int out_len;
nsresult res = pipeline_->rtcp_.send_srtp_->ProtectRtcp(inner_data,
data->len(),
max_len,
&out_len);
if (!NS_SUCCEEDED(res))
return res;
MOZ_MTLOG(ML_DEBUG, pipeline_->description_ << " sending RTCP packet.");
pipeline_->increment_rtcp_packets_sent();
return pipeline_->SendPacket(pipeline_->rtcp_.transport_, inner_data,
out_len);
}
void MediaPipelineTransmit::PipelineListener::
UnsetTrackId(MediaStreamGraphImpl* graph) {
#ifndef USE_FAKE_MEDIA_STREAMS
class Message : public ControlMessage {
public:
Message(PipelineListener* listener) :
explicit Message(PipelineListener* listener) :
ControlMessage(nullptr), listener_(listener) {}
virtual void Run() override
{
@@ -919,19 +893,19 @@ NewData(MediaStreamGraph* graph, TrackID tid,
return;
}
if (track_id_ != TRACK_INVALID) {
if (tid != track_id_) {
return;
}
} else if (conduit_->type() !=
(media.GetType() == MediaSegment::AUDIO ? MediaSessionConduit::AUDIO :
MediaSessionConduit::VIDEO)) {
// Ignore data in case we have a muxed stream
if (conduit_->type() !=
(media.GetType() == MediaSegment::AUDIO ? MediaSessionConduit::AUDIO :
MediaSessionConduit::VIDEO)) {
// Ignore data of wrong kind in case we have a muxed stream
return;
} else {
}
if (track_id_ == TRACK_INVALID) {
// Don't lock during normal media flow except on first sample
MutexAutoLock lock(mMutex);
track_id_ = track_id_external_ = tid;
} else if (tid != track_id_) {
return;
}
// TODO(ekr@rtfm.com): For now assume that we have only one
@@ -974,97 +948,80 @@ void MediaPipelineTransmit::PipelineListener::ProcessAudioChunk(
AudioSessionConduit *conduit,
TrackRate rate,
AudioChunk& chunk) {
// TODO(ekr@rtfm.com): Do more than one channel
nsAutoArrayPtr<int16_t> samples(new int16_t[chunk.mDuration]);
if (enabled_ && chunk.mBuffer) {
switch (chunk.mBufferFormat) {
case AUDIO_FORMAT_FLOAT32:
{
const float* buf = static_cast<const float *>(chunk.mChannelData[0]);
ConvertAudioSamplesWithScale(buf, static_cast<int16_t*>(samples),
chunk.mDuration, chunk.mVolume);
}
break;
case AUDIO_FORMAT_S16:
{
const short* buf = static_cast<const short *>(chunk.mChannelData[0]);
ConvertAudioSamplesWithScale(buf, samples, chunk.mDuration, chunk.mVolume);
}
break;
case AUDIO_FORMAT_SILENCE:
memset(samples, 0, chunk.mDuration * sizeof(samples[0]));
break;
default:
MOZ_ASSERT(PR_FALSE);
return;
break;
}
// Convert to interleaved, 16-bits integer audio, with a maximum of two
// channels (since the WebRTC.org code below makes the assumption that the
// input audio is either mono or stereo).
uint32_t outputChannels = chunk.ChannelCount() == 1 ? 1 : 2;
const int16_t* samples = nullptr;
nsAutoArrayPtr<int16_t> convertedSamples;
// If this track is not enabled, simply ignore the data in the chunk.
if (!enabled_) {
chunk.mBufferFormat = AUDIO_FORMAT_SILENCE;
}
// We take advantage of the fact that the common case (microphone directly to
// PeerConnection, that is, a normal call), the samples are already 16-bits
// mono, so the representation in interleaved and planar is the same, and we
// can just use that.
if (outputChannels == 1 && chunk.mBufferFormat == AUDIO_FORMAT_S16) {
samples = chunk.ChannelData<int16_t>().Elements()[0];
} else {
// This means silence.
memset(samples, 0, chunk.mDuration * sizeof(samples[0]));
convertedSamples = new int16_t[chunk.mDuration * outputChannels];
switch (chunk.mBufferFormat) {
case AUDIO_FORMAT_FLOAT32:
DownmixAndInterleave(chunk.ChannelData<float>(),
chunk.mDuration, chunk.mVolume, outputChannels,
convertedSamples.get());
break;
case AUDIO_FORMAT_S16:
DownmixAndInterleave(chunk.ChannelData<int16_t>(),
chunk.mDuration, chunk.mVolume, outputChannels,
convertedSamples.get());
break;
case AUDIO_FORMAT_SILENCE:
PodZero(convertedSamples.get(), chunk.mDuration * outputChannels);
break;
}
samples = convertedSamples.get();
}
MOZ_ASSERT(!(rate%100)); // rate should be a multiple of 100
// Check if the rate has changed since the last time we came through
// I realize it may be overkill to check if the rate has changed, but
// I believe it is possible (e.g. if we change sources) and it costs us
// very little to handle this case
// Check if the rate or the number of channels has changed since the last time
// we came through. I realize it may be overkill to check if the rate has
// changed, but I believe it is possible (e.g. if we change sources) and it
// costs us very little to handle this case.
if (samplenum_10ms_ != rate/100) {
// Determine number of samples in 10 ms from the rate:
samplenum_10ms_ = rate/100;
// If we switch sample rates (e.g. if we switch codecs),
// we throw away what was in the sample_10ms_buffer at the old rate
samples_10ms_buffer_ = new int16_t[samplenum_10ms_];
buffer_current_ = 0;
uint32_t audio_10ms = rate / 100;
if (!packetizer_ ||
packetizer_->PacketSize() != audio_10ms ||
packetizer_->Channels() != outputChannels) {
// It's ok to drop the audio still in the packetizer here.
packetizer_ = new AudioPacketizer<int16_t, int16_t>(audio_10ms, outputChannels);
}
packetizer_->Input(samples, chunk.mDuration);
while (packetizer_->PacketsAvailable()) {
uint32_t samplesPerPacket = packetizer_->PacketSize() *
packetizer_->Channels();
// We know that webrtc.org's code going to copy the samples down the line,
// so we can just use a stack buffer here instead of malloc-ing.
// Max size given stereo is 480*2*2 = 1920 (10ms of 16-bits stereo audio at
// 48KHz)
const size_t AUDIO_SAMPLE_BUFFER_MAX = 1920;
int16_t packet[AUDIO_SAMPLE_BUFFER_MAX];
packetizer_->Output(packet);
conduit->SendAudioFrame(packet,
samplesPerPacket,
rate, 0);
}
// Vars to handle the non-sunny-day case (where the audio chunks
// we got are not multiples of 10ms OR there were samples left over
// from the last run)
int64_t chunk_remaining;
int64_t tocpy;
int16_t *samples_tmp = samples.get();
chunk_remaining = chunk.mDuration;
MOZ_ASSERT(chunk_remaining >= 0);
if (buffer_current_) {
tocpy = std::min(chunk_remaining, samplenum_10ms_ - buffer_current_);
memcpy(&samples_10ms_buffer_[buffer_current_], samples_tmp, tocpy * sizeof(int16_t));
buffer_current_ += tocpy;
samples_tmp += tocpy;
chunk_remaining -= tocpy;
if (buffer_current_ == samplenum_10ms_) {
// Send out the audio buffer we just finished filling
conduit->SendAudioFrame(samples_10ms_buffer_, samplenum_10ms_, rate, 0);
buffer_current_ = 0;
} else {
// We still don't have enough data to send a buffer
return;
}
}
// Now send (more) frames if there is more than 10ms of input left
tocpy = (chunk_remaining / samplenum_10ms_) * samplenum_10ms_;
if (tocpy > 0) {
conduit->SendAudioFrame(samples_tmp, tocpy, rate, 0);
samples_tmp += tocpy;
chunk_remaining -= tocpy;
}
// Copy what remains for the next run
MOZ_ASSERT(chunk_remaining < samplenum_10ms_);
if (chunk_remaining) {
memcpy(samples_10ms_buffer_, samples_tmp, chunk_remaining * sizeof(int16_t));
buffer_current_ = chunk_remaining;
}
}
#if !defined(MOZILLA_EXTERNAL_LINKAGE)
@@ -1392,19 +1349,18 @@ NotifyPull(MediaStreamGraph* graph, StreamTime desired_time) {
// This comparison is done in total time to avoid accumulated roundoff errors.
while (source_->TicksToTimeRoundDown(track_rate_, played_ticks_) <
desired_time) {
// TODO(ekr@rtfm.com): Is there a way to avoid mallocating here? Or reduce the size?
// Max size given mono is 480*2*1 = 960 (48KHz)
#define AUDIO_SAMPLE_BUFFER_MAX 1000
MOZ_ASSERT((track_rate_/100)*sizeof(uint16_t) <= AUDIO_SAMPLE_BUFFER_MAX);
// Max size given stereo is 480*2*2 = 1920 (48KHz)
const size_t AUDIO_SAMPLE_BUFFER_MAX = 1920;
MOZ_ASSERT((track_rate_/100)*sizeof(uint16_t) * 2 <= AUDIO_SAMPLE_BUFFER_MAX);
int16_t scratch_buffer[AUDIO_SAMPLE_BUFFER_MAX];
nsRefPtr<SharedBuffer> samples = SharedBuffer::Create(AUDIO_SAMPLE_BUFFER_MAX);
int16_t *samples_data = static_cast<int16_t *>(samples->Data());
int samples_length;
// This fetches 10ms of data
// This fetches 10ms of data, either mono or stereo
MediaConduitErrorCode err =
static_cast<AudioSessionConduit*>(conduit_.get())->GetAudioFrame(
samples_data,
scratch_buffer,
track_rate_,
0, // TODO(ekr@rtfm.com): better estimate of "capture" (really playout) delay
samples_length);
@@ -1415,23 +1371,46 @@ NotifyPull(MediaStreamGraph* graph, StreamTime desired_time) {
<< ") to return data @ " << played_ticks_
<< " (desired " << desired_time << " -> "
<< source_->StreamTimeToSeconds(desired_time) << ")");
samples_length = (track_rate_/100)*sizeof(uint16_t); // if this is not enough we'll loop and provide more
memset(samples_data, '\0', samples_length);
samples_length = track_rate_/100; // if this is not enough we'll loop and provide more
PodArrayZero(scratch_buffer);
}
MOZ_ASSERT(samples_length < AUDIO_SAMPLE_BUFFER_MAX);
MOZ_ASSERT(samples_length * sizeof(uint16_t) < AUDIO_SAMPLE_BUFFER_MAX);
MOZ_MTLOG(ML_DEBUG, "Audio conduit returned buffer of length "
<< samples_length);
nsRefPtr<SharedBuffer> samples = SharedBuffer::Create(samples_length * sizeof(uint16_t));
int16_t *samples_data = static_cast<int16_t *>(samples->Data());
AudioSegment segment;
nsAutoTArray<const int16_t*,1> channels;
channels.AppendElement(samples_data);
segment.AppendFrames(samples.forget(), channels, samples_length);
// We derive the number of channels of the stream from the number of samples
// the AudioConduit gives us, considering it gives us packets of 10ms and we
// know the rate.
uint32_t channelCount = samples_length / (track_rate_ / 100);
nsAutoTArray<int16_t*,2> channels;
nsAutoTArray<const int16_t*,2> outputChannels;
size_t frames = samples_length / channelCount;
channels.SetLength(channelCount);
size_t offset = 0;
for (size_t i = 0; i < channelCount; i++) {
channels[i] = samples_data + offset;
offset += frames;
}
DeinterleaveAndConvertBuffer(scratch_buffer,
frames,
channelCount,
channels.Elements());
outputChannels.AppendElements(channels);
segment.AppendFrames(samples.forget(), outputChannels, frames);
// Handle track not actually added yet or removed/finished
if (source_->AppendToTrack(track_id_, &segment)) {
played_ticks_ += track_rate_/100; // 10ms in TrackTicks
played_ticks_ += frames;
} else {
MOZ_MTLOG(ML_ERROR, "AppendToTrack failed");
// we can't un-read the data, but that's ok since we don't want to
media/webrtc/signaling/src/mediapipeline/MediaPipeline.h
+5 -15
View File
@@ -26,6 +26,7 @@
#include "databuffer.h"
#include "runnable_utils.h"
#include "transportflow.h"
#include "AudioPacketizer.h"
#if defined(MOZILLA_INTERNAL_API)
#include "VideoSegment.h"
@@ -191,8 +192,8 @@ class MediaPipeline : public sigslot::has_slots<> {
virtual nsresult SendRtcpPacket(const void* data, int len);
private:
virtual nsresult SendRtpPacket_s(nsAutoPtr<DataBuffer> data);
virtual nsresult SendRtcpPacket_s(nsAutoPtr<DataBuffer> data);
nsresult SendRtpRtcpPacket_s(nsAutoPtr<DataBuffer> data,
bool is_rtp);
MediaPipeline *pipeline_; // Raw pointer to avoid cycles
nsCOMPtr<nsIEventTarget> sts_thread_;
@@ -446,9 +447,7 @@ public:
active_(false),
enabled_(false),
direct_connect_(false),
samples_10ms_buffer_(nullptr),
buffer_current_(0),
samplenum_10ms_(0)
packetizer_(nullptr)
#if !defined(MOZILLA_EXTERNAL_LINKAGE)
, last_img_(-1)
#endif // MOZILLA_INTERNAL_API
@@ -526,16 +525,7 @@ public:
bool direct_connect_;
// These vars handle breaking audio samples into exact 10ms chunks:
// The buffer of 10ms audio samples that we will send once full
// (can be carried over from one call to another).
nsAutoArrayPtr<int16_t> samples_10ms_buffer_;
// The location of the pointer within that buffer (in units of samples).
int64_t buffer_current_;
// The number of samples in a 10ms audio chunk.
int64_t samplenum_10ms_;
nsAutoPtr<AudioPacketizer<int16_t, int16_t>> packetizer_;
#if !defined(MOZILLA_EXTERNAL_LINKAGE)
int32_t last_img_; // serial number of last Image
#endif // MOZILLA_INTERNAL_API
media/webrtc/signaling/src/mediapipeline/SrtpFlow.h
+3 -3
View File
@@ -22,9 +22,9 @@ namespace mozilla {
#define SRTP_MASTER_SALT_LENGTH 14
#define SRTP_TOTAL_KEY_LENGTH (SRTP_MASTER_KEY_LENGTH + SRTP_MASTER_SALT_LENGTH)
// For some reason libsrtp increases packet size by > 12 for RTCP even though
// the doc claims otherwise.
#define SRTP_MAX_EXPANSION 20
// SRTCP requires an auth tag *plus* a 4-byte index-plus-'E'-bit value (see
// RFC 3711)
#define SRTP_MAX_EXPANSION (SRTP_MAX_TRAILER_LEN+4)
class SrtpFlow {
media/webrtc/signaling/src/sdp/sipcc/sdp_attr.c
-4
View File
@@ -5101,13 +5101,11 @@ sdp_result_e sdp_parse_attr_setup(sdp_t *sdp_p,
"%s Warning: Unknown setup attribute",
sdp_p->debug_str);
return SDP_INVALID_PARAMETER;
break;
default:
/* This is an internal error, not a parsing error */
CSFLogError(logTag, "%s Error: Invalid setup enum (%d)",
sdp_p->debug_str, attr_p->attr.setup);
return SDP_FAILURE;
break;
}
return SDP_SUCCESS;
@@ -5161,13 +5159,11 @@ sdp_result_e sdp_parse_attr_connection(sdp_t *sdp_p,
"%s Warning: Unknown connection attribute",
sdp_p->debug_str);
return SDP_INVALID_PARAMETER;
break;
default:
/* This is an internal error, not a parsing error */
CSFLogError(logTag, "%s Error: Invalid connection enum (%d)",
sdp_p->debug_str, attr_p->attr.connection);
return SDP_FAILURE;
break;
}
return SDP_SUCCESS;
}
media/webrtc/signaling/src/sdp/sipcc/sdp_attr_access.c
-6
View File
@@ -1626,8 +1626,6 @@ sdp_result_e sdp_attr_get_ice_attribute (sdp_t *sdp_p, uint16_t level,
sdp_p->conf_p->num_invalid_param++;
return (SDP_INVALID_PARAMETER);
}
return (SDP_FAILURE);
}
/* Function: sdp_attr_is_present
@@ -1694,8 +1692,6 @@ sdp_result_e sdp_attr_get_rtcp_mux_attribute (sdp_t *sdp_p, uint16_t level,
sdp_p->conf_p->num_invalid_param++;
return (SDP_INVALID_PARAMETER);
}
return (SDP_FAILURE);
}
/* Function: sdp_attr_get_setup_attribute
@@ -1799,8 +1795,6 @@ sdp_result_e sdp_attr_get_dtls_fingerprint_attribute (sdp_t *sdp_p, uint16_t lev
sdp_p->conf_p->num_invalid_param++;
return (SDP_INVALID_PARAMETER);
}
return (SDP_FAILURE);
}
/* Function: sdp_attr_sprtmap_payload_valid
media/webrtc/trunk/webrtc/voice_engine/output_mixer.cc
+5
View File
@@ -297,6 +297,11 @@ OutputMixer::GetOutputVolumePan(float& left, float& right)
return 0;
}
int OutputMixer::GetOutputChannelCount()
{
return _audioFrame.num_channels_;
}
int OutputMixer::StartRecordingPlayout(const char* fileName,
const CodecInst* codecInst)
{
media/webrtc/trunk/webrtc/voice_engine/output_mixer.h
+2
View File
@@ -117,6 +117,8 @@ public:
// so ExternalPlayoutData() can insert far-end audio from the audio drivers
void APMAnalyzeReverseStream(AudioFrame &audioFrame);
int GetOutputChannelCount();
private:
OutputMixer(uint32_t instanceId);
int InsertInbandDtmfTone();
media/webrtc/trunk/webrtc/voice_engine/voe_external_media_impl.cc
+15 -5
View File
@@ -214,7 +214,10 @@ int VoEExternalMediaImpl::ExternalRecordingInsertData(
}
uint16_t blockSize = samplingFreqHz / 100;
uint32_t nBlocks = lengthSamples / blockSize;
// We know the number of samples for 10ms of audio, so we can derive the
// number of channels here:
uint32_t channels = lengthSamples * 100 / samplingFreqHz;
uint32_t nBlocks = lengthSamples / blockSize / channels;
int16_t totalDelayMS = 0;
uint16_t playoutDelayMS = 0;
@@ -242,7 +245,7 @@ int VoEExternalMediaImpl::ExternalRecordingInsertData(
shared_->transmit_mixer()->PrepareDemux(
(const int8_t*)(&speechData10ms[i*blockSize]),
blockSize,
1,
channels,
samplingFreqHz,
totalDelayMS,
0,
@@ -380,16 +383,23 @@ int VoEExternalMediaImpl::ExternalPlayoutGetData(
AudioFrame audioFrame;
uint32_t channels = shared_->output_mixer()->GetOutputChannelCount();
// If we have not received any data yet, consider it's mono since it's the
// most common case.
if (channels == 0) {
channels = 1;
}
// Retrieve mixed output at the specified rate
shared_->output_mixer()->MixActiveChannels();
shared_->output_mixer()->DoOperationsOnCombinedSignal(true);
shared_->output_mixer()->GetMixedAudio(samplingFreqHz, 1, &audioFrame);
shared_->output_mixer()->GetMixedAudio(samplingFreqHz, channels, &audioFrame);
// Deliver audio (PCM) samples to the external sink
memcpy(speechData10ms,
audioFrame.data_,
sizeof(int16_t)*(audioFrame.samples_per_channel_));
lengthSamples = audioFrame.samples_per_channel_;
sizeof(int16_t)*audioFrame.samples_per_channel_*channels);
lengthSamples = audioFrame.samples_per_channel_ * channels;
// Store current playout delay (to be used by ExternalRecordingInsertData).
playout_delay_ms_ = current_delay_ms;
toolkit/components/telemetry/Histograms.json
+6
View File
@@ -5095,6 +5095,12 @@
"n_buckets": "1000",
"description": "The time (in milliseconds) that it took a 'enumProperties' request to go round trip."
},
"MEDIA_WMF_DECODE_ERROR": {
"expires_in_version": "50",
"kind": "enumerated",
"n_values": 256,
"description": "WMF media decoder error or success (0) codes."
},
"VIDEO_CANPLAYTYPE_H264_CONSTRAINT_SET_FLAG": {
"expires_in_version": "40",
"kind": "enumerated",

Some files were not shown because too many files have changed in this diff Show More