KotJohansson/palemoon27
1
0
Fork 0
mirror of https://github.com/roytam1/palemoon27.git synced 2026-05-26 14:30:27 +00:00
Code Issues Projects Releases Wiki Activity

import changes from `dev' branch of rmottola/Arctic-Fox:

Browse Source 
- Bug 1150437 - Make gmp-clearkey consistently Apache2 licensed. r=edwin,jwwang,kentuckyfriedtakahe (3c954c73dd)
- Bug 1223486 - Ensure WrapTexCoord returns 0.0 instead of 1.0 when given a negative whole number. r=Bas (b4af8eb524)
- Bug 1160914 - Make gmp-clearkey's decoders threadsafe refcounted, to handle DecodingComplete while GMPVideoHost::CreateFrame() is waiting. r=edwin (03359c7730)
- Bug 1206763 - Enable SkiaGL canvas on gonk r=mattwoodrow (1588079b23)
- Decouple CairoImage from ImageContainer. (bug 1222910, r=roc) (a93a4dad82)
- Decouple D3D11ShareHandleImage from ImageContainer. (bug 1222910, r=mattwoodrow) (9642ef1c53)
- Bug 1194753 - Wait longer when decoding the first frame of D3D9 DXVA video before deciding it is invalid. r=cpearce (865fe1d2f3)
- Decouple D3D9SurfaceImage from ImageContainer. (bug 1222910, r=mattwoodrow) (55bc5abffc)
- Bug 1191534 - Implement BlitImageToFramebuffer for IOSurface. r=jgilbert (0da6aff77f)
- Decouple EGLImageImage from ImageContainer. (bug 1222910, r=snorp) (cffb7aadcb)
- Decouple SurfaceTextureImage from ImageContainer. (bug 1222910, r=snorp) (5b920014d0)
- Decouple MacIOSurfaceImage from ImageContainer. (bug 1222910, r=mstange) (ffab69155c)
- Decouple SharedRGBImage and PlanarYCbCrImage from ImageContainer. (bug 1222910, r=mattwoodrow) (a9f6e23788)
- Remove Gonk usage of ImageContainer::CreateImage. (bug 1222910, r=sotaro) (3f460b901d)
- Remove CreateImage() from ImageContainer, ImageFactory, and ImageClient. (bug 1222910, r=roc) (873d5d5e47)
- Bug 1212795: P1. Ensure the check if HW acceleration is allowed is performed on the main thread. r=jwwang (7cac257e5a)
- Bug 1137529: Prefer Apple's VDA hardware acceleration for Hi-Def videos. r=rillian (a3ffc48e5e)
- Bug 1212795: P2. Ensure all frameworks required for video decoding on mac are loaded. r=jwwang (b903b94c60)
- Bug 1212795: P3. Make AppleDecoderModule detects if the required modules are loaded. r=jwwang (fb132e36b0)
- Bug 1221991 - [1.3] Make SupportsMimeType a const function. r=jya (2450030808)
- Bug 1221991 - [2.3] Split DecoderLoop into functions, simplify decoding state and clean up style. r=snorp (7c0e665cf7)
- Bug 1221991 - [3.1] Manage module state via accessor functions. r=snorp (491b00f74b)
- Bug 1221991 - [4.2] Add AndroidDecoderModule logging. r=snorp (7213505016)
- Bug 1153110: Don't prefer VDA for 720p and above. r=rillian (50d5fbfa7c)
- remove PR_LOGGING (44f4efbaa2)
- Bug 1169653 - Limit use of the speech recognition API in JS to certified apps or apps with the proper flags set. r=smaug (c93a8949b8)
- Bug 1133633: Part2. Enable async decoding on mac. r=mattmoodrow (c9740168bd)
- Bug 1154896: Report decoding errors back to MP4Reader. r=rillian (26291f72e1)
- Bug 1154896: Part2. Ignore kVTDecodeInfo_FrameDropped flag. r=rillian (bfca4c264e)
- Bug 1198094: P1. Limit rate at which InputExhausted could be called by mac decoder. r=rillian (e5819c1666)
- Bug 1214678 - Allow Apple media decoders to work on iOS r=jya (c475eff0be)
- Bug 1219140 - Remove AskMediaCodecAndWait() r=bwu (dbe8d364ec)
- fix includes (432c3419b0)
- adapted Bug 1159509 - Support audio AMR-NB for Gonk in MP4Reader. r=jya (adaddc5c21)
- Bug 1154512 - Remove MediaTaskQueue::SyncDispatch() from PDM. r=cpearce (5bddd082bd)
- Bug 1204622 - release codec listener at reader task queue. r=jya (72b062b5d7)
- Bug 1097498 - Wait fence for the graphic buffer. r=sotaro (c72f301939)
- Bug 1198664 - Refactor: move common behaviors to base class. r=bwu,jya (a754813c5b)
- Bug 1198664 - Use looper to process decoder tasks. r=bwu (768315de13)
- Bug 1174721 - Use AudioCompactor for GonkAudioDecoderManager. r=sotaro (768e68c946)
- Bug 1199809 - Refactor: use RAII to help manage output buffer lifecycle. r=sotaro (9a6e0dceb7)
- Bug 1133955 - Record the last seek time to decide the direction to seek. r=sotaro (d0bcdbe68b)
- Bug 1210045 - Fix GonkVideoDecoderManager shutdown during initialization r=bwu (b379248ff1)
- Bug 1214997 - Use MozPromise in MediaCodecProxy and OMXCodecProxy r=bwu (ab9a92d640)
- Bug 1199809 - Make a copy of output buffer after flush(). r=sotaro (17465ecaea)
- Bug 1216895 - assert that decoder methods are run on correct thread. r=jya (b8c106c2ca)
- Bug 1185018 - Part 1 of 1 - Made speech recognition services language dependent and removed assumption of a single service. r=smaug (f83acf82b4)
- Bug 1207416: Add Telemetry for time spent in different Load states in WebRTC r=gcp (e3959b6760)
- Bug 1219480 - Replace PRLogModuleInfo with LazyLogModule in the media directory. r=rillian (a9b13ef334)
- Bug 1196558: Don't assert should output task not be dispatched. r=edwin (a2569bca73)
- Bug 1199193: Ensure DrainComplete() is called once all decoded frames have been output. r=rillian (760fb42753)
- Bug 1079621 - Change non-fatal errors to warnings. r=rillian (20c85e7e56)
- Bug 1185234 - Implement SpeechRecognition::interimResults. r=smaug (b3d635d8c0)
- Bug 1171850 - Remove 'models' dir from MODELSPS_DEST as the dir 'models' now comes from MODELSPS_FILES. r=gps (5515511e3a)
- Bug 1183503 - Rename model and dictionary files for recognition in preparation for localization. r=smaug (32c78610af)
- Bug 1185235 - Implement SpeechRecognition::maxAlternatives. r=smaug (91eac7f30f)
- Bug 1177514 - Remove final text of 'ERROR' on recognition error, should be signaled by SpeechRecognitionError. r=smaug (c8d7b198e0)
- Bug 1187791 - Part 1 of 1 - SpeechRecognition::maxAlternatives can't throw; so, rm webidl throws specifier. r=smaug (e295b2bde5)
- Bug 1213131: [vpx] Configure libvpx decoder to use multi-threads decoding. r=kentuckyfriedtakahe (f38ba0ef82)
- Bug 1206845: Prevent overflows in MediaCache. r=roc (e8a228dcd3)
- Bug 1205825 - part 1 - don't reacquire the media cache's monitor in MediaCacheStream::FlushPartialBlockInternal; r=roc (6342592be3)
- Bug 1205825 - part 2 - call Get*CachedData*Internal variants from MediaCacheStream::GetCachedRanges; r=roc (f229c8f757)
This commit is contained in:
roytam1
2023-01-10 13:45:00 +08:00
parent f29aba536a
commit 147c5fa2eb
251 changed files with 3263 additions and 2534 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dom/media/platforms/android/AndroidDecoderModule.cpp
+392 -267
View File
@@ -12,18 +12,34 @@
#include "GLLibraryEGL.h"
#include "MediaData.h"
#include "MediaInfo.h"
#include "nsThreadUtils.h"
#include "nsAutoPtr.h"
#include "nsPromiseFlatString.h"
#include "prlog.h"
#include <jni.h>
static PRLogModuleInfo* AndroidDecoderModuleLog()
{
static PRLogModuleInfo* sLogModule = nullptr;
if (!sLogModule) {
sLogModule = PR_NewLogModule("AndroidDecoderModule");
}
return sLogModule;
}
#undef LOG
#define LOG(arg, ...) MOZ_LOG(AndroidDecoderModuleLog(), \
mozilla::LogLevel::Debug, ("AndroidDecoderModule(%p)::%s: " arg, \
this, __func__, ##__VA_ARGS__))
using namespace mozilla;
using namespace mozilla::gl;
using namespace mozilla::widget::sdk;
using media::TimeUnit;
namespace mozilla {
@@ -31,10 +47,11 @@ namespace mozilla {
if (mCallback) { \
mCallback->Func(__VA_ARGS__); \
} else { \
NS_WARNING("callback not set"); \
NS_WARNING("Callback not set"); \
}
static const char* TranslateMimeType(const nsACString& aMimeType)
static const char*
TranslateMimeType(const nsACString& aMimeType)
{
if (aMimeType.EqualsLiteral("video/webm; codecs=vp8")) {
return "video/x-vnd.on2.vp8";
@@ -44,7 +61,8 @@ static const char* TranslateMimeType(const nsACString& aMimeType)
return PromiseFlatCString(aMimeType).get();
}
static MediaCodec::LocalRef CreateDecoder(const nsACString& aMimeType)
static MediaCodec::LocalRef
CreateDecoder(const nsACString& aMimeType)
{
MediaCodec::LocalRef codec;
NS_ENSURE_SUCCESS(MediaCodec::CreateDecoderByType(TranslateMimeType(aMimeType),
@@ -52,19 +70,23 @@ static MediaCodec::LocalRef CreateDecoder(const nsACString& aMimeType)
return codec;
}
class VideoDataDecoder : public MediaCodecDataDecoder {
class VideoDataDecoder : public MediaCodecDataDecoder
{
public:
VideoDataDecoder(const VideoInfo& aConfig,
MediaFormat::Param aFormat, MediaDataDecoderCallback* aCallback,
MediaFormat::Param aFormat,
MediaDataDecoderCallback* aCallback,
layers::ImageContainer* aImageContainer)
: MediaCodecDataDecoder(MediaData::Type::VIDEO_DATA, aConfig.mMimeType, aFormat, aCallback)
: MediaCodecDataDecoder(MediaData::Type::VIDEO_DATA, aConfig.mMimeType,
aFormat, aCallback)
, mImageContainer(aImageContainer)
, mConfig(aConfig)
{
}
RefPtr<InitPromise> Init() override {
RefPtr<InitPromise> Init() override
{
mSurfaceTexture = AndroidSurfaceTexture::Create();
if (!mSurfaceTexture) {
NS_WARNING("Failed to create SurfaceTexture for video decode\n");
@@ -78,26 +100,36 @@ public:
return InitPromise::CreateAndResolve(TrackInfo::kVideoTrack, __func__);
}
void Cleanup() override {
void Cleanup() override
{
mGLContext = nullptr;
}
nsresult Input(MediaRawData* aSample) override {
nsresult Input(MediaRawData* aSample) override
{
return MediaCodecDataDecoder::Input(aSample);
}
bool WantCopy() {
// Allocating a texture is incredibly slow on PowerVR
return mGLContext->Vendor() != GLVendor::Imagination;
bool WantCopy() const
{
// Allocating a texture is incredibly slow on PowerVR and may fail on
// emulators, see bug 1190379.
return mGLContext->Vendor() != GLVendor::Imagination &&
mGLContext->Renderer() != GLRenderer::AndroidEmulator;
}
EGLImage CopySurface(layers::Image* img) {
EGLImage CopySurface(layers::Image* img)
{
mGLContext->MakeCurrent();
GLuint tex = CreateTextureForOffscreen(mGLContext, mGLContext->GetGLFormats(), img->GetSize());
GLuint tex = CreateTextureForOffscreen(mGLContext, mGLContext->GetGLFormats(),
img->GetSize());
GLBlitHelper helper(mGLContext);
if (!helper.BlitImageToTexture(img, img->GetSize(), tex, LOCAL_GL_TEXTURE_2D)) {
auto helper = mGLContext->BlitHelper();
const gl::OriginPos destOrigin = gl::OriginPos::TopLeft;
if (!helper->BlitImageToTexture(img, img->GetSize(), tex,
LOCAL_GL_TEXTURE_2D, destOrigin))
{
mGLContext->fDeleteTextures(1, &tex);
return nullptr;
}
@@ -108,28 +140,24 @@ public:
};
EGLContext eglContext = static_cast<GLContextEGL*>(mGLContext.get())->GetEGLContext();
EGLImage eglImage = sEGLLibrary.fCreateImage(EGL_DISPLAY(), eglContext,
LOCAL_EGL_GL_TEXTURE_2D_KHR,
(EGLClientBuffer)tex, attribs);
EGLImage eglImage = sEGLLibrary.fCreateImage(
EGL_DISPLAY(), eglContext, LOCAL_EGL_GL_TEXTURE_2D_KHR,
reinterpret_cast<EGLClientBuffer>(tex), attribs);
mGLContext->fDeleteTextures(1, &tex);
return eglImage;
}
nsresult PostOutput(BufferInfo::Param aInfo, MediaFormat::Param aFormat,
const media::TimeUnit& aDuration) override {
const TimeUnit& aDuration) override
{
if (!EnsureGLContext()) {
return NS_ERROR_FAILURE;
}
RefPtr<layers::Image> img = mImageContainer->CreateImage(ImageFormat::SURFACE_TEXTURE);
layers::SurfaceTextureImage::Data data;
data.mSurfTex = mSurfaceTexture.get();
data.mSize = mConfig.mDisplay;
data.mOriginPos = gl::OriginPos::BottomLeft;
layers::SurfaceTextureImage* stImg = static_cast<layers::SurfaceTextureImage*>(img.get());
stImg->SetData(data);
RefPtr<layers::Image> img =
new SurfaceTextureImage(mSurfaceTexture.get(), mConfig.mDisplay,
gl::OriginPos::BottomLeft);
if (WantCopy()) {
EGLImage eglImage = CopySurface(img);
@@ -151,16 +179,8 @@ public:
NS_WARNING("No EGL fence support detected, rendering artifacts may occur!");
}
img = mImageContainer->CreateImage(ImageFormat::EGLIMAGE);
layers::EGLImageImage::Data data;
data.mImage = eglImage;
data.mSync = eglSync;
data.mOwns = true;
data.mSize = mConfig.mDisplay;
data.mOriginPos = gl::OriginPos::BottomLeft;
layers::EGLImageImage* typedImg = static_cast<layers::EGLImageImage*>(img.get());
typedImg->SetData(data);
img = new layers::EGLImageImage(eglImage, eglSync, mConfig.mDisplay,
gl::OriginPos::TopLeft, true /* owns */);
}
nsresult rv;
@@ -192,7 +212,8 @@ public:
}
protected:
bool EnsureGLContext() {
bool EnsureGLContext()
{
if (mGLContext) {
return true;
}
@@ -207,33 +228,33 @@ protected:
RefPtr<GLContext> mGLContext;
};
class AudioDataDecoder : public MediaCodecDataDecoder {
private:
uint8_t csd0[2];
class AudioDataDecoder : public MediaCodecDataDecoder
{
public:
AudioDataDecoder(const AudioInfo& aConfig, MediaFormat::Param aFormat, MediaDataDecoderCallback* aCallback)
: MediaCodecDataDecoder(MediaData::Type::AUDIO_DATA, aConfig.mMimeType, aFormat, aCallback)
AudioDataDecoder(const AudioInfo& aConfig, MediaFormat::Param aFormat,
MediaDataDecoderCallback* aCallback)
: MediaCodecDataDecoder(MediaData::Type::AUDIO_DATA, aConfig.mMimeType,
aFormat, aCallback)
{
JNIEnv* env = GetJNIForThread();
JNIEnv* const env = jni::GetEnvForThread();
jni::Object::LocalRef buffer(env);
NS_ENSURE_SUCCESS_VOID(aFormat->GetByteBuffer(NS_LITERAL_STRING("csd-0"), &buffer));
NS_ENSURE_SUCCESS_VOID(aFormat->GetByteBuffer(NS_LITERAL_STRING("csd-0"),
&buffer));
if (!buffer && aConfig.mCodecSpecificConfig->Length() >= 2) {
csd0[0] = (*aConfig.mCodecSpecificConfig)[0];
csd0[1] = (*aConfig.mCodecSpecificConfig)[1];
buffer = jni::Object::LocalRef::Adopt(env, env->NewDirectByteBuffer(csd0, 2));
NS_ENSURE_SUCCESS_VOID(aFormat->SetByteBuffer(NS_LITERAL_STRING("csd-0"), buffer));
buffer = jni::Object::LocalRef::Adopt(
env, env->NewDirectByteBuffer(aConfig.mCodecSpecificConfig->Elements(),
aConfig.mCodecSpecificConfig->Length()));
NS_ENSURE_SUCCESS_VOID(aFormat->SetByteBuffer(NS_LITERAL_STRING("csd-0"),
buffer));
}
}
nsresult Output(BufferInfo::Param aInfo, void* aBuffer,
MediaFormat::Param aFormat,
const media::TimeUnit& aDuration) {
MediaFormat::Param aFormat, const TimeUnit& aDuration)
{
// The output on Android is always 16-bit signed
nsresult rv;
int32_t numChannels;
NS_ENSURE_SUCCESS(rv =
@@ -250,7 +271,7 @@ public:
NS_ENSURE_SUCCESS(rv = aInfo->Offset(&offset), rv);
#ifdef MOZ_SAMPLE_TYPE_S16
int32_t numSamples = size / 2;
const int32_t numSamples = size / 2;
#else
#error We only support 16-bit integer PCM
#endif
@@ -258,8 +279,9 @@ public:
const int32_t numFrames = numSamples / numChannels;
auto audio = MakeUnique<AudioDataValue[]>(numSamples);
uint8_t* bufferStart = static_cast<uint8_t*>(aBuffer) + offset;
PodCopy(audio.get(), reinterpret_cast<AudioDataValue*>(bufferStart), numSamples);
const uint8_t* bufferStart = static_cast<uint8_t*>(aBuffer) + offset;
PodCopy(audio.get(), reinterpret_cast<const AudioDataValue*>(bufferStart),
numSamples);
int64_t presentationTimeUs;
NS_ENSURE_SUCCESS(rv = aInfo->PresentationTimeUs(&presentationTimeUs), rv);
@@ -275,26 +297,32 @@ public:
}
};
bool AndroidDecoderModule::SupportsMimeType(const nsACString& aMimeType)
bool
AndroidDecoderModule::SupportsMimeType(const nsACString& aMimeType) const
{
if (!AndroidBridge::Bridge() || (AndroidBridge::Bridge()->GetAPIVersion() < 16)) {
if (!AndroidBridge::Bridge() ||
(AndroidBridge::Bridge()->GetAPIVersion() < 16)) {
return false;
}
if (aMimeType.EqualsLiteral("video/mp4") ||
aMimeType.EqualsLiteral("video/avc")) {
return true;
}
return static_cast<bool>(mozilla::CreateDecoder(aMimeType));
MediaCodec::LocalRef ref = mozilla::CreateDecoder(aMimeType);
if (!ref) {
return false;
}
ref->Release();
return true;
}
already_AddRefed<MediaDataDecoder>
AndroidDecoderModule::CreateVideoDecoder(
const VideoInfo& aConfig,
layers::LayersBackend aLayersBackend,
layers::ImageContainer* aImageContainer,
FlushableTaskQueue* aVideoTaskQueue,
MediaDataDecoderCallback* aCallback)
const VideoInfo& aConfig, layers::LayersBackend aLayersBackend,
layers::ImageContainer* aImageContainer, FlushableTaskQueue* aVideoTaskQueue,
MediaDataDecoderCallback* aCallback)
{
MediaFormat::LocalRef format;
@@ -311,9 +339,9 @@ AndroidDecoderModule::CreateVideoDecoder(
}
already_AddRefed<MediaDataDecoder>
AndroidDecoderModule::CreateAudioDecoder(const AudioInfo& aConfig,
FlushableTaskQueue* aAudioTaskQueue,
MediaDataDecoderCallback* aCallback)
AndroidDecoderModule::CreateAudioDecoder(
const AudioInfo& aConfig, FlushableTaskQueue* aAudioTaskQueue,
MediaDataDecoderCallback* aCallback)
{
MOZ_ASSERT(aConfig.mBitDepth == 16, "We only handle 16-bit audio!");
@@ -329,7 +357,6 @@ AndroidDecoderModule::CreateAudioDecoder(const AudioInfo& aConfig,
new AudioDataDecoder(aConfig, format, aCallback);
return decoder.forget();
}
PlatformDecoderModule::ConversionRequired
@@ -337,9 +364,8 @@ AndroidDecoderModule::DecoderNeedsConversion(const TrackInfo& aConfig) const
{
if (aConfig.IsVideo()) {
return kNeedAnnexB;
} else {
return kNeedNone;
}
return kNeedNone;
}
MediaCodecDataDecoder::MediaCodecDataDecoder(MediaData::Type aType,
@@ -353,9 +379,7 @@ MediaCodecDataDecoder::MediaCodecDataDecoder(MediaData::Type aType,
, mInputBuffers(nullptr)
, mOutputBuffers(nullptr)
, mMonitor("MediaCodecDataDecoder::mMonitor")
, mFlushing(false)
, mDraining(false)
, mStopping(false)
, mState(kDecoding)
{
}
@@ -365,7 +389,8 @@ MediaCodecDataDecoder::~MediaCodecDataDecoder()
Shutdown();
}
RefPtr<MediaDataDecoder::InitPromise> MediaCodecDataDecoder::Init()
RefPtr<MediaDataDecoder::InitPromise>
MediaCodecDataDecoder::Init()
{
nsresult rv = InitDecoder(nullptr);
@@ -375,10 +400,12 @@ RefPtr<MediaDataDecoder::InitPromise> MediaCodecDataDecoder::Init()
return NS_SUCCEEDED(rv) ?
InitPromise::CreateAndResolve(type, __func__) :
InitPromise::CreateAndReject(MediaDataDecoder::DecoderFailureReason::INIT_ERROR, __func__);
InitPromise::CreateAndReject(
MediaDataDecoder::DecoderFailureReason::INIT_ERROR, __func__);
}
nsresult MediaCodecDataDecoder::InitDecoder(Surface::Param aSurface)
nsresult
MediaCodecDataDecoder::InitDecoder(Surface::Param aSurface)
{
mDecoder = CreateDecoder(mMimeType);
if (!mDecoder) {
@@ -399,224 +426,312 @@ nsresult MediaCodecDataDecoder::InitDecoder(Surface::Param aSurface)
return NS_OK;
}
// This is in usec, so that's 10ms
#define DECODER_TIMEOUT 10000
// This is in usec, so that's 10ms.
static const int64_t kDecoderTimeout = 10000;
#define HANDLE_DECODER_ERROR() \
#define BREAK_ON_DECODER_ERROR() \
if (NS_FAILED(res)) { \
NS_WARNING("exiting decoder loop due to exception"); \
if (mDraining) { \
NS_WARNING("Exiting decoder loop due to exception"); \
if (State() == kDrainDecoder) { \
INVOKE_CALLBACK(DrainComplete); \
mDraining = false; \
State(kDecoding); \
} \
INVOKE_CALLBACK(Error); \
break; \
}
nsresult MediaCodecDataDecoder::GetInputBuffer(JNIEnv* env, int index, jni::Object::LocalRef* buffer)
nsresult
MediaCodecDataDecoder::GetInputBuffer(
JNIEnv* aEnv, int aIndex, jni::Object::LocalRef* aBuffer)
{
bool retried = false;
while (!*buffer) {
*buffer = jni::Object::LocalRef::Adopt(env->GetObjectArrayElement(mInputBuffers.Get(), index));
if (!*buffer) {
if (!retried) {
// Reset the input buffers and then try again
nsresult res = ResetInputBuffers();
if (NS_FAILED(res)) {
return res;
}
retried = true;
} else {
// We already tried resetting the input buffers, return an error
return NS_ERROR_FAILURE;
}
MOZ_ASSERT(aEnv);
MOZ_ASSERT(!*aBuffer);
int numTries = 2;
while (numTries--) {
*aBuffer = jni::Object::LocalRef::Adopt(
aEnv->GetObjectArrayElement(mInputBuffers.Get(), aIndex));
if (*aBuffer) {
return NS_OK;
}
nsresult res = ResetInputBuffers();
if (NS_FAILED(res)) {
return res;
}
}
return NS_ERROR_FAILURE;
}
bool
MediaCodecDataDecoder::WaitForInput()
{
MonitorAutoLock lock(mMonitor);
while (State() == kDecoding && mQueue.empty()) {
// Signal that we require more input.
INVOKE_CALLBACK(InputExhausted);
lock.Wait();
}
return State() != kStopping;
}
MediaRawData*
MediaCodecDataDecoder::PeekNextSample()
{
MonitorAutoLock lock(mMonitor);
if (State() == kFlushing) {
mDecoder->Flush();
ClearQueue();
State(kDecoding);
lock.Notify();
return nullptr;
}
if (mQueue.empty()) {
if (State() == kDrainQueue) {
State(kDrainDecoder);
}
return nullptr;
}
// We're not stopping or flushing, so try to get a sample.
return mQueue.front();
}
nsresult
MediaCodecDataDecoder::QueueSample(const MediaRawData* aSample)
{
MOZ_ASSERT(aSample);
AutoLocalJNIFrame frame(jni::GetEnvForThread(), 1);
// We have a sample, try to feed it to the decoder.
int32_t inputIndex = -1;
nsresult res = mDecoder->DequeueInputBuffer(kDecoderTimeout, &inputIndex);
if (NS_FAILED(res)) {
return res;
}
if (inputIndex < 0) {
// There is no valid input buffer available.
return NS_ERROR_FAILURE;
}
jni::Object::LocalRef buffer(frame.GetEnv());
res = GetInputBuffer(frame.GetEnv(), inputIndex, &buffer);
if (NS_FAILED(res)) {
return res;
}
void* directBuffer = frame.GetEnv()->GetDirectBufferAddress(buffer.Get());
MOZ_ASSERT(frame.GetEnv()->GetDirectBufferCapacity(buffer.Get()) >=
aSample->Size(),
"Decoder buffer is not large enough for sample");
PodCopy(static_cast<uint8_t*>(directBuffer), aSample->Data(), aSample->Size());
res = mDecoder->QueueInputBuffer(inputIndex, 0, aSample->Size(),
aSample->mTime, 0);
if (NS_FAILED(res)) {
return res;
}
mDurations.push(TimeUnit::FromMicroseconds(aSample->mDuration));
return NS_OK;
}
nsresult
MediaCodecDataDecoder::QueueEOS()
{
mMonitor.AssertCurrentThreadOwns();
nsresult res = NS_OK;
int32_t inputIndex = -1;
res = mDecoder->DequeueInputBuffer(kDecoderTimeout, &inputIndex);
if (NS_FAILED(res) || inputIndex < 0) {
return res;
}
res = mDecoder->QueueInputBuffer(inputIndex, 0, 0, 0,
MediaCodec::BUFFER_FLAG_END_OF_STREAM);
if (NS_SUCCEEDED(res)) {
State(kDrainWaitEOS);
mMonitor.Notify();
}
return res;
}
void
MediaCodecDataDecoder::HandleEOS(int32_t aOutputStatus)
{
MonitorAutoLock lock(mMonitor);
if (State() == kDrainWaitEOS) {
State(kDecoding);
mMonitor.Notify();
INVOKE_CALLBACK(DrainComplete);
}
mDecoder->ReleaseOutputBuffer(aOutputStatus, false);
}
TimeUnit
MediaCodecDataDecoder::GetOutputDuration()
{
MOZ_ASSERT(!mDurations.empty(), "Should have had a duration queued");
const TimeUnit duration = mDurations.front();
mDurations.pop();
return duration;
}
nsresult
MediaCodecDataDecoder::ProcessOutput(
BufferInfo::Param aInfo, MediaFormat::Param aFormat, int32_t aStatus)
{
AutoLocalJNIFrame frame(jni::GetEnvForThread(), 1);
const TimeUnit duration = GetOutputDuration();
const auto buffer = jni::Object::LocalRef::Adopt(
frame.GetEnv()->GetObjectArrayElement(mOutputBuffers.Get(), aStatus));
if (buffer) {
// The buffer will be null on Android L if we are decoding to a Surface.
void* directBuffer = frame.GetEnv()->GetDirectBufferAddress(buffer.Get());
Output(aInfo, directBuffer, aFormat, duration);
}
// The Surface will be updated at this point (for video).
mDecoder->ReleaseOutputBuffer(aStatus, true);
PostOutput(aInfo, aFormat, duration);
return NS_OK;
}
void MediaCodecDataDecoder::DecoderLoop()
void
MediaCodecDataDecoder::DecoderLoop()
{
bool outputDone = false;
bool draining = false;
bool waitingEOF = false;
AutoLocalJNIFrame frame(GetJNIForThread(), 1);
bool isOutputDone = false;
AutoLocalJNIFrame frame(jni::GetEnvForThread(), 1);
RefPtr<MediaRawData> sample;
MediaFormat::LocalRef outputFormat(frame.GetEnv());
nsresult res;
nsresult res = NS_OK;
while (WaitForInput()) {
sample = PeekNextSample();
for (;;) {
{
MonitorAutoLock lock(mMonitor);
while (!mStopping && !mDraining && !mFlushing && mQueue.empty()) {
if (mQueue.empty()) {
// We could be waiting here forever if we don't signal that we need more input
INVOKE_CALLBACK(InputExhausted);
}
lock.Wait();
}
if (mStopping) {
// Get out of the loop. This is the only exit point.
break;
}
if (mFlushing) {
mDecoder->Flush();
ClearQueue();
mFlushing = false;
lock.Notify();
continue;
}
// We're not stopping or draining, so try to get a sample
if (!mQueue.empty()) {
sample = mQueue.front();
}
if (mDraining && !sample && !waitingEOF) {
draining = true;
}
}
if (draining && !waitingEOF) {
MOZ_ASSERT(!sample, "Shouldn't have a sample when pushing EOF frame");
int32_t inputIndex;
res = mDecoder->DequeueInputBuffer(DECODER_TIMEOUT, &inputIndex);
HANDLE_DECODER_ERROR();
if (inputIndex >= 0) {
res = mDecoder->QueueInputBuffer(inputIndex, 0, 0, 0, MediaCodec::BUFFER_FLAG_END_OF_STREAM);
HANDLE_DECODER_ERROR();
waitingEOF = true;
if (State() == kDrainDecoder) {
MOZ_ASSERT(!sample, "Shouldn't have a sample when pushing EOF frame");
res = QueueEOS();
BREAK_ON_DECODER_ERROR();
}
}
if (sample) {
// We have a sample, try to feed it to the decoder
int inputIndex;
res = mDecoder->DequeueInputBuffer(DECODER_TIMEOUT, &inputIndex);
HANDLE_DECODER_ERROR();
if (inputIndex >= 0) {
jni::Object::LocalRef buffer(frame.GetEnv());
res = GetInputBuffer(frame.GetEnv(), inputIndex, &buffer);
HANDLE_DECODER_ERROR();
void* directBuffer = frame.GetEnv()->GetDirectBufferAddress(buffer.Get());
MOZ_ASSERT(frame.GetEnv()->GetDirectBufferCapacity(buffer.Get()) >= sample->Size(),
"Decoder buffer is not large enough for sample");
{
// We're feeding this to the decoder, so remove it from the queue
MonitorAutoLock lock(mMonitor);
mQueue.pop();
}
PodCopy((uint8_t*)directBuffer, sample->Data(), sample->Size());
res = mDecoder->QueueInputBuffer(inputIndex, 0, sample->Size(),
sample->mTime, 0);
HANDLE_DECODER_ERROR();
mDurations.push(media::TimeUnit::FromMicroseconds(sample->mDuration));
sample = nullptr;
outputDone = false;
res = QueueSample(sample);
if (NS_SUCCEEDED(res)) {
// We've fed this into the decoder, so remove it from the queue.
MonitorAutoLock lock(mMonitor);
mQueue.pop();
isOutputDone = false;
}
}
if (!outputDone) {
BufferInfo::LocalRef bufferInfo;
res = BufferInfo::New(&bufferInfo);
HANDLE_DECODER_ERROR();
if (isOutputDone) {
continue;
}
int32_t outputStatus;
res = mDecoder->DequeueOutputBuffer(bufferInfo, DECODER_TIMEOUT, &outputStatus);
HANDLE_DECODER_ERROR();
BufferInfo::LocalRef bufferInfo;
nsresult res = BufferInfo::New(&bufferInfo);
BREAK_ON_DECODER_ERROR();
if (outputStatus == MediaCodec::INFO_TRY_AGAIN_LATER) {
// We might want to call mCallback->InputExhausted() here, but there seems to be
// some possible bad interactions here with the threading
} else if (outputStatus == MediaCodec::INFO_OUTPUT_BUFFERS_CHANGED) {
res = ResetOutputBuffers();
HANDLE_DECODER_ERROR();
} else if (outputStatus == MediaCodec::INFO_OUTPUT_FORMAT_CHANGED) {
res = mDecoder->GetOutputFormat(ReturnTo(&outputFormat));
HANDLE_DECODER_ERROR();
} else if (outputStatus < 0) {
NS_WARNING("unknown error from decoder!");
INVOKE_CALLBACK(Error);
int32_t outputStatus = -1;
res = mDecoder->DequeueOutputBuffer(bufferInfo, kDecoderTimeout,
&outputStatus);
BREAK_ON_DECODER_ERROR();
// Don't break here just in case it's recoverable. If it's not, others stuff will fail later and
// we'll bail out.
} else {
int32_t flags;
res = bufferInfo->Flags(&flags);
HANDLE_DECODER_ERROR();
if (outputStatus == MediaCodec::INFO_TRY_AGAIN_LATER) {
// We might want to call mCallback->InputExhausted() here, but there seems
// to be some possible bad interactions here with the threading.
} else if (outputStatus == MediaCodec::INFO_OUTPUT_BUFFERS_CHANGED) {
res = ResetOutputBuffers();
BREAK_ON_DECODER_ERROR();
} else if (outputStatus == MediaCodec::INFO_OUTPUT_FORMAT_CHANGED) {
res = mDecoder->GetOutputFormat(ReturnTo(&outputFormat));
BREAK_ON_DECODER_ERROR();
} else if (outputStatus < 0) {
NS_WARNING("Unknown error from decoder!");
INVOKE_CALLBACK(Error);
// Don't break here just in case it's recoverable. If it's not, other
// stuff will fail later and we'll bail out.
} else {
// We have a valid buffer index >= 0 here.
int32_t flags;
nsresult res = bufferInfo->Flags(&flags);
BREAK_ON_DECODER_ERROR();
// We have a valid buffer index >= 0 here
if (flags & MediaCodec::BUFFER_FLAG_END_OF_STREAM) {
if (draining) {
draining = false;
waitingEOF = false;
mMonitor.Lock();
mDraining = false;
mMonitor.Notify();
mMonitor.Unlock();
INVOKE_CALLBACK(DrainComplete);
}
mDecoder->ReleaseOutputBuffer(outputStatus, false);
outputDone = true;
// We only queue empty EOF frames, so we're done for now
continue;
}
MOZ_ASSERT(!mDurations.empty(), "Should have had a duration queued");
media::TimeUnit duration;
if (!mDurations.empty()) {
duration = mDurations.front();
mDurations.pop();
}
auto buffer = jni::Object::LocalRef::Adopt(
frame.GetEnv()->GetObjectArrayElement(mOutputBuffers.Get(), outputStatus));
if (buffer) {
// The buffer will be null on Android L if we are decoding to a Surface
void* directBuffer = frame.GetEnv()->GetDirectBufferAddress(buffer.Get());
Output(bufferInfo, directBuffer, outputFormat, duration);
}
// The Surface will be updated at this point (for video)
mDecoder->ReleaseOutputBuffer(outputStatus, true);
PostOutput(bufferInfo, outputFormat, duration);
if (flags & MediaCodec::BUFFER_FLAG_END_OF_STREAM) {
HandleEOS(outputStatus);
isOutputDone = true;
// We only queue empty EOF frames, so we're done for now.
continue;
}
res = ProcessOutput(bufferInfo, outputFormat, outputStatus);
BREAK_ON_DECODER_ERROR();
}
}
Cleanup();
// We're done
// We're done.
MonitorAutoLock lock(mMonitor);
mStopping = false;
State(kShutdown);
mMonitor.Notify();
}
void MediaCodecDataDecoder::ClearQueue()
const char*
MediaCodecDataDecoder::ModuleStateStr(ModuleState aState) {
static const char* kStr[] = {
"Decoding", "Flushing", "DrainQueue", "DrainDecoder", "DrainWaitEOS",
"Stopping", "Shutdown"
};
MOZ_ASSERT(aState < sizeof(kStr) / sizeof(kStr[0]));
return kStr[aState];
}
MediaCodecDataDecoder::ModuleState
MediaCodecDataDecoder::State() const
{
return mState;
}
void
MediaCodecDataDecoder::State(ModuleState aState)
{
LOG("%s -> %s", ModuleStateStr(mState), ModuleStateStr(aState));
if (aState == kDrainDecoder) {
MOZ_ASSERT(mState == kDrainQueue);
} else if (aState == kDrainWaitEOS) {
MOZ_ASSERT(mState == kDrainDecoder);
}
mState = aState;
}
void
MediaCodecDataDecoder::ClearQueue()
{
mMonitor.AssertCurrentThreadOwns();
while (!mQueue.empty()) {
mQueue.pop();
}
@@ -625,7 +740,9 @@ void MediaCodecDataDecoder::ClearQueue()
}
}
nsresult MediaCodecDataDecoder::Input(MediaRawData* aSample) {
nsresult
MediaCodecDataDecoder::Input(MediaRawData* aSample)
{
MonitorAutoLock lock(mMonitor);
mQueue.push(aSample);
lock.NotifyAll();
@@ -633,53 +750,61 @@ nsresult MediaCodecDataDecoder::Input(MediaRawData* aSample) {
return NS_OK;
}
nsresult MediaCodecDataDecoder::ResetInputBuffers()
nsresult
MediaCodecDataDecoder::ResetInputBuffers()
{
return mDecoder->GetInputBuffers(ReturnTo(&mInputBuffers));
}
nsresult MediaCodecDataDecoder::ResetOutputBuffers()
nsresult
MediaCodecDataDecoder::ResetOutputBuffers()
{
return mDecoder->GetOutputBuffers(ReturnTo(&mOutputBuffers));
}
nsresult MediaCodecDataDecoder::Flush() {
nsresult
MediaCodecDataDecoder::Flush()
{
MonitorAutoLock lock(mMonitor);
mFlushing = true;
State(kFlushing);
lock.Notify();
while (mFlushing) {
while (State() == kFlushing) {
lock.Wait();
}
return NS_OK;
}
nsresult MediaCodecDataDecoder::Drain() {
nsresult
MediaCodecDataDecoder::Drain()
{
MonitorAutoLock lock(mMonitor);
if (mDraining) {
if (State() == kDrainDecoder || State() == kDrainQueue) {
return NS_OK;
}
mDraining = true;
State(kDrainQueue);
lock.Notify();
return NS_OK;
}
nsresult MediaCodecDataDecoder::Shutdown() {
nsresult
MediaCodecDataDecoder::Shutdown()
{
MonitorAutoLock lock(mMonitor);
if (!mThread || mStopping) {
if (!mThread || State() == kStopping) {
// Already shutdown or in the process of doing so
return NS_OK;
}
mStopping = true;
State(kStopping);
lock.Notify();
while (mStopping) {
while (State() == kStopping) {
lock.Wait();
}