FFMPEG录屏（15）---- WGC 捕获桌面(三) WGC(Windows Graphics Capture)采集

admin管理员组

文章数量:1576278

前言

前两篇已经通过官网Demo对WGC采集方式进行了验证和扩展，现在开始正片~

FFMPEG录屏（13）---- WGC 捕获桌面(一) 改造官网Demo
FFMPEG录屏（14）---- WGC 捕获桌面(二) Copy数据到CPU

参考资料

New Ways to do Screen Capture
Windows.UI.Composition-Win32-Samples
WebRtc WGC

限制

WindowsGraphicsCapture APIs first shipped in the Windows 10 April 2018 Update (1803). These APIs were built for developers who depended on screen capture functionality for their modern applications without depending on restricted capabilities. These APIs enable capture of application windows, displays, and environments in a secure, easy to use way with the use of a system picker UI control.

C++/WinRT is an entirely standard modern C++17 language projection for Windows Runtime (WinRT) APIs, implemented as a header-file-based library, and designed to provide you with first-class access to the modern Windows API. With C++/WinRT, you can author and consume Windows Runtime APIs using any standards-compliant C++17 compiler. The Windows SDK includes C++/WinRT; it was introduced in version 10.0.17134.0 (Windows 10, version 1803).

综上想要基于最新的捕获技术WindowsGraphicsCapture进行图像捕获有以下限制

• 系统版本不低于10.0.17134.0 (Windows 10, version 1803)
• 相关接口均基于微软的新一代运行时库接口C++/WinRT，而且是最低要求 C++17

目前大多数项目和很多成熟项目中一般C++版本最高也才到C++14，所以我们一般会把WGC功能封装进一个动态库中，在使用时进行动态加载。

导出

1. 在export.h中声明宏用以导出函数

#ifdef AMRECORDER_IMPORT
#define AMRECORDER_API extern "C" __declspec(dllimport)
#else
#define AMRECORDER_API extern "C" __declspec(dllexport)
#endif

2. 在export.h中定义WGC模块接口类wgc_session

namespace am {

class wgc_session {
public:
  struct wgc_session_frame {
    unsigned int width;
    unsigned int height;
    unsigned int row_pitch;

    const unsigned char *data;
  };

  class wgc_session_observer {
  public:
    virtual ~wgc_session_observer() {}
    virtual void on_frame(const wgc_session_frame &frame) = 0;
  };

public:
  virtual void release() = 0;

  virtual int initialize(HWND hwnd) = 0;
  virtual int initialize(HMONITOR hmonitor) = 0;

  virtual void register_observer(wgc_session_observer *observer) = 0;

  virtual int start() = 0;
  virtual int stop() = 0;

  virtual int pause() = 0;
  virtual int resume() = 0;

protected:
  virtual ~wgc_session(){};
};

} // namespace am

3. 在export.h中定义函数用以判断当前系统是否支持使用WGC进行采集

AMRECORDER_API bool wgc_is_supported();

4. 在export.h中定义接口函数用以创建wgc_session类实例

AMRECORDER_API am::wgc_session *wgc_create_session();

实现

1. 在预编译头文件pch.h中引入使用到的头文件

// pch.h: This is a precompiled header file.
// Files listed below are compiled only once, improving build performance for
// future builds. This also affects IntelliSense performance, including code
// completion and many code browsing features. However, files listed here are
// ALL re-compiled if any one of them is updated between builds. Do not add
// files here that you will be updating frequently as this negates the
// performance advantage.

#ifndef PCH_H
#define PCH_H

// add headers that you want to pre-compile here
#define WIN32_LEAN_AND_MEAN             // Exclude rarely-used stuff from Windows headers
// Windows Header Files

#include <Unknwn.h>
#include <inspectable.h>

// WinRT
#include <winrt/Windows.Foundation.h>
#include <winrt/Windows.System.h>
#include <winrt/Windows.Graphics.DirectX.h>
#include <winrt/Windows.Graphics.DirectX.Direct3d11.h>
#include <winrt/Windows.Graphics.Capture.h>
#include <Windows.Graphics.Capture.Interop.h>

#include <DispatcherQueue.h>


// STL
#include <atomic>
#include <memory>

// D3D
#include <d3d11_4.h>
#include <dxgi1_6.h>
#include <d2d1_3.h>
#include <wincodec.h>

// windowws

#include <Windows.h>

#include "../Recorder/error_define.h"
#include "export.h"
#include "wgc_session_impl.h"

#endif // PCH_H

2. 在export.cpp中实现函数wgc_is_supported()

#include "pch.h"

#include <winrt/Windows.Foundation.Metadata.h>

bool wgc_is_supported() {
  try {
    /* no contract for IGraphicsCaptureItemInterop, verify 10.0.18362.0 */
    return winrt::Windows::Foundation::Metadata::ApiInformation::
        IsApiContractPresent(L"Windows.Foundation.UniversalApiContract", 8);
  } catch (const winrt::hresult_error &) {
    return false;
  } catch (...) {
    return false;
  }
}

3. 在wgc_session_impl.h中声明派生类 wgc_session_impl

#include <mutex>
#include <thread>

namespace am {
class wgc_session_impl : public wgc_session {
  struct __declspec(uuid("A9B3D012-3DF2-4EE3-B8D1-8695F457D3C1"))
      IDirect3DDxgiInterfaceAccess : ::IUnknown {
    virtual HRESULT __stdcall GetInterface(GUID const &id, void **object) = 0;
  };

  struct {
    union {
      HWND hwnd;
      HMONITOR hmonitor;
    };
    bool is_window;
  } target_{0};

public:
  wgc_session_impl();
  ~wgc_session_impl();

public:
  void release() override;

  int initialize(HWND hwnd) override;
  int initialize(HMONITOR hmonitor) override;

  void register_observer(wgc_session_observer *observer) override;

  int start() override;
  int stop() override;

  int pause() override;
  int resume() override;

private:
  auto create_d3d11_device();
  auto create_capture_item(HWND hwnd);
  auto create_capture_item(HMONITOR hmonitor);
  template <typename T>
  auto
  get_dxgi_interface(winrt::Windows::Foundation::IInspectable const &object);
  HRESULT create_mapped_texture(winrt::com_ptr<ID3D11Texture2D> src_texture,
                                unsigned int width = 0,
                                unsigned int height = 0);
  void
  on_frame(winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool const
               &sender,
           winrt::Windows::Foundation::IInspectable const &args);
  void on_closed(winrt::Windows::Graphics::Capture::GraphicsCaptureItem const &,
                 winrt::Windows::Foundation::IInspectable const &);

  int initialize();
  void cleanup();

  void message_func();

private:
  std::mutex lock_;
  bool is_initialized_ = false;
  bool is_running_ = false;
  bool is_paused_ = false;

  wgc_session_observer *observer_ = nullptr;

  // wgc
  winrt::Windows::Graphics::Capture::GraphicsCaptureItem capture_item_{nullptr};
  winrt::Windows::Graphics::Capture::GraphicsCaptureSession capture_session_{
      nullptr};
  winrt::Windows::Graphics::SizeInt32 capture_frame_size_;

  winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice
      d3d11_direct_device_{nullptr};
  winrt::com_ptr<ID3D11DeviceContext> d3d11_device_context_{nullptr};
  winrt::com_ptr<ID3D11Texture2D> d3d11_texture_mapped_{nullptr};

  std::atomic<bool> cleaned_ = false;
  winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool
      capture_framepool_{nullptr};
  winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool::
      FrameArrived_revoker capture_framepool_trigger_;
  winrt::Windows::Graphics::Capture::GraphicsCaptureItem::Closed_revoker
      capture_close_trigger_;

  // message loop
  std::thread loop_;
  HWND hwnd_ = nullptr;
};

template <typename T>
inline auto wgc_session_impl::get_dxgi_interface(
    winrt::Windows::Foundation::IInspectable const &object) {
  auto access = object.as<IDirect3DDxgiInterfaceAccess>();
  winrt::com_ptr<T> result;
  winrt::check_hresult(
      access->GetInterface(winrt::guid_of<T>(), result.put_void()));
  return result;
}


} // namespace am

4. 在中实现wgc_session_impl相关逻辑

#include "pch.h"

#include <functional>
#include <memory>

#define CHECK_INIT                                                             \
  if (!is_initialized_)                                                        \
  return AM_ERROR::AE_NEED_INIT

#define CHECK_CLOSED                                                           \
  if (cleaned_.load() == true) {                                               \
    throw winrt::hresult_error(RO_E_CLOSED);                                   \
  }

extern "C" {
HRESULT __stdcall CreateDirect3D11DeviceFromDXGIDevice(
    ::IDXGIDevice *dxgiDevice, ::IInspectable **graphicsDevice);
}

namespace am {

wgc_session_impl::wgc_session_impl() {}

wgc_session_impl::~wgc_session_impl() {
  stop();
  cleanup();
}

void wgc_session_impl::release() { delete this; }

int wgc_session_impl::initialize(HWND hwnd) {
  std::lock_guard locker(lock_);

  target_.hwnd = hwnd;
  target_.is_window = true;
  return initialize();
}

int wgc_session_impl::initialize(HMONITOR hmonitor) {
  std::lock_guard locker(lock_);

  target_.hmonitor = hmonitor;
  target_.is_window = false;
  return initialize();
}

void wgc_session_impl::register_observer(wgc_session_observer *observer) {
  std::lock_guard locker(lock_);
  observer_ = observer;
}

int wgc_session_impl::start() {
  std::lock_guard locker(lock_);

  if (is_running_)
    return AM_ERROR::AE_NO;

  int error = AM_ERROR::AE_WGC_CREATE_CAPTURER_FAILED;

  CHECK_INIT;
  try {
    if (!capture_session_) {
      auto current_size = capture_item_.Size();
      capture_framepool_ =
          winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool::
              CreateFreeThreaded(d3d11_direct_device_,
                                 winrt::Windows::Graphics::DirectX::
                                     DirectXPixelFormat::B8G8R8A8UIntNormalized,
                                 2, current_size);
      capture_session_ = capture_framepool_.CreateCaptureSession(capture_item_);
      capture_frame_size_ = current_size;
      capture_framepool_trigger_ = capture_framepool_.FrameArrived(
          winrt::auto_revoke, {this, &wgc_session_impl::on_frame});
      capture_close_trigger_ = capture_item_.Closed(
          winrt::auto_revoke, {this, &wgc_session_impl::on_closed});
    }

    if (!capture_framepool_)
      throw std::exception();

    is_running_ = true;

    // we do not need to crate a thread to enter a message loop coz we use
    // CreateFreeThreaded instead of Create to create a capture frame pool,
    // we need to test the performance later
    // loop_ = std::thread(std::bind(&wgc_session_impl::message_func, this));

    capture_session_.StartCapture();

    error = AM_ERROR::AE_NO;
  } catch (winrt::hresult_error) {
    return AM_ERROR::AE_WGC_CREATE_CAPTURER_FAILED;
  } catch (...) {
    return AM_ERROR::AE_WGC_CREATE_CAPTURER_FAILED;
  }

  return error;
}

int wgc_session_impl::stop() {
  std::lock_guard locker(lock_);

  CHECK_INIT;

  is_running_ = false;

  if (loop_.joinable())
    loop_.join();

  if (capture_framepool_trigger_)
    capture_framepool_trigger_.revoke();

  if (capture_session_) {
    capture_session_.Close();
    capture_session_ = nullptr;
  }

  return AM_ERROR::AE_NO;
}

int wgc_session_impl::pause() {
  std::lock_guard locker(lock_);

  CHECK_INIT;
  return AM_ERROR::AE_NO;
}

int wgc_session_impl::resume() {
  std::lock_guard locker(lock_);

  CHECK_INIT;
  return AM_ERROR::AE_NO;
}

auto wgc_session_impl::create_d3d11_device() {
  auto create_d3d_device = [](D3D_DRIVER_TYPE const type,
                              winrt::com_ptr<ID3D11Device> &device) {
    WINRT_ASSERT(!device);

    UINT flags = D3D11_CREATE_DEVICE_BGRA_SUPPORT;

    //#ifdef _DEBUG
    //	flags |= D3D11_CREATE_DEVICE_DEBUG;
    //#endif

    return ::D3D11CreateDevice(nullptr, type, nullptr, flags, nullptr, 0,
                               D3D11_SDK_VERSION, device.put(), nullptr,
                               nullptr);
  };
  auto create_d3d_device_wrapper = [&create_d3d_device]() {
    winrt::com_ptr<ID3D11Device> device;
    HRESULT hr = create_d3d_device(D3D_DRIVER_TYPE_HARDWARE, device);

    if (DXGI_ERROR_UNSUPPORTED == hr) {
      hr = create_d3d_device(D3D_DRIVER_TYPE_WARP, device);
    }

    winrt::check_hresult(hr);
    return device;
  };

  auto d3d_device = create_d3d_device_wrapper();
  auto dxgi_device = d3d_device.as<IDXGIDevice>();

  winrt::com_ptr<::IInspectable> d3d11_device;
  winrt::check_hresult(CreateDirect3D11DeviceFromDXGIDevice(
      dxgi_device.get(), d3d11_device.put()));
  return d3d11_device
      .as<winrt::Windows::Graphics::DirectX::Direct3D11::IDirect3DDevice>();
}

auto wgc_session_impl::create_capture_item(HWND hwnd) {
  auto activation_factory = winrt::get_activation_factory<
      winrt::Windows::Graphics::Capture::GraphicsCaptureItem>();
  auto interop_factory = activation_factory.as<IGraphicsCaptureItemInterop>();
  winrt::Windows::Graphics::Capture::GraphicsCaptureItem item = {nullptr};
  interop_factory->CreateForWindow(
      hwnd,
      winrt::guid_of<ABI::Windows::Graphics::Capture::IGraphicsCaptureItem>(),
      reinterpret_cast<void **>(winrt::put_abi(item)));
  return item;
}

auto wgc_session_impl::create_capture_item(HMONITOR hmonitor) {
  auto activation_factory = winrt::get_activation_factory<
      winrt::Windows::Graphics::Capture::GraphicsCaptureItem>();
  auto interop_factory = activation_factory.as<IGraphicsCaptureItemInterop>();
  winrt::Windows::Graphics::Capture::GraphicsCaptureItem item = {nullptr};
  interop_factory->CreateForMonitor(
      hmonitor,
      winrt::guid_of<ABI::Windows::Graphics::Capture::IGraphicsCaptureItem>(),
      reinterpret_cast<void **>(winrt::put_abi(item)));
  return item;
}

HRESULT wgc_session_impl::create_mapped_texture(
    winrt::com_ptr<ID3D11Texture2D> src_texture, unsigned int width,
    unsigned int height) {

  D3D11_TEXTURE2D_DESC src_desc;
  src_texture->GetDesc(&src_desc);
  D3D11_TEXTURE2D_DESC map_desc;
  map_desc.Width = width == 0 ? src_desc.Width : width;
  map_desc.Height = height == 0 ? src_desc.Height : height;
  map_desc.MipLevels = src_desc.MipLevels;
  map_desc.ArraySize = src_desc.ArraySize;
  map_desc.Format = src_desc.Format;
  map_desc.SampleDesc = src_desc.SampleDesc;
  map_desc.Usage = D3D11_USAGE_STAGING;
  map_desc.BindFlags = 0;
  map_desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
  map_desc.MiscFlags = 0;

  auto d3dDevice = get_dxgi_interface<ID3D11Device>(d3d11_direct_device_);

  return d3dDevice->CreateTexture2D(&map_desc, nullptr,
                                    d3d11_texture_mapped_.put());
}

void wgc_session_impl::on_frame(
    winrt::Windows::Graphics::Capture::Direct3D11CaptureFramePool const &sender,
    winrt::Windows::Foundation::IInspectable const &args) {
  std::lock_guard locker(lock_);

  auto is_new_size = false;

  {
    auto frame = sender.TryGetNextFrame();
    auto frame_size = frame.ContentSize();

    if (frame_size.Width != capture_frame_size_.Width ||
        frame_size.Height != capture_frame_size_.Height) {
      // The thing we have been capturing has changed size.
      // We need to resize our swap chain first, then blit the pixels.
      // After we do that, retire the frame and then recreate our frame pool.
      is_new_size = true;
      capture_frame_size_ = frame_size;
    }

    // copy to mapped texture
    {
      auto frame_captured =
          get_dxgi_interface<ID3D11Texture2D>(frame.Surface());

      if (!d3d11_texture_mapped_ || is_new_size)
        create_mapped_texture(frame_captured);

      d3d11_device_context_->CopyResource(d3d11_texture_mapped_.get(),
                                          frame_captured.get());

      D3D11_MAPPED_SUBRESOURCE map_result;
      HRESULT hr = d3d11_device_context_->Map(
          d3d11_texture_mapped_.get(), 0, D3D11_MAP_READ,
          0 /*coz we use CreateFreeThreaded, so we cant use flags
               D3D11_MAP_FLAG_DO_NOT_WAIT*/
          ,
          &map_result);
      if (FAILED(hr)) {
        OutputDebugStringW(
            (L"map resource failed: " + std::to_wstring(hr)).c_str());
      }

      // copy data from map_result.pData
      if (map_result.pData && observer_) {
        observer_->on_frame(wgc_session_frame{
            static_cast<unsigned int>(frame_size.Width),
            static_cast<unsigned int>(frame_size.Height), map_result.RowPitch,
            const_cast<const unsigned char *>(
                (unsigned char *)map_result.pData)});
      }


      d3d11_device_context_->Unmap(d3d11_texture_mapped_.get(), 0);
    }
  }

  if (is_new_size) {
    capture_framepool_.Recreate(d3d11_direct_device_,
                                winrt::Windows::Graphics::DirectX::
                                    DirectXPixelFormat::B8G8R8A8UIntNormalized,
                                2, capture_frame_size_);
  }
}

void wgc_session_impl::on_closed(
    winrt::Windows::Graphics::Capture::GraphicsCaptureItem const &,
    winrt::Windows::Foundation::IInspectable const &) {
  OutputDebugStringW(L"wgc_session_impl::on_closed");
}

int wgc_session_impl::initialize() {
  if (is_initialized_)
    return AM_ERROR::AE_NO;

  if (!(d3d11_direct_device_ = create_d3d11_device()))
    return AM_ERROR::AE_D3D_CREATE_DEVICE_FAILED;

  try {
    if (target_.is_window)
      capture_item_ = create_capture_item(target_.hwnd);
    else
      capture_item_ = create_capture_item(target_.hmonitor);

    // Set up
    auto d3d11_device = get_dxgi_interface<ID3D11Device>(d3d11_direct_device_);
    d3d11_device->GetImmediateContext(d3d11_device_context_.put());

  } catch (winrt::hresult_error) {
    return AM_ERROR::AE_WGC_CREATE_CAPTURER_FAILED;
  } catch (...) {
    return AM_ERROR::AE_WGC_CREATE_CAPTURER_FAILED;
  }

  is_initialized_ = true;

  return AM_ERROR::AE_NO;
}

void wgc_session_impl::cleanup() {
  std::lock_guard locker(lock_);

  auto expected = false;
  if (cleaned_.compare_exchange_strong(expected, true)) {
    capture_close_trigger_.revoke();
    capture_framepool_trigger_.revoke();

    if (capture_framepool_)
      capture_framepool_.Close();

    if (capture_session_)
      capture_session_.Close();

    capture_framepool_ = nullptr;
    capture_session_ = nullptr;
    capture_item_ = nullptr;

    is_initialized_ = false;
  }
}

LRESULT CALLBACK WindowProc(HWND window, UINT message, WPARAM w_param,
                            LPARAM l_param) {
  return DefWindowProc(window, message, w_param, l_param);
}

void wgc_session_impl::message_func() {
  const std::wstring kClassName = L"am_fake_window";

  WNDCLASS wc = {};

  wc.style = CS_HREDRAW | CS_VREDRAW;
  wc.lpfnWndProc = DefWindowProc;
  wc.hCursor = LoadCursor(nullptr, IDC_ARROW);
  wc.hbrBackground = reinterpret_cast<HBRUSH>(COLOR_WINDOW);
  wc.lpszClassName = kClassName.c_str();

  if (!::RegisterClassW(&wc))
    return;

  hwnd_ = ::CreateWindowW(kClassName.c_str(), nullptr, WS_OVERLAPPEDWINDOW, 0,
                          0, 0, 0, nullptr, nullptr, nullptr, nullptr);
  MSG msg;
  while (is_running_) {
    while (PeekMessage(&msg, NULL, 0, 0, PM_REMOVE)) {
      if (!is_running_)
        break;
      TranslateMessage(&msg);
      DispatchMessage(&msg);
    }
    Sleep(10);
  }

  ::CloseWindow(hwnd_);
  ::DestroyWindow(hwnd_);
}

} // namespace am

需要注意的是，在创建frame_pool时，我们使用的是

Direct3D11CaptureFramePool::CreateFreeThreaded

而非官网Demo中使用的

Direct3D11CaptureFramePool::Create

因此并没有启动message_loop的线程，解释可以参考 Direct3D11CaptureFramePool.CreateFreeThreaded Method

5. 在export.cpp中实现wgc_create_session

am::wgc_session *wgc_create_session() { return new am::wgc_session_impl(); }

至此我们已经完成了对WGC功能模块的封装，支持采集指定的桌面或窗口。

美中不足

1. 鼠标支持，自Windows 10, version 2004 (introduced in 10.0.19041.0)才开始支持捕获鼠标。

   GraphicsCaptureSession.IsCursorCaptureEnabled Property

2. 黄色边框去除，自Windows 10, version 2104 (introduced in 10.0.20348.0)才开始去除采集目标的黄色边框。

   GraphicsCaptureSession.IsBorderRequired

结尾

完整DEMO已经上传，还是老地方 screen-recorder

本文标签： 桌面WGCffmpegGraphicsWindows