Access to Rendered Frame buffer of undocked window

[shouvik1984]

Greetings,

Following are my objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels

I took a look at two samples provided in the supplied PDK
A. CameraPDK - Where I am facing the following limitations
Even if I get handle to that window handle possibly using Window services of PDK P3D::IWindowV440 and the details of the custom camera being rendered in it using GetCameraSystem(), there are no interfaces which I can implement to get the access to rendered frame buffer.


B. OpenCVStereoCamera - There are two limitations I am facing there
1. The OpenCVInterface is subscribed to a IVRPlugin interface which limits me from opening a custom camera mentioned in the aircraft.cfg file
2. The openCVRenderEffect plugin which implements OpenCVRenderEffect::UpdateImageData() prevents capturing the DX11 framebuffer from an undocked window

SimConnect APIs are a much more higher level abstraction and does not allow me to tap into the rendering service.

I tried searching the forum an the learning center but could not gather much.

I am in need of urgent help where I am able to meet the objectives mentioned at the beginning of this post. Awaiting for an earliest reply at the earliest.

[shouvik1984]

Any suggestions please.

[ztzhan]

Hi

I think this is a great question. I am also playing around with the OpenCV example but now I have some confusion. I've successfullly loaded the plugin on p3d but I don't know how to use it as I didn't find any usage information of this example. After I launched p3d the plugin was loaded but nothing happens when I enabled VR. Do you know how its functionality can be intrigued in p3d?

Thanks in advance

[shouvik1984]

@ztzhan

You have to follow the code with the OpenCV_DLLMain.cpp file from the DLL Start function. Any callback service that is subscribed to will call the ::Invoke interface method (during an event) implemented in the class which inherits the PDK service classes.

In my opinion you will be able to get the VR stream out once the VR headset is connected and you enable the VR HMD.

Unfortunately this is what I have concluded from the OpenCVStreoCamera example.

[shouvik1984]

I further investigated but could not succeed in getting the Undocked Window displaying a camera from aircraft.cfg rendering into OpenCV Mat.
Can someone please help.

[shouvik1984]

Greetings,

This issue is posing as a major roadblock in our development. Requesting urgent help from the LM P3D team.

Thanks in advance.

[Rob McCarthy]

Hi Shouvik,

Have you looked at the available Rendering Services in the PDK? The D3D12Texture sample shows how to use these services to draw into an existing view.

Regards,
Rob McCarthy

[shouvik1984]

Thanks Rob for the prompt reply.

I am actually not trying to overlay/draw any texture on an existing view. I am looking forward to accomplishing the following three objectives
1. Open a custom camera that I have stored in an aircraft.cfg file in an undocked view
2. Get a realtime access to the DX11 rendered frame buffer from the undocked window (possibly using EVENTID_Frame event)
3. Get the rendered frame buffer in the undocked window to be stored in an OpenCV cv::Mat object as RGBA channels

In summary I have a custom camera with a mount location, FoV,Sensor mode etc. defined in the aircraft.cfg file and I need to get that very view (undocked) into OpenCV Mat object.

Hope I could explain it properly

[shouvik1984]

Any insights yet. Need urgent help.

[Beau Hollis]

You would need to create a post process effect in order to get access to the windows render target. From there you would need to create your own readback buffer and copy the texture into it. Our OpenCV sample does what you need in revers, i think, so it should at least provide the info you need regarding the proper image format and memory layout for reading to/from a DX RGBA8 texture.

Alternatively, we have SimConnect functions for saving the contents of any view to an image or streaming it over the network. In that case, we use a similar internal process where we readback the rt image a frame later and run it through encoding in background job. We don't currently provide PDK access to our internal request system for reading back frame data.