[Question] Monitor the actual camera images as received by the observation

[torydebra]

Question

Hi,

I noticed that the camera images as seen in the GUI are not the ones that are received as observation.
For example, in the Isaac-Repose-Cube-Shadow-Vision-Direct-v0 environment the camera resolution is 120x120, indeed this print(self._tiled_camera.data.output["rgb"].shape) prints (torch.Size([1, 120, 120, 3])) (with --num_envs 1). But looking at the camera view in the GUI, resolution is surely higher:

Indeed the camera view seems affected only by the other parameters (focal_length, focal_distance, apertures, and so on, or the intrinsic matrix ), but the resolution is always the same.

I would like to monitor the actual ouput of the camera, taken into the observations. Is there a simple way to do this?
My main concern is that the camera view in the GUI is very different (like cropped differently) from both Gazebo (ROS simulator) and real world, given the same resolution and intrinsic parameters. So I would like to understand what the observation images are actually like

@garylvov Sorry to ping you but I saw #1074

[garylvov]

~~
Hi I'm on vacation until the fall before starting my PhD so I apologize for the brief response I don't have my computer with me.

You can try to use the camera tool I made https://isaac-sim.github.io/IsaacLab/main/source/how-to/estimate_how_many_cameras_can_run.html or the simple camera tool https://isaac-sim.github.io/IsaacLab/main/source/how-to/save_camera_output.html to easily save images to file with your env, but tbh it's been a while since it's been updated so I'm not sure if it will work.

Alternatively,

I'd recommend simply saving the image observation to file in the direct environment with something like cv2 or PIL or matplotlib. I'm not aware how to set viewport maybe @kellyguo11 or @jtigue-bdai have better ideas

For the manager based environments, check out the examples I made for cartpole for the image observation term. It's in like mdp observations the source itself, check out how I inherit the obs term with image features, and do something similar where you just write to file.

~~

Edit: see James' comment

[jtigue-bdai]

You have the camera images (rgb or depth) as observation outputs you should be able to view them in the debug visualizers: #893

[garylvov]

Didn't realize debug visualizers merged I agree with James over my response

[torydebra]

Thanks for the answers, I missed that how-to guides, anyway useful for also other things. For some reason in the scene debug visualization I can not see anything, but I managed to get the observation and output the image values into a file as suggested.

[garylvov]

Feel free to open a new bug report for the debug visualizer if it's giving you trouble!

I'd also try running things from within the latest docker image as it is the most reproducible environment (in my opinion).