Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ add 360 and ortho cameras #705

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ThomasParistech
Copy link

Update 'pixel_to_ray' method to handle orthographic and environment (360°) cameras

@Tom94
Copy link
Collaborator

Tom94 commented Jun 21, 2022

Thank you for the contribution. It's great to have support for additional camera modes.

Two small things I think would be prudent to have before I hit the merge button:

  • the inverse of the new camera modes should be implemented in pos_to_pixel. This will allow correct motion vectors to be computed, and thus DLSS to work.
  • a GUI drop-down within the Advanced camera settings section would be great.
  • code style: the new code uses different spacing and newline rules from the rest of the codebase -- that needs to be aligned.

You don't have to do these if you don't want to; I'd be happy to implement these changes/additions myself in the coming weeks as cycles come up. Cheers!

@ThomasParistech
Copy link
Author

Thanks for your quick reply!

  • Since Vulkan isn't installed on my laptop I can't check if DLSS is still working correctly
  • I've skipped the parallax in the Environment camera mode because the function may not be invertible if the parallax shift goes outside of the sphere of radius parallax.z. What do you think?
  • I've replaced the tabulations at the beginning of new lines

@ThomasParistech ThomasParistech force-pushed the add-orthographic-and-environment-cameras branch from 9963578 to b9cf301 Compare July 6, 2022 14:05
@ThomasParistech ThomasParistech force-pushed the add-orthographic-and-environment-cameras branch from b9cf301 to 30fe0fc Compare July 18, 2022 08:58
@ThomasParistech
Copy link
Author

Let me know if you still have some fixes in mind !

@gateway
Copy link

gateway commented Aug 8, 2022

Hate to jump on this but has this been added, I have a insta 360 1 Inch camera I wanted to test out, also if @ThomasParistech could provide some documentation on best usage of 360 data that would be awesome.

@ThomasParistech
Copy link
Author

Hate to jump on this but has this been added, I have a insta 360 1 Inch camera I wanted to test out, also if @ThomasParistech could provide some documentation on best usage of 360 data that would be awesome.

The goal of this PR is to support the rendering of 360 views once the model has been trained with normal images

Camera models are implemented, but input data is currently considered as coming from a standard perspective camera

But we could as well add the possibility to define the camera type to use during training, or even have multimodal training.
For instance by handling an optional camera_type flag in the transform.json

@colinpeng-datascience
Copy link

Hi! Thanks for the great work. How does it work headlessly, can I generate a 360 video from the perspective camera path file?

@gateway
Copy link

gateway commented Aug 11, 2022

any chance to merg this into the branch or how can I test this.. documentation would be appreciated :)

@ThomasParistech
Copy link
Author

The 360 camera model is actually quite simple.
if it helps, I have just written a Medium article which explains well how to map between 3D and 2D in the case of a spherical/360 image :)
https://medium.com/check-visit-computer-vision/understanding-360-images-8e0fcf0ee861

@aaronhdgolf
Copy link

Adding 360 support would be great, Agisoft currently supports 360 images.

As well would be great to get agisoft support direct.

@Tom94 Tom94 force-pushed the master branch 3 times, most recently from b04280e to c1bd002 Compare January 8, 2023 08:34
@ThomasParistech
Copy link
Author

@Tom94 Any suggestion / improvement to do to help this PR being merged ? :)
(other than resolving update conflicts)

Many thanks !

@Tom94
Copy link
Collaborator

Tom94 commented Apr 4, 2023

Hi, sorry for the radio silence -- I actually wanted to merge it a long time ago, but ended up deciding against it in favor of a different approach.

Instant NGP nowadays has a Lens parameter to control how pixels are mapped to directions, and, importantly, this parameter works for both training and rendering. Among others, there already is a latlong and a equirectangular lens for 360 rendering, so only the orthogonal lens from this PR is missing. If you'd like, you could implement it on top of the latest version of the code, but otherwise I'm afraid the codebase has diverged too much from this PR to merge it.

@ThomasParistech
Copy link
Author

Hi, sorry for the radio silence -- I actually wanted to merge it a long time ago, but ended up deciding against it in favor of a different approach.

Instant NGP nowadays has a Lens parameter to control how pixels are mapped to directions, and, importantly, this parameter works for both training and rendering. Among others, there already is a latlong and a equirectangular lens for 360 rendering, so only the orthogonal lens from this PR is missing. If you'd like, you could implement it on top of the latest version of the code, but otherwise I'm afraid the codebase has diverged too much from this PR to merge it.

Thanks for the reply 👍
Ok, then I might open a PR for the orthographic lens when I have time :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants