{"id":624658,"date":"2024-06-18T17:30:04","date_gmt":"2024-06-18T14:30:04","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/researchers-leverage-shadows-to-model-3d-scenes-including-objects-blocked-from-view\/"},"modified":"2024-06-18T17:30:04","modified_gmt":"2024-06-18T14:30:04","slug":"researchers-leverage-shadows-to-model-3d-scenes-including-objects-blocked-from-view","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/researchers-leverage-shadows-to-model-3d-scenes-including-objects-blocked-from-view\/","title":{"rendered":"#Researchers leverage shadows to model 3D scenes, including objects blocked from view"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2624576d56e\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2624576d56e\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/researchers-leverage-shadows-to-model-3d-scenes-including-objects-blocked-from-view\/#Shedding_light_on_the_problem\" >Shedding light on the problem<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/researchers-leverage-shadows-to-model-3d-scenes-including-objects-blocked-from-view\/#A_winning_combination\" >A winning combination<\/a><\/li><\/ul><\/nav><\/div>\n<div>\n<div class=\"article-gallery lightGallery\">\n<div data-thumb=\"https:\/\/scx1.b-cdn.net\/csz\/news\/tmb\/2024\/researchers-leverage-s.jpg\" data-src=\"https:\/\/scx2.b-cdn.net\/gfx\/news\/2024\/researchers-leverage-s.jpg\" data-sub-html=\"Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credit: Massachusetts Institute of Technology\">\n<figure class=\"article-img\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/scx1.b-cdn.net\/csz\/news\/800a\/2024\/researchers-leverage-s.jpg\" alt=\"Researchers leverage shadows to model 3D scenes, including objects blocked from view\" title=\"Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credit: Massachusetts Institute of Technology\" width=\"800\" height=\"500\"\/><figcaption class=\"text-darken text-low-up text-truncate-js text-truncate mt-3\">\n                Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credit: Massachusetts Institute of <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">Technology<\/a><br \/>\n            <\/figcaption><\/figure>\n<\/div>\n<\/div>\n<p>Imagine driving through a tunnel in an autonomous vehicle, but unbeknownst to you, a crash has stopped traffic up ahead. Normally, you&#8217;d need to rely on the car in front of you to know you should start braking. But what if your vehicle could see around the car ahead and <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ly the brakes even sooner?<\/p>\n<p>Researchers from MIT and Meta have developed a computer vision technique that could someday enable an autonomous vehicle to do just that.<\/p>\n<p>They have introduced a <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/openaccess.thecvf.com\/content\/CVPR2024\/html\/Klinghoffer_PlatoNeRF_3D_Reconstruction_in_Platos_Cave_via_Single-View_Two-Bounce_Lidar_CVPR_2024_paper.html\">method<\/a> that creates physically accurate, 3D models of an entire scene, including areas blocked from view, using images from a single camera position. Their technique uses shadows to determine what lies in obstructed portions of the scene.<\/p>\n<p>They call their approach PlatoNeRF, based on Plato&#8217;s allegory of the cave, a passage from the Greek philosopher&#8217;s &#8220;Republic&#8221; in which prisoners chained in a cave discern the reality of the outside world based on shadows cast on the cave wall.<\/p>\n<p>By combining lidar (light detection and ranging) technology with machine learning, PlatoNeRF can generate more accurate reconstructions of 3D geometry than some existing AI techniques. Additionally, PlatoNeRF is better at smoothly reconstructing scenes where shadows are hard to see, such as those with high ambient light or dark backgrounds.<\/p>\n<p>In addition to improving the safety of autonomous vehicles, PlatoNeRF could make AR\/VR headsets more efficient by enabling a user to model the geometry of a room without the need to walk around taking measurements. It could also help warehouse robots find items in cluttered environments faster.<\/p>\n<p>&#8220;Our key idea was taking these two things that have been done in different disciplines before and pulling them together\u2014multibounce lidar and machine learning. It turns out that when you bring these two together, that is when you find a lot of new opportunities to explore and get the best of both worlds,&#8221; says Tzofi Klinghoffer, an MIT graduate student in <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a> arts and <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/sciencee\/\" data-internallinksmanager029f6b8e52c=\"5\" title=\"Science\" target=\"_blank\" rel=\"noopener\">science<\/a>s, affiliate of the MIT Media Lab, and lead author of the paper on PlatoNeRF.<\/p>\n<p>Klinghoffer wrote the paper with his advisor, Ramesh Raskar, associate professor of media arts and sciences and leader of the Camera Culture Group at MIT; senior author Rakesh Ranjan, a director of AI research at Meta Reality Labs; as well as Siddharth Somasundaram at MIT, and Xiaoyu Xiang, Yuchen Fan, and Christian Richardt at Meta. The research is being presented at the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/cvpr.thecvf.com\/\">Conference on Computer Vision and Pattern Recognition<\/a>, held 17\u201321 June.<\/p>\n<p>                                                                                                        <!-- TechX - News - In-article --><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Shedding_light_on_the_problem\"><\/span>Shedding light on the problem<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Reconstructing a full 3D scene from one camera viewpoint is a complex problem.<\/p>\n<p>Some machine-learning approaches employ generative AI models that try to guess what lies in the occluded regions, but these models can hallucinate objects that aren&#8217;t really there. Other approaches attempt to infer the shapes of hidden objects using shadows in a color image, but these methods can struggle when shadows are hard to see.<\/p>\n<p>For PlatoNeRF, the MIT researchers built off these approaches using a new sensing modality called single-photon lidar. Lidars map a 3D scene by emitting pulses of light and measuring the time it takes that light to bounce back to the sensor. Because single-photon lidars can detect individual photons, they provide higher-resolution data.<\/p>\n<p>The researchers use a single-photon lidar to illuminate a target point in the scene. Some light bounces off that point and returns directly to the sensor. However, most of the light scatters and bounces off other objects before returning to the sensor. PlatoNeRF relies on these second bounces of light.<\/p>\n<p>By calculating how long it takes light to bounce twice and then return to the lidar sensor, PlatoNeRF captures additional information about the scene, including depth. The second bounce of light also contains information about shadows.<\/p>\n<p>The system traces the secondary rays of light\u2014those that bounce off the target point to other points in the scene\u2014to determine which points lie in shadow (due to an absence of light). Based on the location of these shadows, PlatoNeRF can infer the geometry of hidden objects.<\/p>\n<p>The lidar sequentially illuminates 16 points, capturing multiple images that are used to reconstruct the entire 3D scene.<\/p>\n<p>&#8220;Every time we illuminate a point in the scene, we are creating new shadows. Because we have all these different illumination sources, we have a lot of light rays shooting around, so we are carving out the region that is occluded and lies beyond the visible eye,&#8221; Klinghoffer says.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_winning_combination\"><\/span>A winning combination<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Key to PlatoNeRF is the combination of multibounce lidar with a special type of machine-learning model known as a neural radiance field (NeRF). A NeRF encodes the geometry of a scene into the weights of a neural network, which gives the model a strong ability to interpolate, or estimate, novel views of a scene.<\/p>\n<p>This ability to interpolate also leads to highly accurate scene reconstructions when combined with multibounce lidar, Klinghoffer says.<\/p>\n<p>&#8220;The biggest challenge was figuring out how to combine these two things. We really had to think about the physics of how light is transporting with multibounce lidar and how to model that with machine learning,&#8221; he says.<\/p>\n<p>They compared PlatoNeRF to two common alternative methods, one that only uses lidar and the other that only uses a NeRF with a color image.<\/p>\n<p>They found that their method was able to outperform both techniques, especially when the lidar sensor had lower resolution. This would make their approach more practical to deploy in the real world, where lower resolution sensors are common in commercial devices.<\/p>\n<p>&#8220;About 15 years ago, our group invented the first camera to &#8216;see&#8217; around corners, that works by exploiting multiple bounces of light, or &#8216;echoes of light.&#8217; Those techniques used special lasers and sensors, and used three bounces of light. Since then, lidar technology has become more mainstream, that led to our research on cameras that can see through fog,&#8221; Raskar says.<\/p>\n<p>                                                                                                        <!-- TechX - News - In-article --><\/p>\n<p>                                                                                                                                            &#8220;This new work uses only two bounces of light, which means the signal to noise ratio is very high, and 3D reconstruction quality is impressive.&#8221;<\/p>\n<p>In the future, the researchers want to try tracking more than two bounces of light to see how that could improve scene reconstructions. In addition, they are interested in applying more deep learning techniques and combining PlatoNeRF with color image measurements to capture texture information.<\/p>\n<p>&#8220;While camera images of shadows have long been studied as a means to 3D reconstruction, this work revisits the problem in the context of lidar, demonstrating significant improvements in the accuracy of reconstructed hidden geometry. The work shows how clever algorithms can enable extraordinary capabilities when combined with ordinary sensors\u2014including the lidar systems that many of us now carry in our pocket,&#8221; says David Lindell, an assistant professor in the Department of Computer Science at the University of Toronto, who was not involved with this work.<\/p>\n<div class=\"article-main__more p-4\">\n                                                                                                <strong>More information:<\/strong><br \/>\n                                                PlatoNeRF: <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/openaccess.thecvf.com\/content\/CVPR2024\/html\/Klinghoffer_PlatoNeRF_3D_Reconstruction_in_Platos_Cave_via_Single-View_Two-Bounce_Lidar_CVPR_2024_paper.html\">3D Reconstruction in Plato&#8217;s Cave via Single-View Two-Bounce Lidar<\/a><\/p>\n<\/div>\n<div class=\"d-inline-block text-medium my-4\">\n                                                Provided by<br \/>\n                                                                                                    Massachusetts Institute of Technology<br \/>\n                                                                                                        <a rel=\"nofollow noopener\" target=\"_blank\" class=\"icon_open\" href=\"http:\/\/web.mit.edu\/\"><br \/>\n                                                        <svg><use href=\"https:\/\/techx.b-cdn.net\/tmpl\/v2\/img\/svg\/sprite.svg#icon_open\" x=\"0\" y=\"0\"\/><\/svg><\/a><\/p><\/div>\n<p>                                                                                                                            <i>This story is republished courtesy of MIT <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/news\/\" data-internallinksmanager029f6b8e52c=\"2\" title=\"News\" target=\"_blank\" rel=\"noopener\">News<\/a> (<a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/web.mit.edu\/newsoffice\/\">web.mit.edu\/newsoffice\/<\/a>), a popular site that covers news about MIT research, innovation and teaching.<\/i><\/p>\n<p>                                        <!-- print only --><\/p>\n<div class=\"d-none d-print-block\">\n<p>                                                <strong>Citation<\/strong>:<br \/>\n                                                Researchers leverage shadows to model 3D scenes, including objects blocked from view (2024, June 18)<br \/>\n                                                retrieved 18 June 2024<br \/>\n                                                from https:\/\/techxplore.com\/news\/2024-06-leverage-shadows-3d-scenes-blocked.html<\/p>\n<p>                                            This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no<br \/>\n                                            part may be reproduced without the written permission. The content is provided for information purposes only.<\/p><\/div>\n<\/p><\/div>\n<p><script id=\"facebook-jssdk\" async=\"\" src=\"https:\/\/connect.facebook.net\/en_US\/sdk.js\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more Like this articles, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/science\/\" target=\"_blank\" rel=\"noopener\">Science category.<\/a><\/span><\/strong>\n<\/p><\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/techxplore.com\/news\/2024-06-leverage-shadows-3d-scenes-blocked.html\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credit: Massachusetts Institute of Technology Imagine driving through&#8230;<\/p>\n","protected":false},"author":1,"featured_media":624659,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/scx2.b-cdn.net\/gfx\/news\/2024\/researchers-leverage-s.jpg","fifu_image_alt":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-624658","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-sciencee"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/624658","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=624658"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/624658\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/624659"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=624658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=624658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=624658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}