{"id":291291,"date":"2021-07-05T15:47:37","date_gmt":"2021-07-05T12:47:37","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/self-driving-cars-dont-need-lidar\/"},"modified":"2021-07-05T15:47:37","modified_gmt":"2021-07-05T12:47:37","slug":"self-driving-cars-dont-need-lidar","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/","title":{"rendered":"#Self-driving cars don\u2019t need LiDAR"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a34f40411038\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a34f40411038\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#A_general_computer_vision_system\" >A general computer vision system<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#Supervised_learning\" >Supervised learning<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#Auto-labeled_dataset\" >Auto-labeled dataset<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#Hierarchical_deep_learning_architecture\" >Hierarchical deep learning architecture<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#Vertical_integration\" >Vertical integration<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/self-driving-cars-dont-need-lidar\/#Open_issues\" >Open issues<\/a><\/li><\/ul><\/nav><\/div>\n<p>&#8220;<strong>#Self-driving cars don\u2019t need LiDAR<\/strong>&#8221;<\/p>\n<div>What is the <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">technology<\/a> stack you need to create fully autonomous vehicles? Companies and researchers are divided on the answer to that question. <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">App<\/a>roaches to autonomous driving range from just cameras and<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2019\/01\/14\/what-is-computer-vision\/\">computer vision<\/a><span>\u00a0<\/span>to a combination of computer vision and advanced sensors.<\/p>\n<p>Tesla has been a vocal champion for the<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/07\/29\/self-driving-tesla-car-deep-learning\/\">pure vision-based approach to autonomous driving<\/a>, and in this year\u2019s Conference on Computer Vision and Pattern Recognition (CVPR), its chief AI scientist Andrej Karpathy explained why.<\/p>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.youtube.com\/watch?v=eOL_rCK59ZI&amp;t=28286s\">Speaking at CVPR 2021 Workshop on Autonomous Driving<\/a>, Karpathy, who has been leading Tesla\u2019s self-driving efforts in the past years, detailed how the company is developing deep learning systems that only need video input to make sense of the car\u2019s surroundings. He also explained why Tesla is in the best position to make vision-based self-driving cars a reality.<\/p>\n<blockquote class=\"twitter-tweet\" data-width=\"500\" data-dnt=\"true\">\n<p lang=\"en\" dir=\"ltr\">Gave a talk at CVPR over the weekend on our recent work at Tesla Autopilot to estimate very accurate depth, velocity, acceleration with neural nets from vision. Necessary ingredients include: 1M car fleet data engine, strong AI team and a Supercomputer <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/t.co\/osmEEgkgtL\">https:\/\/t.co\/osmEEgkgtL<\/a><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/t.co\/A3F4i948pD\">pic.twitter.com\/A3F4i948pD<\/a><\/p>\n<p>\u2014 Andrej Karpathy (@karpathy) <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/twitter.com\/karpathy\/status\/1407017269083865095?ref_src=twsrc%5Etfw\">June 21, 2021<\/a><\/p>\n<\/blockquote>\n<h2><span class=\"ez-toc-section\" id=\"A_general_computer_vision_system\"><\/span>A general computer vision system<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/01\/28\/deep-learning-explainer\/\">Deep neural networks<\/a><span>\u00a0<\/span>are one of the main components of the self-driving technology stack. Neural networks analyze on-car camera feeds for roads, signs, cars, obstacles, and people.<\/p>\n<p>But deep learning can also make mistakes in detecting objects in images. This is why most self-driving car companies, including Alphabet subsidiary<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/04\/08\/waymo-ceo-reshuffling-self-driving-car-industry\/\">Waymo<\/a>, use lidars, a device that creates 3D maps of the car\u2019s surrounding by emitting laser beams in all directions. Lidars provided added information that can fill the gaps of the neural networks.<\/p>\n<p>However, adding lidars to the self-driving stack comes with its own complications. \u201cYou have to pre-map the environment with the lidar, and then you have to create a high-definition map, and you have to insert all the lanes and how they connect and all the traffic lights,\u201d Karpathy said. \u201cAnd at test time, you are simply localizing to that map to drive around.\u201d<\/p>\n<p>It is extremely difficult to create a precise mapping of every location the self-driving car will be <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/trip-and-travel\/\" data-internallinksmanager029f6b8e52c=\"10\" title=\"Trip &amp; Travel\" target=\"_blank\" rel=\"noopener\">travel<\/a>ing. \u201cIt\u2019s unscalable to collect, build, and maintain these high-definition lidar maps,\u201d Karpathy said. \u201cIt would be extremely difficult to keep this infrastructure up to date.\u201d<\/p>\n<p>Tesla does not use lidars and high-definition maps in its self-driving stack. \u201cEverything that happens, happens for the first time, in the car, based on the videos from the eight cameras that surround the car,\u201d Karpathy said.<\/p>\n<p>The self-driving technology must figure out where the lanes are, where the traffic lights are, what is their status, and which ones are relevant to the vehicle. And it must do all of this without having any predefined information about the roads it is navigating.<\/p>\n<p>Karpathy acknowledged that vision-based autonomous driving is technically more difficult because it requires neural networks that function incredibly well based on the video feeds only. \u201cBut once you actually get it to work, it\u2019s a <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/general\/\" data-internallinksmanager029f6b8e52c=\"3\" title=\"General\" target=\"_blank\" rel=\"noopener\">general<\/a> vision system, and can principally be deployed anywhere on earth,\u201d he said.<\/p>\n<p>With the general vision system, you will no longer need any complementary gear on your car. And Tesla is already moving in this direction, Karpathy says. Previously, the company\u2019s cars used a combination of radar and cameras for self-driving. But it has recently<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.reuters.com\/business\/autos-transportation\/tesla-drops-radar-is-autopilot-system-safe-2021-06-02\/\">started shipping cars without radars<\/a>.<\/p>\n<p>\u201cWe deleted the radar and are driving on vision alone in these cars,\u201d Karpathy said, adding that the reason is that Tesla\u2019s deep learning system has reached the point where it is a hundred times better than the radar, and now the radar is starting to hold things back and is \u201cstarting to contribute noise.\u201d<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Supervised_learning\"><\/span>Supervised learning<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359266 js-lazy\" alt=\"\" width=\"643\" height=\"469\" sizes=\"auto, (max-width: 643px) 100vw, 643px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD.jpeg\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-280x204.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-370x270.jpeg 370w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-185x135.jpeg 185w\"\/><noscript><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359266\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD.jpeg\" alt=\"\" width=\"643\" height=\"469\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-280x204.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-370x270.jpeg 370w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-object-detectionBD-185x135.jpeg 185w\"\/><\/noscript><\/figure>\n<p>The main argument against the pure computer vision approach is that there is uncertainty on whether neural networks can do range-finding and depth estimation without help from lidar depth maps.<\/p>\n<p>\u201cObviously humans drive around with vision, so our neural net is able to process visual input to understand the depth and velocity of objects around us,\u201d Karpathy said. \u201cBut the big question is can the synthetic neural networks do the same. And I think the answer to us internally, in the last few months that we\u2019ve worked on this, is an unequivocal yes.\u201d<\/p>\n<p>Tesla\u2019s engineers wanted to create a deep learning system that could perform<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/06\/21\/object-detection-deep-learning\/\">object detection<\/a><span>\u00a0<\/span>along with depth, velocity, and acceleration. They decided to treat the challenge as a<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/02\/10\/unsupervised-learning-vs-supervised-learning\/\">supervised learning<\/a><span>\u00a0<\/span>problem, in which a neural network learns to detect objects and their associated properties after training on annotated data.<\/p>\n<p>To train their deep learning architecture, the Tesla team needed a massive dataset of millions of videos, carefully annotated with the objects they contain and their properties. Creating datasets for self-driving cars is especially tricky, and the engineers must make sure to include a diverse set of road settings and edge cases that don\u2019t happen very often.<\/p>\n<p>\u201cWhen you have a large, clean, diverse datasets, and you train a large neural network on it, what I\u2019ve seen in practice is\u2026 success is guaranteed,\u201d Karpathy said.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Auto-labeled_dataset\"><\/span>Auto-labeled dataset<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359267 js-lazy\" alt=\"\" width=\"650\" height=\"451\" sizes=\"auto, (max-width: 650px) 100vw, 650px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2.png 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-280x194.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-389x270.png 389w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-195x135.png 195w\"\/><noscript><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359267\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2.png\" alt=\"\" width=\"650\" height=\"451\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2.png 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-280x194.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-389x270.png 389w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Tesla-data-engineering-cycleBD2-195x135.png 195w\"\/><\/noscript><\/figure>\n<p>With millions of camera-equipped cars sold across the world, Tesla is in a great position to collect the data required to train the car vision deep learning model. The Tesla self-driving team accumulated 1.5 petabytes of data consisting of one million 10-second videos and 6 billion objects annotated with bounding boxes, depth, and velocity.<\/p>\n<p>But labeling such a dataset is a great challenge. One approach is to have it annotated manually through<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/fortune.com\/2020\/02\/04\/artificial-intelligence-data-labeling-labelbox\/\">data-labeling companies<\/a><span>\u00a0<\/span>or online platforms such as Amazon Turk. But this would require a massive manual effort, could cost a fortune, and become a very slow process.<\/p>\n<p>Instead, the Tesla team used an auto-labeling technique that involves a combination of neural networks, radar data, and human reviews. Since the dataset is being annotated offline, the neural networks can run the videos back in forth, compare their predictions with the ground truth, and adjust their parameters. This contrasts with test-time inference, where everything happens in real-time and the deep learning models can\u2019t make recourse.<\/p>\n<p>Offline labeling also enabled the engineers to apply very powerful and compute-intensive object detection networks that can\u2019t be deployed on cars and used in real-time, low-latency applications. And they used radar sensor data to further verify the neural network\u2019s inferences. All of this improved the precision of the labeling network.<\/p>\n<p>\u201cIf you\u2019re offline, you have the benefit of hindsight, so you can do a much better job of calmly fusing [different sensor data],\u201d Karpathy said. \u201cAnd in addition, you can involve humans, and they can do cleaning, verification, editing, and so on.\u201d<\/p>\n<p>According to videos Karpathy showed at CVPR, the object detection network remains consistent through debris, dust, and snow clouds.<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1359269 js-lazy\" alt=\"\" width=\"480\" height=\"124\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD3tesla-object-tracking-auto-labeling.gif\"\/><noscript><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-1359269\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD3tesla-object-tracking-auto-labeling.gif\" alt=\"\" width=\"480\" height=\"124\" srcset=\"\"\/><\/noscript><\/figure>\n<\/p>\n<div class=\"wp-block-image\">\n<p>Karpathy did not say how much human effort was required to make the final corrections to the auto-labeling system. But human cognition played a key role in steering the auto-labeling system in the right direction.<\/p>\n<p>While developing the dataset, the Tesla team found more than 200 triggers that indicated the object detection needed adjustments. These included problems such as inconsistency between detection results in different cameras or between the camera and the radar. They also identified scenarios that might need special care such as tunnel entry and exit and cars with objects on top.<\/p>\n<p>It took four months to develop and master all these triggers. As the labeling network became better, it was deployed in \u201cshadow mode,\u201d which means it is installed in consumer vehicles and run silently without issuing commands to the car. The network\u2019s output is compared to that of the legacy network, the radar, and the driver\u2019s behavior.<\/p>\n<p>The Tesla team went through seven iterations of data engineering. They started with an initial dataset on which they trained their neural network. They then deployed the deep learning in shadow mode on real cars and used the triggers to detect inconsistencies, errors, and special scenarios. The errors were then revised, corrected, and if necessary, new data was added to the dataset.<\/p>\n<p>\u201cWe spin this loop over and over again until the network becomes incredibly good,\u201d Karpathy said.<\/p>\n<p>So, the architecture can better be described as a semi-auto labeling system with an<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/06\/14\/google-reinforcement-learning-ai-chip-design\/\">ingenious division of labor<\/a>, in which the neural networks do the repetitive work and humans take care of the high-level cognitive issues and corner cases.<\/p>\n<p>Interestingly, when one of the attendees asked Karpathy whether the generation of the triggers could be automated, he said, \u201c[Automating the trigger] is a very tricky scenario, because you can have general triggers, but they will not correctly represent the error modes. It would be very hard to, for example, automatically have a trigger that triggers for entering and exiting tunnels.<span>\u00a0<\/span><strong>That\u2019s something semantic that you as a person have to intuit<\/strong><span>\u00a0<\/span>[emphasis mine] that this is a challenge\u2026 It\u2019s not clear how that would work.\u201d<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Hierarchical_deep_learning_architecture\"><\/span>Hierarchical deep learning architecture<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359271 js-lazy\" alt=\"\" width=\"655\" height=\"347\" sizes=\"auto, (max-width: 655px) 100vw, 655px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car.jpeg\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-280x148.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-509x270.jpeg 509w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-255x135.jpeg 255w\"\/><noscript><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359271\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car.jpeg\" alt=\"\" width=\"655\" height=\"347\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-280x148.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-509x270.jpeg 509w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD4Tesla-neural-network-self-driving-car-255x135.jpeg 255w\"\/><\/noscript><\/figure>\n<p>Tesla\u2019s self-driving team needed a very efficient and well-designed neural network to make the most out of the high-quality dataset they had gathered.<\/p>\n<p>The company created a hierarchical deep learning architecture composed of different neural networks that process information and feed their output to the next set of networks.<\/p>\n<p>The deep learning model uses<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/01\/06\/convolutional-neural-networks-cnn-convnets\/\">convolutional neural networks<\/a><span>\u00a0<\/span>to extract features from the videos of eight cameras installed around the car and fuses them together using<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/jalammar.github.io\/illustrated-transformer\/\">transformer networks<\/a>. It then fuses them across time, which is important for tasks such as trajectory-prediction and to smooth out inference inconsistencies.<\/p>\n<p>The spatial and temporal features are then fed into a branching structure of neural networks that Karpathy described as heads, trunks, and terminals.<\/p>\n<p>\u201cThe reason you want this branching structure is because there\u2019s a huge amount of outputs that you\u2019re interested in, and you can\u2019t afford to have a single neural network for every one of the outputs,\u201d Karpathy said.<\/p>\n<p>The hierarchical structure makes it possible to reuse components for different tasks and enable feature-sharing between the different inference pathways.<\/p>\n<p>Another benefit of the modular architecture of the network is the possibility of distributed development. Tesla is currently employing a large team of machine learning engineers working on the self-driving neural network. Each of them works on a small component of the network and they plug in their results into the larger network.<\/p>\n<p>\u201cWe have a team of roughly 20 people who are training neural networks full time. They\u2019re all cooperating on a single neural network,\u201d Karpathy said.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Vertical_integration\"><\/span>Vertical integration<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359272 js-lazy\" alt=\"\" width=\"643\" height=\"362\" sizes=\"auto, (max-width: 643px) 100vw, 643px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers.jpeg\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-280x158.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-479x270.jpeg 479w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-240x135.jpeg 240w\"\/><noscript><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-1359272\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers.jpeg\" alt=\"\" width=\"643\" height=\"362\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers.jpeg 696w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-280x158.jpeg 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-479x270.jpeg 479w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/BD5Tesla-AI-computers-240x135.jpeg 240w\"\/><\/noscript><\/figure>\n<p>In his presentation at CVPR, Karpathy shared some details about the supercomputer Tesla is using to train and finetune its deep learning models.<\/p>\n<p>The compute cluster is composed of 80 nodes, each containing eight Nvidia A100 GPUs with 80 gigabytes of video memory, amounting to 5,760 GPUs and more than 450 terabytes of VRAM. The supercomputer also has 10 petabytes of NVME superfast storage and 640 tbps networking capacity to connect all the nodes and allow efficient distributed training of the neural networks.<\/p>\n<p>Tesla also owns and builds the AI chips installed inside its cars. \u201cThese chips are specifically designed for the neural networks we want to run for [full self-driving] applications,\u201d Karpathy said.<\/p>\n<p>Tesla\u2019s big advantage is its vertical integration. Tesla owns the entire self-driving car stack. It manufactures the car and the hardware for self-driving capabilities. It is in a unique position to collect a wide variety of telemetry and video data from the millions of cars it has sold. It also creates and trains its neural networks on its proprietary datasets, its special in-house compute clusters, and validates and finetunes the networks through shadow testing on its cars. And, of course, it has a very talented team of machine learning engineers, researchers, and hardware designers to put all the pieces together.<\/p>\n<p>\u201cYou get to co-design and engineer at all the layers of that stack,\u201d Karpathy said. \u201cThere\u2019s no third party that is holding you back. You\u2019re fully in charge of your own destiny, which I think is incredible.\u201d<\/p>\n<p>This vertical integration and repeating cycle of creating data, tuning machine learning models, and deploying them on many cars puts Tesla in a unique position to implement vision-only self-driving car capabilities. In his presentation, Karpathy showed several examples where the new neural network alone outmatched the legacy ML model that worked in combination with radar information.<\/p>\n<p>And if the system continues to improve, as Karpathy says, Tesla might be on the track of making lidars obsolete. And I don\u2019t see any other company being able to reproduce Tesla\u2019s approach.<\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=2blLi3T4EGw\">https:\/\/www.youtube.com\/watch?v=2blLi3T4EGw<\/a><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Open_issues\"><\/span>Open issues<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>But the question remains as to whether deep learning in its current state will be enough to overcome all the challenges of self-driving. Surely, object detection and velocity and range estimation play a big part in driving. But human vision also performs many other complex functions, which scientists call the \u201cdark matter\u201d of vision. Those are all important components in the conscious and subconscious analysis of visual input and navigation of different environments.<\/p>\n<p>Deep learning models also<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/03\/15\/machine-learning-causality\/\">struggle with making causal inference<\/a>, which can be a huge barrier when the models face new situations they haven\u2019t seen before. So, while Tesla has managed to create a very huge and diverse dataset, open roads are also very complex environments where new and unpredicted things can happen all the time.<\/p>\n<p>The AI community is divided over whether you need to explicitly integrate causality and reasoning into deep neural networks or if you can overcome the causality barrier through \u201c<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/06\/22\/direct-fit-artificial-neural-networks\/\">direct fit<\/a>,\u201d where a large and well-distributed dataset will be enough to reach general-purpose deep learning. Tesla\u2019s vision-based self-driving team seems to favor the latter (though given their full control over the stack, they could always try new neural network architectures in the future). It will be interesting to how the technology fares against the test of time.<\/p>\n<p><i><span>This article was originally published by Ben Dickson on\u00a0<\/span><\/i><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/\"><i><span>TechTalks<\/span><\/i><\/a><i><span>, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original article\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/06\/28\/tesla-computer-vision-autonomous-driving\/\">here<\/a>.<\/span><\/i><\/p>\n<\/div>\n<\/div>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong>\n<\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/tesla-ai-chief-explains-self-driving-cars-dont-need-lidar-syndication\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#Self-driving cars don\u2019t need LiDAR&#8221; What is the technology stack you need to create fully autonomous vehicles? Companies and researchers are divided on the answer to that question. Approaches to autonomous driving range from just cameras and\u00a0computer vision\u00a0to a combination of computer vision and advanced sensors. Tesla has been a vocal champion for the\u00a0pure vision-based&#8230;<\/p>\n","protected":false},"author":1,"featured_media":291292,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/img-cdn.tnwcdn.com\/image\/shift?filter_last=1&fit=1280,640&url=https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/07\/Teslahed.jpg&signature=b7f31a3678104e26e5bfdcca0a02aee5","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-291291","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/291291","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=291291"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/291291\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/291292"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=291291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=291291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=291291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}