{"id":381478,"date":"2021-12-15T16:00:51","date_gmt":"2021-12-15T13:00:51","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/"},"modified":"2021-12-15T16:00:51","modified_gmt":"2021-12-15T13:00:51","slug":"how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/","title":{"rendered":"#How to Use an NVIDIA GPU with Docker Containers \u2013 CloudSavvy IT"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a35a04532f74\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a35a04532f74\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Making_GPUs_Work_In_Docker\" >Making GPUs Work In Docker<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Adding_the_NVIDIA_Drivers\" >Adding the NVIDIA Drivers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Starting_a_Container_With_GPU_Access\" >Starting a Container With GPU Access<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Selecting_a_Base_Image\" >Selecting a Base Image<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Manually_Configuring_an_Image\" >Manually Configuring an Image<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#How_Does_It_Work\" >How Does It Work?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/buradabiliyorum.com\/en\/how-to-use-an-nvidia-gpu-with-docker-containers-cloudsavvy-it\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<p><strong>&#8220;#How to Use an NVIDIA GPU with Docker Containers \u2013 CloudSavvy IT&#8221;<\/strong><\/p>\n<div id=\"article-content-area\">\n<img loading=\"lazy\" decoding=\"async\" class=\"type:primaryImage aligncenter size-full wp-image-14946\" data-pagespeed-lazy-srcset=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/4352907f.jpeg?width=398&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 400w, https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/4352907f.jpeg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1 1200w\" sizes=\"auto, 400w, 1200w\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/4352907f.jpeg?width=1198&amp;trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"Graphic showing the NVIDIA and Docker logos\" width=\"1202\" height=\"677\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>Docker containers don\u2019t see your system\u2019s GPU automatically. This causes reduced performance in GPU-dependent workloads such as machine learning frameworks. Here\u2019s how to expose your host\u2019s NVIDIA GPU to your containers.<\/p>\n<h2 id=\"making-gpus-work-in-docker\"><span class=\"ez-toc-section\" id=\"Making_GPUs_Work_In_Docker\"><\/span>Making GPUs Work In Docker<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Docker containers share your host\u2019s kernel but bring along their own operating system and software packages. This means they lack the NVIDIA drivers used to interface with your GPU. Docker doesn\u2019t even add GPUs to containers by default so a plain <code>docker run<\/code> won\u2019t see your hardware at all.<\/p>\n<p>At a high level, getting your GPU to work is a two-step procedure: install the drivers within your image, then instruct Docker to add GPU devices to your containers at runtime.<\/p>\n<p>This guide focuses on modern versions of CUDA and Docker. The latest release of NVIDIA Container Toolkit is designed for combinations of CUDA 10 and Docker Engine 19.03 and later. Older builds of CUDA, Docker, and the NVIDIA drivers may require additional steps.<\/p>\n<h2 id=\"adding-the-nvidia-drivers\"><span class=\"ez-toc-section\" id=\"Adding_the_NVIDIA_Drivers\"><\/span>Adding the NVIDIA Drivers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Make sure you\u2019ve got the NVIDIA drivers working properly on your host before you continue with your Docker configuration. You should be able to successfully run <code>nvidia-smi<\/code> and see your GPU\u2019s name, driver version, and CUDA version.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-14944\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/7ca3462c.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"1045\" height=\"563\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<p>To use your GPU with Docker, begin by adding the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/github.com\/NVIDIA\/nvidia-docker\">NVIDIA Container Toolkit<\/a> to your host. This integrates into Docker Engine to automatically configure your containers for GPU support.<\/p>\n<p>Add the toolkit\u2019s package repository to your system using the example command:<\/p>\n<pre>distribution=$(. \/etc\/os-release;echo $ID$VERSION_ID) &#13;\n   &amp;&amp; curl -s -L https:\/\/nvidia.github.io\/nvidia-docker\/gpgkey | sudo apt-key add - &#13;\n   &amp;&amp; curl -s -L https:\/\/nvidia.github.io\/nvidia-docker\/$distribution\/nvidia-docker.list | sudo tee \/etc\/apt\/sources.list.d\/nvidia-docker.list<\/pre>\n<p>Next install the <code>nvidia-docker2<\/code> package on your host:<\/p>\n<pre>apt-get update&#13;\napt-get install -y nvidia-docker2<\/pre>\n<p>Restart the Docker daemon to complete the installation:<\/p>\n<pre>sudo systemctl restart docker<\/pre>\n<p>The Container Toolkit should now be operational. You\u2019re ready to start a test container.<\/p>\n<h2 id=\"starting-a-container-with-gpu-access\"><span class=\"ez-toc-section\" id=\"Starting_a_Container_With_GPU_Access\"><\/span>Starting a Container With GPU Access<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>As Docker doesn\u2019t provide your system\u2019s GPUs by default, you need to create containers with the <code>--gpus<\/code> flag for your hardware to show up. You can either specify specific devices to enable or use the <code>all<\/code> keyword.<\/p>\n<p>The <code>nvidia\/cuda<\/code> images are preconfigured with the CUDA binaries and GPU tools. Start a container and run the <code>nvidia-smi<\/code> command to check your GPU\u2019s accessible. The output should match what you saw when using <code>nvidia-smi<\/code> on your host. The CUDA version could be different depending on the toolkit versions on your host and in your selected container image.<\/p>\n<pre>docker run -it --gpus all nvidia\/cuda:11.4.0-base-ubuntu20.04 nvidia-smi<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-14945\" data-pagespeed-lazy-src=\"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/c5d24934.png?trim=1,1&amp;bg-color=000&amp;pad=1,1\" alt=\"\" width=\"1331\" height=\"372\" src=\"\/pagespeed_static\/1.JiBnMqyl6S.gif\" onload=\"pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\" onerror=\"this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);\"\/><\/p>\n<h2 id=\"selecting-a-base-image\"><span class=\"ez-toc-section\" id=\"Selecting_a_Base_Image\"><\/span>Selecting a Base Image<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Using one of the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/hub.docker.com\/r\/nvidia\/cuda\/tags\"><code>nvidia\/cuda<\/code><\/a> tags is the quickest and easiest way to get your GPU workload running in Docker. Many different variants are available; they provide a matrix of operating system, CUDA version, and NVIDIA software options. The images are built for multiple architectures.<\/p>\n<p>Each tag has this format:<\/p>\n<pre>11.4.0-base-ubuntu20.04<\/pre>\n<ul>\n<li><code>11.4.0<\/code> \u2013 CUDA version.<\/li>\n<li><code>base<\/code> \u2013 Image flavor.<\/li>\n<li><code>ubuntu20.04<\/code> \u2013 Operating system version.<\/li>\n<\/ul>\n<p>Three different image flavors are available. The <code>base<\/code> image is a minimal option with the essential CUDA runtime binaries. <code>runtime<\/code> is a more fully-featured option that includes the CUDA math libraries and NCCL <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/developer.nvidia.com\/nccl\">for cross-GPU communication<\/a>. The third variant is <code>devel<\/code> which gives you everything from <code>runtime<\/code> as well as headers and development tools for creating custom CUDA images.<\/p>\n<p>If one of the images will work for you, aim to use it as your base in your <code>Dockerfile<\/code>. You can then use regular Dockerfile instructions to install your programming languages, copy in your source code, and configure your <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>lication. It removes the complexity of manual GPU set up steps.<\/p>\n<pre>FROM nvidia\/cuda:11.4.0-base-ubuntu20.04&#13;\nRUN apt update&#13;\nRUN apt-get install -y python3 python3-pip&#13;\nRUN pip install tensorflow-gpu&#13;\n&#13;\nCOPY tensor-code.py .&#13;\nENTRYPONT [\"python3\", \"tensor-code.py\"]<\/pre>\n<p>Building and running this image with the <code>--gpus<\/code> flag would start your Tensor workload with GPU acceleration.<\/p>\n<h2 id=\"manually-configuring-an-image\"><span class=\"ez-toc-section\" id=\"Manually_Configuring_an_Image\"><\/span>Manually Configuring an Image<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You can manually add CUDA support to your image if you need to choose a different base. The best way to achieve this is to reference the <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/gitlab.com\/nvidia\/container-images\/cuda\/-\/blob\/master\/dist\/11.5.0\/ubuntu2004\/base\/Dockerfile\">official NVIDIA Dockerfiles<\/a>.<\/p>\n<p>Copy the instructions used to add the CUDA package repository, install the library, and link it into your path. We\u2019re not reproducing all the steps in this guide as they vary by CUDA version and operating system.<\/p>\n<p>Pay attention to the environment variables at the end of the Dockerfile \u2013 these define how containers using your image integrate with the NVIDIA Container Runtime:<\/p>\n<pre>ENV NVIDIA_VISIBLE_DEVICES all&#13;\nENV NVIDIA_DRIVER_CAPABILITIES compute,utility<\/pre>\n<p>Your image should detect your GPU once CUDA\u2019s installed and the environment variables have been set. This gives you more control over the contents of your image but leaves you liable to adjust the instructions as new CUDA versions release.<\/p>\n<h2 id=\"how-does-it-work\"><span class=\"ez-toc-section\" id=\"How_Does_It_Work\"><\/span>How Does It Work?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The NVIDIA Container Toolkit is <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/docs.nvidia.com\/datacenter\/cloud-native\/container-toolkit\/arch-overview.html\">a collection of packages<\/a> which wrap container runtimes like Docker with an interface to the NVIDIA driver on the host. The <code>libnvidia-container<\/code> library is responsible for providing an API and CLI that automatically provides your system\u2019s GPUs to containers via the runtime wrapper.<\/p>\n<p>The <code>nvidia-container-toolkit<\/code> component implements a container runtime <code>prestart<\/code> hook. This means it\u2019s notified when a new container is about to start. It looks at the GPUs you want to attach and invokes <code>libnvidia-container<\/code> to handle container creation.<\/p>\n<p>The hook is enabled by <code>nvidia-container-runtime<\/code>. This wraps your \u201creal\u201d container runtime such as containerd or runc to ensure the NVIDIA <code>prestart<\/code> hook is run. Your existing runtime continues the container start process after the hook has executed. When the container toolkit is installed, you\u2019ll see the NVIDIA runtime selected in your Docker daemon config file.<\/p>\n<h2 id=\"summary\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Using an NVIDIA GPU inside a Docker container requires you to add the NVIDIA Container Toolkit to the host. This integrates the NVIDIA drivers with your container runtime.<\/p>\n<p>Calling <code>docker run<\/code> with the <code>--gpu<\/code> flag makes your hardware visible to the container. This must be set on each container you launch, after the Container Toolkit has been installed.<\/p>\n<p>NVIDIA provides preconfigured CUDA Docker images that you can use as a quick starter for your application. If you need something more specific, refer to the official Dockerfiles to assemble your own that\u2019s still compatible with the Container Toolkit.\n<\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/www.cloudsavvyit.com\/14942\/how-to-use-an-nvidia-gpu-with-docker-containers\/\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#How to Use an NVIDIA GPU with Docker Containers \u2013 CloudSavvy IT&#8221; Docker containers don\u2019t see your system\u2019s GPU automatically. This causes reduced performance in GPU-dependent workloads such as machine learning frameworks. Here\u2019s how to expose your host\u2019s NVIDIA GPU to your containers. Making GPUs Work In Docker Docker containers share your host\u2019s kernel but&#8230;<\/p>\n","protected":false},"author":1,"featured_media":381479,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/www.cloudsavvyit.com\/p\/uploads\/2021\/11\/4352907f.jpeg","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-381478","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/381478","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=381478"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/381478\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/381479"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=381478"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=381478"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=381478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}