{"id":450726,"date":"2022-05-21T19:00:29","date_gmt":"2022-05-21T16:00:29","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/dont-expect-large-language-models-like-the-next-gpt-to-be-democratized\/"},"modified":"2022-05-21T19:00:29","modified_gmt":"2022-05-21T16:00:29","slug":"dont-expect-large-language-models-like-the-next-gpt-to-be-democratized","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/dont-expect-large-language-models-like-the-next-gpt-to-be-democratized\/","title":{"rendered":"#Don\u2019t expect large language models like the next GPT to be democratized"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_89 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6aee229a0c3a5\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6aee229a0c3a5\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/dont-expect-large-language-models-like-the-next-gpt-to-be-democratized\/#%E2%80%9CDont_expect_large_language_models_like_the_next_GPT_to_be_democratized%E2%80%9D\" >&#8220;Don\u2019t expect large language models like the next GPT to be democratized&#8221;<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/dont-expect-large-language-models-like-the-next-gpt-to-be-democratized\/#Greetings_humanoids\" >Greetings humanoids<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"%E2%80%9CDont_expect_large_language_models_like_the_next_GPT_to_be_democratized%E2%80%9D\"><\/span>&#8220;Don\u2019t expect large language models like the next GPT to be democratized&#8221;<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<div>\n                            <em>This article is part of our\u00a0coverage of the latest in\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/tag\/ai-research-papers\/\">AI research<\/a>.<\/em><\/p>\n<p>In early May, Meta released Open Pretrained Transformer (OPT-175B), a large language model (LLM) that can perform various tasks. Large language models have become one of the hottest areas of research in artificial intelligence in the past few years.<\/p>\n<p>OPT-175B is the latest entrant in the LLM arms race triggered by OpenAI\u2019s\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/08\/17\/openai-gpt-3-commercial-ai\/\">GPT-3<\/a>, a deep neural network with 175 billion parameters. GPT-3 showed that LLMs can perform many tasks without undergoing extra training and only seeing a few examples (<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/08\/12\/what-is-one-shot-learning\/\">zero- or few-shot learning<\/a>). Microsoft later integrated GPT-3 into several of its products, showing not only the scientific but also the commercial promises of LLMs.<\/p>\n<div class=\"corona-wrapper neural-cta hs-embed-tnw\">\n<div class=\"neural-cta-wrapper\">\n<div class=\"neural-cta-img\"><img decoding=\"async\" src=\"https:\/\/s3.amazonaws.com\/events.tnw\/hardfork-2018\/uploads\/companies\/neural-newsletter_header.gif\"\/><\/div>\n<p><noscript><img decoding=\"async\" src=\"https:\/\/s3.amazonaws.com\/events.tnw\/hardfork-2018\/uploads\/companies\/neural-newsletter_header.gif\"\/><\/noscript><\/p>\n<div class=\"neural-cta-input\">\n<h2 class=\"neural-cta-title\"><span class=\"ez-toc-section\" id=\"Greetings_humanoids\"><\/span>Greetings humanoids<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"neural-cta-tagline\">Subscribe now for a weekly recap of our favorite AI stories<\/p>\n<p><!--[if lte IE 8]><![endif]--><\/div>\n<\/div>\n<\/div>\n<p>What makes OPT-175B unique is Meta\u2019s commitment to \u201copenness,\u201d as the model\u2019s name implies. Meta has made the model available to the public (with some caveats). It has also released a ton of details about the training and development process. In a post published on the\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/ai.facebook.com\/blog\/democratizing-access-to-large-scale-language-models-with-opt-175b\/\">Meta AI blog<\/a>, the company described its release of OPT-175B as \u201cDemocratizing access to large-scale language models.\u201d<\/p>\n<p>Meta\u2019s move toward transparency is commendable. However, the competition over large language models has reached a point where it can no longer be democratized.<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" class=\"aligncenter js-lazy\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/12\/large-language-models.jpg?resize=696%2C435&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/12\/large-language-models.jpg?resize=696%2C435&amp;ssl=1\"\/><noscript><img decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/12\/large-language-models.jpg?resize=696%2C435&amp;ssl=1\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/12\/large-language-models.jpg?resize=696%2C435&amp;ssl=1\" srcset=\"\"\/><\/noscript><\/figure>\n<p>Meta\u2019s release of OPT-175B has some key features. It includes both pretrained models as well as the code needed to train and use the LLM. Pretrained models are especially useful for organizations that do not have the computational resources for training the model (training neural networks is much more resource-intensive than running them). It will also help reduce the massive\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.pcmag.com\/news\/ai-could-save-the-world-if-it-doesnt-ruin-the-environment-first\">carbon footprint<\/a>\u00a0caused by the computational resources needed to train large neural networks.<\/p>\n<p>Like GPT-3, OPT comes in different sizes, ranging from 125 million to 175 billion parameters (models with more parameters have more capacity for learning). At the time of this writing, all models up to OPT-30B are accessible for <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">download<\/a>. The full 175-billion-parameter model will be made available to select researchers and institutions that fill a request form.<\/p>\n<p>According to the Meta AI blog, \u201cTo maintain integrity and prevent misuse, we are releasing our model under a noncommercial license to focus on research use cases. Access to the model will be granted to academic researchers; those affiliated with organizations in government, civil society, and academia; along with industry research laboratories around the world.\u201d<\/p>\n<p>In addition to the models, Meta has released a full logbook that provides a detailed technical timeline of the development and training process of large language models. Published papers usually only include information about the final model. The logbook gives valuable insights about \u201chow much compute was used to train OPT-175B and the human overhead required when underlying infrastructure or the training process itself becomes unstable at scale,\u201d according to Meta.<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" class=\"aligncenter js-lazy\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/09\/microsoft-openai-gpt-3-license.jpg?resize=696%2C464&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/09\/microsoft-openai-gpt-3-license.jpg?resize=696%2C464&amp;ssl=1\"\/><noscript><img decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/09\/microsoft-openai-gpt-3-license.jpg?resize=696%2C464&amp;ssl=1\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/09\/microsoft-openai-gpt-3-license.jpg?resize=696%2C464&amp;ssl=1\" srcset=\"\"\/><\/noscript><\/figure>\n<p>In its blog post, Meta states that large language models are mostly accessible through \u201cpaid APIs\u201d and that restricted access to LLMs has \u201climited researchers\u2019 ability to understand how and why these large language models work, hindering progress on efforts to improve their robustness and mitigate known issues such as bias and toxicity.\u201d<\/p>\n<p>This is a jab at OpenAI (and\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/09\/24\/microsoft-openai-gpt-3-license\/\">by extension Microsoft<\/a>), which released GPT-3 as a black-box API service instead of making its model\u2019s weights and source code available to the public. Among the reasons OpenAI stated for not making GPT-3 public was controlling misuse and development of harmful applications.<\/p>\n<p>Meta believes that by making the models available to a wider audience, it will be in a better position to study and prevent any harm they can cause.<\/p>\n<p>Here\u2019s how Meta describes the effort: \u201cWe hope that OPT-175B will bring more voices to the frontier of large language model creation, help the community collectively design responsible release strategies, and add an unprecedented level of transparency and openness to the development of large language models in the field.\u201d<\/p>\n<p>However, it is worth noting that \u201ctransparency and openness\u201d is not the equivalent of \u201cdemocratizing large language models.\u201d The costs of training, configuring, and running large language models remain prohibitive and are likely to grow in the future.<\/p>\n<p>According to Meta\u2019s blog post, its researchers have managed to considerably reduce the costs of training large language models. The company says that the model\u2019s carbon footprint has been reduced to a seventh of GPT-3. Experts I had previously spoken to estimated GPT-3\u2019s training costs to be\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/09\/21\/gpt-3-economy-business-model\/\">up to $27.6 million<\/a>.<\/p>\n<p>This means that OPT-175B will still cost several million dollars to train. Fortunately, the pretrained model will obviate the need to train the model, and Meta says it will provide the codebase used to train and deploy the full model \u201cusing only 16 NVIDIA V100 GPUs.\u201d This is the equivalent of an Nvidia DGX-2, which costs about $400,000, not a small sum for a cash-constrained research lab or an individual researcher. (According to a\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/pdf\/2205.01068.pdf\">paper<\/a>\u00a0that provides more details on OPT-175B, Meta trained their own model with 992 80GB A100 GPUs, which are\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/lambdalabs.com\/blog\/nvidia-a100-vs-v100-benchmarks\/\">significantly faster than the V100<\/a>.)<\/p>\n<p>Meta AI\u2019s logbook further confirms that training large language models is a very complicated task. The timeline of OPT-175B is filled with server crashes, hardware failures, and other complications that require a highly technical staff. The researchers also had to restart the training process several times, tweak hyperparameters, and change loss functions. All of these incur extra costs that small labs can\u2019t afford.<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" class=\"aligncenter js-lazy\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/09\/tech-giants-artificial-intelligence.jpg?resize=696%2C392&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/09\/tech-giants-artificial-intelligence.jpg?resize=696%2C392&amp;ssl=1\"\/><noscript><img decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/09\/tech-giants-artificial-intelligence.jpg?resize=696%2C392&amp;ssl=1\" alt=\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2021\/09\/tech-giants-artificial-intelligence.jpg?resize=696%2C392&amp;ssl=1\" srcset=\"\"\/><\/noscript><\/figure>\n<p>Language models such as OPT and GPT are based on the\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2022\/05\/02\/what-is-the-transformer\/\">transformer architecture<\/a>. One of the key features of transformers is their ability to process large sequential data (e.g., text) in parallel and at scale.<\/p>\n<p>In recent years, researchers have shown that by adding more layers and parameters to transformer models, they can improve their performance on language tasks. Some researchers believe that reaching higher levels of intelligence is only a scale problem. Accordingly, cash-rich research labs like Meta AI, DeepMind (owned by Alphabet), and OpenAI (backed by Microsoft) are moving toward creating\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2019\/11\/25\/ai-research-neural-networks-compute-costs\/\">larger and larger neural networks<\/a>.<\/p>\n<p>Last year, Microsoft and Nvidia created a\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/developer.nvidia.com\/blog\/using-deepspeed-and-megatron-to-train-megatron-turing-nlg-530b-the-worlds-largest-and-most-powerful-generative-language-model\/\">530-billion parameter language model<\/a>\u00a0called Megatron-Turing (MT-NLG). Last month, Google introduced the\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/ai.googleblog.com\/2022\/04\/pathways-language-model-palm-scaling-to.html\">Pathways Language Model (PaLM)<\/a>, an LLM with 540 billion parameters. And there are rumors that OpenAI will release GPT-4 in the next few months.<\/p>\n<p>However, larger neural networks also require larger financial and technical resources. And while larger language models will have new bells and whistles (and\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.technologyreview.com\/2020\/08\/22\/1007539\/gpt3-openai-language-generator-artificial-intelligence-ai-opinion\/\">new failures<\/a>), they will inevitably centralize power within the hands of a few wealthy companies by making it even harder for smaller research labs and independent researchers to work on large language models.<\/p>\n<p>On the commercial side, big tech companies will have an even greater advantage. Running large language models is very expensive and challenging. Companies like Google and Microsoft have special servers and processors that allow them to run these models at scale and in a profitable way. For smaller companies, the overhead of running their own version of an LLM like GPT-3 is too prohibitive. Just as most businesses use cloud hosting services instead of setting up their own servers and data centers,\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/01\/25\/gpt-3-startups-businesses\/\">out-of-the-box systems<\/a>\u00a0like the GPT-3 API will gain more traction as large language models become more popular.<\/p>\n<p>This in turn will further centralize AI in the hands of big tech companies. More AI research labs will have to enter partnerships with big tech to fund their research. And this will give big tech more power to decide the future directions of AI research (which will probably be aligned with their financial interests). This can come at the cost of areas of research that do not have a short-term return on investment.<\/p>\n<p>The bottom line is that, while we celebrate Meta\u2019s move to bring transparency to LLMs, let\u2019s not forget that the very nature of large language models is undemocratic and in favor of the very companies that are publicizing them.<\/p>\n<p><em>This article was originally written by Ben Dickson and published by Ben Dickson on<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/\">TechTalks<\/a>, a publication that examines trends in <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">technology<\/a>, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original article\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2022\/05\/16\/opt-175b-large-language-models\/\">here<\/a>.<\/em>\n                        <\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/dont-expect-large-language-models-like-the-next-gpt-to-be-democratized\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;Don\u2019t expect large language models like the next GPT to be democratized&#8221; This article is part of our\u00a0coverage of the latest in\u00a0AI research. In early May, Meta released Open Pretrained Transformer (OPT-175B), a large language model (LLM) that can perform various tasks. Large language models have become one of the hottest areas of research in&#8230;<\/p>\n","protected":false},"author":1,"featured_media":450727,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/img-cdn.tnwcdn.com\/image\/neural?filter_last=1&fit=1280,640&url=https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/learning-language-models-hed.jpg&signature=7aff3dd351694d1e80e1e2cb91c17a04","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-450726","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/450726","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=450726"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/450726\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/450727"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=450726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=450726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=450726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}