{"id":473305,"date":"2022-07-12T22:39:28","date_gmt":"2022-07-12T19:39:28","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/"},"modified":"2022-07-12T22:39:28","modified_gmt":"2022-07-12T19:39:28","slug":"an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/","title":{"rendered":"#An open-source model that dwarfs GPT-3 aims to free AI from Big Tech"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a287e6d03b1e\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a287e6d03b1e\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/#%E2%80%9CAn_open-source_model_that_dwarfs_GPT-3_aims_to_free_AI_from_Big_Tech%E2%80%9D\" >&#8220;An open-source model that dwarfs GPT-3 aims to free AI from Big Tech&#8221;<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/#Greetings_humanoids\" >Greetings, humanoids<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/#Opening_AI\" >Opening AI<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/an-open-source-model-that-dwarfs-gpt-3-aims-to-free-ai-from-big-tech\/#The_seeds_of_BLOOM\" >The seeds of BLOOM<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"%E2%80%9CAn_open-source_model_that_dwarfs_GPT-3_aims_to_free_AI_from_Big_Tech%E2%80%9D\"><\/span>&#8220;An open-source model that dwarfs GPT-3 aims to free AI from Big Tech&#8221;<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<div id=\"article-main-content\">\n                            A\u00a0language model bigger than GPT-3 has arrived with a bold\u00a0 ambition: freeing AI from Big Tech\u2019s clutches.<\/p>\n<p>Named BLOOM, the large language model (LLM) promises a similar performance to Silicon Valley\u2019s leading systems \u2014 but with a radically different <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>roach to access.<\/p>\n<p>While tech giants tend to keep their vaunted LLMs hidden from the public, BLOOM is available to anyone and free.<\/p>\n<div class=\"inarticle-wrapper neural channel-cta hs-embed-tnw\">\n<div id=\"hs-embed-tnw\" class=\"channel-cta-wrapper\">\n<div class=\"channel-cta-img\"><img decoding=\"async\" src=\"https:\/\/s3.amazonaws.com\/uploads.tnw\/uploads\/neural-newsletter_header-1.gif\"\/><\/div>\n<p><noscript><img decoding=\"async\" src=\"https:\/\/thenextweb.com\/news\/src=\" https:=\"\"\/><\/noscript><\/p>\n<div class=\"channel-cta-input\">\n<h2 class=\"channel-cta-title\"><span class=\"ez-toc-section\" id=\"Greetings_humanoids\"><\/span>Greetings, humanoids<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p class=\"channel-cta-tagline\">Subscribe to our <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/news\/\" data-internallinksmanager029f6b8e52c=\"2\" title=\"News\" target=\"_blank\" rel=\"noopener\">news<\/a>letter now for a weekly recap of our favorite AI stories in your inbox.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p>It\u2019s also multilingual \u2014 unlike Google\u2019s LaMDA and OpenAI\u2019s GPT-3 \u2014 an unusual feature in an English-dominated field.<\/p>\n<p>These features could democratize access to <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">technology<\/a> that\u2019s set to make a deep impact on society.<\/p>\n<blockquote class=\"c-richText__pullQuote\">\n<div class=\"c-richText__pullQuoteGradient\">\n<p class=\"c-richText__pullQuoteQuote\">Powerful AI models can be trained and released in an open way.<\/p>\n<\/p><\/div>\n<\/blockquote>\n<p>LLMs are proving proficient at a growing range of tasks, including writing essays, generating code, and translating languages.<\/p>\n<p>Yet they\u2019re also adept at producing harmful content \u2014 and their future capabilities are <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/hai.stanford.edu\/news\/how-large-language-models-will-transform-science-society-and-ai\">difficult to predict<\/a>.<\/p>\n<p>BLOOM gives researchers a unique chance to explore their risks and benefits.<\/p>\n<p>\u201cBLOOM is a demonstration that the most powerful AI models can be trained and released by the broader research community with accountability and in an actual open way, in contrast to the typical secrecy of industrial AI research labs.\u201d said Teven Le Scao, co-lead of BLOOM\u2019s training, in a statement.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Opening_AI\"><\/span>Opening AI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>LLM\u2019s are prohibitively expensive to create and run. Training GPT-3, for instance, was estimated to cost\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/09\/21\/gpt-3-economy-business-model\/\">up to $27.6 million<\/a>.<\/p>\n<p>Inevitably, tech companies want to protect such large investments \u2014 particularly when they provide competitive advantages.<\/p>\n<p>It\u2019s therefore unsurprising that LLMs are rarely open-sourced \u2014 with some notable exceptions.<\/p>\n<p>Meta has produced the most prominent anomaly. In May, the company offered access to the 175-billion parameter OPT <span style=\"color: #ffffff;\"><span style=\"background-color: #a4a3a3;\">sytem<\/span><\/span>.<\/p>\n<p>The full model, however, is only available upon request and for non-commercial use.<\/p>\n<p>BLOOM ramps up the accessibility.<\/p>\n<p>The 176-billion-parameter model is available for free to any individual or institution who agrees to <span style=\"font-weight: 400;\">the system\u2019s<\/span><span style=\"font-weight: 400;\"><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bigscience.huggingface.co\/blog\/the-bigscience-rail-license\"> Responsible AI License<\/a>. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Anyone can also <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bigscience.notion.site\/bigscience\/BigScience-214dc9a8c1434d7bbcddb391c383922a\">publicly view<\/a>\u00a0the meeting notes, discussions, and code behind the model.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_seeds_of_BLOOM\"><\/span>The seeds of BLOOM<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>BLOOM\u00a0 was created by Big<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/sciencee\/\" data-internallinksmanager029f6b8e52c=\"5\" title=\"Science\" target=\"_blank\" rel=\"noopener\">Science<\/a>, a research project that launched in early 2021. The initiative is bootstrapped and led by AI startup<span style=\"font-weight: 400;\">\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/huggingface.co\"><span style=\"font-weight: 400;\">Hugging Face<\/span><\/a><span style=\"font-weight: 400;\">.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u201cLarge ML models have changed the world of AI research over the last to years but the huge compute cost necessary to train them resulted in very few teams actually having the ability to train and research them,\u201d said Thomas Wolf, the BigScience co-lead and Hugging Face co-founder<\/span><\/p>\n<blockquote class=\"c-richText__pullQuote\">\n<div class=\"c-richText__pullQuoteGradient\">\n<p class=\"c-richText__pullQuoteQuote\">The training corpus aligned with our values.<\/p>\n<\/p><\/div>\n<\/blockquote>\n<p>Wolf\u2019s team of 100,000 researchers from more than 60 countries and 250 institutions developed BLOOM to promote inclusion and responsibility in LLMs.<\/p>\n<p>They trained the model <span style=\"font-weight: 400;\">on the <\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/www.idris.fr\/eng\/info\/missions-eng.html\"><span style=\"font-weight: 400;\">Jean Zay supercomputer<\/span><\/a><span style=\"font-weight: 400;\"> in Paris, France.<\/span><span style=\"font-weight: 400;\"\/><\/p>\n<p><span style=\"font-weight: 400;\">\u201cWe adopted a data-first approach to make sure the training corpus was aligned with our values,\u201d said Christopher Akiki, a research scientist at Leipzig University and a BigScience researcher.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u201cThe multidisciplinary and international makeup of BigScience enabled us to critically reflect on every step of the process from multiple vantage points: ethical, legal, environmental, linguistic, and technical.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">\u201cThat meant we were able to mitigate ethical concerns without compromising on performance or scale.\u201d<\/span><\/p>\n<p>The size is certainly imposing. <span style=\"font-weight: 400;\">At 176 billion parameters, BLOOM is larger than OpenAI\u2019s GPT-3 and MetaAI\u2019s OPT.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The model can generate text in 46 natural languages and dialects and 13 programming languages. For many of them, it\u2019s the first-ever language model with over 100B parameters.<\/span><span style=\"font-weight: 400;\"><br \/><\/span><\/p>\n<p>It\u2019s also uniquely affordable. <span style=\"font-weight: 400;\">BigScience says <\/span><span style=\"font-weight: 400;\">researchers can use BLOOM for less than $40\/hr on a cloud provider.<\/span><\/p>\n<p>The model isn\u2019t likely to compete with those built by Big Tech \u2014 but it at least provides a way to scrutinize them.\n                        <\/p><\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/bloom-new-open-source-ai-model-bigger-than-gpt-3-large-language-model-llm\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;An open-source model that dwarfs GPT-3 aims to free AI from Big Tech&#8221; A\u00a0language model bigger than GPT-3 has arrived with a bold\u00a0 ambition: freeing AI from Big Tech\u2019s clutches. Named BLOOM, the large language model (LLM) promises a similar performance to Silicon Valley\u2019s leading systems \u2014 but with a radically different approach to access&#8230;.<\/p>\n","protected":false},"author":1,"featured_media":473306,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/img-cdn.tnwcdn.com\/image\/neural?filter_last=1&fit=1280,640&url=https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/07\/Untitled-design-1-2.jpg&signature=2b3ea2c0f7c6537b588fda34f8b22cbc","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-473305","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/473305","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=473305"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/473305\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/473306"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=473305"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=473305"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=473305"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}