{"id":256347,"date":"2021-05-22T13:00:52","date_gmt":"2021-05-22T10:00:52","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/"},"modified":"2021-05-22T13:00:52","modified_gmt":"2021-05-22T10:00:52","slug":"understanding-transformers-the-machine-learning-model-behind-gpt-3","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/","title":{"rendered":"#Understanding Transformers, the machine learning model behind GPT-3"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a882768c9ece\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a882768c9ece\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#Enter_Transformers\" >Enter Transformers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#How_do_Transformers_work\" >How do Transformers work?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#Positional_Encodings\" >Positional Encodings<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#Attention\" >Attention<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#Self-Attention\" >Self-Attention<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#What_Can_Transformers_Do\" >What Can Transformers Do?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/buradabiliyorum.com\/en\/understanding-transformers-the-machine-learning-model-behind-gpt-3\/#How_can_I_use_Transformers\" >How can I use Transformers?<\/a><\/li><\/ul><\/nav><\/div>\n<p>&#8220;<strong>#Understanding Transformers, the machine learning model behind GPT-3<\/strong>&#8221;<\/p>\n<div>You know that expression<span>\u00a0<\/span><em>When you have a hammer, everything looks like a nail<\/em>? Well, in machine learning, it seems like we really have discovered a magical hammer for which everything is, in fact, a nail, and they\u2019re called Transformers. Transformers are models that can be designed to translate text, write\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.gwern.net\/GPT-3\">poems and op eds<\/a>, and<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.wired.com\/story\/ai-latest-trick-writing-computer-code\/\">even generate computer code<\/a>. In fact, lots of the amazing research I write about on daleonai.com is built on Transformers, like\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/daleonai.com\/how-alphafold-works\">AlphaFold 2<\/a>, the model that predicts the structures of proteins from their genetic sequences, as well as powerful natural language processing (NLP) models like\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/daleonai.com\/how-alphafold-works\">GPT-3<\/a>, BERT, T5, Switch, Meena, and others. You might say they\u2019re more than meets the\u2026 ugh, forget it.<\/p>\n<p>If you want to stay hip in machine learning and especially NLP, you have to know at least a bit about Transformers. So in this post, we\u2019ll talk about what they are, how they work, and why they\u2019ve been so impactful.<\/p>\n<p>A Transformer is a type of neural network architecture. To recap, neural nets are a very effective type of model for analyzing complex data types like images, videos, audio, and text. But there are different types of neural networks optimized for different types of data. For example, for analyzing images, we\u2019ll typically use\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/en.wikipedia.org\/wiki\/Convolutional_neural_network\">convolutional neural networks<\/a><span>\u00a0<\/span>or \u201cCNNs.\u201d Vaguely, they mimic the way the human brain processes visual information.<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351850 js-lazy\" alt=\"A typical Convolutional Neural Network\" width=\"640\" height=\"230\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons..png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons..png 640w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-280x101.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-540x194.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-270x97.png 270w\"\/><figcaption>Credit: Renanar2 \/ Wikicommons<\/figcaption><figcaption><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/thenextweb.com\/news\/#\" data-url=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Feditorial.thenextweb.com%2Fneural%2F2021%2F05%2F22%2Funderstanding-transformers-the-machine-learning-model-behind-gpt-3-machine-learning-ai-syndication%2F&amp;via=thenextweb&amp;related=thenextweb&amp;text=Check out this picture on: A typical Convolutional Neural Network\" data-title=\"Share A typical Convolutional Neural Network on Twitter\" data-width=\"685\" data-height=\"500\" class=\"post-image-share popitup\" title=\"Share A typical Convolutional Neural Network on Twitter\"><i class=\"icon icon--inline icon--twitter--dark\"\/><\/a>A typical Convolutional Neural Network<\/figcaption><noscript><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351850\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons..png\" alt=\"A typical Convolutional Neural Network\" width=\"640\" height=\"230\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons..png 640w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-280x101.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-540x194.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Convolutional-Neural-Network-courtesy-Renanar2-at-Wikicommons.-270x97.png 270w\"\/><\/noscript><\/figure>\n<p>And<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/qz.com\/1034972\/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world\/\">since around 2012<\/a>, we\u2019ve been quite successful at solving vision problems with CNNs, like identifying objects in photos, recognizing faces, and reading handwritten digits. But for a long time, nothing comparably good existed for language tasks (translation, text summarization, text generation, named entity recognition, etc). That was unfortunate, because language is the main way we humans communicate.<\/p>\n<p>Before Transformers were introduced in 2017, the way we used deep learning to understand text was with a type of model called a Recurrent Neural Network or RNN that looked something like this:<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351853 js-lazy\" alt=\"A typical Recurrent Neural Network (RNN)\" width=\"2880\" height=\"960\" sizes=\"auto, (max-width: 2880px) 100vw, 2880px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn.png 2880w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-280x93.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-540x180.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-270x90.png 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-796x265.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-1592x531.png 1592w\"\/><figcaption>Credit: Wiki<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">media<\/a><\/figcaption><figcaption><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/thenextweb.com\/news\/#\" data-url=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Feditorial.thenextweb.com%2Fneural%2F2021%2F05%2F22%2Funderstanding-transformers-the-machine-learning-model-behind-gpt-3-machine-learning-ai-syndication%2F&amp;via=thenextweb&amp;related=thenextweb&amp;text=Check out this picture on: A typical Recurrent Neural Network (RNN)\" data-title=\"Share A typical Recurrent Neural Network (RNN) on Twitter\" data-width=\"685\" data-height=\"500\" class=\"post-image-share popitup\" title=\"Share A typical Recurrent Neural Network (RNN) on Twitter\"><i class=\"icon icon--inline icon--twitter--dark\"\/><\/a>A typical Recurrent Neural Network (RNN)<\/figcaption><noscript><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351853\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn.png\" alt=\"A typical Recurrent Neural Network (RNN)\" width=\"2880\" height=\"960\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn.png 2880w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-280x93.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-540x180.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-270x90.png 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-796x265.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/renn-1592x531.png 1592w\"\/><\/noscript><\/figure>\n<p>Let\u2019s say you wanted to translate a sentence from English to French. An RNN would take as input an English sentence, process the words one at a time, and then, sequentially, spit out their French counterparts. The key word here is \u201csequential.\u201d In language, the order of words matters and you can\u2019t just shuffle them around. The sentence:<\/p>\n<p>\u201cJane went looking for trouble.\u201d<\/p>\n<p>means something very different from the sentence:<\/p>\n<p>\u201cTrouble went looking for Jane\u201d<\/p>\n<p>So any model that\u2019s going to understand language must capture word order, and recurrent neural networks did this by processing one word at a time, in a sequence.<\/p>\n<p>But RNNs had issues. First, they struggled to handle large sequences of text, like long paragraphs or essays. By the time got to the end of a paragraph, they\u2019d forget what h<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>ened at the beginning. An RNN-based translation model, for example, might have trouble remembering the gender of the subject of a long paragraph.<\/p>\n<p>Worse, RNNs were hard to train. They were notoriously susceptible to what\u2019s called the<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/towardsdatascience.com\/the-exploding-and-vanishing-gradients-problem-in-time-series-6b87d558d22\">vanishing\/exploding gradient problem<\/a><span>\u00a0<\/span>(sometimes you simply had to restart training and cross your fingers). Even more problematic, because they processed words sequentially, RNNs were hard to parallelize. This meant you couldn\u2019t just speed up training by throwing more GPUs at the them, which meant, in turn, you couldn\u2019t train them on all that much data.<\/p>\n<h2 id=\"enter-transformers\"><span class=\"ez-toc-section\" id=\"Enter_Transformers\"><\/span>Enter Transformers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This is where Transformers changed everything. They were developed in 2017 by researchers at Google and the University of Toronto, initially designed to do translation. But unlike recurrent neural networks, Transformers could be very efficiently parallelized. And that meant, with the right hardware, you could train some really big models.<\/p>\n<p>How big?<\/p>\n<p>Bigly big.<\/p>\n<p>GPT-3, the especially impressive text-generation model that writes almost as well as a human was trained on some<span>\u00a0<\/span><em>45 TB<\/em>\u00a0of text data, including almost all of the public web.<\/p>\n<p>So if you remember anything about Transformers, let it be this: combine a model that scales well with a huge dataset and the results will likely blow you away.<\/p>\n<h2 id=\"how-do-transformers-work\"><span class=\"ez-toc-section\" id=\"How_do_Transformers_work\"><\/span>How do Transformers work?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351860 js-lazy\" alt=\"Transformer diagram from the original paper\" width=\"1024\" height=\"1416\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1.png 1024w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-152x210.png 152w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-195x270.png 195w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-98x135.png 98w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-796x1101.png 796w\"\/><figcaption><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/thenextweb.com\/news\/#\" data-url=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Feditorial.thenextweb.com%2Fneural%2F2021%2F05%2F22%2Funderstanding-transformers-the-machine-learning-model-behind-gpt-3-machine-learning-ai-syndication%2F&amp;via=thenextweb&amp;related=thenextweb&amp;text=Check out this picture on: Transformer diagram from the original paper\" data-title=\"Share Transformer diagram from the original paper on Twitter\" data-width=\"685\" data-height=\"500\" class=\"post-image-share popitup\" title=\"Share Transformer diagram from the original paper on Twitter\"><i class=\"icon icon--inline icon--twitter--dark\"\/><\/a>Transformer diagram from the original paper<\/figcaption><noscript><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351860\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1.png\" alt=\"Transformer diagram from the original paper\" width=\"1024\" height=\"1416\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1.png 1024w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-152x210.png 152w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-195x270.png 195w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-98x135.png 98w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/screen-shot-2021-05-06-at-12.12.21-pm-1-796x1101.png 796w\"\/><\/noscript><\/figure>\n<p>While the diagram from the<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/1706.03762\">original paper<\/a><span>\u00a0<\/span>is a little scary, the innovation behind Transformers boils down to three main concepts:<\/p>\n<ol>\n<li>Positional Encodings<\/li>\n<li>Attention<\/li>\n<li>Self-Attention<\/li>\n<\/ol>\n<h2 id=\"positional-encodings\"><span class=\"ez-toc-section\" id=\"Positional_Encodings\"><\/span>Positional Encodings<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Let\u2019s start with the first one, positional encodings. Let\u2019s say we\u2019re trying to translate text from English to French. Remember that RNNs, the old way of doing translation, understood word order by processing words sequentially. But this is also what made them hard to parallelize.<\/p>\n<p>Transformers get around this barrier via an innovational called positional encodings. The idea is to take all of the words in your input sequence\u2013an English sentence, in this case\u2013and append each word with a number it\u2019s order. So, you feed your network a sequence like:<\/p>\n<p><code>[(\"Dale\", 1), (\"says\", 2), (\"hello\", 3), (\"world\", 4)]<\/code><\/p>\n<p>Conceptually, you can think of this as moving the burden of understanding word order from the structure of the neural network to the data itself.<\/p>\n<p>At first, before the Transformer has been trained on any data, it doesn\u2019t know how to interpret these positional encodings. But as the model sees more and more examples of sentences and their encodings, it learns how to use them effectively.<\/p>\n<p>I\u2019ve done a bit of over-simplification here\u2013the original authors used sine functions to come up with positional encodings, not the simple integers 1, 2, 3, 4\u2013but the point is the same. Store word order as data, not structure, and your neural network becomes easier to train.<\/p>\n<h2 id=\"attention\"><span class=\"ez-toc-section\" id=\"Attention\"><\/span>Attention<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>THE NEXT IMPORTANT PART OF TRANSFORMERS IS CALLED ATTENTION.<\/p>\n<p>Got that?<\/p>\n<p>Attention is a neural network structure that you\u2019ll hear about all over the place in machine learning these days. In fact, the title of the 2017 paper that introduced Transformers wasn\u2019t called,<span>\u00a0<\/span><em>We Present You the Transformer.<\/em><span>\u00a0<\/span>Instead it was called<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/pdf\/1706.03762.pdf\">Attention is All You Need<\/a>.<\/p>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/pdf\/1409.0473.pdf\">Attention<\/a><span>\u00a0<\/span>was introduced in the context of translation two years earlier, in 2015. To understand it, take this example sentence from the original paper:<\/p>\n<p><em>The agreement on the European Economic Area was signed in August 1992.<\/em><\/p>\n<p>Now imagine trying to translate that sentence into its French equivalent:<\/p>\n<p><em>L\u2019accord sur la zone \u00e9conomique europ\u00e9enne a \u00e9t\u00e9 sign\u00e9 en ao\u00fbt 1992.<\/em><\/p>\n<p>One bad way to try to translate that sentence would be to go through each word in the English sentence and try to spit out its French equivalent, one word at a time. That wouldn\u2019t work well for several reasons, but for one, some words in the French translation are flipped: it\u2019s \u201cEuropean Economic Area\u201d in English, but \u201cla zone \u00e9conomique europ\u00e9enne\u201d in French. Also, French is a language with gendered words. The adjectives \u201c\u00e9conomique\u201d and \u201ceurop\u00e9enne\u201d must be in feminine form to match the feminine object \u201cla zone.\u201d<\/p>\n<p>Attention is a mechanism that allows a text model to \u201clook at\u201d every single word in the original sentence when making a decision about how to translate words in the output sentence. Here\u2019s a nice visualization from that original attention paper:<\/p>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351862 js-lazy\" alt=\"Figure from the paper, 'Neural Machine Translation by Jointly Learning to Align and Translate (2015)'\" width=\"558\" height=\"548\" sizes=\"auto, (max-width: 558px) 100vw, 558px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015.png 558w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-214x210.png 214w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-275x270.png 275w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-137x135.png 137w\"\/><figcaption><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/thenextweb.com\/news\/#\" data-url=\"https:\/\/twitter.com\/intent\/tweet?url=https%3A%2F%2Feditorial.thenextweb.com%2Fneural%2F2021%2F05%2F22%2Funderstanding-transformers-the-machine-learning-model-behind-gpt-3-machine-learning-ai-syndication%2F&amp;via=thenextweb&amp;related=thenextweb&amp;text=Check out this picture on: Figure from the paper, \u2018Neural Machine Translation by Jointly Learning to Align and Translate (2015)\u2019\" data-title=\"Share Figure from the paper, \u2018Neural Machine Translation by Jointly Learning to Align and Translate (2015)\u2019 on Twitter\" data-width=\"685\" data-height=\"500\" class=\"post-image-share popitup\" title=\"Share Figure from the paper, \u2018Neural Machine Translation by Jointly Learning to Align and Translate (2015)\u2019 on Twitter\"><i class=\"icon icon--inline icon--twitter--dark\"\/><\/a>Figure from the paper, \u2018Neural Machine Translation by Jointly Learning to Align and Translate (2015)\u2019<\/figcaption><noscript><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1351862\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015.png\" alt=\"Figure from the paper, 'Neural Machine Translation by Jointly Learning to Align and Translate (2015)'\" width=\"558\" height=\"548\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015.png 558w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-214x210.png 214w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-275x270.png 275w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/Figure-from-the-paper-Neural-Machine-Translation-by-Jointly-Learning-to-Align-and-Translate-2015-137x135.png 137w\"\/><\/noscript><\/figure>\n<p>It\u2019s a sort of heat map that shows where the model is \u201cattending\u201d when it outputs each word in the French sentence. As you might expect, w<strong>h<\/strong>en the model outputs the word \u201ceurop\u00e9enne,\u201d it\u2019s attending heavily to both the input words \u201cEuropean\u201d and \u201cEconomic.\u201d<\/p>\n<p>And how does the model know which words it should be \u201cattending\u201d to at each time step? It\u2019s something that\u2019s learned from training data. By seeing thousands of examples of French and English sentences, the model learns what types of words are interdependent. It learns how to respect gender, plurality, and other rules of grammar.<\/p>\n<p>The attention mechanism has been an extremely useful tool for natural language processing since its discovery in 2015, but in its original form, it was used alongside recurrent neural networks. So, the innovation of the 2017 Transformers paper was, in part, to ditch RNNs entirely. That\u2019s why the 2017 paper was called \u201cAttention is<span>\u00a0<\/span><em>all<\/em><span>\u00a0<\/span>you need.\u201d<\/p>\n<h2 id=\"self-attention\"><span class=\"ez-toc-section\" id=\"Self-Attention\"><\/span>Self-Attention<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The last (and maybe most impactful) piece of the Transformer is a twist on attention called \u201cself-attention.\u201d<\/p>\n<p>The type of \u201cvanilla\u201d attention we just talked about helped align words across English and French sentences, which is important for translation. But what if you\u2019re not trying to translate words but instead build a model that understands underlying meaning and patterns in language\u2013a type of model that could be used to do any number of language tasks?<\/p>\n<p>In <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/general\/\" data-internallinksmanager029f6b8e52c=\"3\" title=\"General\" target=\"_blank\" rel=\"noopener\">general<\/a>, what makes neural networks powerful and exciting and cool is that they often automatically build up meaningful internal representations of the data they\u2019re trained on. When you inspect the layers of a vision neural network, for example, you\u2019ll find sets of neurons that \u201crecognize\u201d edges, shapes, and even high-level structures like eyes and mouths. A model trained on text data might automatically learn parts of speech, rules of grammar, and whether words are synonymous.<\/p>\n<p>The better the internal representation of language a neural network learns, the better it will be at any language task. And it turns out that attention can be a very effective way of doing just this, if it\u2019s turned on the input text itself.<\/p>\n<p>For example, take these two sentence:<\/p>\n<p>\u201cServer, can I have the check?\u201d<\/p>\n<p>\u201cLooks like I just crashed the server.\u201d<\/p>\n<p>The word server here means two very different things, which we humans can easily disambiguate by looking at surrounding words. Self-attention allows a neural network to understand a word in the context of the words around it.<\/p>\n<p>So when a model processes the word \u201cserver\u201d in the first sentence, it might be \u201cattending\u201d to the word \u201ccheck,\u201d which helps disambiguate a human server from a metal one.<\/p>\n<p>In the second sentence, the model might attend to the word \u201ccrashed\u201d to determine<span>\u00a0<\/span><em>this<\/em><span>\u00a0<\/span>\u201cserver\u201d refers to a machine.<\/p>\n<p>Self-attention help neural networks disambiguate words, do part-of-speech tagging, entity resolution, learn semantic roles and<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/1905.05950\">a lot more<\/a>.<\/p>\n<p>So, here we are.: Transformers, explained at 10,000 feet, boil down to:<\/p>\n<ol>\n<li>Position Encodings<\/li>\n<li>Attention<\/li>\n<li>Self-Attention<\/li>\n<\/ol>\n<p>If you want a deeper technical explanation, I\u2019d highly recommend checking out Jay Alammar\u2019s blog post<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"http:\/\/jalammar.github.io\/illustrated-transformer\/\">The Illustrated Transformer<\/a>.<\/p>\n<h2 id=\"what-can-transformers-do\"><span class=\"ez-toc-section\" id=\"What_Can_Transformers_Do\"><\/span>What Can Transformers Do?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>One of the most popular Transformer-based models is called BERT, short for \u201cBidirectional Encoder Representations from Transformers.\u201d It was introduced by researchers at Google around the time I joined the company, in 2018, and soon made its way into almost every NLP project\u2013including<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/blog.google\/products\/search\/search-language-understanding-bert\/\">Google Search<\/a>.<\/p>\n<p>BERT refers not just a model architecture but to a trained model itself, which you can download and use for free<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/github.com\/google-research\/bert\">here<\/a>. It was trained by Google researchers on a massive text corpus and has become something of a general-purpose pocket knife for NLP. It can be extended solve a bunch of different tasks, like:<\/p>\n<p>\u2013 text summarization<\/p>\n<p>\u2013 question answering<\/p>\n<p>\u2013 classification<\/p>\n<p>\u2013 named entity resolution<\/p>\n<p>\u2013 text similarity<\/p>\n<p>\u2013 offensive message\/profanity detection<\/p>\n<p>\u2013 understanding user queries<\/p>\n<p>\u2013 a whole lot more<\/p>\n<p>BERT proved that you could build very good language models trained on unlabeled data, like text scraped from Wikipedia and Reddit, and that these large \u201cbase\u201d models could then be adapted with domain-specific data to lots of different use cases.<\/p>\n<p>More recently, the model\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/daleonai.com\/gpt3-explained-fast\">GPT-3<\/a>, created by OpenAI, has been blowing people\u2019s minds with its ability to generate realistic text.<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/ai.googleblog.com\/2020\/01\/towards-conversational-agent-that-can.html\">Meena<\/a>, introduced by Google Research last year, is a Transformer-based chatbot (akhem, \u201cconversational agent\u201d) that can have compelling conversations about almost any topic (this author once spent twenty minutes arguing with Meena about what it means to be human).<\/p>\n<p>Transformers have also been making waves outside of NLP, by<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/magenta.tensorflow.org\/music-transformer\">composing music<\/a>,<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/daleonai.com\/dalle-5-mins\">generating images from text descriptions<\/a>, and<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/deepmind.com\/blog\/article\/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology\">predicting protein structure<\/a>.<\/p>\n<h2 id=\"how-can-i-use-transformers\"><span class=\"ez-toc-section\" id=\"How_can_I_use_Transformers\"><\/span>How can I use Transformers?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Now that you\u2019re sold on the power of Transformers, you might want to know how you can start using them in your own app. No problemo.<\/p>\n<p>You can download common Transformer-based models like BERT from<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/tfhub.dev\/\">TensorFlow Hub<\/a>. For a code tutorial, check out<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/daleonai.com\/semantic-ml\">this one<\/a><span>\u00a0<\/span>I wrote on building apps powered by semantic language.<\/p>\n<p>But if you want to be really trendy and you write Python, I\u2019d highly recommend the popular \u201cTransformers\u201d library maintained by the company\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/huggingface.co\/\">HuggingFace<\/a>. The platform allow you to train and use most of today\u2019s popular NLP models, like BERT, Roberta, T5, GPT-2, in a very developer-friendly way.<\/p>\n<\/div>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong>\n<\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/understanding-transformers-the-machine-learning-model-behind-gpt-3-machine-learning-ai-syndication\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;#Understanding Transformers, the machine learning model behind GPT-3&#8221; You know that expression\u00a0When you have a hammer, everything looks like a nail? Well, in machine learning, it seems like we really have discovered a magical hammer for which everything is, in fact, a nail, and they\u2019re called Transformers. Transformers are models that can be designed to&#8230;<\/p>\n","protected":false},"author":1,"featured_media":256348,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/img-cdn.tnwcdn.com\/image\/neural?filter_last=1&fit=1280,640&url=https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2021\/05\/AI-Transformers-abstract-hed.jpg&signature=7dad3d13328b212e6fc49afed1eb2819","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-256347","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/256347","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=256347"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/256347\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/256348"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=256347"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=256347"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=256347"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}