{"id":441295,"date":"2022-05-03T17:26:38","date_gmt":"2022-05-03T14:26:38","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/"},"modified":"2022-05-03T17:26:38","modified_gmt":"2022-05-03T14:26:38","slug":"whats-the-transformer-machine-learning-model-and-why-should-you-care","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/","title":{"rendered":"#What\u2019s the transformer machine learning model? And why should you care?"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a284c8d05e84\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a284c8d05e84\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-1'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#%E2%80%9CWhats_the_transformer_machine_learning_model_And_why_should_you_care%E2%80%9D\" >&#8220;What\u2019s the transformer machine learning model? And why should you care?&#8221;<\/a><ul class='ez-toc-list-level-2' ><li class='ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Processing_sequences_with_neural_networks\" >Processing sequences with neural networks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Processing_sequences_with_neural_networks-2\" >Processing sequences with neural networks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Tokens_and_embeddings\" >Tokens and embeddings<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Attention_layers\" >Attention layers<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Training_the_transformer\" >Training the transformer<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/buradabiliyorum.com\/en\/whats-the-transformer-machine-learning-model-and-why-should-you-care\/#Variations_of_the_transformer\" >Variations of the transformer<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h1><span class=\"ez-toc-section\" id=\"%E2%80%9CWhats_the_transformer_machine_learning_model_And_why_should_you_care%E2%80%9D\"><\/span>&#8220;What\u2019s the transformer machine learning model? And why should you care?&#8221;<span class=\"ez-toc-section-end\"><\/span><\/h1>\n<div>\n                            <em>This article is part of <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/tag\/demystifying-ai\/\">Demystifying AI<\/a>, a <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/watch-movies-tv-seriess\/\" data-internallinksmanager029f6b8e52c=\"8\" title=\"Watch Movies &amp; TV Series\" target=\"_blank\" rel=\"noopener\">series<\/a> of posts that (try to) disambiguate the jargon and myths surrounding AI. (In partnership with <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/www.paperspace.com\/\">Paperspace<\/a>)<\/em><\/p>\n<p>In recent years, the transformer model has become one of the main highlights of advances in <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2019\/02\/15\/what-is-deep-learning-neural-networks\/\">deep learning<\/a> and deep neural networks. It is mainly used for advanced <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>lications in natural language processing. Google is using it to enhance its search engine results. OpenAI has used transformers to create its famous GPT-2 and <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/08\/17\/openai-gpt-3-commercial-ai\/\">GPT-3<\/a> models.<\/p>\n<p>Since its debut in 2017, the transformer architecture has evolved and branched out into many different variants, expanding beyond language tasks into other areas. They have been used for time series forecasting. They are the key innovation behind AlphaFold, DeepMind\u2019s protein structure prediction model. <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/07\/15\/openai-codex-ai-programming\/\">Codex<\/a>, OpenAI\u2019s source code\u2013generation model, is based on transformers. More recently, transformers have found their way into computer vision, where they are slowly replacing <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/01\/06\/convolutional-neural-networks-cnn-convnets\/\">convolutional neural networks<\/a> (CNN) in many complicated tasks.<\/p>\n<p>Researchers are still exploring ways to improve transformers and use them in new applications. Here is a brief explainer about what makes transformers exciting and how they work.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Processing_sequences_with_neural_networks\"><\/span>Processing sequences with neural networks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385916 js-lazy\" alt=\"feedforward-neural-net-vs-recurrent-neural-network\" width=\"796\" height=\"209\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-796x209.webp\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-796x209.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-280x74.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-270x71.webp 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-540x142.webp 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network.webp 1390w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385916\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-796x209.webp\" alt=\"feedforward-neural-net-vs-recurrent-neural-network\" width=\"796\" height=\"209\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-796x209.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-280x74.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-270x71.webp 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network-540x142.webp 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/feedforward-neural-net-vs-recurrent-neural-network.webp 1390w\"\/><\/noscript><\/figure>\n<p>The classic <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/01\/28\/deep-learning-explainer\/\">feed-forward neural network<\/a> is not designed to keep track of sequential data and maps each input into an output. This works for tasks such as classifying images but fails on sequential data such as text. A machine learning model that processes text must not only compute every word but also take into consideration how words come in sequences and relate to each other. The meaning of words can change depending on other words that come before and after them in the sentence.<\/p>\n<p>Before transformers, <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/06\/08\/what-is-recurrent-neural-network-rnn\/\">recurrent neural networks<\/a> (RNN) were the go-to solution for natural language processing. When provided with a sequence of words, an RNN processes the first word and feeds back the result into the layer that processes the next word. This enables it to keep track of the entire sentence instead of processing each word separately.<\/p>\n<p>Recurrent neural nets had disadvantages that limited their usefulness. First, they were very slow. Since they had to process data sequentially, they could not take advantage of parallel computing hardware and graphics processing units (GPU) in training and inference. Second, they could not handle long sequences of text. As the RNN got deeper into a text excerpt, the effects of the first words of the sentence gradually faded. This problem, known as \u201cvanishing gradients,\u201d was problematic when two linked words were very far apart in the text. And third, they only captured the relations between a word and the words that came before it. In reality, the meaning of words depends on the words that come both before and after them.<\/p>\n<p>Long short-term memory (LSTM) networks, the successor to RNNs, were able to solve the vanishing gradients problem to some degree and were able to handle larger sequences of text. But LSTMs were even slower to train than RNNs and still couldn\u2019t take full advantage of parallel computing. They still relied on the serial processing of text sequences.<\/p>\n<p>Transformers, introduced in the 2017 paper \u201c<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/arxiv.org\/abs\/1706.03762\">Attention Is All You Need<\/a>,\u201d made two key contributions. First, they made it possible to process entire sequences in parallel, making it possible to scale the speed and capacity of sequential deep learning models to unprecedented rates. And second, they introduced \u201cattention mechanisms\u201d that made it possible to track the relations between words across very long text sequences in both forward and reverse directions.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Processing_sequences_with_neural_networks-2\"><\/span>Processing sequences with neural networks<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385917 js-lazy\" alt=\"rnn-types\" width=\"796\" height=\"495\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-796x495.webp\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-796x495.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-280x174.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-217x135.webp 217w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-434x270.webp 434w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types.webp 1126w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385917\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-796x495.webp\" alt=\"rnn-types\" width=\"796\" height=\"495\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-796x495.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-280x174.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-217x135.webp 217w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types-434x270.webp 434w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/rnn-types.webp 1126w\"\/><\/noscript><\/figure>\n<p>Before we discuss how the transformer model works, it is worth discussing the types of problems that sequential neural networks solve.<\/p>\n<p>A \u201cvector to sequence\u201d model takes a single input, such as an image, and produces a sequence of data, such as a description.<\/p>\n<p>A \u201csequence to vector\u201d model takes a sequence as input, such as a product review or a <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/social-mediaa\/\" data-internallinksmanager029f6b8e52c=\"1\" title=\"Social Media\" target=\"_blank\" rel=\"noopener\">social media<\/a> post, and outputs a single value, such as a sentiment score.<\/p>\n<p>A \u201csequence to sequence\u201d model takes a sequence as input, such as an English sentence, and outputs another sequence, such as the French translation of the sentence.<\/p>\n<p>Despite their differences, all these types of models have one thing in common. They learn representations. The job of a neural network is to transform one type of data into another. During training, the hidden layers of the neural network (the layers that stand between the input and output) tune their parameters in a way that best represents the features of the input data type and maps it to the output.<\/p>\n<p>The original transformer was designed as a sequence-to-sequence (seq2seq) model for machine translation (of course, seq2seq models are not limited to translation tasks). It is composed of an encoder module that compresses an input string from the source language into a vector that represents the words and their relations to each other. The decoder module transforms the encoded vector into a string of text in the destination language.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Tokens_and_embeddings\"><\/span>Tokens and embeddings<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385918 js-lazy\" alt=\"Tokens and embedding\" width=\"796\" height=\"408\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-796x408.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-796x408.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-280x143.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-264x135.png 264w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-527x270.png 527w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47.png 1402w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385918\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-796x408.png\" alt=\"Tokens and embedding\" width=\"796\" height=\"408\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-796x408.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-280x143.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-264x135.png 264w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47-527x270.png 527w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.44.47.png 1402w\"\/><\/noscript><\/figure>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385919 js-lazy\" alt=\"Tokens and embedding\" width=\"796\" height=\"268\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-796x268.png\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-796x268.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-280x94.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-270x91.png 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-540x182.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36.png 1380w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385919\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-796x268.png\" alt=\"Tokens and embedding\" width=\"796\" height=\"268\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-796x268.png 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-280x94.png 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-270x91.png 270w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36-540x182.png 540w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/Screenshot-2022-05-03-at-15.45.36.png 1380w\"\/><\/noscript><\/figure>\n<p>The input text must be processed and transformed into a unified format before being fed to the transformer. First, the text goes through a \u201ctokenizer,\u201d which breaks it down into chunks of characters that can be processed separately. The tokenization algorithm can depend on the application. In most cases, every word and punctuation mark roughly counts as one token. Some suffixes and prefixes count as separate tokens (e.g., \u201cize,\u201d \u201cly,\u201d and \u201cpre\u201d). The tokenizer produces a list of numbers that represent the token IDs of the input text.<\/p>\n<p>The tokens are then converted into \u201cword embeddings.\u201d A word embedding is a vector that tries to capture the value of words in a multi-dimensional space. For example, the words \u201ccat\u201d and \u201cdog\u201d can have similar values across some dimensions because they are both used in sentences that are about animals and house pets. However, \u201ccat\u201d is closer to \u201clion\u201d than \u201cwolf\u201d across some other dimension that separates felines from canids. Similarly, \u201cParis\u201d and \u201cLondon\u201d might be close to each other because they are both cities. However, \u201cLondon\u201d is closer to \u201cEngland\u201d and \u201cParis\u201d to \u201cFrance\u201d on a dimension that separates countries. Word embeddings usually have hundreds of dimensions.<\/p>\n<p>Word embeddings are created by embedding models, which are trained separately from the transformer. There are several pre-trained embedding models that are used for language tasks.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Attention_layers\"><\/span>Attention layers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385920 js-lazy\" alt=\"transformer-architecture\" width=\"796\" height=\"497\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-796x497.webp\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-796x497.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-280x175.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-216x135.webp 216w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-432x270.webp 432w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture.webp 1392w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385920\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-796x497.webp\" alt=\"transformer-architecture\" width=\"796\" height=\"497\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-796x497.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-280x175.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-216x135.webp 216w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture-432x270.webp 432w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/transformer-architecture.webp 1392w\"\/><\/noscript><\/figure>\n<p>Once the sentence is transformed into a list of word embeddings, it is fed into the transformer\u2019s encoder module. Unlike RNN and LSTM models, the transformer does not receive one input at a time. It can receive an entire sentence\u2019s worth of embedding values and process them in parallel. This makes transformers more compute-efficient than their predecessors and also enables them to examine the context of the text in both forward and backward sequences.<\/p>\n<p>To preserve the sequential nature of the words in the sentence, the transformer applies \u201cpositional encoding,\u201d which basically means that it modifies the values of each embedding vector to represent its location in the text.<\/p>\n<p>Next, the input is passed to the first encoder block, which processes it through an \u201cattention layer.\u201d The attention layer tries to capture the relations between the words in the sentence. For example, consider the sentence \u201cThe big black cat crossed the road after it dropped a bottle on its side.\u201d Here, the model must associate \u201cit\u201d with \u201ccat\u201d and \u201cits\u201d with \u201cbottle.\u201d Accordingly, it should establish other associations such as \u201cbig\u201d and \u201ccat\u201d or \u201ccrossed\u201d and \u201ccat.\u201d Otherwise put, the attention layer receives a list of word embeddings that represent the values of individual words and produces a list of vectors that represent both individual words and their relations to each other. The attention layer contains multiple \u201cattention heads,\u201d each of which can capture different kinds of relations between words.<\/p>\n<p>The output of the attention layer is fed to a feed-forward neural network that transforms it into a vector representation and sends it to the next attention layer. Transformers contain several blocks of attention and feed-forward layers to gradually capture more complicated relationships.<\/p>\n<p>The task of the decoder module is to translate the encoder\u2019s attention vector into the output data (e.g., the translated version of the input text). During the training phase, the decoder has access both to the attention vector produced by the encoder and the expected outcome (e.g., the translated string).<\/p>\n<p>The decoder uses the same tokenization, word embedding, and attention mechanism to process the expected outcome and create attention vectors. It then passes this attention vector and the attention layer in the encoder module, which establishes relations between the input and output values. In the translation application, this is the part where the words from the source and destination languages are mapped to each other. Like the encoder module, the decoder attention vector is passed through a feed-forward layer. Its result is then mapped to a very large vector which is the size of the target data (in the case of language translation, this can span across tens of thousands of words).<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Training_the_transformer\"><\/span>Training the transformer<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"post-image post-mediaBleed aligncenter\"><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385921 js-lazy\" alt=\"large-language-models\" width=\"796\" height=\"497\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-796x497.webp\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-796x497.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-280x175.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-216x135.webp 216w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-432x270.webp 432w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models.webp 1392w\"\/><noscript><img decoding=\"async\" loading=\"lazy\" class=\"aligncenter size-featured_img wp-image-1385921\" src=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-796x497.webp\" alt=\"large-language-models\" width=\"796\" height=\"497\" srcset=\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-796x497.webp 796w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-280x175.webp 280w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-216x135.webp 216w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models-432x270.webp 432w, https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/large-language-models.webp 1392w\"\/><\/noscript><\/figure>\n<p>During training, the transformer is provided with a very large corpus of paired examples (e.g., English sentences and their corresponding French translations). The encoder module receives and processes the full input string. The decoder, however, receives a masked version of the output string, one word at a time, and tries to establish the mappings between the encoded attention vector and the expected outcome. The encoder tries to predict the next word and makes corrections based on the difference between its output and the expected outcome. This feedback enables the transformer to modify the parameters of the encoder and decoder and gradually create the right mappings between the input and output languages.<\/p>\n<p>The more training data and parameters the transformer has, the more capacity it gains to maintain coherence and consistency across long sequences of text.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Variations_of_the_transformer\"><\/span>Variations of the transformer<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In the machine translation example that we examined above, the encoder module of the transformer learned the relations between English words and sentences, and the decoder learns the mappings between English and French.<\/p>\n<p>But not all transformer applications require both the encoder and decoder module. For example, the GPT family of large language models uses stacks of decoder modules to generate text. BERT, another variation of the transformer model developed by researchers at Google, only uses encoder modules.<\/p>\n<p>The advantage of some of these architectures is that they can be trained through <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/03\/23\/yann-lecun-self-supervised-learning\/\">self-supervised learning<\/a> or <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2020\/02\/10\/unsupervised-learning-vs-supervised-learning\/\">unsupervised methods<\/a>. BERT, for example, does much of its training by taking large corpora of unlabeled text, masking parts of it, and trying to predict the missing parts. It then tunes its parameters based on how much its predictions were close to or far from the actual data. By continuously going through this process, BERT captures the statistical relations between different words in different contexts. After this pretraining phase, BERT can be finetuned for a downstream task such as question answering, text summarization, or sentiment analysis by training it on a small number of labeled examples.<\/p>\n<p>Using unsupervised and self-supervised pretraining reduces the manual effort required to annotate training data.<\/p>\n<p>A lot more can be said about transformers and the new applications they are unlocking, which is out of the scope of this article. Researchers are still finding ways to squeeze more out of transformers.<\/p>\n<p>Transformers have also created discussions about <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/12\/20\/artificial-intelligence-large-language-understanding\/\">language understanding and artificial general intelligence<\/a>. What is clear is that transformers, like other neural networks, are statistical models that capture regularities in data in clever and complicated ways. <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2021\/07\/12\/linguistics-for-the-age-of-ai\/\">They do not \u201cunderstand\u201d language<\/a> in the way that humans do. But they are exciting and useful nonetheless and have a lot to offer.<\/p>\n<p><em>This article was originally written by Ben Dickson and published by Ben Dickson on<span>\u00a0<\/span><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/\">TechTalks<\/a>, a publication that examines trends in <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">technology<\/a>, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original article\u00a0<a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/bdtechtalks.com\/2022\/05\/02\/what-is-the-transformer\/\">here<\/a>.<\/em>\n                        <\/div>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMLG0nwswvr63Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\">For forums sites go to <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/forum.buradabiliyorum.com\/\" target=\"_blank\" rel=\"noopener\">Forum.BuradaBiliyorum.Com<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more like this article, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/technology\/\" target=\"_blank\" rel=\"noopener\">Technology category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/thenextweb.com\/news\/whats-the-transformer-machine-learning-model\" target=\"_blank\" rel=\"noopener\">Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>&#8220;What\u2019s the transformer machine learning model? And why should you care?&#8221; This article is part of Demystifying AI, a series of posts that (try to) disambiguate the jargon and myths surrounding AI. (In partnership with Paperspace) In recent years, the transformer model has become one of the main highlights of advances in deep learning and&#8230;<\/p>\n","protected":false},"author":1,"featured_media":441296,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"https:\/\/img-cdn.tnwcdn.com\/image\/neural?filter_last=1&fit=1280,640&url=https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2022\/05\/TransformerMM.jpg&signature=92f8b94cdce98911cec71e3d778b563f","fifu_image_alt":"","footnotes":""},"categories":[18],"tags":[],"class_list":["post-441295","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/441295","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=441295"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/441295\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media\/441296"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=441295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=441295"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=441295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}