{"id":676841,"date":"2025-06-23T07:35:26","date_gmt":"2025-06-23T04:35:26","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/bilinear-sequence-regression-model-shows-why-ai-excels-at-learning-from-word-sequences\/"},"modified":"2025-06-23T07:35:26","modified_gmt":"2025-06-23T04:35:26","slug":"bilinear-sequence-regression-model-shows-why-ai-excels-at-learning-from-word-sequences","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/bilinear-sequence-regression-model-shows-why-ai-excels-at-learning-from-word-sequences\/","title":{"rendered":"Bilinear sequence regression model shows why AI excels at learning from word sequences"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a2f5f74b8193\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #dd3333;color:#dd3333\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #dd3333;color:#dd3333\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a2f5f74b8193\" checked aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buradabiliyorum.com\/en\/bilinear-sequence-regression-model-shows-why-ai-excels-at-learning-from-word-sequences\/#A_black_box_even_for_experts\" >A black box, even for experts<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buradabiliyorum.com\/en\/bilinear-sequence-regression-model-shows-why-ai-excels-at-learning-from-word-sequences\/#A_clear_mathematical_benchmark\" >A clear mathematical benchmark<\/a><\/li><\/ul><\/nav><\/div>\n<div>\n<div class=\"article-gallery lightGallery\">\n<div data-thumb=\"https:\/\/scx1.b-cdn.net\/csz\/news\/tmb\/2024\/chatgpt-4.jpg\" data-src=\"https:\/\/scx2.b-cdn.net\/gfx\/news\/hires\/2024\/chatgpt-4.jpg\" data-sub-html=\"Credit: Unsplash\/CC0 Public Domain\">\n<figure class=\"article-img\">\n            <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/scx1.b-cdn.net\/csz\/news\/800a\/2024\/chatgpt-4.jpg\" alt=\"ChatGPT\" title=\"Credit: Unsplash\/CC0 Public Domain\" width=\"800\" height=\"530\"\/><figcaption class=\"text-darken text-low-up text-truncate-js text-truncate mt-3\">\n                Credit: Unsplash\/CC0 Public Domain<br \/>\n            <\/figcaption><\/figure>\n<\/p><\/div>\n<\/div>\n<p>Researchers at EPFL have created a mathematical model that helps explain how breaking language into sequences makes modern AI-like chatbots so good at understanding and using words. The work is <a rel=\"nofollow\" target=\"_blank\" href=\"https:\/\/link.aps.org\/doi\/10.1103\/l4p2-vrxt\" target=\"_blank\">published<\/a> in the journal <i>Physical Review X<\/i>.<\/p>\n<p>There is no doubt that AI <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/technology\/\" data-internallinksmanager029f6b8e52c=\"4\" title=\"Technology\" target=\"_blank\" rel=\"noopener\">technology<\/a> is dominating our world today. Progress seems to be moving in leaps and bounds, especially focused on large language models (LLMs) like chatGPT.<\/p>\n<p>But how do they work? LLMs are made up of neural networks that process long sequences of &#8220;tokens.&#8221; Each token is typically a word or part of a word and is represented by a list of hundreds or thousands of numbers\u2014what researchers call a &#8220;high-dimensional vector.&#8221; This list captures the word&#8217;s meaning and how it&#8217;s used.<\/p>\n<p>For example, the word &#8220;cat&#8221; might become a list like [0.15, -0.22, 0.47, \u2026, 0.09], while &#8220;dog&#8221; is encoded in a similar way but with its own unique numbers. Words with similar meanings get similar lists, so the LLM can recognize that &#8220;cat&#8221; and &#8220;dog&#8221; are more alike than &#8220;cat&#8221; and &#8220;banana.&#8221;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_black_box_even_for_experts\"><\/span>A black box, even for experts<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Processing language as sequences of these vectors is clearly effective, but, ironically, we don&#8217;t really understand why. Simple mathematical models for long sequences of these high-dimensional tokens are still mostly unexplored.<\/p>\n<p>This leaves a gap in our understanding: Why does this <a href=\"https:\/\/buradabiliyorum.com\/en\/category\/download-scripts-themes-apps\/\" data-internallinksmanager029f6b8e52c=\"9\" title=\"Download Scripts &amp; Themes &amp; Apps\" target=\"_blank\" rel=\"noopener\">app<\/a>roach work so well, and what makes it fundamentally different from older methods? Why is it better to present data to neural networks as sequences of high-dimensional tokens rather than as a single, long list of numbers? While today&#8217;s AI can write stories or answer questions impressively, the inner workings that make this possible are still a black box\u2014even for experts.<\/p>\n<p>Now, a team of scientists led by Lenka Zdeborov\u00e1 at EPFL has built the simplest possible mathematical model that still captures the heart of learning from tokens as LLMs do.<\/p>\n<p>Their model, called bilinear sequence regression (BSR), s<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/trip-and-travel\/\" data-internallinksmanager029f6b8e52c=\"10\" title=\"Trip &amp; Travel\" target=\"_blank\" rel=\"noopener\">trip<\/a>s away the complexity of real-world AI but keeps some of its essential structure and acts as a &#8220;theoretical playground&#8221; for studying how AI models learn from sequences.<\/p>\n<p>How does BSR work? Imagine a sentence where you can turn each word into a list of numbers that captures its meaning\u2014just like LLMs do. You line these lists up into a table, with one row per word. This table keeps track of the whole sequence and all the details packed into each word.<\/p>\n<p>                                                                                                        <!-- TechX - News - In-article --><\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_clear_mathematical_benchmark\"><\/span>A clear mathematical benchmark<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Instead of processing all the information at once like older AI models, BSR looks at the rows of the table in one way and at the column in another. The model then uses this information to predict a single outcome, such as the sentiment of the sentence.<\/p>\n<p>The power of BSR is that it is simple enough to be fully solved with mathematics. This lets researchers see exactly when sequence-based learning starts to work, and how much data is needed for a model to reliably learn from patterns in sequences.<\/p>\n<p>BSR sheds light on why we get better results using a sequence of embeddings rather than flattening all the data into one big vector. The model revealed sharp thresholds where learning jumps from useless to effective once it &#8220;sees&#8221; enough examples.<\/p>\n<p>This research offers a new lens for understanding the inner workings of large language models. By solving BSR exactly, the team provides a clear mathematical benchmark that takes a step toward a theory that can guide the design of future AI systems.<\/p>\n<p>These insights could help scientists build models that are simpler, more efficient, and possibly more transparent.<\/p>\n<div class=\"article-main__more p-4\">\n<p><strong>More information:<\/strong><br \/>\n\t\t\t\t\t\t\t\t\t\t\t\tVittorio Erba et al, Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-Dimensional Tokens, <i>Physical Review X<\/i> (2025). <a rel=\"nofollow\" target=\"_blank\" data-doi=\"1\" href=\"https:\/\/dx.doi.org\/10.1103\/l4p2-vrxt\" target=\"_blank\">DOI: 10.1103\/l4p2-vrxt<\/a><\/p>\n<\/p><\/div>\n<div class=\"d-inline-block text-medium my-4\">\n                                                Provided by<br \/>\n                                                                                                    Ecole Polytechnique Federale de Lausanne<br \/>\n                                                    \t\t\t\t\t\t\t\t\t\t\t\t\t<a rel=\"nofollow\" target=\"_blank\" class=\"icon_open\" href=\"http:\/\/www.epfl.ch\/\" target=\"_blank\" rel=\"nofollow\"><br \/>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<svg>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<use href=\"https:\/\/techx.b-cdn.net\/tmpl\/v2\/img\/svg\/sprite.svg#icon_open\" x=\"0\" y=\"0\"\/>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/svg><br \/>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/a><\/p><\/div>\n<p>                                        <!-- print only --><\/p>\n<div class=\"d-none d-print-block\">\n<p>\n                                                <strong>Citation<\/strong>:<br \/>\n                                                Bilinear sequence regression model shows why AI excels at learning from word sequences (2025, June 20)<br \/>\n                                                retrieved 23 June 2025<br \/>\n                                                from https:\/\/techxplore.com\/<a href=\"https:\/\/buradabiliyorum.com\/en\/category\/news\/\" data-internallinksmanager029f6b8e52c=\"2\" title=\"News\" target=\"_blank\" rel=\"noopener\">news<\/a>\/2025-06-bilinear-sequence-regression-ai-excels.html\n                                            <\/p>\n<p>\n                                            This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no<br \/>\n                                            part may be reproduced without the written permission. The content is provided for information purposes only.\n                                            <\/p>\n<\/p><\/div>\n<\/p><\/div>\n<p><script id=\"facebook-jssdk\" async=\"\" src=\"https:\/\/connect.facebook.net\/en_US\/sdk.js\"><\/script><\/p>\n<blockquote><p><strong><span style=\"color: #ff6600;\">If you liked the article, do not forget to share it with your friends. Follow us on\u00a0<span style=\"color: #ff0000;\"><a style=\"color: #ff0000;\" href=\"https:\/\/news.google.com\/publications\/CAAqBwgKMN63nwsw68G3Aw\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Google News<\/a><\/span>\u00a0too, click on the star and choose us from your favorites.<\/span><\/strong><\/p><\/blockquote>\n<blockquote>\n<p style=\"text-align: center;\"><strong>If you want to read more Like this articles, you can visit our <span style=\"color: #ff9900;\"><a style=\"color: #ff9900;\" href=\"https:\/\/en.buradabiliyorum.com\/category\/sciencee\/\" target=\"_blank\" >Science category.<\/a><\/span><\/strong><\/p>\n<\/blockquote>\n<p><span style=\"color: black;\"><a style=\"color: #ff9900;\" href=\"https:\/\/techxplore.com\/news\/2025-06-bilinear-sequence-regression-ai-excels.html\" target=\"_blank\" >Source<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Credit: Unsplash\/CC0 Public Domain Researchers at EPFL have created a mathematical model that helps explain how breaking language into sequences makes modern AI-like chatbots so good at understanding and using words. The work is published in the journal Physical Review X. There is no doubt that AI technology is dominating our world today. Progress seems&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"fifu_image_url":"","fifu_image_alt":"","footnotes":""},"categories":[16],"tags":[],"class_list":["post-676841","post","type-post","status-publish","format-standard","hentry","category-sciencee"],"_links":{"self":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/676841","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/comments?post=676841"}],"version-history":[{"count":0,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/posts\/676841\/revisions"}],"wp:attachment":[{"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/media?parent=676841"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/categories?post=676841"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buradabiliyorum.com\/en\/wp-json\/wp\/v2\/tags?post=676841"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}