{"id":89762,"date":"2020-10-15T14:21:54","date_gmt":"2020-10-15T11:21:54","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/microsofts-image-captioning-ai-is-pretty-darn-good-at-describing-pictures\/"},"modified":"2020-10-15T14:21:54","modified_gmt":"2020-10-15T11:21:54","slug":"microsofts-image-captioning-ai-is-pretty-darn-good-at-describing-pictures","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/microsofts-image-captioning-ai-is-pretty-darn-good-at-describing-pictures\/","title":{"rendered":"#Microsoft’s image-captioning AI is pretty darn good at describing pictures"},"content":{"rendered":"

“#Microsoft’s image-captioning AI is pretty darn good at describing pictures<\/strong>”<\/p>\n

\n Microsoft has built a new AI image-captioning system that described photos more accurately than humans in limited tests.<\/p>\n

The model has been added to Seeing AI<\/a>, a free app<\/a> for people with visual impairments that uses a smartphone camera to read text, identify people, and describe objects and surroundings.<\/p>\n

It\u2019s also now available to\u00a0app developers through\u00a0the Computer Vision API in Azure Cognitive Services, and will start rolling out in Microsoft Word, Outlook, and PowerPoint later this year.<\/span><\/p>\n

The model can generate \u201calt text\u201d image descriptions for web pages and documents, an important feature for people with limited vision that\u2019s all-too-often unavailable.<\/p>\n

\n
\n
\n

\u201cIdeally, everyone would include alt text for all images in documents, on the web, in social media<\/a> \u2013 as this enables people who are blind to access the content and participate in the conversation,\u201d said\u00a0Saqib Shaikh, a software engineering manager at Microsoft\u2019s AI platform group. \u201cBut, alas, people don\u2019t. So, there are several apps that use image captioning as [a] way to fill in alt text when it\u2019s missing.\u201d<\/p>\n

[Read:\u00a0Microsoft unveils efforts to make AI more accessible to people with disabilities]<\/em>\n<\/div>\n<\/div>\n<\/section>\n

The algorithm now tops the leaderboard of an image-captioning benchmark called\u00a0nocaps<\/a>.\u00a0Microsoft achieved this by\u00a0pre-training a large AI model on a dataset of images paired with word tags \u2014 rather than full captions, which are less efficient to create. Each of the tags was mapped to a specific object in an image.<\/span><\/p>\n

The pre-trained model was then fine-tuned on a dataset of captioned images, which enabled it to compose sentences. It then used its \u201cvisual vocabulary\u201d to create captions for images containing novel objects.<\/p>\n

Microsoft said the model is twice as good as the one it\u2019s used in products since 2015. The\u00a0image below shows how these improvements work in practice:<\/p>\n

\"\"
Credit: Microsoft<\/figcaption>
<\/a>The legacy AI captioned this image as \u201cA person sitting at a table using a laptop.\u201d The new model described it as \u201cA person using a microscope.\u201d<\/figcaption><\/figure>\n

However, the benchmark performance achievement doesn\u2019t mean the model will be better than humans at image captioning in the real world. Harsh Agrawal, one of the creators of the benchmark, told The Verge<\/a> that its evaluation metrics \u201conly roughly correlate with human preferences\u201d and that it \u201conly covers a small percentage of all the possible visual concepts.\u201d<\/p>\n

\n Published October 15, 2020 \u2014 11:21 UTC\n <\/p>\n<\/p><\/div>\n