{"id":289395,"date":"2021-07-01T20:29:11","date_gmt":"2021-07-01T17:29:11","guid":{"rendered":"https:\/\/en.buradabiliyorum.com\/new-data-science-platform-speeds-up-python-queries\/"},"modified":"2021-07-01T20:29:11","modified_gmt":"2021-07-01T17:29:11","slug":"new-data-science-platform-speeds-up-python-queries","status":"publish","type":"post","link":"https:\/\/buradabiliyorum.com\/en\/new-data-science-platform-speeds-up-python-queries\/","title":{"rendered":"#New data science platform speeds up Python queries"},"content":{"rendered":"

“#New data science<\/a> platform speeds up Python queries<\/strong>”<\/p>\n

\n
\n
\n
\"coding\"
\n Credit: CC0 Public Domain
\n <\/figcaption><\/figure>\n<\/div>\n<\/div>\n

Researchers from Brown University and MIT have developed a new data science framework that allows users to process data with the programming language Python\u2014without paying the ‘performance tax’ normally associated with a user-friendly language.<\/p>\n

The new framework, called Tuplex, is able to process data queries written in Python up to 90 times faster than industry-standard data systems like Apache Spark or Dask. The research team unveiled the system in research presented at SIGMOD 2021, a premier data processing conference, and have made the software freely available to all.<\/p>\n

“Python is the primary programming language used by people doing data science,” said Malte Schwarzkopf, an assistant professor of computer science at Brown and one of the developers of Tuplex. “That makes a lot of sense. Python is widely taught in universities, and it’s an easy language to get started with. But when it comes to data science, there’s a huge performance tax associated with Python because platforms can’t process Python efficiently on the back end.”<\/p>\n

Platforms like Spark perform data analytics by distributing tasks across multiple processor cores or machines in a data center. That parallel processing allows users to deal with giant data sets that would choke a single computer to death. Users interact with these platforms by inputting their own queries, which contain custom logic written as “user-defined functions” or UDFs. UDFs specify custom logic, like extracting the number of bedrooms from the text of a real estate listing for a query that searches all of the real estate listings in the U.S. and selects all the ones with three bedrooms.<\/p>\n

Because of its simplicity, Python is the language of choice for creating UDFs in the data science community. In fact, the Tuplex team cites a recent poll showing that 66% of data platform users utilize Python as their primary language. The problem is that analytics platforms have trouble dealing with those bits of Python code efficiently.<\/p>\n


\n