The future of Python: Concurrency devoured, Node.js next on menu
Programming language keeps getting fatter amid awkward version 3 split
Analysis The PyBay 2017 conference, held in San Francisco over the weekend, began with a keynote about concurrency.
Though hardly a draw for a general interest audience, the topic – an examination of multithreaded and multiprocess programming techniques – turns out to be central to the future of Python.
Since 2008, the Python community has tried to reconcile incompatibility between Python 2 and newly introduced Python 3.
For years, adoption of Python 3 was slow and some even dared to suggest Python didn't have a future. As late as last year, Zed Shaw, an accomplished developer and author of the popular Learn Python the Hard Way, even ventured to opine, "There is a high probability that Python 3 is such a failure it will kill Python."
Despite these unsubstantiated odds, Shaw – a polarizing figure for some Pythonistas – this year released a version of his book for Python 3.
Python remains one of the most popular programming languages. It ranked first in IEEE Spectrum's top programming languages for 2017, third in Redmonk's June 2017 ranking, and fifth in Tiobe's August index. And it continues to be recommended for those new to writing code.
It might even be argued that Python is resurgent, thanks to its utility for data science and machine learning projects. Now that almost all the popular Python packages have been ported to Python 3, doubts about the language have receded into the background.
It's too simplistic to say that the two languages are competing with one another, but there's some of that, particularly now that Python is focusing on event-driven programming.
In his keynote, Python core developer, consultant and educator Raymond Hettinger outlined the advantages and disadvantages of threads and processes for implementing concurrency in programs.
Concurrency is important, he said, because it's how the real world operates. Things don't always happen in a predictable sequence.
Code that implements concurrency can handle multiple tasks that complete at different times. It's essential for writing applications that scale and it remains an area of active discussion among Python programmers.
Last December, Python version 3.6 arrived, bringing with it non-provisional support for the asyncio module introduced in Python 3.4. The module provides a mechanism for writing single-threaded concurrent code.
"It's insanely difficult to get large multi-threaded programs correct," Hettinger explained. "For complex systems, async is much easier to get right than threads with locks."
'There's twice as much Python as you know'
"There's twice as much Python as you know," as Hettinger put it.
Simeon Franklin, a Python technical instructor at Twitter, said there's a lot of enthusiasm about new features, but suggested it's hard to keep up. He said he used to know the entire standard library very well. Now, he said, there's a lot of new material.
Event-driven programming was available in Python long before Node existed, through the Twisted framework, created by Glyph Lefkowitz in 2002, and later the Tornado framework. Now, with Python 3 showing signs of strong adoption, it has become a central focus among Python developers.
Event-driven programming relies on an event loop that runs continuously. Presented with asynchronous requests, it can process them without blocking other requests.
Certain applications may be suited for handling requests in a series, but this approach isn't ideal, say, if you're trying to serve a web page – you don't want a site visitor to wait for each image and file to be downloaded, one after the other, if your code and infrastructure can handle simultaneous transmission.
"I think async is the future," said Hettinger. "Threading is so hard to get right."
Lefkowitz, in an email to The Register, said, "Based on my own experience at PyCon, asyncio is really bringing the community together around event-driven concurrency as the main, blessed way to do concurrency at a language level in Python."
Lefkowitz pointed to a blog post he wrote in 2014 and said his predictions have largely come true. "By adopting 'async def' and 'await' into the language, the core team has largely ended the debate about whether it is 'pythonic' or not, and cooperative concurrency has become the dominant coordination paradigm," he said.
The adoption of asyncio has created a library ecosystem boom, Lefkowitz said, but it has not made projects like Twisted or Tornado obsolete. "Relieved of the need to convince people that 'event-driven might be a good idea,' we now have people showing up for the functionality, ready-made ecosystem, and consistent design sensibility that Twisted provides rather than just 'it's the only way to do async,'" he explained.
Lefkowitz believes Twisted helped shape Python community consensus. "There's no one dramatic thing we did," he said. "Asyncio largely adopted Twisted's lowest-level core interfaces (protocols and transports, the reactor) and learned a lot from the higher-level request-response pattern (Deferreds) without adopting it outright. A lot of care was taken in asyncio's design to ensure it could be compatible with existing frameworks and give them a turbo-boost of community interest rather than hitting the reset button."