飞跃式发展的后现代 Python 世界_python教程_开发教程

如果现代Python有一个标志性特性，那么简单说来便是Python对自身定义的越来越模糊。在过去的几年的许多项目都极大拓展了Python，并重建了“Python”本身的意义。

与此同时新技术的涌现侵占了Python的份额，并带来了新的优势：

Go – ( Goroutines, Types, Interfaces )
Rust – ( Traits, Speed, Types )
Julia – ( Speed, Types, Multiple Dispatch )
Scala – ( Traits, Speed, Types )
Clojure ( Metaprogramming, DSLs, Protocols )

这是一篇Python对这些新技术、新库及模型响应的简短指南：

元编程

MacroPy 是一个元编程框架，它提供了多种语法结构，将现代语言元素编译成标准的Python代码，扩展了Python AST。举个例子，我们可以实现对代数数据类型的衡量:

from macropy.case_classes import case

@case
class Nil():
    pass

@case
class Cons(x, xs):
    pass

Cons(1, Cons(2, Cons(3, Nil())))

frommacropy.case_classesimportcase
 
@case
class Nil():
    pass
 
@case
class Cons(x, xs):
    pass
 
Cons(1, Cons(2, Cons(3, Nil())))

然后模式和声明的类型相匹配了:

def reduce(op, my_list):
    with switch(my_list):
        if Cons(x, Nil()):
            return x
        elif Cons(x, xs):
            return op(x, reduce(op, xs))

defreduce(op, my_list):
    withswitch(my_list):
        if Cons(x, Nil()):
            return x
        elifCons(x, xs):
            return op(x, reduce(op, xs))

消失的部分仍然是一个沿着 camlp4 路线，可扩展阶段的元编程系统。但是Mython 提供了一个 pgen2 解析框架，给引用块定义了新的语法，来解决这个问题。

my[namedtupledef] Point(x, y): pass

my[c]:
    int add (int x, int y) {
        return x + y;
    }

print "Regular Python"

my[namedtupledef] Point(x, y): pass
 
my[c]:
    int add (int x, int y) {
        return x + y;
    }
 
print "Regular Python"

类型

Python 是动态类型语言，并且引以为傲。我当然不希望对类型的“圣战”煽风点火，但同时肯定有大学派认为构建可靠的应用程序需要有比只使用单元测试更加有力的保障。Benjamin Pierce对类型系统的定义如下：

…一种易于处理的语法，通过根据计算值的类型对词组分类证明了缺少了特定的程序行为

重点是证明有关运行空间的属性，所有程序行为的运行空间替代了只是简单地罗列有限种情况的运行空间。全静态类型对于Python是否是正确的选择让人十分疑惑，但是在过度的动态类型和静态类型保证之间肯定有更加合适的方案。 MyPy project 找到了一个不错的平衡点，允许有类型的和没有类型的代码能够同时存于语言的超集中。例如：

def simple_typed(x : int, y : int) -> int:
    return x + y

simple_typed(1, 2)     # Type-checks succesfully

# Fails: Argument 2 to "simple_typed" has incompatible type # "float"
simple_typed(1, 2.0)

# Fails: Argument 2 to "simple_typed" has incompatible type "str"
simple_typed(1, "foo")

defsimple_typed(x : int, y : int) -> int:
    return x + y
 
simple_typed(1, 2)    # Type-checks succesfully
 
# Fails: Argument 2 to "simple_typed" has incompatible type # "float"
simple_typed(1, 2.0)
 
# Fails: Argument 2 to "simple_typed" has incompatible type "str"
simple_typed(1, "foo")

当然对C语言没有太多的用处。所以我们不只限于简单类型的函数，参数类型也有泛型，指针类型和各种各样内建的类型级的函数。

from typing import Iterator, typevar, Generic, Function, List

T = typevar('T')

def example_typed(x : Iterator[int]) -> Iterator[str]:
    for i in x:
        yield str(i)

def example_generic(x : Iterator[T]) -> Iterator[T]:
    for i in x:
        yield i

fromtypingimportIterator, typevar, Generic, Function, List
 
T = typevar('T')
 
defexample_typed(x : Iterator[int]) -> Iterator[str]:
    for i in x:
        yieldstr(i)
 
defexample_generic(x : Iterator[T]) -> Iterator[T]:
    for i in x:
        yield i

我们也能定义更加高级的泛型结构例如函子和单元

a = typevar('a')
b = typevar('b')

class Functor(Generic[a]):
    def __init__(self, xs : List[a]) -> None:
        self._storage = xs

    def iter(self) -> Iterator[a]:
        return iter(self._storage)

def fmap(f : Function[[a], b], xs : Functor[a]) -> Functor[b]:
    return Functor([f(x) for x in xs.iter()])

class Monad(Generic[a]):
    def __init__(self, val : a) -> None:
        self.val = val

class IdMonad(Monad):

    # Monad m => a -> m a
    def unit(self, x : a) -> Monad[b]:
        return IdMonad(x)

    # Monad m => m a -> (a -> m b) -> m b
    def bind(self, x : Monad[a], f : Function[[a], Monad[b]]) -> Monad[b]:
        return f(x.val)

    # Monad m => m (m a) -> m a
    def join(self, x : Monad[Monad[a]]) -> Monad[a]:
        return x.val

a = typevar('a')
b = typevar('b')
 
class Functor(Generic[a]):
    def__init__(self, xs : List[a]) -> None:
        self._storage = xs
 
    defiter(self) -> Iterator[a]:
        return iter(self._storage)
 
deffmap(f : Function[[a], b], xs : Functor[a]) -> Functor[b]:
    return Functor([f(x) for x in xs.iter()])
 
class Monad(Generic[a]):
    def__init__(self, val : a) -> None:
        self.val = val
 
class IdMonad(Monad):
 
    # Monad m => a -> m a
    defunit(self, x : a) -> Monad[b]:
        return IdMonad(x)
 
    # Monad m => m a -> (a -> m b) -> m b
    defbind(self, x : Monad[a], f : Function[[a], Monad[b]]) -> Monad[b]:
        return f(x.val)
 
    # Monad m => m (m a) -> m a
    defjoin(self, x : Monad[Monad[a]]) -> Monad[a]:
        return x.val

速度

“高性能”Python最近最重要的进展是 Pandas库提供的更高等级DataFrame容器的开发。Pandas混合各种Python进行操作，对于某些操作使用NumPy，其它的使用Cython，对于某些内部哈希表甚至使用C语言。Panda底层架构非教条式的方法已经让它成为数据分析领域的标准库。Pandas的开发体现了很多让数值Python生态系统成功的东西。

In [1]: from pandas import DataFrame

In [2]: titanic = DataFrame.from_csv('titanic.csv')

In [3]: titanic.groupby('pclass').survived.mean()
pclass
1st       0.619195
2nd       0.429603
3rd       0.255289
Name: survived

In [1]: frompandasimportDataFrame
 
In [2]: titanic = DataFrame.from_csv('titanic.csv')
 
In [3]: titanic.groupby('pclass').survived.mean()
pclass
1st      0.619195
2nd      0.429603
3rd      0.255289
Name: survived

然而改善Python性能最近的尝试是利用LLVM编译器有选择的编译某些Python代码段为本地代码。虽然不同的技术的实现方式不同，但是大部分与下述方式类似：

在函数上添加@jit或@compile这样的装饰器。
函数的AST或者bytecode被提取出来放入编译器流水线，在流水线中被映射到内部AST，给定特定的输入类型集合决定如何将给定的函数逻辑降低为机器代码。
编译过的函数与一组类型一起被调用，参数被检查过，代码在给定类型下生成。生成的代码连同参数被缓存使得接下来的调用直接分发到本地代码。

这些项目增加了大家对Python语言技术和 llvmpy 项目开发的兴趣，我猜测llvmpy在Python的历史上比特定的JIT编译器更重要。

最简单的例子（来自极好的 Kaleidescope 教程）是创建一个简单的本地乘加函数，然后通过解箱三个Python整数调用它：

import llvm.core as lc
import llvm.ee as le

mod = lc.Module.new('mymodule')

i32 = lc.Type.int(32)
funty = lc.Type.function(lc.Type.int(), [i32, i32, i32])

madd = lc.Function.new(mod, funty, "multiply")
x = madd.args[0]
y = madd.args[1]
z = madd.args[2]

block = madd.append_basic_block("L1")

builder = lc.Builder.new(block)
x0 = builder.mul(x, y)
x1 = builder.add(x0, z)

builder.ret(x1)

print mod

tm = le.TargetMachine.new(features='', cm=le.CM_JITDEFAULT)
eb = le.EngineBuilder.new(mod)
engine = eb.create(tm)

ax = le.GenericValue.int(i32, 1024)
ay = le.GenericValue.int(i32, 1024)
az = le.GenericValue.int(i32, 1024)

ret = engine.run_function(madd, [ax, ay, az])

print ret.as_int()
print mod.to_native_assembly()

importllvm.coreas lc
importllvm.eeas le
 
mod = lc.Module.new('mymodule')
 
i32 = lc.Type.int(32)
funty = lc.Type.function(lc.Type.int(), [i32, i32, i32])
 
madd = lc.Function.new(mod, funty, "multiply")
x = madd.args[0]
y = madd.args[1]
z = madd.args[2]
 
block = madd.append_basic_block("L1")
 
builder = lc.Builder.new(block)
x0 = builder.mul(x, y)
x1 = builder.add(x0, z)
 
builder.ret(x1)
 
printmod
 
tm = le.TargetMachine.new(features='', cm=le.CM_JITDEFAULT)
eb = le.EngineBuilder.new(mod)
engine = eb.create(tm)
 
ax = le.GenericValue.int(i32, 1024)
ay = le.GenericValue.int(i32, 1024)
az = le.GenericValue.int(i32, 1024)
 
ret = engine.run_function(madd, [ax, ay, az])
 
printret.as_int()
printmod.to_native_assembly()

上述代码编译生成下述LLVM IR。

define i32 @multiply(i32, i32, i32) {
L1:
  %3 = mul i32 %0, %1
  %4 = add i32 %3, %2
  ret i32 %4
}

definei32 @multiply(i32, i32, i32) {
L1:
  %3 = muli32 %0, %1
  %4 = addi32 %3, %2
  reti32 %4
}

虽然这个例子不太直观，但是可以生成很快的JIT’d函数，与NumPy这样的库集成的很好，把数据做为大块的解箱内存存储。

接口

分解行为到可组合的单元，而不是显式的继承层次结构是一个Python没有解决好的问题，经常导致噩梦般的复杂的使用mixin。然而通过使用ABC模组模仿静态定义的接口可以缓解这个问题。

import heapq
import collections

class Heap(collections.Sized):
   def __init__(self, initial=None, key=lambda x:x):
       self.key = key
       if initial:
           self._data = [(key(item), item) for item in initial]
           heapq.heapify(self._data)
       else:
           self._data = []

   def pop(self):
       return heapq.heappop(self._data)[1]

   def push(self, item):
       heapq.heappush(self._data, (self.key(item), item))

   def len(self):
       return len(self._data)

importheapq
importcollections
 
class Heap(collections.Sized):
  def__init__(self, initial=None, key=lambda x:x):
      self.key = key
      if initial:
          self._data = [(key(item), item) for itemin initial]
          heapq.heapify(self._data)
      else:
          self._data = []
 
  defpop(self):
      return heapq.heappop(self._data)[1]
 
  defpush(self, item):
      heapq.heappush(self._data, (self.key(item), item))
 
  deflen(self):
      return len(self._data)

例如建立一个等价类，让所有类的实例实现eq()方法。我们可以这样做：:

from abc import ABCMeta, abstractmethod

class Eq(object):

    __metaclass__ = ABCMeta

    @classmethod
    def __subclasshook__(cls, C):
        if cls is Eq:
            for B in C.__mro__:
                if "eq" in B.__dict__:
                    if B.__dict__["eq"]:
                        return True
                    break
        return NotImplemented

def eq(a, b):
    if isinstance(a, Eq) and isinstance(b, Eq) and type(a) == type(b):
        return a.eq(b)
    else:
        raise NotImplementedError

class Foo(object):
    def eq(self, other):
        return True

class Fizz(Foo):
    pass

class Bar(object):
    def __init__(self, val):
        self.val = val

    def eq(self, other):
        return self.val == other.val

print eq(Foo(), Foo())
print eq(Bar(1), Bar(1))
print eq(Foo(), Bar(1))
print eq(Foo(), Fizz())

fromabcimportABCMeta, abstractmethod
 
class Eq(object):
 
    __metaclass__ = ABCMeta
 
    @classmethod
    def__subclasshook__(cls, C):
        if clsis Eq:
            for B in C.__mro__:
                if "eq" in B.__dict__:
                    if B.__dict__["eq"]:
                        return True
                    break
        return NotImplemented
 
defeq(a, b):
    if isinstance(a, Eq) and isinstance(b, Eq) and type(a) == type(b):
        return a.eq(b)
    else:
        raiseNotImplementedError
 
class Foo(object):
    defeq(self, other):
        return True
 
class Fizz(Foo):
    pass
 
class Bar(object):
    def__init__(self, val):
        self.val = val
 
    defeq(self, other):
        return self.val == other.val
 
printeq(Foo(), Foo())
printeq(Bar(1), Bar(1))
printeq(Foo(), Bar(1))
printeq(Foo(), Fizz())

然后扩展这种类型的接口概念到多参数的函数，使得查询__dict__越来越可能发生，在组合的情况下很脆弱。问题的关键是分解所有的事情到单一类型不同的接口，当我们真正想要的是声明涵盖一组多类型的接口时。OOP中的这种缺点是

表达式问题的关键。

诸如Scala、Haskell和Rust这样的语言以trait和typeclass这样的形式提供该问题的解决方案。例如Haskell可以自动地为所有类型的交叉产品推导出微分方程。

instance (Floating a, Eq a) => Floating (Dif a) where pi = C pi

exp (C x)        = C (exp x)
exp (D x x')     = r where r = D (exp x) (x' * r)

log (C x)        = C (log x)
log p@(D x x')   = D (log x) (x' / p)

sqrt (C x)       = C (sqrt x)
sqrt (D x x')    = r where r = D (sqrt x) (x' / (2 * r))

instance (Floating a, Eq a) => Floating (Dif a) where
    pi              = C pi
 
    exp (C x)        = C (exp x)
    exp (D x x')     = r where r = D (exp x) (x' * r)
 
    log (C x)        = C (log x)
    log p@(D x x')   = D (log x) (x' / p)
 
    sqrt (C x)      = C (sqrt x)
    sqrt (D x x')    = r where r = D (sqrt x) (x' / (2 * r))

异步编程

在这个主题下，我们还是有很多缝缝补补的解决方案，解决了部分的问题，但是引入了一整与常规Python背道而驰的套限制和模式。Gevent通过剪接底层C堆栈保持了Python自己的一致性。生成的API非常优雅，但是使得推理控制流和异常非常复杂。

import gevent

def foo():
    print('Running in foo')
    gevent.sleep(0)
    print('Explicit context switch to foo again')

def bar():
    print('Explicit context to bar')
    gevent.sleep(0)
    print('Implicit context switch back to bar')

gevent.joinall([
    gevent.spawn(foo),
    gevent.spawn(bar),
])

importgevent
 
deffoo():
    print('Running in foo')
    gevent.sleep(0)
    print('Explicit context switch to foo again')
 
defbar():
    print('Explicit context to bar')
    gevent.sleep(0)
    print('Implicit context switch back to bar')
 
gevent.joinall([
    gevent.spawn(foo),
    gevent.spawn(bar),
])

控制流展示在下面：

通过对标准库相当不优美的缝缝补补（monkey-patching），我们可以模仿Erlang式带有异步进入点和内部状态的actor行为：

import gevent
from gevent.queue import Queue
from SimpleXMLRPCServer import SimpleXMLRPCServer

class Actor(object):
    _export = [
        'push',
    ]

    def __init__(self, address):
        self.queue = Queue()

        self._serv = SimpleXMLRPCServer(address, allow_none=True, logRequests=False)
        self.address = address

        for name in self._export:
            self._serv.register_function(getattr(self, name))

    def push(self, thing):
        self.queue.put(thing)

    def poll(self):
        while True:
            print(self.queue.get())

    def periodic(self):
        while True:
            print('PING')
            gevent.sleep(5)

    def serve_forever(self):
        gevent.spawn(self.periodic)
        gevent.spawn(self.poll)
        self._serv.serve_forever()

def main():
    from gevent.monkey import patch_all
    patch_all()

    serve = Actor(('', 8000))
    serve.serve_forever()

importgevent
fromgevent.queueimportQueue
fromSimpleXMLRPCServerimportSimpleXMLRPCServer
 
class Actor(object):
    _export = [
        'push',
    ]
 
    def__init__(self, address):
        self.queue = Queue()
 
        self._serv = SimpleXMLRPCServer(address, allow_none=True, logRequests=False)
        self.address = address
 
        for namein self._export:
            self._serv.register_function(getattr(self, name))
 
    defpush(self, thing):
        self.queue.put(thing)
 
    defpoll(self):
        while True:
            print(self.queue.get())
 
    defperiodic(self):
        while True:
            print('PING')
            gevent.sleep(5)
 
    defserve_forever(self):
        gevent.spawn(self.periodic)
        gevent.spawn(self.poll)
        self._serv.serve_forever()
 
defmain():
    fromgevent.monkeyimportpatch_all
    patch_all()
 
    serve = Actor(('', 8000))
    serve.serve_forever()

DSLs

Z3工程是嵌在Python对象层的扩展API。用Z3的实例来解决N皇后问题可以被描述为Python表达式和扩展SMT来解决问题:

from Z3 import *

Q = [ Int('Q_%i' % (i + 1)) for i in range(8) ]

# Each queen is in a column {1, ... 8 }
val_c = [ And(1 <= Q[i], Q[i] <= 8) for i in range(8) ]
# At most one queen per column
col_c = [ Distinct(Q) ]

# Diagonal constraint
diag_c = [ If(i == j,
              True,
              And(Q[i] - Q[j] != i - j, Q[i] - Q[j] != j - i))
           for i in range(8) for j in range(i) ]

solve(val_c + col_c + diag_c)

fromZ3import *
 
Q = [ Int('Q_%i' % (i + 1)) for i in range(8) ]
 
# Each queen is in a column {1, ... 8 }
val_c = [ And(1 <= Q[i], Q[i] <= 8) for i in range(8) ]
# At most one queen per column
col_c = [ Distinct(Q) ]
 
# Diagonal constraint
diag_c = [ If(i == j,
              True,
              And(Q[i] - Q[j] != i - j, Q[i] - Q[j] != j - i))
          for i in range(8) for j in range(i) ]
 
solve(val_c + col_c + diag_c)

在Theano,SymPy,PySpark中的其它工程大量使用基于Python表达式的重载操作符的方式。

Python

from sympy import Symbol
from sympy.logic.inference import satisfiable

x = Symbol('x')
y = Symbol('y')
satisfiable((x | y) & (x | ~y) & (~x | y))

from sympyimport Symbol
from sympy.logic.inferenceimport satisfiable
 
x = Symbol('x')
y = Symbol('y')
satisfiable((x | y) & (x | ~y) & (~x | y))

(责任编辑：好模板)