Designing a JIT API for CPython
- For CoreCLR
- For CPython
- Git
- TortoiseSVN (required to get external dependencies)
- Visual Studio
This repository uses Git submodules, which means the best way to clone this repository is with the --recursive flag:
git clone --recursive https://github.com/Microsoft/Pyjion.gitRun PatchDeps.bat to patch Python to have JIT support and CoreCLR to disable COM support.
Run BuildDeps.cmd to build CoreCLR and Python (which includes downloading Python's dependencies).
- From Visual Studio
- Open the
pyjion.slnfile - Build the solution
- Open the
- Run
CopyFiles.batto copy files to key locations
- Run
x64\Debug\Test.exe - Run
x64\Debug\Tests.exe
- Copy
x64\Debug\pyjit.dlltoPython\PCbuild\amd64\(initially done byCopyFiles.bat, so only do as necessary after rebuilding Pyjion) - Go into the
Pythondirectory and launchpython.bat
You'll need to run git clean -d -f -x in CoreCLR when switching between release and debug builds.
There are three goals for this project.
- Add a C API to CPython for plugging in a JIT
- Develop a JIT module using CoreCLR utilizing the C API mentioned in goal #1
- Develop a C++ framework
Goal #1 is to make it so that CPython have a JIT plugged in as desired (CPython is the Python implementation you download from https://www.python.org/). That would allow for an ecosystem of JIT implementations for Python where users can choose the JIT that works best for their use-case. And by using CPython we hope to have compatibility with all code that it can run (both Python code as well as C extension modules).
Goal #2 is to develop a JIT for CPython using the JIT provided by the CoreCLR. It's cross-platform, liberally licensed, and the original creator of Pyjion has a lot of experience with it.
Goal #3 is to abstract out all of the common bits required to write a JIT implementation for CPython. The idea is to create a framework where JIT implementations only have to worry about JIT-specific stuff like how to do addition and not when to do addition.
Like the word "pigeon". @DinoV wanted a name that had something with "Python" -- the "Py" part -- and something with "JIT" -- the "JI" part -- and have it be pronounceable.
PyPy?
PyPy is an implementation of Python with its own JIT. The biggest difference compared to Pyjion is that PyPy doesn't support C extension modules without modification unless they use CFFI. Pyjion also aims to support many JIT compilers while PyPy only supports their own custom JIT compiler.
Pyston is an implementation of Python using LLVM as a JIT compiler. Compared to Pyjion, Pyston has partial CPython C API support but not complete support. Pyston also only supports LLVM as a JIT compiler.
Numba is a JIT compiler for "array-oriented and math-heavy Python code". This means that Numba is focused on scientific computing while Pyjion tries to optimize all Python code. Numba also only supports LLVM.
IronPython is an implementation of Python that is implemented using .NET. While IronPython tries to be usable from within .NET, Pyjion does not have an compatibility with .NET. This also means IronPython cannot use C extension modules while Pyjion can.
Psyco was a module that monkeypatched CPython to add a custom JIT compiler. Pyjion wants to introduce a proper C API for adding a JIT compiler to CPython instead of monkeypatching it. It should be noted the creator of Psyco went on to be one of the co-founders of PyPy.
Unladen Swallow was an attempt to make LLVM be a JIT compiler for CPython. Unfortunately the project lost funding before finishing their work after having to spend a large amount of time fixing issues in LLVM's JIT compiler (which has greatly improved over the subsequent years).
Yes! Goals #1 and #3 are entirely platform-agnostic while goal #2 of using CoreCLR as a JIT compiler is not an impedence to supporting OS X or Linux as it already supports the major OSs. The only reason Pyjion doesn't directly support Linux or OS X is entirely momentum/laziness: since the work is being driven by Microsoft employees, it simply meant it was easier to get going on Windows.
Goal #1 is explicitly to add a C API to CPython to support JIT compilers. There is no expectation, though, to ship a JIT compiler with CPython. This is because CPython compiles with nothing more than a C89 compiler, which allows it to run on many platforms. But adding a JIT compiler to CPython would immediately limit it to only the platforms that the JIT supports.
No.