Skip to content

IronPython

Jump to bottom Edit New page
Aaron Todd edited this page Sep 17, 2016 · 1 revision

I've looked into this and unfortunately the numpy/scipy/pandas story on IronPython is a sad state of affairs at the moment.

From what I gather there were essentially 2 major attempts that were taken in the past to getting these modules, which are heavy C/Cython extensions, to work on IronPython. The approaches are detailed below.

Approach 1: Port Numpy/Scipy/Pandas

This is essentially the approach taken by Enthrought from the tutorial Jared referenced.

I have followed that tutorial and was able to successfully install the packages referenced. I am including the steps for completeness but see my notes below.

  1. Install IronPython 2.7.5

  2. Add IronPython to your PATH

  3. Open Command prompt and verify it works ipy -c "print('hello, world!')"

  4. Download ironpkg

    1. Open https://store.enthought.com/repo/.iron/
    2. Should see a page not found displayed
    3. Click the link in the top right corner to create an account. Follow the steps and register
    4. Once you are registered you should be able to download ironpkg-1.0.0.py from above link

  5. Run ipy ironpkg-1.0.0.py --install

  6. Download all of the eggs from the repo link above

  7. Edit the .ironpkg file. Mine was at C:\Users\Aaron\.ironpkg. Change the location to wherever you placed the downloaded eggs. e.g. IndexedRepos = ['file://C:\Users\Aaron\Documents\eggs']

  8. Run the following to install numpy and scipy

    ironpkg scipy
ironpkg numpy

  9. Check whether the install worked

ipy -X:Frames
IronPython 2.7.5 (2.7.5.0) on .NET 4.0.30319.42000 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__version__
'2.0.0'
>>> import scipy
>>> scipy.__version__
'1.0.0.dev'

Notes

This first approach was based off of now abandoned C# ports/refactors of numpy and scipy to work on .NET. This is fairly obvious just from the versions (latest numpy and scipy versions for CPYthon are 1.11.1 and 0.18.0 respectively). The following articles indicate this as well though.

http://blog.enthought.com/python/scipy-for-net/#.V8rZPvkrJD8 http://pytools.codeplex.com/wikipage?title=NumPy%20and%20SciPy%20for%20.Net

Likely sources for some of these tools/modules: https://github.com/numpy/numpy-refactor https://github.com/jasonmccampbell/scipy-refactor https://github.com/ilanschnell/ironpkg

These haven't been touched since 2011 and as far as I can see don't include Pandas. This is not a viable approach in my mind and not likely to gain any support from the community at this point.

Approach 2: Stub CPython API

This second approach was taken by William Reade and Resolver Systems for a spreadsheet product that integrated tightly with Python. Essentially they stub out the CPython API and when the DLL is loaded they substitute IronPython C# objects and manage the lifetimes back and forth. This approach is interesting because theoretically you can talk to any C python extension. Example: You install numpy (or whatever C extension) into a normal CPython installation, play around with import paths, and ironclad takes care of the rest and detects when an attempt is made to a C extension. I'm sure I've glossed over the finer details a bit but that's the gist of the approach I believe.

I followed along with the build steps here. I ran into a few issues and targeted newer versions in some cases. The steps below reflect my changes.

  1. Download and install IronPython 2.7.6.3

    1. Add ironpython to your PATH. NOTE: If you followed along with approach 1 you'll want to remove IronPython 2.7.5 from your PATH.
    2. Install pip/setuptools ipy -X:Frames -m ensurepip

  2. Download and Install CPython 2.7.12

  3. Download and Install SCons 2.5.0

  4. Install mingw

    1. install and start mingw-get
    2. Under Basic setup select mingw32-base, mingw32-gcc-g++, msys-base
    3. Under All Packages select mingw32-pexports
      1. NOTE: I also had to select mingw32-pthreads-w32
    4. Apply Changes (upper left "Installation" dropdown)
    5. Add C:\MinGW\bin to PATH
    6. If you have cygwin ensure PATH has no references to it or at least make sure it is after MinGW

  5. Download and install CMake 3.0.2

  6. Download and install GCCXML

    1. Get the source git clone https://github.com/gccxml/gccxml

    2. Create a new directory for building.

      mkdir gccxml-build
cd gccxml-build

    3. Generate makefiles with cmake

      cmake -G "MinGW Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-fgnu89-inline -std=gnu++0x" ..\gccxml

    4. Compile mingw32-make.exe

      1. Note: On my system I ended up having to patch gccxml to get around strcasecmp undeclared in this scope:

        diff --git a/GCC_XML/KWSys/SystemTools.cxx b/GCC_XML/KWSys/SystemTools.cxx
index 4d83293..1137539 100644
--- a/GCC_XML/KWSys/SystemTools.cxx
+++ b/GCC_XML/KWSys/SystemTools.cxx
@@ -51,6 +51,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#include <strings.h>
#include <sys/stat.h>
#include <time.h>

    5. Install mingw32-make.exe install. Defaults to C:\Program Files (x86)\gccxml Note: might need administrative privilege

  7. Install pyexpat

    1. Download pyexpat.py

    2. Rename it to expat.py and save it to your iron python installation C:\Program Files (x86)\IronPython-2.7.6.3\Lib\xml\parsers

    3. Copy the following expat files from CPython installation to IronPython installation

      copy "C:/Python27/lib/xml/dom/expatbuilder.py" "C:/Program Files (x86)/IronPython-2.7.6.3/Lib/xml/dom"
copy "C:/Python27/lib/xml/sax/expatreader.py" "C:/Program Files (x86)/IronPython-2.7.6.3/Lib/xml/sax"

  8. pygccxml 1.6.2

    1. Download from https://pypi.python.org/packages/source/p/pygccxml/pygccxml-v1.6.2.tar.gz
    2. Extract with something like 7zip
    3. After unpacking install into ironpython site-packages 
      cd C:\Users\Aaron\Downloads\pygccxml-v1.6.2
ipy setup.py install --user

  9. Download and install nasm 2.11

    1. Once setup is complete, copy nasm.exe to MinGW/bin

      copy C:\Users\Aaron\AppData\Local\nasm\nasm.exe C:\MinWG\bin

  10. Build Ironcload

    1. Download IronClad

      git clone https://github.com/IronLanguages/ironclad.git

    2. Edit SConstruct

      1. Update paths that may differ on your system. Of particular note is MSVCR90_DLL Here is a diff of the changes I made
      diff --git a/SConstruct b/SConstruct
index eb68ff5..4ea6275 100644
--- a/SConstruct
+++ b/SConstruct
@@ -49,8 +49,8 @@ if WIN32:
GCCXML_CC1PLUS = r'"C:\Program Files (x86)\gccxml\bin\gccxml_cc1plus.exe"'

    # standard location
-    IPY = r'"C:\Program Files (x86)\IronPython 2.7\ipy.exe"'
-    IPY_DIR = r'"C:\Program Files (x86)\IronPython 2.7"'
+    IPY = r'"C:\Program Files (x86)\IronPython-2.7.6.3\ipy.exe"'
+    IPY_DIR = r'"C:\Program Files (x86)\IronPython-2.7.6.3"'
# private build
# IPY = r'"C:\github\IronLanguages\bin\Debug\ipy.exe"'
# IPY_DIR = r'"C:\github\IronLanguages\bin\Debug"'
@@ -69,7 +69,7 @@ if WIN32:
COPY_CMD = 'copy $SOURCE $TARGET'
DLLTOOL_CMD = 'dlltool -D $NAME -d $SOURCE -l $TARGET'
LINK_MSVCR90_FLAGS = '-specs=stub/use-msvcr90.spec'
-    MSVCR90_DLL = r'C:\Windows\winsxs\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.6161_none_50934f2ebcb7eb57\msvcr90.dll'
+    MSVCR90_DLL = r'C:\Windows\winsxs\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_5094ca96bcb6b2bb\msvcr90.dll'
PEXPORTS_CMD = 'pexports $SOURCE > $TARGET'
RES_CMD = 'windres --input $SOURCE --output $TARGET --output-format=coff'

@@ -77,7 +77,7 @@ if WIN32:
MINGW_DIR = r'C:\MinGW'
MINGW_LIB = os.path.join(MINGW_DIR, 'lib')
MINGW_INCLUDE = os.path.join(MINGW_DIR, 'include')
-    GCCXML_INSERT = '-isystem "%s" -isystem "%s"' % (MINGW_INCLUDE,          os.path.join(MINGW_LIB, 'gcc', 'mingw32', '4.8.1', 'include'))
+    GCCXML_INSERT = '-isystem "%s" -isystem "%s"' % (MINGW_INCLUDE,         os.path.join(MINGW_LIB, 'gcc', 'mingw32', '5.3.0', 'include'))

     # Calculate DLLs dir of cpython - assume this is run from the cpython
     # If not, change to match your instalation, defaults to C:\Python27\DLLs

    3. Build using CPython C:\Python27\Scripts\scons.bat

      1. Clean with C:\Python27\Scripts\scons.bat -c

    4. Run the test suite Note: there are dependencies on pysvn and numpy for the test suite, you would install these into the CPYthon installation e.g. C:\Python27\python.exe -m pip install numpy==1.11.1

      1. To run the full test suite: C:\Python27\Scripts\scons.bat test

      2. To run a subset (I haven't tried this)

      set IRONPYTHONPATH=.;C:\Python27\DLLs;C:\Python27\Lib\site-packages
ipy runtest.py tests.functionalitytest.BZ2Test.testFunctionsWork

      1. Try to import a C extension

        set IRONPYTHONPATH=C:\Users\Aaron\Documents\ironclad\build;C:\Python27\DLLs;C:\Python27\Lib\site-packages
ipy
IronPython 2.7.6.3 (2.7.6.3) on .NET 4.0.30319.42000 (32-bit)
Type "help", "copyright", "credits" or "license" for more information.
>>> import ironclad
>>> import numpy
detected unsupported member type HAVE_INPLACEOPS; ignoring
Error: PyFrozenSet_New is not yet implemented
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\Lib\site-packages\numpy\__init__.py", line 180, in <module>
  File "C:\Python27\Lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
  File "C:\Python27\Lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
  File "C:\Python27\Lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
  File "C:\Python27\Lib\site-packages\numpy\core\__init__.py", line 58, in <module>
  File "C:\Python27\Lib\site-packages\numpy\testing\__init__.py", line 12, in <module>
  File "C:\Python27\Lib\site-packages\numpy\testing\decorators.py", line 21, in <module>
  File "C:\Python27\Lib\site-packages\numpy\testing\utils.py", line 15, in <module>
  File "C:\Program Files (x86)\IronPython-2.7.6.3\Lib\tempfile.py", line 35, in <module>
  File "C:\Program Files (x86)\IronPython-2.7.6.3\Lib\random.py", line 49, in <module>
  File "C:\Program Files (x86)\IronPython-2.7.6.3\Lib\hashlib.py", line 134, in <module>
  File "<string>", line 21, in load_module
  NotImplementedError: PyFrozenSet_New

This is about as far as I expected to make it considering the same build wiki indicates numpy fails to import (likely a different error as the versions are different). Likely would need to install Python 2.7.8 as that looks like the last version of the C API that was updated or try to update the source to 2.7.12.

Overall I think this approach has the most merit but it would be good to reach out to the IronPython maintainers to see how/if they plan to address C extensions going forward. You'll want to align with that approach.

http://www.voidspace.org.uk/python/weblog/arch_d7_2009_01_24.shtml#e1055 (Interesting read if you want to get an idea of the black magic they are pulling around lifetimes and crossing boundaries) http://www.johndcook.com/blog/2009/03/19/ironclad-ironpytho/

Add a custom footer
Add a custom sidebar

Clone this wiki locally