Add support for PYO3

[gbonnefille]

Currently, PDL can generate a fully native Python API.

Rust and Python can be wrapped together with POY3.

Does it make sense to decorate, via a "python" feature, the generated classes in order to have a code usable both by Rust and Python?

[gbonnefille]

Note: I did not see any hack to inject the expected PYO3 declaration without modifying the generated code.

[hchataing]

There are substantial differences in how the pure python code and the rust code are being generated. That comes from two aspects:

1. the two languages have different typing constraints, there are things that python will let us do (even with typing annotations, I am just referring to sub-typing here), that Rust will block us from doing.

2. the python output I am typically using more for test applications than actual hardened code. i.e. I care less about handling specific kind of parsing errors and the decoder takes a yolo approach to parsing :D

I am aware that having an external script do the generation for python is painful, and I have a backlog task to rewrite the generator in rust to be fully integrated in the pdlc binary. But lack of time...

Still I am curious how you would bind the generated rust to python, have done experiments to test how that would work in practice ?

[gbonnefille]

My big picture is to produce a codec for up to 10 CCSDS packets structures. The upper layer is expected in Python and it is the opportunity to challenge Rust+Python.

I have to admit that currently I'm still evaluating the state of the art.
Rust+Python is quite well tooled with PYO3.
PDL seems a good solution for pure Rust toolchain to generate codecs.
But after minor experiments, mixing PYO3 and a generated code is not obvious, while I tend to limit the manuals actions to adjust the code when structure evolves.

I will try to inject some PYO3 attributes before the end of the week, just to validate my idea and discover the next challenges.

[hchataing]

Ok ! Let me know how your experiment goes 👍

[gbonnefille]

After a really simple hack I obtain a PDL compiler generating struct with pyo3::pyclass attribute and pyo3::methods on the implementation.

But such construction failed on nested structures.

Initially I used

struct Header {
...
}
packet Packet {
header: Header
}

Then the compiler generated something like

    pub fn header(&self) -> &Header {
        &self.header
    }

But the pyo3 macros generated an impossible code as the borrowed reference cannot be mutate in Python as the two opposite memory logics conflict: the borrowing on one side and the reference counting on the other side.

As a solution, I'm trying to convert nested structures as group as the resulting structure for the packet is flat

group Header {
...
}
packet Packet {
 Header
}

I now have to ensure this is compatible with the real complexity of the packets I want to describe.

[hchataing]

That sounds nice !
Could you share some code snippets from the client python side to see you use the generated helpers ?

[gbonnefille]

Here is some preliminary results:

>>> help(my_module)
Help on package my_module:

NAME
    my_module

DESCRIPTION
    A Python module implemented in Rust. The name of this function must match
    the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
    import the module.

PACKAGE CONTENTS
    my_module

CLASSES
    builtins.object
        builtins.CCSDS_Space_Packet
    
    class CCSDS_Space_Packet(object)
     |  Methods defined here:
     |  
     |  APID(self, /)
     |  
     |  packet_data_length(self, /)
     |  
     |  packet_sequence_count(self, /)
     |  
     |  packet_type(self, /)
     |  
     |  packet_version_number(self, /)
     |  
     |  secondary_header_flag(self, /)
     |  
     |  sequence_flags(self, /)
     |  
     |  ----------------------------------------------------------------------
     |  Static methods defined here:
     |  
     |  __new__(*args, **kwargs)
     |      Create and return a new object.  See help(type) for accurate signature.

DATA
    __all__ = ['CCSDS_Space_Packet']

FILE
    /workspaces/my_module/.venv/lib/python3.11/site-packages/my_module/__init__.py

[hchataing]

Looks interesting 👍
I am not opposed to adding the pyo3 derivations behind a flag, similarly to the serde serialization.
Well that's assuming that it produces reasonable bindings for a reasonably large share of the language constructs :)

[gbonnefille]

Initially, I tagged the methods:

#[cfg_attr(feature = "python", pyo3::pymethods)]
impl CCSDS_Space_Packet {
    pub fn primary_header(&self) -> &PrimaryHeader {
        &self.primary_header
    }
}

But this solution does not support nested structures as the generated methods returns a reference to the embedded structure.

So I changed my mind for another approach: generating getter and setter from PYO3:

#[cfg_attr(feature = "python", pyo3::pyclass)]
pub struct CCSDS_Space_Packet {
    #[cfg_attr(feature = "python", pyo3(get,set))]
    pub primary_header: PrimaryHeader,
}

And it's working.

I will continue my experiments before proposing any pull-request.

[hchataing]

Sounds good, thank you for the help 👍

[gbonnefille]

I published my change as a PR. But I will not be able to sign the CLA (at least in a reasonable time).
Feel free to recode this change in your side if you think it is valuable enough.

[hchataing]

Thanks, I will check it out given some time 👍

[gbonnefille]

Doing more tests, I realized that my minor change is far from enough. I've just commited the activation of the python feature in pdl-tests as it reveals the problem.

The issue is related to other generated elements like enums. As they are not declared to pyo3, they cannot be used in structs related to packet.