[Pdl-porters] PDL OO thoughts and Prima OO

Hey everyone,

I know I originally started rumblings along these lines (at least on the
PDL end) a couple of years back, so obviously I've thought about it. In an
effort to not spam the list with a lengthy, disorganized response, I'm
going to try to collect my thoughts and be a bit more systematic about it.

Didn't want my silence to indicate indifference. :-)

David

Post by Chris Marshall
Hi Dmitry-
I wanted to bring your attention to a number of items tied
into PDL3 development/plans (at least mine) that could be
of interest to you and might be an opportunity for some
synergy with Prima.
* David has been working on TinyCC based JIT compiling
with his C::Blocks development. This would allow for
runtime compilation of PDL+C code.
* My current thoughts for the PDL3 kernel is that it would
be lightweight, support modern perl OO similar to Moose
and perl5-mop ideas, *and* be symmetrically usable from
C and from Perl (this is where the JIT really helps!).
* I think it is important that the PDL3 kernel be C-based
for portability and compatibility across platforms, libraries,
compilers... With C++ you lose that.
* Going with C I've come across a number of techniques
http://www.cs.rit.edu/~ats/books/ooc.pdf (uses a custom
code generator from a specification language)
https://wiki.gnome.org/Projects/Vala (the Vala OOC-to-C
compiler based on the Gnome Object system)
Prima (I finally was able to work through the internals
API and saw that it is very similar to the above and it
support symmetrically access to methods from either
C or perl layers)

David Mertens

2014-06-10 23:54:52 UTC

Hey everyone,

If we just wanted a C object system that supported inheritance, I would say
we should take Prima's object system as a starting point and build new
classes based on Prima's Object class. I spent a weekend on this a while
ago, but it'll take quite a bit more work: Prima's purpose as a GUI toolkit
is pretty deeply embedded.

However, if you want roles as well, that's a different beast, and the bulk
of my thoughts lie there. I think we can do it, but it'll be tricky. My
concerns and ideas, in brief:

In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. As a result, in C an object
can
be cast into another class if it has the same binary layout. The problem
with
roles in C is that there is no promise of binary layout. If two different
classes implement the same role, you don't know where in the vtable to find
the same method. This can be solved in a handful of ways. The most efficient
way I can think of is to make role methods a different kind of method than a
class method.

Thoughts?
David

---

In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. Typically, the first element
of a struct definin a class is a vtable pointer, and the rest of the items
are the data elements.

struct my_string_vtable;

struct my_string {
struct my_string_vtable * methods;
char * string;
STRLEN allocated_length;
}

struct my_string_vtable {
struct my_string * copy(struct my_string *);
struct my_string * delete(sruct my_string *);
STRLEN get_length (struct my_string *);
char * get_string(struct my_string*);
}

You then invoke the methods by calling the function pointers directly from
the vtable. To get this string's length, I would say:

STRLEN the_len = the_string->methods->get_length(the_string);

(Presumably it might iterate through the characters byte-by-byte until it
found the null byte. We might consider storing this offset in the object
itself, but I use it this way for illustrative purposes.) The C compiler
knows that the first element of the_string is a pointer to the vtable. The C
compiler also knows where the get_length method resides in this vtable, so
it
can locate the function to call with two pointer dereferences.

As a result, in C an object can be cast into another class if it has the
same
binary layout. Suppose, for example, that the length method is supposed to
return the number of printable characters. If I had a different (i.e.
Unicode)
implementation of a string, then it would need its own length method. We
might want to add a few additional methods, but as long as the first four
elements in our my_unicode_string_vtable are a copy, delete, get_length, and
get_string function, we can cast our unicode object pointer to be a string
object pointer. Any functions that operate on the string through provided
methods will Just Work. This is how C inheritance works. (Introspection
requires an additonal wrinkle, but it's not important here.)

The problem with roles in C is that there is no promise of binary layout. If
two different classes implement the same role, you don't know where in the
vtable to find the same method. Suppose I have a role, does_logger.
Obviously,
if I compose the role into the class, then the class knows where those
methods live and can call them out of the vtable without trouble. But what
if I write a function that can take *anything* that does_logger? (So long as
it sticks just to the logger role's functions, it should just work.) If I
have a widget class and I create a derived widget class that does_logger,
where in the vtable are the logger methods? The derived widget must be
binary
compatible with the base widget, so the logger methods must go at the end of
the vtable. The binary location of these methods will certainly be different
from the binary location of the logger methods composed into a derivative of
my_string because that class only has four methods. We are left with a
conundrum: the function cannot invoke logger commands as if they were normal
object methods because the role does_logger makes no promises about binary
layout. The compiler does not know where to look for them.

This can be solved in a handful of ways. One way is to fetch role methods
via
a string hash or other index lookup. This method lookup could return a void
pointer that is the function pointer, and it would have to be cast to the
appropriate function type. Another mechanism which I think would be more
efficient at the cost of an extra argument, would involve looking up a role
vtable (once) and including that as a separate argument to role methods.

For comparison, here is a normal object method invocation:

object->vtable->method(object, arg1, arg2, ...)

Here is an index lookup, which relies upon a previously defined typedef and
global index "does_logger_log_something_idx":

typedef does_logger_log_something(void * objec);
does_logger_log_something log_func
= object->vtable->lookup_role_method(does_logger_log_something_idx);
log_func(object, arg1, arg2, ...);

And here is the role vtable lookup, which relies upon a global index for the
role, not the methods:

struct * does_logger_vtable logger_vtable;
logger_vtable = object->vtable->get_role_vtable(object,
does_logger_role_idx)
logger_vtable->log_something(logger_vtable, object, arg1, arg2, ...);

The advantage of this third approach is that you perform the vtable lookup
once. Any role-based function calls within log_something have the vtable on
hand and can perform role method lookups with a simple struct member lookup.

One more thing about role vtables: they don't need to be separate vtables.
They can simply be offsets into the class's vtable. You could implement
roles so that if the passed role vtable is null, the first thing the role
method does is lookup the role vtable. In this way, if you know the class of
your object, you could simply invoke the logger method as

object->vtable->log_something(0, object, arg1, arg2, ...)

Post by David Mertens
Hey everyone,
I know I originally started rumblings along these lines (at least on the
PDL end) a couple of years back, so obviously I've thought about it. In an
effort to not spam the list with a lengthy, disorganized response, I'm
going to try to collect my thoughts and be a bit more systematic about it.
Didn't want my silence to indicate indifference. :-)
David

Ed .

2014-06-11 03:25:04 UTC

David is quite right that roles are problematic in OO C.

I think Glib / Vala is a working system that is quite good. I also believe that if you can accept âinterfacesâ rather than full, formal ârolesâ, that Glib can do that. How important is it to have âproperâ roles?

From: David Mertens
Sent: Wednesday, June 11, 2014 12:54 AM
To: Chris Marshall
Cc: Dmitry Karasik ; pdl-porters
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO

Hey everyone,

If we just wanted a C object system that supported inheritance, I would say we should take Prima's object system as a starting point and build new classes based on Prima's Object class. I spent a weekend on this a while ago, but it'll take quite a bit more work: Prima's purpose as a GUI toolkit is pretty deeply embedded.

However, if you want roles as well, that's a different beast, and the bulk of my thoughts lie there. I think we can do it, but it'll be tricky. My concerns and ideas, in brief:

In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. As a result, in C an object can
be cast into another class if it has the same binary layout. The problem with
roles in C is that there is no promise of binary layout. If two different
classes implement the same role, you don't know where in the vtable to find
the same method. This can be solved in a handful of ways. The most efficient
way I can think of is to make role methods a different kind of method than a
class method.

Thoughts?
David

---

In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. Typically, the first element
of a struct definin a class is a vtable pointer, and the rest of the items
are the data elements.

struct my_string_vtable;

struct my_string {
struct my_string_vtable * methods;
char * string;
STRLEN allocated_length;
}

struct my_string_vtable {
struct my_string * copy(struct my_string *);
struct my_string * delete(sruct my_string *);
STRLEN get_length (struct my_string *);
char * get_string(struct my_string*);
}

You then invoke the methods by calling the function pointers directly from
the vtable. To get this string's length, I would say:

STRLEN the_len = the_string->methods->get_length(the_string);

(Presumably it might iterate through the characters byte-by-byte until it
found the null byte. We might consider storing this offset in the object
itself, but I use it this way for illustrative purposes.) The C compiler
knows that the first element of the_string is a pointer to the vtable. The C
compiler also knows where the get_length method resides in this vtable, so it
can locate the function to call with two pointer dereferences.

As a result, in C an object can be cast into another class if it has the same
binary layout. Suppose, for example, that the length method is supposed to
return the number of printable characters. If I had a different (i.e. Unicode)
implementation of a string, then it would need its own length method. We
might want to add a few additional methods, but as long as the first four
elements in our my_unicode_string_vtable are a copy, delete, get_length, and
get_string function, we can cast our unicode object pointer to be a string
object pointer. Any functions that operate on the string through provided
methods will Just Work. This is how C inheritance works. (Introspection
requires an additonal wrinkle, but it's not important here.)

The problem with roles in C is that there is no promise of binary layout. If
two different classes implement the same role, you don't know where in the
vtable to find the same method. Suppose I have a role, does_logger. Obviously,
if I compose the role into the class, then the class knows where those
methods live and can call them out of the vtable without trouble. But what
if I write a function that can take *anything* that does_logger? (So long as
it sticks just to the logger role's functions, it should just work.) If I
have a widget class and I create a derived widget class that does_logger,
where in the vtable are the logger methods? The derived widget must be binary
compatible with the base widget, so the logger methods must go at the end of
the vtable. The binary location of these methods will certainly be different
from the binary location of the logger methods composed into a derivative of
my_string because that class only has four methods. We are left with a
conundrum: the function cannot invoke logger commands as if they were normal
object methods because the role does_logger makes no promises about binary
layout. The compiler does not know where to look for them.

This can be solved in a handful of ways. One way is to fetch role methods via
a string hash or other index lookup. This method lookup could return a void
pointer that is the function pointer, and it would have to be cast to the
appropriate function type. Another mechanism which I think would be more
efficient at the cost of an extra argument, would involve looking up a role
vtable (once) and including that as a separate argument to role methods.

For comparison, here is a normal object method invocation:

object->vtable->method(object, arg1, arg2, ...)

Here is an index lookup, which relies upon a previously defined typedef and
global index "does_logger_log_something_idx":

typedef does_logger_log_something(void * objec);
does_logger_log_something log_func
= object->vtable->lookup_role_method(does_logger_log_something_idx);
log_func(object, arg1, arg2, ...);

And here is the role vtable lookup, which relies upon a global index for the
role, not the methods:

struct * does_logger_vtable logger_vtable;
logger_vtable = object->vtable->get_role_vtable(object, does_logger_role_idx)
logger_vtable->log_something(logger_vtable, object, arg1, arg2, ...);

The advantage of this third approach is that you perform the vtable lookup
once. Any role-based function calls within log_something have the vtable on
hand and can perform role method lookups with a simple struct member lookup.

One more thing about role vtables: they don't need to be separate vtables.
They can simply be offsets into the class's vtable. You could implement
roles so that if the passed role vtable is null, the first thing the role
method does is lookup the role vtable. In this way, if you know the class of
your object, you could simply invoke the logger method as

object->vtable->log_something(0, object, arg1, arg2, ...)

On Tue, Jun 10, 2014 at 2:52 PM, David Mertens <***@gmail.com> wrote:

Hey everyone,

I know I originally started rumblings along these lines (at least on the PDL end) a couple of years back, so obviously I've thought about it. In an effort to not spam the list with a lengthy, disorganized response, I'm going to try to collect my thoughts and be a bit more systematic about it.

Didn't want my silence to indicate indifference. :-)

David

On Wed, Jun 4, 2014 at 10:59 AM, Chris Marshall <***@gmail.com> wrote:

Hi Dmitry-

I wanted to bring your attention to a number of items tied
into PDL3 development/plans (at least mine) that could be
of interest to you and might be an opportunity for some
synergy with Prima.

* David has been working on TinyCC based JIT compiling
with his C::Blocks development. This would allow for
runtime compilation of PDL+C code.

* My current thoughts for the PDL3 kernel is that it would
be lightweight, support modern perl OO similar to Moose
and perl5-mop ideas, *and* be symmetrically usable from
C and from Perl (this is where the JIT really helps!).

* I think it is important that the PDL3 kernel be C-based
for portability and compatibility across platforms, libraries,
compilers... With C++ you lose that.

* Going with C I've come across a number of techniques
for implementation:

http://www.cs.rit.edu/~ats/books/ooc.pdf (uses a custom
code generator from a specification language)

https://wiki.gnome.org/Projects/Vala (the Vala OOC-to-C
compiler based on the Gnome Object system)

Prima (I finally was able to work through the internals
API and saw that it is very similar to the above and it
support symmetrically access to methods from either
C or perl layers)

Dmitry Karasik

2014-06-11 06:48:15 UTC

Post by David Mertens
Hey everyone,
If we just wanted a C object system that supported inheritance, I would say
we should take Prima's object system as a starting point and build new
classes based on Prima's Object class. I spent a weekend on this a while
ago, but it'll take quite a bit more work: Prima's purpose as a GUI toolkit
is pretty deeply embedded.

Hi all,

I think the biggest question I cannot find answer to, is why do we need a
separate object system (with or without roles) for PDL3. In my rather limited
use of PDL standard perl OO was more than enough, but I cannot see the big
picture, so would like to ask for some clarifications, what is the reason
behind it.

I think that if we want to modernize the OO then the best approach would be to
select Moose or Mouse (the latter is a lightweight Moose written in XS), and
that's basically it. The syntax is well-known, it's modern, and we get roles
etc for free.

Prima's object system does not support roles, but it's rather easy to hack
support for it on C level. Prima's OO indeed is built over a vtable pointer
scheme, and has already build_dynamic_vmt() function that hacks into class'
vmt, which is enough to support roles. However on Perl level one must reinvent
the Moose semantics (with roles/after/before etc keywords) and that's lot of
work.

Another reason we might want Prima-based OO is because of the way it allows to
declare C/Perl wrappers for methods and properties, if that is needed (I don't
know if it is).

Finally, it needs to be decoupled from Prima because it seems wrong to me to build
PDL with help of a GUI toolkit .. a separate OO system is ok though.

Post by David Mertens
In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. Typically, the first element
of a struct definin a class is a vtable pointer, and the rest of the items
are the data elements.

--
Sincerely,
Dmitry Karasik

Christian Walde

2014-06-11 08:33:46 UTC

Post by Dmitry Karasik
I think that if we want to modernize the OO then the best approach would
be to select Moose or Mouse (the latter is a lightweight Moose written in
XS), and that's basically it. The syntax is well-known, it's modern, and
we get roles etc for free.

Strongly objecting to Mouse, on the basis of it not playing nice with any
of the other OO systems and having in the past caused a lot of trouble,
and also because the original author never intended it to get into any
production, but only to have it be an experiment so he could learn how
Moose style OO works.

A counter recommendation is Moo, which is pure perl, plays nice with all
the things (as far as possible), and supports almost everything Moose
provides, excepting meta, without the startup cost.

--
With regards,
Christian Walde

David Mertens

2014-06-11 11:24:43 UTC

@Dmitry, Christian,

The issue is not about Perl-level subclasses of PDL. Rather, PDL's own
implementation could be helped a great, great deal by building on a well
designed object system. PDL's guts include its own quasi-object system
which is a essentially a huge pile of spaghetti. Second, PDL currently only
supports basic C types (byte, int, float, double, etc), but it would be
nice to extend PDL so that it could hold arrays of objects, not just
numbers. Primary among those would be support for complex numbers. So
really, we need a C-level object system. Moose and Moo will not work. In
principle, p5-mop-xs might work, once it arrives. This discussion is in
many ways parallel to that one.

@Ed,

Glib is good. Do you know enough to compare it to Prima's object system?
Prima's object system supports single inheritance. It also has support for
runtime declaration of derived classes at the Perl level that know how to
build the correct vtable on the fly. This level of C-to-Perl coupling the
Does What You Mean at the C level is *amazing*.

David

Post by Dmitry Karasik
I think that if we want to modernize the OO then the best approach would

Post by Dmitry Karasik
be to select Moose or Mouse (the latter is a lightweight Moose written in
XS), and that's basically it. The syntax is well-known, it's modern, and
we get roles etc for free.

Strongly objecting to Mouse, on the basis of it not playing nice with any
of the other OO systems and having in the past caused a lot of trouble, and
also because the original author never intended it to get into any
production, but only to have it be an experiment so he could learn how
Moose style OO works.
A counter recommendation is Moo, which is pure perl, plays nice with all
the things (as far as possible), and supports almost everything Moose
provides, excepting meta, without the startup cost.
--
With regards,
Christian Walde

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan

Christian Walde

2014-06-11 11:26:32 UTC

On Wed, 11 Jun 2014 13:24:43 +0200, David Mertens

Post by David Mertens
The issue is not about Perl-level subclasses of PDL. Rather, PDL's own
implementation could be helped a great, great >deal by building on a
well designed object system. PDL's guts include its own quasi-object
system which is a >essentially a huge pile of spaghetti.

Noted. Sadly i can't help with that outside of yelling from the sidelines.
:(

--
With regards,
Christian Walde

Chris Marshall

2014-06-11 14:10:02 UTC

Here is a quick response to the discussion so far (I've
extracted some of the related points for reference below),
in bullet form for brevity. I hope that clarity doesn't
suffer! :-)

Why yet another OO system for PDL?

The performance of PDL is due to the C-level support
for computation. We need to keep that but it needs
to be more general (e.g., support new data types
including user datatypes). We _could_ do the OO work
by popping up to the perl level but then we end up
with the PDL computational engine tied to the existence
of perl. I would like to see it separate to better
support embedding of PDL capabilities in other
environments and, as a specific example, to allow
for full support for GPGPU computation. If we had
to have perl for PDL computation that would require
perl on a GPU or some kludgy data/control shuffling.

What type of OO system is needed? Why roles? Why Prima
or XXX?

First off, the plan is a new OO implementation taking
the best practices of a number of existing implementations
but I don't think we want to use any of them directly.
Think Moo versus Moose. The PDL compute core needs to
be simple, flexible, and provide support for higher
level abstractions (data types, methods). I would
like the PDL OO/compute core to support symmetric
method calls as in Prima (C<->C, C<->Perl, Perl<->C)
but we definitely don't want unrelated GUI stuff.
That said, it would be ideal if the new PDL OO Core
could be used to refactor a new Prima implementation.

Regarding Roles: that seems to me to be one of the
outstanding new OO paridigms for its ability to fix
problems of multiple inheritance. It would offer
a natural framework for new computational capability
as roles to apply to a class or even an object
(think JIT to apply a new GPU compute role to a
piddle which would allow GPGPU acceleration)

Moose vs Mouse vs Moo?

Per previous discussion on this topic, Moo is the
preferred perl-level OO for a number of reasons:
* perl only === portable!!!
* Moose compatable when full meta is needed
* small and simple which seems appropriate as a model for our OO
C PDL core.
* http://shadow.cat/blog/matt-s-trout/moo-versus-any-moose/

Nice to see an active discussion on this topic.

--Chris

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point and build new classes based on
Prima's Object class. I spent a weekend on this a while ago,
but it'll take quite a bit more work: Prima's purpose as a
GUI toolkit is pretty deeply embedded.
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we can
In C, object method invocation is efficient because the
compiler knows exactly where to find your function pointers.
As a result, in C an object can be cast into another class
if it has the same binary layout. The problem with roles
in C is that there is no promise of binary layout. If two
different classes implement the same role, you don't know
where in the vtable to find the same method. This can be
solved in a handful of ways. The most efficient way I can
think of is to make role methods a different kind of method
than a class method.
I think Glib / Vala is a working system that is quite good.
I also believe that if you can accept "interfaces" rather
than full, formal "roles", that Glib can do that. How
important is it to have "proper" roles?
I think the biggest question I cannot find answer to, is
why do we need a separate object system (with or without
roles) for PDL3. In my rather limited use of PDL standard
perl OO was more than enough, but I cannot see the big
picture, so would like to ask for some clarifications,
what is the reason behind it. Another reason we might want
Prima-based OO is because of the way it allows to declare
C/Perl wrappers for methods and properties, if that is
needed (I don't know if it is).
Finally, it needs to be decoupled from Prima because it
seems wrong to me to build PDL with help of a GUI toolkit ..
a separate OO system is ok though.

Ed .

2014-06-11 23:42:37 UTC

Nice round-up!

David, I can't add anything further about Prima since I know nothing about
it.

All, what do "roles" provide to PDL that "interfaces" (which Glib has,
Java-style) don't?

Ed

-----Original Message-----
From: Chris Marshall
Sent: Wednesday, June 11, 2014 3:10 PM
To: Christian Walde
Cc: Dmitry Karasik ; pdl-porters
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO

Here is a quick response to the discussion so far (I've
extracted some of the related points for reference below),
in bullet form for brevity. I hope that clarity doesn't
suffer! :-)

Why yet another OO system for PDL?

The performance of PDL is due to the C-level support
for computation. We need to keep that but it needs
to be more general (e.g., support new data types
including user datatypes). We _could_ do the OO work
by popping up to the perl level but then we end up
with the PDL computational engine tied to the existence
of perl. I would like to see it separate to better
support embedding of PDL capabilities in other
environments and, as a specific example, to allow
for full support for GPGPU computation. If we had
to have perl for PDL computation that would require
perl on a GPU or some kludgy data/control shuffling.

What type of OO system is needed? Why roles? Why Prima
or XXX?

First off, the plan is a new OO implementation taking
the best practices of a number of existing implementations
but I don't think we want to use any of them directly.
Think Moo versus Moose. The PDL compute core needs to
be simple, flexible, and provide support for higher
level abstractions (data types, methods). I would
like the PDL OO/compute core to support symmetric
method calls as in Prima (C<->C, C<->Perl, Perl<->C)
but we definitely don't want unrelated GUI stuff.
That said, it would be ideal if the new PDL OO Core
could be used to refactor a new Prima implementation.

Regarding Roles: that seems to me to be one of the
outstanding new OO paridigms for its ability to fix
problems of multiple inheritance. It would offer
a natural framework for new computational capability
as roles to apply to a class or even an object
(think JIT to apply a new GPU compute role to a
piddle which would allow GPGPU acceleration)

Moose vs Mouse vs Moo?

Per previous discussion on this topic, Moo is the
preferred perl-level OO for a number of reasons:
* perl only === portable!!!
* Moose compatable when full meta is needed
* small and simple which seems appropriate as a model for our OO
C PDL core.
* http://shadow.cat/blog/matt-s-trout/moo-versus-any-moose/

Nice to see an active discussion on this topic.

--Chris

Chris Marshall

2014-06-12 11:17:36 UTC

I'm not a Java-ist but here is a post discussing the topic:
http://modernperlbooks.com/mt/2009/05/perl-roles-versus-interfaces-and-abcs.html

--Chris

Post by Ed .
Nice round-up!
David, I can't add anything further about Prima since I know nothing about
it.
All, what do "roles" provide to PDL that "interfaces" (which Glib has,
Java-style) don't?
Ed
-----Original Message----- From: Chris Marshall
Sent: Wednesday, June 11, 2014 3:10 PM
To: Christian Walde
Cc: Dmitry Karasik ; pdl-porters
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO
Here is a quick response to the discussion so far (I've
extracted some of the related points for reference below),
in bullet form for brevity. I hope that clarity doesn't
suffer! :-)
Why yet another OO system for PDL?
The performance of PDL is due to the C-level support
for computation. We need to keep that but it needs
to be more general (e.g., support new data types
including user datatypes). We _could_ do the OO work
by popping up to the perl level but then we end up
with the PDL computational engine tied to the existence
of perl. I would like to see it separate to better
support embedding of PDL capabilities in other
environments and, as a specific example, to allow
for full support for GPGPU computation. If we had
to have perl for PDL computation that would require
perl on a GPU or some kludgy data/control shuffling.
What type of OO system is needed? Why roles? Why Prima
or XXX?
First off, the plan is a new OO implementation taking
the best practices of a number of existing implementations
but I don't think we want to use any of them directly.
Think Moo versus Moose. The PDL compute core needs to
be simple, flexible, and provide support for higher
level abstractions (data types, methods). I would
like the PDL OO/compute core to support symmetric
method calls as in Prima (C<->C, C<->Perl, Perl<->C)
but we definitely don't want unrelated GUI stuff.
That said, it would be ideal if the new PDL OO Core
could be used to refactor a new Prima implementation.
Regarding Roles: that seems to me to be one of the
outstanding new OO paridigms for its ability to fix
problems of multiple inheritance. It would offer
a natural framework for new computational capability
as roles to apply to a class or even an object
(think JIT to apply a new GPU compute role to a
piddle which would allow GPGPU acceleration)
Moose vs Mouse vs Moo?
Per previous discussion on this topic, Moo is the
* perl only === portable!!!
* Moose compatable when full meta is needed
* small and simple which seems appropriate as a model for our OO
C PDL core.
* http://shadow.cat/blog/matt-s-trout/moo-versus-any-moose/
Nice to see an active discussion on this topic.
--Chris

_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

David Mertens

2014-06-13 03:21:02 UTC

Ed,

To be very brief, Roles let you actually define Role methods, and so
provide a unit of code reuse that you don't have with interfaces. However,
Roles are treated differently (by the object system) than the Base Class,
and help avoid confusion.

Interfaces say "If your class implements this Interface, it must have
methods X, Y, and Z, and member W." Full stop. There's no reuse here, just
a guarantee, a contract. Roles are similar with one key difference: "If
your class implements this Role, it must have methods X, Y, and Z, and
member W, and by here are some handy methods that all Roles of this sort
will be able to do (built upon the required Role methods)." Roles, in
short, provide a mechanism for code reuse. With Moose, you get not only
code reuse, but the awesome, awesome manipulations. A Role in Moose can
provide method modifiers that act on the class. Very powerful stuff.

You can think of this visually: if inheritance is a vertically ordered
hierarchy, Roles are a means for code reuse bolted on to the side. The
future of Perl's OO system includes Roles, so we should give very careful
thought to whether and how we might implement them.

David

_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2014-06-12 14:23:33 UTC

Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

Post by David Mertens
and build new classes based on Prima's Object class.
I spent a weekend on this a while ago, but it'll take
quite a bit more work: Prima's purpose
as a GUI toolkit is pretty deeply embedded.

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

Post by David Mertens
In C, object method invocation is efficient because
the compiler knows exactly where to find your function
pointers. As a result, in C an object can be cast into
another class if it has the same binary layout. The
problem with roles in C is that there is no promise of
binary layout. If two different classes implement the same
role, you don't know where in the vtable to find the same
method. This can be solved in a handful of ways. The most
efficient way I can think of is to make role methods a
different kind of method than a class method.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.

I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

Post by David Mertens
Thoughts?
David
---
In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. Typically, the first element
of a struct definin a class is a vtable pointer, and the rest of the items
are the data elements.
...snip...
This can be solved in a handful of ways. One way is to
fetch role methods via a string hash or other index
lookup. This method lookup could return a void pointer
that is the function pointer, and it would have to be cast
to the appropriate function type. Another mechanism which
I think would be more efficient at the cost of an extra
argument, would involve looking up a role vtable (once)
and including that as a separate argument to role methods.
object->vtable->method(object, arg1, arg2, ...)
Here is an index lookup, which relies upon a previously defined typedef and
typedef does_logger_log_something(void * objec);
does_logger_log_something log_func
= object->vtable->lookup_role_method(does_logger_log_something_idx);
log_func(object, arg1, arg2, ...);
And here is the role vtable lookup, which relies upon a global index for the
struct * does_logger_vtable logger_vtable;
logger_vtable = object->vtable->get_role_vtable(object,
does_logger_role_idx)
logger_vtable->log_something(logger_vtable, object, arg1, arg2, ...);
The advantage of this third approach is that you perform the vtable lookup
once. Any role-based function calls within log_something have the vtable on
hand and can perform role method lookups with a simple struct member lookup.
One more thing about role vtables: they don't need to be separate vtables.
They can simply be offsets into the class's vtable. You could implement
roles so that if the passed role vtable is null, the first thing the role
method does is lookup the role vtable. In this way, if you know the class of
your object, you could simply invoke the logger method as
object->vtable->log_something(0, object, arg1, arg2, ...)

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.

--Chris

Chris Marshall

2014-06-12 14:33:18 UTC

The thunk idea even works for things like ->apply_roles_to_object()
where you would malloc the new object structure/vtable, populate
from existing object and add in thunks for the role (presumably compiled
already or could JIT/Inline::C/gcc it).

Very cool stuff!
Chris

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.
--Chris

Ed .

2014-06-12 18:27:00 UTC

Looks a lot like reimplementing C++!

My reading of Chris's link is that "roles" provides the same as "interfaces"
except you can also treat an existing "class" as an "interface". To
reiterate my previous question (and the answer may well be yes): putting
aside that "roles" are a neat recent OO innovation: do they actually add
anything that PDL3 *needs*, that interfaces can't?

-----Original Message-----
From: Chris Marshall
Sent: Thursday, June 12, 2014 3:33 PM
To: David Mertens
Cc: Dmitry Karasik ; pdl-porters
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO

The thunk idea even works for things like ->apply_roles_to_object()
where you would malloc the new object structure/vtable, populate
from existing object and add in thunks for the role (presumably compiled
already or could JIT/Inline::C/gcc it).

Very cool stuff!
Chris

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.
--Chris

Chris Marshall

2014-06-12 19:03:57 UTC

Post by Ed .
Looks a lot like reimplementing C++!

Except the goal is a clean OO framework, that is
implemented in C for portability and supports the key
modern OO features of Moo rather than C++.

Post by Ed .
My reading of Chris's link is that "roles" provides the same
as "interfaces" except you can also treat an existing "class"
as an "interface". To reiterate my previous question (and the
answer may well be yes): putting aside that "roles" are a
neat recent OO innovation: do they actually add anything
that PDL3 *needs*, that interfaces can't?

I'm not a Java programmer but you can google more
discussion and docs on the difference between interfaces
and Moo[se] roles in perl. As for PDL3 needs, I think there
is value in having the C-OO PDL layer be compatible or
equivalent to Moo. It could simplify handing off processing
between the perl layer and the C-OO layer.

However, if you can show that all the goals for PDL3 could
be done only using interfaces that could be useful data in
planning the development effort. Is there a specific reason
that you prefer interfaces to roles?

--Chris

Post by Ed .
-----Original Message----- From: Chris Marshall
Sent: Thursday, June 12, 2014 3:33 PM
To: David Mertens
Cc: Dmitry Karasik ; pdl-porters
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO
The thunk idea even works for things like ->apply_roles_to_object()
where you would malloc the new object structure/vtable, populate
from existing object and add in thunks for the role (presumably compiled
already or could JIT/Inline::C/gcc it).
Very cool stuff!
Chris

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.
--Chris

_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2014-06-12 19:34:35 UTC

Here's another couple of links re perl roles:

http://programming.oreilly.com/2014/01/horizontal-reuse-an-alternative-to-inheritance.html
http://en.wikipedia.org/wiki/Trait_%28computer_programming%29

It seems that the main difference is that a role can include methods and not
just require them.

--Chris

Post by Ed .
Looks a lot like reimplementing C++!

Except the goal is a clean OO framework, that is
implemented in C for portability and supports the key
modern OO features of Moo rather than C++.

I'm not a Java programmer but you can google more
discussion and docs on the difference between interfaces
and Moo[se] roles in perl. As for PDL3 needs, I think there
is value in having the C-OO PDL layer be compatible or
equivalent to Moo. It could simplify handing off processing
between the perl layer and the C-OO layer.
However, if you can show that all the goals for PDL3 could
be done only using interfaces that could be useful data in
planning the development effort. Is there a specific reason
that you prefer interfaces to roles?
--Chris

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.
...snip...

Ed .

2014-06-12 20:04:45 UTC

I am not seeking to influence the choice of final OO framework. I am very
keen that whoever DOES make that choice to know for sure what the OO
requirements of PDL3 are. For reasons of simplicity and speed of design and
implementation, I am also keen that the OO requirements be as parsimonious
as possible.

I want the answer to "do we really need proper OO roles" to be clear so the
list of perl/C toolkit options is correctly composed. Glib (of which I am
mildly a fan) does interfaces but not (I believe) roles, so if PDL3's
problems can all be solved using single inheritance and interfaces, Glib
should be on the list.

Separately, I would argue that rolling our own (based on Prima or otherwise)
is a bad idea.

Chris Marshall

2014-06-12 21:03:21 UTC

Agreed.

Post by Ed .
I want the answer to "do we really need proper OO roles" to be clear so the
list of perl/C toolkit options is correctly composed. Glib (of which I am
mildly a fan) does interfaces but not (I believe) roles, so if PDL3's
problems can all be solved using single inheritance and interfaces, Glib
should be on the list.

Would definitely like your input on whether or not interfaces alone
can do it. I expect there will be at least one throw-away implementation
before we settle on the best option.

Post by Ed .
Separately, I would argue that rolling our own (based on Prima or otherwise)
is a bad idea.

David Mertens

2014-06-13 03:40:17 UTC

I agree that it would be unwise to start from scratch. Fortunately, Prima
has already laid great foundations for single-inheritance scheme that
interfaces bi-directionally between C and Perl. The question I see is
whether we can extract the object system from the GUI system (we can, and
it won't take too much work, I think). Once that is in place, we can ask if
we want to support Roles at the C level, or simply at the Perl level. My
initial response indicates how I think we might do this efficiently at the
C level.

Regarding Glib vs Prima, Prima has the distinct advantage that it already
provides the bidirectional Perl/C interface. I know there are Perl bindings
to Glib, and I would have to look closely at those to see if they are as
robust as Prima's. If somebody really wants to use Glib to implement PDL3,
I would not complain, but if I end up throwing my shoulder behind the
effort, Prima's object system would be my preferred starting point.

David

Post by Ed .
I am not seeking to influence the choice of final OO framework. I am very
keen that whoever DOES make that choice to know for sure what the OO
requirements of PDL3 are. For reasons of simplicity and speed of design and
implementation, I am also keen that the OO requirements be as parsimonious
as possible.
I want the answer to "do we really need proper OO roles" to be clear so
the list of perl/C toolkit options is correctly composed. Glib (of which I
am mildly a fan) does interfaces but not (I believe) roles, so if PDL3's
problems can all be solved using single inheritance and interfaces, Glib
should be on the list.
Separately, I would argue that rolling our own (based on Prima or

Ed .

2014-06-13 04:21:27 UTC

David, can you think of killer features for PDL3 that demand roles? Iâll be very pleased if you come up with some â killer features for PDL3 are a good thing.

From: David Mertens
Sent: Friday, June 13, 2014 4:40 AM
To: Ed .
Cc: Chris Marshall ; pdl-porters ; Dmitry Karasik
Subject: Re: [Pdl-porters] PDL OO thoughts and Prima OO

I agree that it would be unwise to start from scratch. Fortunately, Prima has already laid great foundations for single-inheritance scheme that interfaces bi-directionally between C and Perl. The question I see is whether we can extract the object system from the GUI system (we can, and it won't take too much work, I think). Once that is in place, we can ask if we want to support Roles at the C level, or simply at the Perl level. My initial response indicates how I think we might do this efficiently at the C level.

Regarding Glib vs Prima, Prima has the distinct advantage that it already provides the bidirectional Perl/C interface. I know there are Perl bindings to Glib, and I would have to look closely at those to see if they are as robust as Prima's. If somebody really wants to use Glib to implement PDL3, I would not complain, but if I end up throwing my shoulder behind the effort, Prima's object system would be my preferred starting point.

David

On Thu, Jun 12, 2014 at 4:04 PM, Ed . <***@hotmail.com> wrote:

I am not seeking to influence the choice of final OO framework. I am very keen that whoever DOES make that choice to know for sure what the OO requirements of PDL3 are. For reasons of simplicity and speed of design and implementation, I am also keen that the OO requirements be as parsimonious as possible.

I want the answer to "do we really need proper OO roles" to be clear so the list of perl/C toolkit options is correctly composed. Glib (of which I am mildly a fan) does interfaces but not (I believe) roles, so if PDL3's problems can all be solved using single inheritance and interfaces, Glib should be on the list.

Separately, I would argue that rolling our own (based on Prima or

Chris Marshall

2014-06-13 06:45:11 UTC

I can't speak for David but the ability to apply roles to
objects allows for very elegant specification of JIT
compiling as well as a way to cleanly implement
GPGPU computing.

--Chris

David, can you think of killer features for PDL3 that demand roles? I’ll be
very pleased if you come up with some – killer features for PDL3 are a good
thing.

Chris Marshall

2014-06-13 06:42:11 UTC

I took a look at GObject and it is pretty nice but still looks
like a lot of boilerplate implementing the same stufff we have
been talking about but without the support for the perl
specific features (symmetry between C and Perl OO layers
and Moo[se] OO capabilities). It does look like the GLib
closures could be used to implement method modifiers;

One thought to reduce risk to the implementation is that
we could use test driven development with unit tests based
on the perl OO for the reference and then check that
against the C-OO results.

--Chris

On Thu, Jun 12, 2014 at 11:40 PM, David Mertens

Post by David Mertens
I agree that it would be unwise to start from scratch. Fortunately, Prima
has already laid great foundations for single-inheritance scheme that
interfaces bi-directionally between C and Perl. The question I see is
whether we can extract the object system from the GUI system (we can, and it
won't take too much work, I think). Once that is in place, we can ask if we
want to support Roles at the C level, or simply at the Perl level. My
initial response indicates how I think we might do this efficiently at the C
level.
Regarding Glib vs Prima, Prima has the distinct advantage that it already
provides the bidirectional Perl/C interface. I know there are Perl bindings
to Glib, and I would have to look closely at those to see if they are as
robust as Prima's. If somebody really wants to use Glib to implement PDL3, I
would not complain, but if I end up throwing my shoulder behind the effort,
Prima's object system would be my preferred starting point.
David

David Mertens

2014-06-13 04:10:25 UTC

One important point about the does_roles method. I could have a plotting
widget in debug mode, or a database query handle, or a complex number in
debug mode, and all of them could "does_logger". If you hand me a pointer
to an object and tell me that the bject "does_logger", I have no idea where
the thunks live. If "does_role" is a method in the base class at the root
of the inheritance tree, then I can at least call this method with the
"logger" role, in order to retrieve the role vtable. But the lookup will be
a real lookup. It can't be implemented as a switch statement because the
role IDs would not be compile-time constants. If we want to allow
Perl-level subclasses, it can't (solely) be implemented as a series of if
statements, either. That could provide the C-side role lookup, but the
lookup for Perl-implemented roles would have to involve either a hash
lookup or a list search. If any of the role methods are provided from the
role itself, then they will have to call this lookup method for the role
vtable each time they are called, unless you pass in the vtable as an
argument. The drawback of passing the vtable as an argument is that it
increases stack usage and it makes the code a bit more long-winded. Of
course, we can address longer code with a minimal set of code wrappers and
some very light notation.

But to your point, the advantage of looking up the role vtable whenever you
need it is that your role methods can be first-class object methods.

David

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

Post by David Mertens
Thoughts?
David
---
In C, object method invocation is efficient because the compiler knows
exactly where to find your function pointers. Typically, the first

element

Post by David Mertens
of a struct definin a class is a vtable pointer, and the rest of the

items

Post by David Mertens
are the data elements.
...snip...
This can be solved in a handful of ways. One way is to
fetch role methods via a string hash or other index
lookup. This method lookup could return a void pointer
that is the function pointer, and it would have to be cast
to the appropriate function type. Another mechanism which
I think would be more efficient at the cost of an extra
argument, would involve looking up a role vtable (once)
and including that as a separate argument to role methods.
object->vtable->method(object, arg1, arg2, ...)
Here is an index lookup, which relies upon a previously defined typedef

and

Post by David Mertens
typedef does_logger_log_something(void * objec);
does_logger_log_something log_func
= object->vtable->lookup_role_method(does_logger_log_something_idx);
log_func(object, arg1, arg2, ...);
And here is the role vtable lookup, which relies upon a global index for

the

Post by David Mertens
struct * does_logger_vtable logger_vtable;
logger_vtable = object->vtable->get_role_vtable(object,
does_logger_role_idx)
logger_vtable->log_something(logger_vtable, object, arg1, arg2, ...);
The advantage of this third approach is that you perform the vtable

lookup

Post by David Mertens
once. Any role-based function calls within log_something have the vtable

Post by David Mertens
hand and can perform role method lookups with a simple struct member

lookup.

Post by David Mertens
One more thing about role vtables: they don't need to be separate

vtables.

Post by David Mertens
They can simply be offsets into the class's vtable. You could implement
roles so that if the passed role vtable is null, the first thing the role
method does is lookup the role vtable. In this way, if you know the

class of

Post by David Mertens
your object, you could simply invoke the logger method as
object->vtable->log_something(0, object, arg1, arg2, ...)

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.
--Chris

Chris Marshall

2014-06-13 06:38:12 UTC

Hmm. If a class does a role then all the methods/attributes
are flattened into the class---you know what they are and can
put them in the *class* vtable pointing to the thunk. The
thunk could use the does_<role> method (a.k.a. does(role))
to connect the dots with the actual function pointer(s). Am
I missing something?

--Chris

On Fri, Jun 13, 2014 at 12:10 AM, David Mertens

Post by David Mertens
One important point about the does_roles method. I could have a plotting
widget in debug mode, or a database query handle, or a complex number in
debug mode, and all of them could "does_logger". If you hand me a pointer to
an object and tell me that the bject "does_logger", I have no idea where the
thunks live. If "does_role" is a method in the base class at the root of the
inheritance tree, then I can at least call this method with the "logger"
role, in order to retrieve the role vtable. But the lookup will be a real
lookup. It can't be implemented as a switch statement because the role IDs
would not be compile-time constants. If we want to allow Perl-level
subclasses, it can't (solely) be implemented as a series of if statements,
either. That could provide the C-side role lookup, but the lookup for
Perl-implemented roles would have to involve either a hash lookup or a list
search. If any of the role methods are provided from the role itself, then
they will have to call this lookup method for the role vtable each time they
are called, unless you pass in the vtable as an argument. The drawback of
passing the vtable as an argument is that it increases stack usage and it
makes the code a bit more long-winded. Of course, we can address longer code
with a minimal set of code wrappers and some very light notation.
But to your point, the advantage of looking up the role vtable whenever you
need it is that your role methods can be first-class object methods.
David

Post by Chris Marshall
Some more thoughts---point by point.

Post by David Mertens
If we just wanted a C object system that supported
inheritance, I would say we should take Prima's object
system as a starting point

Yes

We want a clean start here. Let's take the best of the
current Prima ideas (the big on is symmetric access from
both Perl and C to call and create methods).

Post by David Mertens
However, if you want roles as well, that's a different
beast, and the bulk of my thoughts lie there. I think we
can do it, but it'll be tricky. My concerns and ideas, in

Roles are one of the best feature of modern OO programming
because of the way they address problems of multiple
inheritance for complex OO system architectures.

One thought that may simplify things: given that a Role
is composed into the Class, we know what the methods are
already---we just are missing the implementation. How
about putting a pointer to a thunk in the Class vtable
which is responsible for calling the actual routine.
I suggest that the does_role() routine could return a
pointer to the role's implementation vtable. Patching
could be done in one of the ways you've described below
and with JIT we could even collapse chained calls into
a single call if needed.

I think the thunk idea would allow the role methods to be the same
as class methods (I think they are) and the thunk pointer could be
"collapsed" into the appropriate role vtable pointer.
--Chris

David Mertens

2014-06-13 11:33:06 UTC

Post by Chris Marshall
Hmm. If a class does a role then all the methods/attributes
are flattened into the class---you know what they are and can
put them in the *class* vtable pointing to the thunk. The
thunk could use the does_<role> method (a.k.a. does(role))
to connect the dots with the actual function pointer(s). Am
I missing something?
--Chris

No, this would work. The more I think about it, the more I think you're
right on this.

My objection is that role lookup is more costly than a struct offset
lookup. This penalty will be most evident in short role-provided methods
that get called frequently, perhaps in a tight loop. If the role-provided
method does not take the role vtable as a parameter, it *must* look it up
every time. On the other hand, if it does take the role method vtable as a
parameter, then it can avoid the lookup process. This can be avoided by
re-implementing the hot method in the role-consuming class (since the class
method knows the struct offset of the thunks), but that means we're
limiting code reuse for short method (or using #define's for short method
code), and consuming more of the CPU's code cache.

To make life a little easier, we could agree that any role method must
accept a null pointer as the role vtable, in which case it should look up
the role vtable itself. This way, if you only use a single role method
call, you keep your code uncluttered by passing NULL as the second
argument. If you make multiple calls to the same role, or if you need to
call role functions repeatedly within a loop, you can elect to look up the
vtable before the looping, and improve performance. Then again, it'll
needlessly consume the stack if 99% of the role method calls pass a null
role vtable pointer. Furthermore, this null-check requirement will also
impose itself on the CPU's code cache.

It's a trade-off, a very low-level trade-off. Having thought about it quite
a bit now, I'd say that we should go forward with your idea---vtable
lookups as needed. If we realize that we *really* need fast role methods,
then we can implement a new version later.

David

Chris Marshall

2014-06-13 11:59:36 UTC

Post by David Mertens

By connect the dots, I meant look up the actual method pointer
for the role and put that in the object vtable so from then on
the role method would be exactly the same as any class method
call. So the timeline is this:

1. with Role puts vtable pointing to thunk routines
2. call Role method looks up Role vtable method
- calls method as desired and
- pokes original vtable with real role method

Lots of room to tweak things but the case of most interest for
PDL (and most OO programs I've seen) is semi-static method
resolution.

--Chris

Post by David Mertens
On the other hand, if it does take the role method vtable as a parameter, then
it can avoid the lookup process. This can be avoided by re-implementing the
hot method in the role-consuming class (since the class method knows the
struct offset of the thunks), but that means we're limiting code reuse for
short method (or using #define's for short method code), and consuming more
of the CPU's code cache.
To make life a little easier, we could agree that any role method must
accept a null pointer as the role vtable, in which case it should look up
the role vtable itself. This way, if you only use a single role method call,
you keep your code uncluttered by passing NULL as the second argument. If
you make multiple calls to the same role, or if you need to call role
functions repeatedly within a loop, you can elect to look up the vtable
before the looping, and improve performance. Then again, it'll needlessly
consume the stack if 99% of the role method calls pass a null role vtable
pointer. Furthermore, this null-check requirement will also impose itself on
the CPU's code cache.
It's a trade-off, a very low-level trade-off. Having thought about it quite
a bit now, I'd say that we should go forward with your idea---vtable lookups
as needed. If we realize that we *really* need fast role methods, then we
can implement a new version later.
David
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan

Dmitry Karasik

2014-06-13 11:06:47 UTC

Post by David Mertens
"logger" role, in order to retrieve the role vtable. But the lookup will be
a real lookup. It can't be implemented as a switch statement because the
role IDs would not be compile-time constants.

Perl uses real lookup for all methods, but caches the lookup results. Same
is valid for role methods, and Prima does the same trick, looking up and
caching method pointers.

The things become more interesting if we want roles also for C classes - the C
part has to emulate whatever multiple inheritance scheme the class has
(c3/dfs). Probably for the first shot we can try to make a proof of concept system
that doesn't have roles on C level (but allows them on Perl level).

The C/Perl part though is not problematic, the mechanism that Prima uses to
build a vmt to access perl methods from C can be used once we have access to
the perl class (there's mro::get_linear_isa() that gets us list of classes to
visit. This however raises question about minimal perl version - if we need to
use mro:: from C, that hash to be at least 5.9.5.

--
Sincerely,
Dmitry Karasik

Chris Marshall

2014-06-13 12:05:10 UTC

Maybe the C-OO could support single inheritance and roles
with multiple inheritance and fancier things delegated to the
perl level?

--Chris

Perl uses real lookup for all methods, but caches the lookup results. Same
is valid for role methods, and Prima does the same trick, looking up and
caching method pointers.
The things become more interesting if we want roles also for C classes - the C
part has to emulate whatever multiple inheritance scheme the class has
(c3/dfs). Probably for the first shot we can try to make a proof of concept system
that doesn't have roles on C level (but allows them on Perl level).
The C/Perl part though is not problematic, the mechanism that Prima uses to
build a vmt to access perl methods from C can be used once we have access to
the perl class (there's mro::get_linear_isa() that gets us list of classes to
visit. This however raises question about minimal perl version - if we need to
use mro:: from C, that hash to be at least 5.9.5.
--
Sincerely,
Dmitry Karasik

Dmitry Karasik

2014-06-13 13:54:43 UTC

Surely, we can do that, easily. IIRC though you wanted to start from a pure-C libpdl.so,
did you plan it to have objects and roles, or only straight functions operating on
primitive data types? Or rather, hom much of the object machinery will be relevant on
the C level?

Post by Chris Marshall
Maybe the C-OO could support single inheritance and roles
with multiple inheritance and fancier things delegated to the
perl level?
--Chris

Perl uses real lookup for all methods, but caches the lookup results. Same
is valid for role methods, and Prima does the same trick, looking up and
caching method pointers.
The things become more interesting if we want roles also for C classes - the C
part has to emulate whatever multiple inheritance scheme the class has
(c3/dfs). Probably for the first shot we can try to make a proof of concept system
that doesn't have roles on C level (but allows them on Perl level).
The C/Perl part though is not problematic, the mechanism that Prima uses to
build a vmt to access perl methods from C can be used once we have access to
the perl class (there's mro::get_linear_isa() that gets us list of classes to
visit. This however raises question about minimal perl version - if we need to
use mro:: from C, that hash to be at least 5.9.5.
--
Sincerely,
Dmitry Karasik

--
Sincerely,
Dmitry Karasik

Chris Marshall

2014-06-13 16:14:51 UTC

Post by Dmitry Karasik
Surely, we can do that, easily. IIRC though you wanted to start from a pure-C libpdl.so,
did you plan it to have objects and roles, or only straight functions operating on
primitive data types?

The idea was to use a C-OO implementation to allow the
PDL routines to be called equivalently from perl or from
C. Since C-OO is written in C we avoid the problem of a
C++ implementation where it requires one to use the same
compiler for linking as for building libraries due to the
lack of standard name mangling (our C-OO implementation is
essentially a clearly defined name mangling of our own at
the C level).

The PDL2 OO framework is a a hand rolled implementation of
the core object processing at the C level together with
PDL::PP to generate PDL-threading routines compiled for
all basic data types. There is the ability to extend PDL
using a hash object with a PDL key that contains the
C-level PDL object.

The PDL3 C-OO framework would allow direct, equivalent
support for PDL objects (piddles) from both Perl and C.
For example we might have all piddles be in a PDL
attribute which could be accessed as the PDL attribute
from the perl level (e.g., has PDL => (...)) so the
machinery to extend PDL would be identical whether done
from Perl or directly from C.

As already presented, Roles allow for better code reuse
and a simpler way to solve the problem of multiple
inheritance. In fact, PDL could be implemented as
a Role. If the PDL C-OO layer were to support Roles
directly with single inheritance, I don't think it would
need to support multiple inheritance even conceptually.
Anyone wanting that could implement in the Perl level
with no need for us to clutter the C-OO level.

Looking at PDL2, I think Roles could be used to
implement: threading, code generation (PDL::PP),
JIT code generation, GPGPU processing, and
maybe new datatypes as well.

--Chris

Post by Dmitry Karasik
Or rather, hom much of the object machinery will be relevant on the C level?

Post by Chris Marshall
Maybe the C-OO could support single inheritance and roles
with multiple inheritance and fancier things delegated to the
perl level?
--Chris

Perl uses real lookup for all methods, but caches the lookup results. Same
is valid for role methods, and Prima does the same trick, looking up and
caching method pointers.
The things become more interesting if we want roles also for C classes - the C
part has to emulate whatever multiple inheritance scheme the class has
(c3/dfs). Probably for the first shot we can try to make a proof of concept system
that doesn't have roles on C level (but allows them on Perl level).
The C/Perl part though is not problematic, the mechanism that Prima uses to
build a vmt to access perl methods from C can be used once we have access to
the perl class (there's mro::get_linear_isa() that gets us list of classes to
visit. This however raises question about minimal perl version - if we need to
use mro:: from C, that hash to be at least 5.9.5.
--
Sincerely,
Dmitry Karasik

--
Sincerely,
Dmitry Karasik

Dmitry Karasik

2014-06-13 18:46:47 UTC

The idea was to use a C-OO implementation to allow the
PDL routines to be called equivalently from perl or from
C.

If that is the only reason there is, then the OO system shouldn't be complex.
It shouldn't really support roles, even: single-inheritance (Prima-like) is
enough for throwing calls between C and Perl. But that means it will be bound
to perl so there won't be pure-C libpdl.so, right?

Post by Chris Marshall
If the PDL C-OO layer were to support Roles
directly with single inheritance, I don't think it would
need to support multiple inheritance even conceptually.

That depends what kinds of Roles are we talking about - Moose roles, f.ex., are
not in @ISA, so C level will be having hard time traversing class inheritance
tree, trying to locate PDL's data on it, figthing with
Moose/Mouse/whatever-specific way to store inheritance tree on a package.

Custom-rolled roles in C, that know nothing about perl, seem to me just the same as multiple inhertance.

Post by Chris Marshall
Looking at PDL2, I think Roles could be used to
implement: threading, code generation (PDL::PP),
JIT code generation, GPGPU processing, and
maybe new datatypes as well.

Yes, thank you. But still I'd like to get a more precise idea about the protential roles use.
My interpretation of you ideas is somewhat like this: I can think of five OO types, please tell me
do you have something similar in mind:

* 1: single-inheritance C/Perl Prima-like OO:

This is basically Prima's OO plus added support for multiple inheritance in
Perl, not in C. We won't get pure-C libpdl.so, but we'll get automatic C/Perl
translation layer and compile-time C classes. A piddle will be represented by
a C structure with pointers to a vmt, parent class, and data.

* 2: pure-C libpdl.so

Simple API with no inheritance ( think "pdl_slice_t* create_piddle_slice( pdl_t * piddle ... )" )
API. Perl bindings come separately. Works completely standalone on C level,
without perl. The question is how much of PDL functionality written in Perl
will be unavailable from libpdl that way.

* 3: pure-C libpdl.so with single inheritance

Same as #2 but basically a C++like system

* 4: pure-C libpdl.so with multiple inheritance

Same as #3, and the role system must emulate perl's c3/dfs.

* 5: extension of #1 with C-level roles.

No libpdl.so, again, must emulate perl's c3/dfs.

In addition to this, I'm thinking do we or do we not want Moose and/or friends in the picture:

* A: No Moose, only pure-perl OO. If users want to extend PDL with Moose
classes it's ok, but that's not core problem. Lightweight, and roles can be done with usual perl multiple inheritance.

* B: Moose on perl level only, i.e. some or all PDL types, piddles also, are formally declared with Moose
with has/with/extends etc, but essentially are wrappers around single-inheritance C/Perl classes.
Great flexibility when creating custom roles in perl. Basically, two different OO systems working together.

* C: Tight integration with Moose/MOP on C level.

I have little competence over PDL architecture and its patterns of use, but if I'd built something
like PDL from scratch, I'd go for A1 which is not far from the existing PDL architecture.

--
Sincerely,
Dmitry Karasik

David Mertens

2014-06-13 18:58:31 UTC

Speaking for myself, I have in mind something essentially like A1, and
would build roles (with some added gencls syntactic sugar) on top of that.
(For those who don't know, gencls is Prima's script to convert class
descriptions into the necessary vtable glue, thunks, etc.)

David

Post by Dmitry Karasik
Surely, we can do that, easily. IIRC though you wanted to start from a

pure-C libpdl.so,

Post by Dmitry Karasik
did you plan it to have objects and roles, or only straight functions

operating on

Post by Dmitry Karasik
primitive data types?

The idea was to use a C-OO implementation to allow the
PDL routines to be called equivalently from perl or from
C.

Post by Chris Marshall
If the PDL C-OO layer were to support Roles
directly with single inheritance, I don't think it would
need to support multiple inheritance even conceptually.

That depends what kinds of Roles are we talking about - Moose roles, f.ex., are
tree, trying to locate PDL's data on it, figthing with
Moose/Mouse/whatever-specific way to store inheritance tree on a package.
Custom-rolled roles in C, that know nothing about perl, seem to me just
the same as multiple inhertance.

Post by Chris Marshall
Looking at PDL2, I think Roles could be used to
implement: threading, code generation (PDL::PP),
JIT code generation, GPGPU processing, and
maybe new datatypes as well.

Yes, thank you. But still I'd like to get a more precise idea about the
protential roles use.
My interpretation of you ideas is somewhat like this: I can think of five
OO types, please tell me
This is basically Prima's OO plus added support for multiple inheritance in
Perl, not in C. We won't get pure-C libpdl.so, but we'll get automatic C/Perl
translation layer and compile-time C classes. A piddle will be represented by
a C structure with pointers to a vmt, parent class, and data.
* 2: pure-C libpdl.so
Simple API with no inheritance ( think "pdl_slice_t* create_piddle_slice(
pdl_t * piddle ... )" )
API. Perl bindings come separately. Works completely standalone on C level,
without perl. The question is how much of PDL functionality written in Perl
will be unavailable from libpdl that way.
* 3: pure-C libpdl.so with single inheritance
Same as #2 but basically a C++like system
* 4: pure-C libpdl.so with multiple inheritance
Same as #3, and the role system must emulate perl's c3/dfs.
* 5: extension of #1 with C-level roles.
No libpdl.so, again, must emulate perl's c3/dfs.
In addition to this, I'm thinking do we or do we not want Moose and/or
* A: No Moose, only pure-perl OO. If users want to extend PDL with Moose
classes it's ok, but that's not core problem. Lightweight, and roles can
be done with usual perl multiple inheritance.
* B: Moose on perl level only, i.e. some or all PDL types, piddles also,
are formally declared with Moose
with has/with/extends etc, but essentially are wrappers around
single-inheritance C/Perl classes.
Great flexibility when creating custom roles in perl. Basically, two
different OO systems working together.
* C: Tight integration with Moose/MOP on C level.
I have little competence over PDL architecture and its patterns of use,
but if I'd built something
like PDL from scratch, I'd go for A1 which is not far from the existing PDL architecture.
--
Sincerely,
Dmitry Karasik

Chris Marshall

2014-06-13 20:09:57 UTC

This is more the direction I was thinking. Had the thought to use
metaobject programming to take a Moo class/role/... and convert
it into the corresponding/equivalent C-OO stuff.

--Chris

Speaking for myself, I have in mind something essentially like A1, and would
build roles (with some added gencls syntactic sugar) on top of that. (For
those who don't know, gencls is Prima's script to convert class descriptions
into the necessary vtable glue, thunks, etc.)
David

Post by David Mertens
...snip...
Yes, thank you. But still I'd like to get a more precise idea about the
I can think of five OO types, please tell me do you have something
This is basically Prima's OO plus added support for multiple inheritance
in Perl, not in C. We won't get pure-C libpdl.so, but we'll get automatic
C/Perl translation layer and compile-time C classes. A piddle will be
represented by a C structure with pointers to a vmt, parent class, and data.
...snip...

Chris Marshall

2014-06-13 20:13:37 UTC

On a related note, it appears that Vala supports interfaces with
actual methods (a.k.a. mixins or roles).

Post by Chris Marshall
This is more the direction I was thinking. Had the thought to use
metaobject programming to take a Moo class/role/... and convert
it into the corresponding/equivalent C-OO stuff.
--Chris

Speaking for myself, I have in mind something essentially like A1, and would
build roles (with some added gencls syntactic sugar) on top of that. (For
those who don't know, gencls is Prima's script to convert class descriptions
into the necessary vtable glue, thunks, etc.)
David

Chris Marshall

2014-06-13 20:06:20 UTC

Post by Dmitry Karasik
Surely, we can do that, easily. IIRC though you wanted
to start from a pure-C libpdl.so, did you plan it to
have objects and roles, or only straight functions
operating on primitive data types?

The idea was to use a C-OO implementation to allow the
PDL routines to be called equivalently from perl or from
C.

If that is the only reason there is, then the OO system
shouldn't be complex. It shouldn't really support roles,

If the perl level has Roles then the idea was that the C-OO
level would have them as well.

Post by Dmitry Karasik
even: single-inheritance (Prima-like) is enough for
throwing calls between C and Perl. But that means it will
be bound to perl so there won't be pure-C libpdl.so,
right?

Thinking of GPGPU computation, I don't think it will be
possible to get perl onto the GPU so the C-level support
is needed to enable JIT for GPU enabled PDL computation.

Post by Chris Marshall
If the PDL C-OO layer were to support Roles directly
with single inheritance, I don't think it would need to
support multiple inheritance even conceptually.

That depends what kinds of Roles are we talking about
will be having hard time traversing class inheritance
tree, trying to locate PDL's data on it, figthing with
Moose/Mouse/whatever-specific way to store inheritance
tree on a package.

At the perl level I propose using Moo as the reference
for Modern Perl OO---it is small, perl-only, and Moose
compatible if you need the whole MOP stuff.

Post by Dmitry Karasik
Custom-rolled roles in C, that know nothing about perl,
seem to me just the same as multiple inhertance.

I don't think roles are the same as multiple inheritance.
In fact the specific differences are what make them more
useful than multiple inheritance for many OO applications.

Post by Chris Marshall
Looking at PDL2, I think Roles could be used to
implement: threading, code generation (PDL::PP),
JIT code generation, GPGPU processing, and
maybe new datatypes as well.

Yes, thank you. But still I'd like to get a more precise
idea about the protential roles use. My interpretation of
you ideas is somewhat like this: I can think of five OO
types, please tell me do you have something similar in
This is basically Prima's OO plus added support for
multiple inheritance in Perl, not in C. We won't get
pure-C libpdl.so, but we'll get automatic C/Perl
translation layer and compile-time C classes. A piddle
will be represented by a C structure with pointers to a
vmt, parent class, and data.

I like the ability to traverse between C and Perl layers
in Prima. The description of a piddle you describe is
pretty much what any of the C-OO frameworks use to
represent their data. In fact, I believe many OO languages
use those types of structures in their implementations.

Post by Dmitry Karasik
* 2: pure-C libpdl.so
Simple API with no inheritance ( think "pdl_slice_t*
create_piddle_slice( pdl_t * piddle ... )" ) API. Perl
bindings come separately. Works completely standalone on
C level, without perl. The question is how much of PDL
functionality written in Perl will be unavailable from
libpdl that way.

The part of PDL3 I'm talking about here is the kernel of
the architecture. Take a look at the stuff in PDL/Basic/Core
and you'll see that not much of PDL actually resides there.
However, it does contain the engine of the module computation.
This C-OO framework is to clean the existing mess up.

Post by Dmitry Karasik
* 3: pure-C libpdl.so with single inheritance
Same as #2 but basically a C++like system

More like a Moo-like system and lightweight as possible.

Post by Dmitry Karasik
* 4: pure-C libpdl.so with multiple inheritance
Same as #3, and the role system must emulate perl's c3/dfs.

No, I'm thinking support for roles at the C level which is
essentially the same as support for classes since the roles
are flattened into the class at creation/compile time.
C-level support for method modifiers allows for easy
extensions at either perl and/or C levels.

Post by Dmitry Karasik
* 5: extension of #1 with C-level roles.
No libpdl.so, again, must emulate perl's c3/dfs.

I don't think emulating perl's c3/dfs is the only way to
implement roles. Maybe the C-OO code for the PDL kernel
and core could be generated from a Moo equivalent.

Post by Dmitry Karasik
* A: No Moose, only pure-perl OO. If users want to extend
PDL with Moose classes it's ok, but that's not core
problem. Lightweight, and roles can be done with usual
perl multiple inheritance.
* B: Moose on perl level only, i.e. some or all PDL types,
piddles also, are formally declared with Moose with
has/with/extends etc, but essentially are wrappers around
single-inheritance C/Perl classes. Great flexibility when
creating custom roles in perl. Basically, two different OO
systems working together.
* C: Tight integration with Moose/MOP on C level.

Moo at perl level and Moo-like enough to support C-level
operations relatively unassisted (think PDL computation on
GPU where perl is *not* directly available).

Post by Dmitry Karasik
I have little competence over PDL architecture and its
patterns of use, but if I'd built something like PDL from
scratch, I'd go for A1 which is not far from the existing
PDL architecture.

I think some support for the OO environment at the C level
will make a big difference in our ability to implement PDL3.
Moo at the perl level is a Moose-compatible, lightweight
implementation of modern OO in perl. The current PDL OO
framework is a hardwired mix of hashes for objects, C only
implementation of PDL runtime and compile time implementation
of threading.

--Chris

unknown

1970-01-01 00:00:00 UTC