[Pdl-porters] How to specify PDL

Discussion:

[Pdl-porters] How to specify PDL_Indx in OtherPars?

Edward Baudrez

2013-09-27 10:36:40 UTC

Dear porters

I'm reading primitive.pd line 1627 and below (the definition of histogram()
and related functions). These functions should be updated for 64-bit. The
indices should be:

register PDL_Indx j;
register PDL_Indx maxj = $SIZE(m)-1;

and a bit further:

j = (PDL_Indx) (($in()-min)/step);

Logically, 'm' should have type PDL_Indx. However, when I want to specify
OtherPars as follows

OtherPars => 'double step; double min; PDL_Indx msize => m'

I get a build error:

The type =PDL_Indx= does not have a typemap entry!

There *is* a typemap entry for PDL_Indx, so I don't understand the error
message. The XS code doesn't even get generated for this function, so I
can't look at that to see what's going wrong. Anybody any idea?

Best regards
Edward

Chris Marshall

2013-09-28 12:30:01 UTC

Permalink

Could you provide a short example of the problem for
debugging? If this feature is not working, it definitely
should be fixed before the official release. Thanks.

--Chris

On Fri, Sep 27, 2013 at 6:36 AM, Edward Baudrez

Post by Edward Baudrez
Dear porters
I'm reading primitive.pd line 1627 and below (the definition of histogram()
and related functions). These functions should be updated for 64-bit. The
register PDL_Indx j;
register PDL_Indx maxj = $SIZE(m)-1;
j = (PDL_Indx) (($in()-min)/step);
Logically, 'm' should have type PDL_Indx. However, when I want to specify
OtherPars as follows
OtherPars => 'double step; double min; PDL_Indx msize => m'
The type =PDL_Indx= does not have a typemap entry!
There *is* a typemap entry for PDL_Indx, so I don't understand the error
message. The XS code doesn't even get generated for this function, so I
can't look at that to see what's going wrong. Anybody any idea?
Best regards
Edward
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2013-09-28 12:55:01 UTC

Permalink

Do you have another PDL install on your system? Maybe an
older PDL typemap is being used instead of the one in the
CPAN developers release you were working with?

--Chris

Post by Chris Marshall
Could you provide a short example of the problem for
debugging? If this feature is not working, it definitely
should be fixed before the official release. Thanks.
--Chris
On Fri, Sep 27, 2013 at 6:36 AM, Edward Baudrez

Edward Baudrez

2013-10-01 08:42:39 UTC

Permalink

Back with more details. I've attached a program which builds and tests fine
on my setup (64-bit Linux machine with PDL 2.006_91). Notice in particular
the two variables of type PDL_Indx in the function body of myhistogram(). I
assume this confirms the typedef in pdl.h works fine.

However, when I change the two 'long's (one in the OtherPars section of
myhistogram() and the other one as the argument of myfunction()) to
PDL_Indx, the program fails to build.

First, the 'long' argument to myfunction(). When I change it to 'PDL_Indx',
the build fails with
--------------
Could not find a typemap for C type 'PDL_Indx'.
The following C types are mapped by the current typemap:
'AV *', 'Boolean', ...
--------------
I found the typemap file 'typemap.pdl' in the PDL/Core directory doesn't
contain the PDL_Indx type. The typemap called 'typemap' in the PDL source
directory, however, does. I assume it's internal to the build, though.
Manually adding PDL_Indx to typemap.pdl in PDL/Core resolves the problem of
PDL_Indx in the argument list of myfunction(). But I think the user
shouldn't be required to do this.

Second, the 'long' argument (in OtherPars) to myhistogram(). When I change
it to 'PDL_Indx', the build fails with
--------------
/home/ebaudrez/opt/perl/5.16.3/bin/perl
-I/home/ebaudrez/opt/perl/5.16.3/lib/site_perl/5.16.3/x86_64-linux
"-MPDL::PP qw/MyHistogram MyHistogram MyHistogram/" histogram.pd
The type =PDL_Indx= does not have a typemap entry!
make: *** [MyHistogram.pm] Error 2
--------------
This error is issued by PDL::PP. So apparently, even after adding PDL_Indx
to typemap.pdl in PDL/Core, PDL::PP is unable to find it. However, adding a
typemap file to the current working directory containing "PDL_Indx T_IV",
the build works and tests fine. So it appears to be a matter of path
lookups ... I think the user shouldn't be required to do this.

Hope this helps
Best regards
Edward

Post by Chris Marshall
Do you have another PDL install on your system? Maybe an
older PDL typemap is being used instead of the one in the
CPAN developers release you were working with?
--Chris

Post by Edward Baudrez
Dear porters
I'm reading primitive.pd line 1627 and below (the definition of

histogram()

Post by Chris Marshall

Post by Edward Baudrez
and related functions). These functions should be updated for 64-bit.

The

Post by Chris Marshall

Post by Edward Baudrez
register PDL_Indx j;
register PDL_Indx maxj = $SIZE(m)-1;
j = (PDL_Indx) (($in()-min)/step);
Logically, 'm' should have type PDL_Indx. However, when I want to

specify

Post by Chris Marshall

Post by Edward Baudrez
OtherPars as follows
OtherPars => 'double step; double min; PDL_Indx msize => m'
The type =PDL_Indx= does not have a typemap entry!
There *is* a typemap entry for PDL_Indx, so I don't understand the error
message. The XS code doesn't even get generated for this function, so I
can't look at that to see what's going wrong. Anybody any idea?
Best regards
Edward
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2013-10-01 10:38:24 UTC

Permalink

Thanks for the further investigation and report. As you can
tell, it is a little hard to follow the typemap stuff. :-)

--Chris

Post by Edward Baudrez
Back with more details. I've attached a program which builds and tests fine
on my setup (64-bit Linux machine with PDL 2.006_91). Notice in particular
the two variables of type PDL_Indx in the function body of myhistogram(). I
assume this confirms the typedef in pdl.h works fine.
However, when I change the two 'long's (one in the OtherPars section of
myhistogram() and the other one as the argument of myfunction()) to
PDL_Indx, the program fails to build.
First, the 'long' argument to myfunction(). When I change it to 'PDL_Indx',
the build fails with
--------------
Could not find a typemap for C type 'PDL_Indx'.
'AV *', 'Boolean', ...
--------------
I found the typemap file 'typemap.pdl' in the PDL/Core directory doesn't
contain the PDL_Indx type. The typemap called 'typemap' in the PDL source
directory, however, does. I assume it's internal to the build, though.
Manually adding PDL_Indx to typemap.pdl in PDL/Core resolves the problem of
PDL_Indx in the argument list of myfunction(). But I think the user
shouldn't be required to do this.
Second, the 'long' argument (in OtherPars) to myhistogram(). When I change
it to 'PDL_Indx', the build fails with
--------------
/home/ebaudrez/opt/perl/5.16.3/bin/perl
-I/home/ebaudrez/opt/perl/5.16.3/lib/site_perl/5.16.3/x86_64-linux
"-MPDL::PP qw/MyHistogram MyHistogram MyHistogram/" histogram.pd
The type =PDL_Indx= does not have a typemap entry!
make: *** [MyHistogram.pm] Error 2
--------------
This error is issued by PDL::PP. So apparently, even after adding PDL_Indx
to typemap.pdl in PDL/Core, PDL::PP is unable to find it. However, adding a
typemap file to the current working directory containing "PDL_Indx T_IV",
the build works and tests fine. So it appears to be a matter of path lookups
... I think the user shouldn't be required to do this.
Hope this helps
Best regards
Edward

Post by Chris Marshall
Do you have another PDL install on your system? Maybe an
older PDL typemap is being used instead of the one in the
CPAN developers release you were working with?
--Chris

Chris Marshall

2013-10-01 21:52:39 UTC

Permalink

I was able to determine that the build works just fine if you have
a proper typemap in the build directory. It appears that the Pdlpp.pm
automatically adds typemap.pdl to the list of typemap files to search.

If that is the case---since that is the file that appears to be installed---I
wonder if we should have Basic/Core/typemap generate the corresponding
typemap.pdl file (or vice-versa)?

Any PDL typemap masters at bat?

Thanks much,
Chris

Post by Chris Marshall
Thanks for the further investigation and report. As you can
tell, it is a little hard to follow the typemap stuff. :-)
--Chris

Post by Chris Marshall
Do you have another PDL install on your system? Maybe an
older PDL typemap is being used instead of the one in the
CPAN developers release you were working with?
--Chris

Chris Marshall

2013-10-01 22:01:26 UTC

Permalink

Looking at the current Basic/Pod/PP.pod is the statement:

That said, there are a couple of important differences from the general
handling of types in XS. The first, and probably most important, is
that at the moment pointers to types are not allowed in the "OtherPars"
section. To get around this limitation you must use the "IV" type
(thanks to Judd Taylor for pointing out that this is necessary for
portability).

Is it still true that pointer types are not allowed in OtherPars?
I did a grep for OtherPars in *.pd files in the PDL distribution an
came up with not only 'IV blah;' but also 'char* blah;' and
'SV* blah;' being used.

Thanks,
Chris

Edward Baudrez

2013-10-14 09:34:50 UTC

Permalink

Chris or other porters,

Has a decision been reached on this issue? I'm inclined to think that a
typemap for PDL_Indx should be "exported" to users, since PDL libraries are
likely to need it.

Best regards
Edward

Post by Chris Marshall
That said, there are a couple of important differences from the general
handling of types in XS. The first, and probably most important, is
that at the moment pointers to types are not allowed in the "OtherPars"
section. To get around this limitation you must use the "IV" type
(thanks to Judd Taylor for pointing out that this is necessary for
portability).
Is it still true that pointer types are not allowed in OtherPars?
I did a grep for OtherPars in *.pd files in the PDL distribution an
came up with not only 'IV blah;' but also 'char* blah;' and
'SV* blah;' being used.
Thanks,
Chris

Chris Marshall

2013-10-14 11:53:52 UTC

Permalink

Edward-

This issue still needs to be addressed. It is likely there are other
areas where the PDL_Indx work is incomplete or inadequate.

A clean-up of PDL::PP and the PDL core implementation for
more performance and easier development is already in the
works for PDL-2.008.

BTW, I thought that having a typemap file in the build directory
was a valid fix to the build problem. Is that not the case?

Regards,
Chris

On Mon, Oct 14, 2013 at 5:34 AM, Edward Baudrez

Post by Edward Baudrez
Chris or other porters,
Has a decision been reached on this issue? I'm inclined to think that a
typemap for PDL_Indx should be "exported" to users, since PDL libraries are
likely to need it.
Best regards
Edward

Post by Chris Marshall
That said, there are a couple of important differences from the
general handling of types in XS. The first, and probably most
important, is that at the moment pointers to types are not
allowed in the "OtherPars" section.
To get around this limitation you must use the "IV" type
(thanks to Judd Taylor for pointing out that this is necessary
for portability).
Is it still true that pointer types are not allowed in OtherPars?
I did a grep for OtherPars in *.pd files in the PDL distribution an
came up with not only 'IV blah;' but also 'char* blah;' and
'SV* blah;' being used.
Thanks,
Chris

Edward Baudrez

2013-10-14 12:11:06 UTC

Permalink

Post by Chris Marshall
Edward-
This issue still needs to be addressed. It is likely there are other
areas where the PDL_Indx work is incomplete or inadequate.

No problem - not trying to rush things, but I thought I'd missed part of
the discussion.

Post by Chris Marshall
A clean-up of PDL::PP and the PDL core implementation for
more performance and easier development is already in the
works for PDL-2.008.
BTW, I thought that having a typemap file in the build directory
was a valid fix to the build problem. Is that not the case?

Yes, including a typemap file in the directory where I build my library
fixes the problem.

Very happy to see all the nice work happening on PDL!

Best regards
Edward

Edward Baudrez

2013-11-12 09:15:25 UTC

Permalink

Hi David & porters

Bringing up an old conversation again, sorry.

Browsing the PDL source code again, I'm now wondering whether PDL_Indx
*always* maps to T_IV? I somehow (pehaps mistakenly) assumed it would map
to different types on different setups, but Basic/Core/typemap is checked
in in the repository, instead of generated on the fly.

@David, I think you meant to forward your reply to the list, right?
Forwarding now ...

Best regards
Edward

<snip>

Post by Chris Marshall
BTW, I thought that having a typemap file in the build directory
was a valid fix to the build problem. Is that not the case?

Yes, including a typemap file in the directory where I build my library
fixes the problem.

And this is the officially supported Perl/XS mechanism for handling custom
types. However, I vote that PP.pm should handle all of this for the "end
user", i.e. the module author. I suspect this means that we will need to
generate the typemap.pdl file at build time, then?
David

Chris Marshall

2013-11-12 17:09:25 UTC

Permalink

On Tue, Nov 12, 2013 at 4:15 AM, Edward Baudrez

Post by Edward Baudrez
Hi David & porters
Bringing up an old conversation again, sorry.
Browsing the PDL source code again, I'm now wondering whether PDL_Indx
*always* maps to T_IV? I somehow (pehaps mistakenly) assumed it would map to
different types on different setups, but Basic/Core/typemap is checked in in
the repository, instead of generated on the fly.

T_IV is different for different setups since the underlying
perls have different ivtype configurations. If you try building
PDL on a perl with 32bit ivsize, you'll only get 32bit index
support.

Post by Edward Baudrez
@David, I think you meant to forward your reply to the list, right?
Forwarding now ...

<snip>

Post by Chris Marshall
BTW, I thought that having a typemap file in the build directory
was a valid fix to the build problem. Is that not the case?

Yes, including a typemap file in the directory where I build my library
fixes the problem.

The "fix" of including a typemap explicitly was a work-around
for something missing in the PDL-2.007 typemap handling.
We definitely need to have PP code for PDL_Indx not require
any more special measures than someone using PDL_Long
or others.

--Chris

Chris Marshall

2013-12-10 19:25:18 UTC

Permalink

Has anyone had the chance to look at what this fix requires?

Thanks,
Chris

Post by Chris Marshall
On Tue, Nov 12, 2013 at 4:15 AM, Edward Baudrez

T_IV is different for different setups since the underlying
perls have different ivtype configurations. If you try building
PDL on a perl with 32bit ivsize, you'll only get 32bit index
support.

Post by Edward Baudrez
@David, I think you meant to forward your reply to the list, right?
Forwarding now ...

<snip>

Post by Chris Marshall
BTW, I thought that having a typemap file in the build directory
was a valid fix to the build problem. Is that not the case?

Yes, including a typemap file in the directory where I build my library
fixes the problem.

David Mertens

2013-10-14 12:26:20 UTC

Permalink

I suspect that this statement is out of date because PDL::Drawing::Prima
uses at least one SV* argument in the OtherPars for every single pp
function definition. Given that this works for SV* arguments, the only
reason we might post such a warning is that somehow this doesn't play well
with custom typemaps. Unfortunately I don't have time to test that
hypothesis at the moment.

David

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan