[Pdl-porters] Strawberry Perl PDL - better user experience

[Chris Marshall]

< adding pdl-porters cc to this discussion >

Hi kmx-

My plan is to provide Alien::XXX modules for all the external
dependencies but, as you are aware, I've been chewing on
the problem for a while without much progress.

Part of the problem is that the current approach to implement
Alien modules from the original manifesto is too vague and
currently results in Alien::XXX modules that often only work
on a singleplatform or type of platform, don't give a clue in the
docs ortest results about whether or not it actually works, and
doesn't even handle consistently, correctly, or at all just
detecting and using an existing dependency!

I have some concrete ideas for a fix with the start here:

>
http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html

I haven't had a lot of time to finish the second post on an
example Alien2 implementation with discussion. The hope
was that we could refactor the existing Alien::Base to
support the Alien2 improvements.

It might be simpler to finish some example "Alien2::XXX"
implementations to support the PDL2 -> PDL3 transition
(with the goal of splitting out non-core dependencies based
modules from the PDL core distribution and replacing the
all-in-one install by a bundle install). The "show me the code"
approach has the plus it will help PDL and provide some
concrete examples of what I mean in Alien2.

Below I've commented on the current status of dependency
detection in the PDL-2.007 distribution. This is off the top of
my head. Clarification and updates welcome.

Cheers,
Chris


On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz> wrote:

> Hi Chris,
>
> as for the ConsoleZ you are right.
>
> Maybe lets focus on external library detection. Here is my observation
> (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):
>
> PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
>
>
Alien::Curses or Alien::NCurses would work but I really want to
have a better data visualization tool for PDL. Text based should
be the fallback, not the default standard approach.

> PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
>
>
I've only just been able to compile the latest PLplot library to
revisit this detection and use in the module. David Mertens
had done some work here but discovered Prima and switched
to his own implementation which is PDL::Graphics::Prima

> PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.a
>
> Definitely looking for an Alien::HDF4 here. Figured I would
do it at the same time as the HDF5 work.


> PDL::GSL::* ### (PDL-2.007) OK
>
> Detection ok, not sure where Alien::GSL stands but work for
that was part of the motivation for Joel Berger's Alien::Base
work. I would like to see a more comprehensive numeric
processing support in PDL, maybe via additional GSL bindings...



> PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't build
>
> PROJ4 pretty much builds out of the box so an Alien::PROJ4
would work and the current detection using Devel::CheckLib seems
to work. If the projection is not found in the test, then probably
a needed environment variable is not set or the datum info has
not been installed.


> PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
>
>
Ditto


> PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
>
>
Alien::HDF5 is in the works to support PDL::IO::HDF5 but
it is still waiting on the Alien2 push...


> PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
>
>
Have a rudimentary, incomplete Alien::FFTW3 that would be
nice to make cross platform but waiting on the Alien2 push...


> PDL::FFTW ### OK
>
> This module is no longer part of the PDL core and is
buggy. Once the PDL::FFTW3 is finished, I plan to
deprecate it completely.


> PDL::Graphics::Gnuplot ### OK
>
> This worked for cygwin and I've had it work for Strawberry
Perl. I assume Rob could say if that is the case for ASPerl.


> Alien::GSL ### does not detect existing library
>
> Alien::FFTW3 ### does not detect existing library
>
>
>
These need the Alien2 rework...


> Should I try to create patches for related modules? Or do you plan to
> unify lib detection over all PDL related modules? (Alien::* approach of
> Devel::CheckLib approach of other?)
>
> If you gimme a short crash course how a build system for PDL modules work
> I can start with some hacking.
>
> --
> kmx
>

[Chris Marshall]

On Tue, Apr 15, 2014 at 6:38 PM, kmx <***@atlas.cz> wrote:

> Chris,
>
> I'll cut out only Alien2 part from your last email. Let us discuss the
> rest in separate thread.
>
> On 15.4.2014 23:20, Chris Marshall wrote:
>
>> < adding pdl-porters cc to this discussion >
>>
>> Hi kmx-
>>
>> My plan is to provide Alien::XXX modules for all the external
>> dependencies but, as you are aware, I've been chewing on
>> the problem for a while without much progress.
>>
>> Part of the problem is that the current approach to implement
>> Alien modules from the original manifesto is too vague and
>> currently results in Alien::XXX modules that often only work
>> on a singleplatform or type of platform, don't give a clue in the
>> docs ortest results about whether or not it actually works, and
>> doesn't even handle consistently, correctly, or at all just
>> detecting and using an existing dependency!
>>
>> I have some concrete ideas for a fix with the start here:
>>
>> > http://blogs.perl.org/users/chris_marshall/2013/12/a-
>> framework-for-alien-modules-the-alien2-manifesto.html
>>
>> I haven't had a lot of time to finish the second post on an
>> example Alien2 implementation with discussion. The hope
>> was that we could refactor the existing Alien::Base to
>> support the Alien2 improvements.
>>
>> It might be simpler to finish some example "Alien2::XXX"
>> implementations to support the PDL2 -> PDL3 transition
>> (with the goal of splitting out non-core dependencies based
>> modules from the PDL core distribution and replacing the
>> all-in-one install by a bundle install). The "show me the code"
>> approach has the plus it will help PDL and provide some
>> concrete examples of what I mean in Alien2.
>>
>
> I have quite enough experiences from authoring Alien::SDL Alien::IUP and
> Alien::Tidyp
>
> Reading your blog post I agree with most of your suggestion.
>
> But: "On installation, make sure the required package is there, otherwise
> install it."
>
>
This was the original Alien manifesto, not "Alien2".

My experience is: detect and use if present (allows the user
to "fix" things and then have it "just work").

Install *if wanted*since in some/many cases, the user
might not want a blind install. One example of this is
OpenGL where you *really* don't want to try to add the
OpenGL drivers and all for a given platform---extremely
difficult to get right and people get really mad if you
make their system unbootable....

[Ed .]

As someone who is interested in getting involved in this Alien*::* business – what do people think about taking the existing PDL 2.007, and migrating that towards using Alien modules, using CPAN underscore releases and the smoke-testing network to make sure things don’t work any worse than before, and hopefully work better?

I believe that an incremental approach is the way forward here – evolution, not revolution. I don’t see it as something that needs to wait for PDL3. What do you think?

Cheers,
Ed J

From: Chris Marshall
Sent: Tuesday, April 15, 2014 10:20 PM
To: kmx
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

< adding pdl-porters cc to this discussion >


Hi kmx-


My plan is to provide Alien::XXX modules for all the external
dependencies but, as you are aware, I've been chewing on
the problem for a while without much progress.

Part of the problem is that the current approach to implement
Alien modules from the original manifesto is too vague and
currently results in Alien::XXX modules that often only work
on a singleplatform or type of platform, don't give a clue in the
docs ortest results about whether or not it actually works, and

doesn't even handle consistently, correctly, or at all just

detecting and using an existing dependency!


I have some concrete ideas for a fix with the start here:

> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html


I haven't had a lot of time to finish the second post on an

example Alien2 implementation with discussion. The hope
was that we could refactor the existing Alien::Base to
support the Alien2 improvements.

It might be simpler to finish some example "Alien2::XXX"
implementations to support the PDL2 -> PDL3 transition
(with the goal of splitting out non-core dependencies based
modules from the PDL core distribution and replacing the

all-in-one install by a bundle install). The "show me the code"
approach has the plus it will help PDL and provide some
concrete examples of what I mean in Alien2.


Below I've commented on the current status of dependency
detection in the PDL-2.007 distribution. This is off the top of
my head. Clarification and updates welcome.


Cheers,
Chris




On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz> wrote:

Hi Chris,

as for the ConsoleZ you are right.

Maybe lets focus on external library detection. Here is my observation (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):


PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
Alien::Curses or Alien::NCurses would work but I really want to
have a better data visualization tool for PDL. Text based should
be the fallback, not the default standard approach.

PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
I've only just been able to compile the latest PLplot library to

revisit this detection and use in the module. David Mertens
had done some work here but discovered Prima and switched

to his own implementation which is PDL::Graphics::Prima

PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.aDefinitely looking for an Alien::HDF4 here. Figured I would

do it at the same time as the HDF5 work.


PDL::GSL::* ### (PDL-2.007) OKDetection ok, not sure where Alien::GSL stands but work for
that was part of the motivation for Joel Berger's Alien::Base
work. I would like to see a more comprehensive numeric
processing support in PDL, maybe via additional GSL bindings...



PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't buildPROJ4 pretty much builds out of the box so an Alien::PROJ4

would work and the current detection using Devel::CheckLib seems

to work. If the projection is not found in the test, then probably

a needed environment variable is not set or the datum info has
not been installed.


PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
Ditto


PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
Alien::HDF5 is in the works to support PDL::IO::HDF5 but

it is still waiting on the Alien2 push...


PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
Have a rudimentary, incomplete Alien::FFTW3 that would be
nice to make cross platform but waiting on the Alien2 push...


PDL::FFTW ### OKThis module is no longer part of the PDL core and is
buggy. Once the PDL::FFTW3 is finished, I plan to
deprecate it completely.


PDL::Graphics::Gnuplot ### OKThis worked for cygwin and I've had it work for Strawberry

Perl. I assume Rob could say if that is the case for ASPerl.


Alien::GSL ### does not detect existing libraryAlien::FFTW3 ### does not detect existing library


These need the Alien2 rework...


Should I try to create patches for related modules? Or do you plan to unify lib detection over all PDL related modules? (Alien::* approach of Devel::CheckLib approach of other?)

If you gimme a short crash course how a build system for PDL modules work I can start with some hacking.

--
kmx




--------------------------------------------------------------------------------

[Chris Marshall]

Hi Ed-

That has always been the plan. The thought was that by
using Alien modules, the reliability of the PDL modules
depending on externals would be improved and simplified.
Then the work to transition from PDL2 core architecture to
PDL3 stuff could focus on *only* the underlying engine for
computation rather than dealing with all the other stuff.

My hope was to put the 64bit indexing support and cleaner
dependency detection and use as part of the support for
PDL3 development which, arguably, will be disruptive. The
idea is that PDL2 would continue to work and be supported
but that the PDL3 effort would focus on the computational
core with a goal of back-compatibility with the PDL2 base.

Should be possible since lots of the code is automatically
generated, see PDL::PP, and a replacement using PDL3
technology instead of the static PDL2 approach should be
transparent....at least that is the goal.

Cheers,
Chris

P.S. There is no waiting for PDL3 except for a lack of
time to develop. I've finally learned enough perl and PDL
to make some significant changes but now am constrained
by the time to implement.

I mentioned starting with PDL3 as a clean point for the
ExtUtils::Depends stuff since I don'tunderstand it and it
should be simpler to fold it in to a more compact code
base than the spaghetti kitchen sink that is the PDL 2.x
core distribution.

The Alien::XXX support would greatly simplify the PDL-2.x
code and is relatively simple to add. No need to wait for
PDL3 for that (especially since I already have a lot of
experience with the PDL-2.x code base). The start for
many of the Alien::XXX modules could be the current
detection code in PDL-2.007. However, I think the Alien2
proposals are essential to have it be truly effective and
successful---not just for PDL but for other perl modules
with external dependencies.


On Tue, Apr 15, 2014 at 7:06 PM, Ed . <***@hotmail.com> wrote:

> As someone who is interested in getting involved in this Alien*::*
> business – what do people think about taking the existing PDL 2.007, and
> migrating that towards using Alien modules, using CPAN underscore releases
> and the smoke-testing network to make sure things don’t work any worse than
> before, and hopefully work better?
>
> I believe that an incremental approach is the way forward here –
> evolution, not revolution. I don’t see it as something that needs to wait
> for PDL3. What do you think?
>
> Cheers,
> Ed J
>
> *From:* Chris Marshall <***@gmail.com>
> *Sent:* Tuesday, April 15, 2014 10:20 PM
> *To:* kmx <***@volny.cz>
> *Cc:* pdl-porters <PDL-***@jach.hawaii.edu>
> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user experience
>
> < adding pdl-porters cc to this discussion >
>
> Hi kmx-
>
> My plan is to provide Alien::XXX modules for all the external
> dependencies but, as you are aware, I've been chewing on
> the problem for a while without much progress.
>
> Part of the problem is that the current approach to implement
> Alien modules from the original manifesto is too vague and
> currently results in Alien::XXX modules that often only work
> on a singleplatform or type of platform, don't give a clue in the
> docs ortest results about whether or not it actually works, and
> doesn't even handle consistently, correctly, or at all just
> detecting and using an existing dependency!
>
> I have some concrete ideas for a fix with the start here:
>
> >
> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html
>
> I haven't had a lot of time to finish the second post on an
> example Alien2 implementation with discussion. The hope
> was that we could refactor the existing Alien::Base to
> support the Alien2 improvements.
>
> It might be simpler to finish some example "Alien2::XXX"
> implementations to support the PDL2 -> PDL3 transition
> (with the goal of splitting out non-core dependencies based
> modules from the PDL core distribution and replacing the
> all-in-one install by a bundle install). The "show me the code"
> approach has the plus it will help PDL and provide some
> concrete examples of what I mean in Alien2.
>
> Below I've commented on the current status of dependency
> detection in the PDL-2.007 distribution. This is off the top of
> my head. Clarification and updates welcome.
>
> Cheers,
> Chris
>
>
> On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz> wrote:
>
>> Hi Chris,
>>
>> as for the ConsoleZ you are right.
>>
>> Maybe lets focus on external library detection. Here is my observation
>> (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):
>>
>> PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
>>
>>
> Alien::Curses or Alien::NCurses would work but I really want to
> have a better data visualization tool for PDL. Text based should
> be the fallback, not the default standard approach.
>
>> PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
>>
>>
> I've only just been able to compile the latest PLplot library to
> revisit this detection and use in the module. David Mertens
> had done some work here but discovered Prima and switched
> to his own implementation which is PDL::Graphics::Prima
>
>> PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.a
>>
>> Definitely looking for an Alien::HDF4 here. Figured I would
> do it at the same time as the HDF5 work.
>
>
>> PDL::GSL::* ### (PDL-2.007) OK
>>
>> Detection ok, not sure where Alien::GSL stands but work for
> that was part of the motivation for Joel Berger's Alien::Base
> work. I would like to see a more comprehensive numeric
> processing support in PDL, maybe via additional GSL bindings...
>
>
>
>> PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't build
>>
>> PROJ4 pretty much builds out of the box so an Alien::PROJ4
> would work and the current detection using Devel::CheckLib seems
> to work. If the projection is not found in the test, then probably
> a needed environment variable is not set or the datum info has
> not been installed.
>
>
>> PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
>>
>>
> Ditto
>
>
>> PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
>>
>>
> Alien::HDF5 is in the works to support PDL::IO::HDF5 but
> it is still waiting on the Alien2 push...
>
>
>> PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
>>
>>
> Have a rudimentary, incomplete Alien::FFTW3 that would be
> nice to make cross platform but waiting on the Alien2 push...
>
>
>> PDL::FFTW ### OK
>>
>> This module is no longer part of the PDL core and is
> buggy. Once the PDL::FFTW3 is finished, I plan to
> deprecate it completely.
>
>
>> PDL::Graphics::Gnuplot ### OK
>>
>> This worked for cygwin and I've had it work for Strawberry
> Perl. I assume Rob could say if that is the case for ASPerl.
>
>
>> Alien::GSL ### does not detect existing library
>>
>> Alien::FFTW3 ### does not detect existing library
>>
>>
>>
> These need the Alien2 rework...
>
>
>> Should I try to create patches for related modules? Or do you plan to
>> unify lib detection over all PDL related modules? (Alien::* approach of
>> Devel::CheckLib approach of other?)
>>
>> If you gimme a short crash course how a build system for PDL modules work
>> I can start with some hacking.
>>
>> --
>> kmx
>>
>
>
> ------------------------------
> _______________________________________________
> PDL-porters mailing list
> PDL-***@jach.hawaii.edu
> http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
>
>

[Ed .]

Hi Chris,

Fair enough on the general plans!

ExtUtils::Depends is not a complex module; supporting it is literally no more than this (in Gimp-Perl’s Makefile.PL):
use ExtUtils::Depends;
my $pkg = new ExtUtils::Depends Gimp, Gtk2;
$pkg->set_inc($cfg{GIMP_CFLAGS});
$pkg->add_pm(map { ($_ => "\$(INST_LIB)/$_") } qw(Gimp.pm));
# there would be a –>set_libs line but it is done in another dir... for now
$pkg->save_config('Gimp/IFiles.pm');
WriteMakefile(
[...]
$pkg->get_makefile_vars,
);
You have no idea how much code with the Gtk2 and Glib config-detection I was able to delete. When (hopefully not if) PDL supports it too, I will be able to delete the PDL-specific parts and replace with just “, PDL” on the “new” line.

You refer to the spaghetti of the current PDL-2.007 – call me strange, but I actually enjoyed untangling the “challenging” (accreted over many years and 3 maintainers) build stuff in GP. I will be pleased to get stuck in here, if I get some buy-in. Evolution, not revolution (for the build stuff).

Cheers,
Ed

From: Chris Marshall
Sent: Wednesday, April 16, 2014 12:20 AM
To: Ed .
Cc: kmx ; pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

Hi Ed-


That has always been the plan. The thought was that by

using Alien modules, the reliability of the PDL modules
depending on externals would be improved and simplified.

Then the work to transition from PDL2 core architecture to
PDL3 stuff could focus on *only* the underlying engine for

computation rather than dealing with all the other stuff.


My hope was to put the 64bit indexing support and cleaner
dependency detection and use as part of the support for

PDL3 development which, arguably, will be disruptive. The
idea is that PDL2 would continue to work and be supported
but that the PDL3 effort would focus on the computational
core with a goal of back-compatibility with the PDL2 base.


Should be possible since lots of the code is automatically

generated, see PDL::PP, and a replacement using PDL3
technology instead of the static PDL2 approach should be

transparent....at least that is the goal.


Cheers,

Chris


P.S. There is no waiting for PDL3 except for a lack of

time to develop. I've finally learned enough perl and PDL
to make some significant changes but now am constrained

by the time to implement.

I mentioned starting with PDL3 as a clean point for the
ExtUtils::Depends stuff since I don'tunderstand it and it
should be simpler to fold it in to a more compact code
base than the spaghetti kitchen sink that is the PDL 2.x
core distribution.


The Alien::XXX support would greatly simplify the PDL-2.x

code and is relatively simple to add. No need to wait for
PDL3 for that (especially since I already have a lot of

experience with the PDL-2.x code base). The start for
many of the Alien::XXX modules could be the current
detection code in PDL-2.007. However, I think the Alien2

proposals are essential to have it be truly effective and
successful---not just for PDL but for other perl modules
with external dependencies.




On Tue, Apr 15, 2014 at 7:06 PM, Ed . <***@hotmail.com> wrote:

As someone who is interested in getting involved in this Alien*::* business – what do people think about taking the existing PDL 2.007, and migrating that towards using Alien modules, using CPAN underscore releases and the smoke-testing network to make sure things don’t work any worse than before, and hopefully work better?

I believe that an incremental approach is the way forward here – evolution, not revolution. I don’t see it as something that needs to wait for PDL3. What do you think?

Cheers,
Ed J

From: Chris Marshall
Sent: Tuesday, April 15, 2014 10:20 PM
To: kmx
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

< adding pdl-porters cc to this discussion >


Hi kmx-


My plan is to provide Alien::XXX modules for all the external
dependencies but, as you are aware, I've been chewing on
the problem for a while without much progress.

Part of the problem is that the current approach to implement
Alien modules from the original manifesto is too vague and
currently results in Alien::XXX modules that often only work
on a singleplatform or type of platform, don't give a clue in the
docs ortest results about whether or not it actually works, and

doesn't even handle consistently, correctly, or at all just

detecting and using an existing dependency!


I have some concrete ideas for a fix with the start here:

> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html


I haven't had a lot of time to finish the second post on an

example Alien2 implementation with discussion. The hope
was that we could refactor the existing Alien::Base to
support the Alien2 improvements.

It might be simpler to finish some example "Alien2::XXX"
implementations to support the PDL2 -> PDL3 transition
(with the goal of splitting out non-core dependencies based
modules from the PDL core distribution and replacing the

all-in-one install by a bundle install). The "show me the code"
approach has the plus it will help PDL and provide some
concrete examples of what I mean in Alien2.


Below I've commented on the current status of dependency
detection in the PDL-2.007 distribution. This is off the top of
my head. Clarification and updates welcome.


Cheers,
Chris




On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz> wrote:

Hi Chris,

as for the ConsoleZ you are right.

Maybe lets focus on external library detection. Here is my observation (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):


PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
Alien::Curses or Alien::NCurses would work but I really want to
have a better data visualization tool for PDL. Text based should
be the fallback, not the default standard approach.

PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
I've only just been able to compile the latest PLplot library to

revisit this detection and use in the module. David Mertens
had done some work here but discovered Prima and switched

to his own implementation which is PDL::Graphics::Prima

PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.aDefinitely looking for an Alien::HDF4 here. Figured I would

do it at the same time as the HDF5 work.


PDL::GSL::* ### (PDL-2.007) OKDetection ok, not sure where Alien::GSL stands but work for
that was part of the motivation for Joel Berger's Alien::Base
work. I would like to see a more comprehensive numeric
processing support in PDL, maybe via additional GSL bindings...



PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't buildPROJ4 pretty much builds out of the box so an Alien::PROJ4

would work and the current detection using Devel::CheckLib seems

to work. If the projection is not found in the test, then probably

a needed environment variable is not set or the datum info has
not been installed.


PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
Ditto


PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
Alien::HDF5 is in the works to support PDL::IO::HDF5 but

it is still waiting on the Alien2 push...


PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
Have a rudimentary, incomplete Alien::FFTW3 that would be
nice to make cross platform but waiting on the Alien2 push...


PDL::FFTW ### OKThis module is no longer part of the PDL core and is
buggy. Once the PDL::FFTW3 is finished, I plan to
deprecate it completely.


PDL::Graphics::Gnuplot ### OKThis worked for cygwin and I've had it work for Strawberry

Perl. I assume Rob could say if that is the case for ASPerl.


Alien::GSL ### does not detect existing libraryAlien::FFTW3 ### does not detect existing library


These need the Alien2 rework...


Should I try to create patches for related modules? Or do you plan to unify lib detection over all PDL related modules? (Alien::* approach of Devel::CheckLib approach of other?)

If you gimme a short crash course how a build system for PDL modules work I can start with some hacking.

--
kmx

[Ed .]

kmx,

You are quite right that EU::D is intended for XS-module use.

In fact, the first argument to ExtUtils::Depends->new is “me” – the code below was cut and pasted from Gimp-Perl, hence “Gimp” (it does actually use the GIMP libraries and includes, of course, and it saves and exports them via EU::D for other hypothetical XS modules that want to do the same). Its functionality is quite close to Alien::Base-derived Alien classes. I intend to make these various classes (EU::D, Alien::*, Inline) work together more smoothly.

I did happen to know what you were saying, but I don’t mind you making sure!

Cheers,
Ed

From: kmx
Sent: Wednesday, April 16, 2014 8:24 AM
To: Ed . ; Chris Marshall
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

Ed,

AFAIK ExtUtils::Depends is slightly different story than Alien::WhatEver

ExtUtils::Depends IMO handles scenario where:
- you have an XS module - let's say PDL::XS::Core (implemented in XS)
- you want to provide an interface for other modules to call directly XS functions of PDL::XS::Core
- if you install PDL::XS::Core you will get
....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
....lib/auto/PDL/XS/Core/Core.a - this extra
- now you have PDL::Something::Special written in XS where you want to directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
- for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core + PDL::Something::Special)

In your example 'my $pkg = new ExtUtils::Depends Gimp, Gtk2;' does not mean you are detecting gimp/gtk2 libraries (like Alien::* does) but you are getting compile/link options for linking with Gimp and Gtk2 perl modules (written in XS). The gtk2 library detection is done via ExtUtils::PkgConfig (see Gtk2's Makefile.PL)

Anyway in PDL land we probably want both:
- ExtUtils::Depends - to allow PDL-related modules directly calling XS functions from PDL core
- Alien::* - to get the correct flags for linking with external libraries
- we even want to be able to combine both

Skip this email if you already know all the above.

--
kmx



On 16.4.2014 1:59, Ed . wrote:

Hi Chris,

Fair enough on the general plans!

ExtUtils::Depends is not a complex module; supporting it is literally no more than this (in Gimp-Perl’s Makefile.PL):
use ExtUtils::Depends;
my $pkg = new ExtUtils::Depends Gimp, Gtk2;
$pkg->set_inc($cfg{GIMP_CFLAGS});
$pkg->add_pm(map { ($_ => "\$(INST_LIB)/$_") } qw(Gimp.pm));
# there would be a –>set_libs line but it is done in another dir... for now
$pkg->save_config('Gimp/IFiles.pm');
WriteMakefile(
[...]
$pkg->get_makefile_vars,
);
You have no idea how much code with the Gtk2 and Glib config-detection I was able to delete. When (hopefully not if) PDL supports it too, I will be able to delete the PDL-specific parts and replace with just “, PDL” on the “new” line.

You refer to the spaghetti of the current PDL-2.007 – call me strange, but I actually enjoyed untangling the “challenging” (accreted over many years and 3 maintainers) build stuff in GP. I will be pleased to get stuck in here, if I get some buy-in. Evolution, not revolution (for the build stuff).

Cheers,
Ed

From: Chris Marshall
Sent: Wednesday, April 16, 2014 12:20 AM
To: Ed .
Cc: kmx ; pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

Hi Ed-


That has always been the plan. The thought was that by

using Alien modules, the reliability of the PDL modules
depending on externals would be improved and simplified.

Then the work to transition from PDL2 core architecture to
PDL3 stuff could focus on *only* the underlying engine for

computation rather than dealing with all the other stuff.


My hope was to put the 64bit indexing support and cleaner
dependency detection and use as part of the support for

PDL3 development which, arguably, will be disruptive. The
idea is that PDL2 would continue to work and be supported
but that the PDL3 effort would focus on the computational
core with a goal of back-compatibility with the PDL2 base.


Should be possible since lots of the code is automatically

generated, see PDL::PP, and a replacement using PDL3
technology instead of the static PDL2 approach should be

transparent....at least that is the goal.


Cheers,

Chris


P.S. There is no waiting for PDL3 except for a lack of

time to develop. I've finally learned enough perl and PDL
to make some significant changes but now am constrained

by the time to implement.

I mentioned starting with PDL3 as a clean point for the
ExtUtils::Depends stuff since I don'tunderstand it and it
should be simpler to fold it in to a more compact code
base than the spaghetti kitchen sink that is the PDL 2.x
core distribution.


The Alien::XXX support would greatly simplify the PDL-2.x

code and is relatively simple to add. No need to wait for
PDL3 for that (especially since I already have a lot of

experience with the PDL-2.x code base). The start for
many of the Alien::XXX modules could be the current
detection code in PDL-2.007. However, I think the Alien2

proposals are essential to have it be truly effective and
successful---not just for PDL but for other perl modules
with external dependencies.




On Tue, Apr 15, 2014 at 7:06 PM, Ed . <***@hotmail.com> wrote:

As someone who is interested in getting involved in this Alien*::* business – what do people think about taking the existing PDL 2.007, and migrating that towards using Alien modules, using CPAN underscore releases and the smoke-testing network to make sure things don’t work any worse than before, and hopefully work better?

I believe that an incremental approach is the way forward here – evolution, not revolution. I don’t see it as something that needs to wait for PDL3. What do you think?

Cheers,
Ed J

From: Chris Marshall
Sent: Tuesday, April 15, 2014 10:20 PM
To: kmx
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

< adding pdl-porters cc to this discussion >


Hi kmx-


My plan is to provide Alien::XXX modules for all the external
dependencies but, as you are aware, I've been chewing on
the problem for a while without much progress.

Part of the problem is that the current approach to implement
Alien modules from the original manifesto is too vague and
currently results in Alien::XXX modules that often only work
on a singleplatform or type of platform, don't give a clue in the
docs ortest results about whether or not it actually works, and

doesn't even handle consistently, correctly, or at all just

detecting and using an existing dependency!


I have some concrete ideas for a fix with the start here:

> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html


I haven't had a lot of time to finish the second post on an

example Alien2 implementation with discussion. The hope
was that we could refactor the existing Alien::Base to
support the Alien2 improvements.

It might be simpler to finish some example "Alien2::XXX"
implementations to support the PDL2 -> PDL3 transition
(with the goal of splitting out non-core dependencies based
modules from the PDL core distribution and replacing the

all-in-one install by a bundle install). The "show me the code"
approach has the plus it will help PDL and provide some
concrete examples of what I mean in Alien2.


Below I've commented on the current status of dependency
detection in the PDL-2.007 distribution. This is off the top of
my head. Clarification and updates welcome.


Cheers,
Chris




On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz> wrote:

Hi Chris,

as for the ConsoleZ you are right.

Maybe lets focus on external library detection. Here is my observation (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):


PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
Alien::Curses or Alien::NCurses would work but I really want to
have a better data visualization tool for PDL. Text based should
be the fallback, not the default standard approach.

PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
I've only just been able to compile the latest PLplot library to

revisit this detection and use in the module. David Mertens
had done some work here but discovered Prima and switched

to his own implementation which is PDL::Graphics::Prima

PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.aDefinitely looking for an Alien::HDF4 here. Figured I would

do it at the same time as the HDF5 work.


PDL::GSL::* ### (PDL-2.007) OKDetection ok, not sure where Alien::GSL stands but work for
that was part of the motivation for Joel Berger's Alien::Base
work. I would like to see a more comprehensive numeric
processing support in PDL, maybe via additional GSL bindings...



PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't buildPROJ4 pretty much builds out of the box so an Alien::PROJ4

would work and the current detection using Devel::CheckLib seems

to work. If the projection is not found in the test, then probably

a needed environment variable is not set or the datum info has
not been installed.


PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
Ditto


PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
Alien::HDF5 is in the works to support PDL::IO::HDF5 but

it is still waiting on the Alien2 push...


PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
Have a rudimentary, incomplete Alien::FFTW3 that would be
nice to make cross platform but waiting on the Alien2 push...


PDL::FFTW ### OKThis module is no longer part of the PDL core and is
buggy. Once the PDL::FFTW3 is finished, I plan to
deprecate it completely.


PDL::Graphics::Gnuplot ### OKThis worked for cygwin and I've had it work for Strawberry

Perl. I assume Rob could say if that is the case for ASPerl.


Alien::GSL ### does not detect existing libraryAlien::FFTW3 ### does not detect existing library


These need the Alien2 rework...


Should I try to create patches for related modules? Or do you plan to unify lib detection over all PDL related modules? (Alien::* approach of Devel::CheckLib approach of other?)

If you gimme a short crash course how a build system for PDL modules work I can start with some hacking.

--
kmx

[kmx]

Just to make it clear what I am afraid of in Alien::* discussion:
- AFAIK none of the 26 Alien::Base based modules available on CPAN work
with strawberry perl
- these Alien::* fail even if there is corresponding library (*.h/*.a)
bundled with strawberry perl
- Alien::Base seems to handle only installation part, detection at least on
MS Windows seems to be poor or none

That's why I suggest to focus on detection part as I exactly know how
complicated it is to build all those libraries on MS Windows and I simply
do not believe it is a good idea to implement it (and support it) in
Alien::* modules

But of course it is just my opinion :)

--
kmx


On 16.4.2014 10:38, Ed . wrote:
> kmx,
> You are quite right that EU::D is intended for XS-module use.
> In fact, the first argument to ExtUtils::Depends->new is "me" -- the code
> below was cut and pasted from Gimp-Perl, hence "Gimp" (it does actually
> use the GIMP libraries and includes, of course, and it saves and exports
> them via EU::D for other hypothetical XS modules that want to do the
> same). Its functionality is quite close to Alien::Base-derived Alien
> classes. I intend to make these various classes (EU::D, Alien::*, Inline)
> work together more smoothly.
> I did happen to know what you were saying, but I don't mind you making sure!
> Cheers,
> Ed
> *From:* kmx <mailto:***@volny.cz>
> *Sent:* Wednesday, April 16, 2014 8:24 AM
> *To:* Ed . <mailto:***@hotmail.com> ; Chris Marshall
> <mailto:***@gmail.com>
> *Cc:* pdl-porters <mailto:PDL-***@jach.hawaii.edu>
> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user experience
> Ed,
>
> AFAIK ExtUtils::Depends is slightly different story than Alien::WhatEver
>
> ExtUtils::Depends IMO handles scenario where:
> - you have an XS module - let's say PDL::XS::Core (implemented in XS)
> - you want to provide an interface for other modules to call directly XS
> functions of PDL::XS::Core
> - if you install PDL::XS::Core you will get
> ....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
> ....lib/auto/PDL/XS/Core/Core.a - this extra
> - now you have PDL::Something::Special written in XS where you want to
> directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
> - for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core
> + PDL::Something::Special)
>
> In your example 'my $pkg = new ExtUtils::Depends Gimp, Gtk2;' does not
> mean you are detecting gimp/gtk2 libraries (like Alien::* does) but you
> are getting compile/link options for linking with Gimp and Gtk2 perl
> modules (written in XS). The gtk2 library detection is done via
> ExtUtils::PkgConfig (see Gtk2's Makefile.PL)
>
> Anyway in PDL land we probably want both:
> - ExtUtils::Depends - to allow PDL-related modules directly calling XS
> functions from PDL core
> - Alien::* - to get the correct flags for linking with external libraries
> - we even want to be able to combine both
>
> Skip this email if you already know all the above.
>
> --
> kmx
>
>
> On 16.4.2014 1:59, Ed . wrote:
>> Hi Chris,
>> Fair enough on the general plans!
>> ExtUtils::Depends is not a complex module; supporting it is literally no
>> more than this (in Gimp-Perl's Makefile.PL):
>>
>> use ExtUtils::Depends;
>> my $pkg = new ExtUtils::Depends Gimp, Gtk2;
>> $pkg->set_inc($cfg{GIMP_CFLAGS});
>> $pkg->add_pm(map { ($_ => "\$(INST_LIB)/$_") } qw(Gimp.pm));
>> # there would be a -->set_libs line but it is done in another dir...
>> for now
>> $pkg->save_config('Gimp/IFiles.pm');
>> WriteMakefile(
>> [...]
>> $pkg->get_makefile_vars,
>> );
>>
>> You have no idea how much code with the Gtk2 and Glib config-detection I
>> was able to delete. When (hopefully not if) PDL supports it too, I will
>> be able to delete the PDL-specific parts and replace with just ", PDL"
>> on the "new" line.
>> You refer to the spaghetti of the current PDL-2.007 -- call me strange,
>> but I actually enjoyed untangling the "challenging" (accreted over many
>> years and 3 maintainers) build stuff in GP. I will be pleased to get
>> stuck in here, if I get some buy-in. Evolution, not revolution (for the
>> build stuff).
>> Cheers,
>> Ed
>> *From:* Chris Marshall <mailto:***@gmail.com>
>> *Sent:* Wednesday, April 16, 2014 12:20 AM
>> *To:* Ed . <mailto:***@hotmail.com>
>> *Cc:* kmx <mailto:***@volny.cz> ; pdl-porters
>> <mailto:PDL-***@jach.hawaii.edu>
>> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user experience
>> Hi Ed-
>>
>> That has always been the plan. The thought was that by
>> using Alien modules, the reliability of the PDL modules
>> depending on externals would be improved and simplified.
>> Then the work to transition from PDL2 core architecture to
>> PDL3 stuff could focus on *only* the underlying engine for
>> computation rather than dealing with all the other stuff.
>>
>> My hope was to put the 64bit indexing support and cleaner
>> dependency detection and use as part of the support for
>> PDL3 development which, arguably, will be disruptive. The
>> idea is that PDL2 would continue to work and be supported
>> but that the PDL3 effort would focus on the computational
>> core with a goal of back-compatibility with the PDL2 base.
>>
>> Should be possible since lots of the code is automatically
>> generated, see PDL::PP, and a replacement using PDL3
>> technology instead of the static PDL2 approach should be
>> transparent....at least that is the goal.
>>
>> Cheers,
>> Chris
>>
>> P.S. There is no waiting for PDL3 except for a lack of
>> time to develop. I've finally learned enough perl and PDL
>> to make some significant changes but now am constrained
>> by the time to implement.
>>
>> I mentioned starting with PDL3 as a clean point for the
>> ExtUtils::Depends stuff since I don'tunderstand it and it
>> should be simpler to fold it in to a more compact code
>> base than the spaghetti kitchen sink that is the PDL 2.x
>> core distribution.
>> The Alien::XXX support would greatly simplify the PDL-2.x
>> code and is relatively simple to add. No need to wait for
>> PDL3 for that (especially since I already have a lot of
>> experience with the PDL-2.x code base). The start for
>> many of the Alien::XXX modules could be the current
>> detection code in PDL-2.007. However, I think the Alien2
>> proposals are essential to have it be truly effective and
>> successful---not just for PDL but for other perl modules
>> with external dependencies.
>>
>>
>> On Tue, Apr 15, 2014 at 7:06 PM, Ed . <***@hotmail.com
>> <mailto:***@hotmail.com>> wrote:
>>
>> As someone who is interested in getting involved in this Alien*::*
>> business -- what do people think about taking the existing PDL
>> 2.007, and migrating that towards using Alien modules, using CPAN
>> underscore releases and the smoke-testing network to make sure
>> things don't work any worse than before, and hopefully work better?
>> I believe that an incremental approach is the way forward here --
>> evolution, not revolution. I don't see it as something that needs to
>> wait for PDL3. What do you think?
>> Cheers,
>> Ed J
>> *From:* Chris Marshall <mailto:***@gmail.com>
>> *Sent:* Tuesday, April 15, 2014 10:20 PM
>> *To:* kmx <mailto:***@volny.cz>
>> *Cc:* pdl-porters <mailto:PDL-***@jach.hawaii.edu>
>> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user
>> experience
>> < adding pdl-porters cc to this discussion >
>>
>> Hi kmx-
>>
>> My plan is to provide Alien::XXX modules for all the external
>> dependencies but, as you are aware, I've been chewing on
>> the problem for a while without much progress.
>>
>> Part of the problem is that the current approach to implement
>> Alien modules from the original manifesto is too vague and
>> currently results in Alien::XXX modules that often only work
>> on a singleplatform or type of platform, don't give a clue in the
>> docs ortest results about whether or not it actually works, and
>> doesn't even handle consistently, correctly, or at all just
>> detecting and using an existing dependency!
>>
>> I have some concrete ideas for a fix with the start here:
>>
>> >
>> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html
>>
>> I haven't had a lot of time to finish the second post on an
>> example Alien2 implementation with discussion. The hope
>> was that we could refactor the existing Alien::Base to
>> support the Alien2 improvements.
>>
>> It might be simpler to finish some example "Alien2::XXX"
>> implementations to support the PDL2 -> PDL3 transition
>> (with the goal of splitting out non-core dependencies based
>> modules from the PDL core distribution and replacing the
>> all-in-one install by a bundle install). The "show me the code"
>> approach has the plus it will help PDL and provide some
>> concrete examples of what I mean in Alien2.
>>
>> Below I've commented on the current status of dependency
>> detection in the PDL-2.007 distribution. This is off the top of
>> my head. Clarification and updates welcome.
>>
>> Cheers,
>> Chris
>>
>>
>> On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz
>> <mailto:***@volny.cz>> wrote:
>>
>> Hi Chris,
>>
>> as for the ConsoleZ you are right.
>>
>> Maybe lets focus on external library detection. Here is my
>> observation (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):
>>
>> PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
>>
>> Alien::Curses or Alien::NCurses would work but I really want to
>> have a better data visualization tool for PDL. Text based should
>> be the fallback, not the default standard approach.
>>
>> PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
>>
>> I've only just been able to compile the latest PLplot library to
>> revisit this detection and use in the module. David Mertens
>> had done some work here but discovered Prima and switched
>> to his own implementation which is PDL::Graphics::Prima
>>
>> PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.a
>>
>> Definitely looking for an Alien::HDF4 here. Figured I would
>> do it at the same time as the HDF5 work.
>>
>> PDL::GSL::* ### (PDL-2.007) OK
>>
>> Detection ok, not sure where Alien::GSL stands but work for
>> that was part of the motivation for Joel Berger's Alien::Base
>> work. I would like to see a more comprehensive numeric
>> processing support in PDL, maybe via additional GSL bindings...
>>
>> PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't build
>>
>> PROJ4 pretty much builds out of the box so an Alien::PROJ4
>> would work and the current detection using Devel::CheckLib seems
>> to work. If the projection is not found in the test, then probably
>> a needed environment variable is not set or the datum info has
>> not been installed.
>>
>> PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
>>
>> Ditto
>>
>> PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
>>
>> Alien::HDF5 is in the works to support PDL::IO::HDF5 but
>> it is still waiting on the Alien2 push...
>>
>> PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
>>
>> Have a rudimentary, incomplete Alien::FFTW3 that would be
>> nice to make cross platform but waiting on the Alien2 push...
>>
>> PDL::FFTW ### OK
>>
>> This module is no longer part of the PDL core and is
>> buggy. Once the PDL::FFTW3 is finished, I plan to
>> deprecate it completely.
>>
>> PDL::Graphics::Gnuplot ### OK
>>
>> This worked for cygwin and I've had it work for Strawberry
>> Perl. I assume Rob could say if that is the case for ASPerl.
>>
>> Alien::GSL ### does not detect existing library
>>
>> Alien::FFTW3 ### does not detect existing library
>>
>>
>> These need the Alien2 rework...
>>
>> Should I try to create patches for related modules? Or do you
>> plan to unify lib detection over all PDL related modules?
>> (Alien::* approach of Devel::CheckLib approach of other?)
>>
>> If you gimme a short crash course how a build system for PDL
>> modules work I can start with some hacking.
>>
>> --
>> kmx
>>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> PDL-porters mailing list
>> PDL-***@jach.hawaii.edu <mailto:PDL-***@jach.hawaii.edu>
>> http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
>>
>
>
>
> _______________________________________________
> PDL-porters mailing list
> PDL-***@jach.hawaii.edu
> http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

[Chris Marshall]

I agree completely. Maybe to avoid confusion, we should
consider the plan to be to provide Alien2::XXX for the various
PDL external dependencies since, by definition, that does
detect and use first. That makes it clear that something
different is being done.

I don't like the idea of propagating a new namespace but
the challenge I'm seeing with accepting a slight course
change in Alien::XXX implementation might be people
using the same name for completely different things and
their assumptions.

For me, the thing an Alien2 module needs to do is to
make available a non-perl dependency for perl to use *and*
be implemented cross-platform. It doesn't do much good
to have a "working" Alien::XXX on a linux with full package
manager support which immediately breaks on windows
with no such infrastructure.

--Chris


On Wed, Apr 16, 2014 at 6:08 AM, kmx <***@atlas.cz> wrote:

> Just to make it clear what I am afraid of in Alien::* discussion:
> - AFAIK none of the 26 Alien::Base based modules available on CPAN work
> with strawberry perl
> - these Alien::* fail even if there is corresponding library (*.h/*.a)
> bundled with strawberry perl
> - Alien::Base seems to handle only installation part, detection at least
> on MS Windows seems to be poor or none
>
> That's why I suggest to focus on detection part as I exactly know how
> complicated it is to build all those libraries on MS Windows and I simply
> do not believe it is a good idea to implement it (and support it) in
> Alien::* modules
>
> But of course it is just my opinion :)
>
> --
> kmx
>
>
> On 16.4.2014 10:38, Ed . wrote:
>
> kmx,
>
> You are quite right that EU::D is intended for XS-module use.
>
> In fact, the first argument to ExtUtils::Depends->new is “me” – the code
> below was cut and pasted from Gimp-Perl, hence “Gimp” (it does actually use
> the GIMP libraries and includes, of course, and it saves and exports them
> via EU::D for other hypothetical XS modules that want to do the same). Its
> functionality is quite close to Alien::Base-derived Alien classes. I intend
> to make these various classes (EU::D, Alien::*, Inline) work together more
> smoothly.
>
> I did happen to know what you were saying, but I don’t mind you making
> sure!
>
> Cheers,
> Ed
>
>
>

[kmx]

Ed,

AFAIK ExtUtils::Depends is slightly different story than Alien::WhatEver

ExtUtils::Depends IMO handles scenario where:
- you have an XS module - let's say PDL::XS::Core (implemented in XS)
- you want to provide an interface for other modules to call directly XS
functions of PDL::XS::Core
- if you install PDL::XS::Core you will get
....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
....lib/auto/PDL/XS/Core/Core.a - this extra
- now you have PDL::Something::Special written in XS where you want to
directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
- for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core +
PDL::Something::Special)

In your example 'my $pkg = new ExtUtils::Depends Gimp, Gtk2;' does not mean
you are detecting gimp/gtk2 libraries (like Alien::* does) but you are
getting compile/link options for linking with Gimp and Gtk2 perl modules
(written in XS). The gtk2 library detection is done via ExtUtils::PkgConfig
(see Gtk2's Makefile.PL)

Anyway in PDL land we probably want both:
- ExtUtils::Depends - to allow PDL-related modules directly calling XS
functions from PDL core
- Alien::* - to get the correct flags for linking with external libraries
- we even want to be able to combine both

Skip this email if you already know all the above.

--
kmx


On 16.4.2014 1:59, Ed . wrote:
> Hi Chris,
> Fair enough on the general plans!
> ExtUtils::Depends is not a complex module; supporting it is literally no
> more than this (in Gimp-Perl’s Makefile.PL):
>
> use ExtUtils::Depends;
> my $pkg = new ExtUtils::Depends Gimp, Gtk2;
> $pkg->set_inc($cfg{GIMP_CFLAGS});
> $pkg->add_pm(map { ($_ => "\$(INST_LIB)/$_") } qw(Gimp.pm));
> # there would be a –>set_libs line but it is done in another dir...
> for now
> $pkg->save_config('Gimp/IFiles.pm');
> WriteMakefile(
> [...]
> $pkg->get_makefile_vars,
> );
>
> You have no idea how much code with the Gtk2 and Glib config-detection I
> was able to delete. When (hopefully not if) PDL supports it too, I will
> be able to delete the PDL-specific parts and replace with just “, PDL” on
> the “new” line.
> You refer to the spaghetti of the current PDL-2.007 – call me strange,
> but I actually enjoyed untangling the “challenging” (accreted over many
> years and 3 maintainers) build stuff in GP. I will be pleased to get
> stuck in here, if I get some buy-in. Evolution, not revolution (for the
> build stuff).
> Cheers,
> Ed
> *From:* Chris Marshall <mailto:***@gmail.com>
> *Sent:* Wednesday, April 16, 2014 12:20 AM
> *To:* Ed . <mailto:***@hotmail.com>
> *Cc:* kmx <mailto:***@volny.cz> ; pdl-porters
> <mailto:PDL-***@jach.hawaii.edu>
> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user experience
> Hi Ed-
>
> That has always been the plan. The thought was that by
> using Alien modules, the reliability of the PDL modules
> depending on externals would be improved and simplified.
> Then the work to transition from PDL2 core architecture to
> PDL3 stuff could focus on *only* the underlying engine for
> computation rather than dealing with all the other stuff.
>
> My hope was to put the 64bit indexing support and cleaner
> dependency detection and use as part of the support for
> PDL3 development which, arguably, will be disruptive. The
> idea is that PDL2 would continue to work and be supported
> but that the PDL3 effort would focus on the computational
> core with a goal of back-compatibility with the PDL2 base.
>
> Should be possible since lots of the code is automatically
> generated, see PDL::PP, and a replacement using PDL3
> technology instead of the static PDL2 approach should be
> transparent....at least that is the goal.
>
> Cheers,
> Chris
>
> P.S. There is no waiting for PDL3 except for a lack of
> time to develop. I've finally learned enough perl and PDL
> to make some significant changes but now am constrained
> by the time to implement.
>
> I mentioned starting with PDL3 as a clean point for the
> ExtUtils::Depends stuff since I don'tunderstand it and it
> should be simpler to fold it in to a more compact code
> base than the spaghetti kitchen sink that is the PDL 2.x
> core distribution.
> The Alien::XXX support would greatly simplify the PDL-2.x
> code and is relatively simple to add. No need to wait for
> PDL3 for that (especially since I already have a lot of
> experience with the PDL-2.x code base). The start for
> many of the Alien::XXX modules could be the current
> detection code in PDL-2.007. However, I think the Alien2
> proposals are essential to have it be truly effective and
> successful---not just for PDL but for other perl modules
> with external dependencies.
>
>
> On Tue, Apr 15, 2014 at 7:06 PM, Ed . <***@hotmail.com
> <mailto:***@hotmail.com>> wrote:
>
> As someone who is interested in getting involved in this Alien*::*
> business – what do people think about taking the existing PDL 2.007,
> and migrating that towards using Alien modules, using CPAN underscore
> releases and the smoke-testing network to make sure things don’t work
> any worse than before, and hopefully work better?
> I believe that an incremental approach is the way forward here –
> evolution, not revolution. I don’t see it as something that needs to
> wait for PDL3. What do you think?
> Cheers,
> Ed J
> *From:* Chris Marshall <mailto:***@gmail.com>
> *Sent:* Tuesday, April 15, 2014 10:20 PM
> *To:* kmx <mailto:***@volny.cz>
> *Cc:* pdl-porters <mailto:PDL-***@jach.hawaii.edu>
> *Subject:* Re: [Pdl-porters] Strawberry Perl PDL - better user experience
> < adding pdl-porters cc to this discussion >
>
> Hi kmx-
>
> My plan is to provide Alien::XXX modules for all the external
> dependencies but, as you are aware, I've been chewing on
> the problem for a while without much progress.
>
> Part of the problem is that the current approach to implement
> Alien modules from the original manifesto is too vague and
> currently results in Alien::XXX modules that often only work
> on a singleplatform or type of platform, don't give a clue in the
> docs ortest results about whether or not it actually works, and
> doesn't even handle consistently, correctly, or at all just
> detecting and using an existing dependency!
>
> I have some concrete ideas for a fix with the start here:
>
> >
> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html
>
> I haven't had a lot of time to finish the second post on an
> example Alien2 implementation with discussion. The hope
> was that we could refactor the existing Alien::Base to
> support the Alien2 improvements.
>
> It might be simpler to finish some example "Alien2::XXX"
> implementations to support the PDL2 -> PDL3 transition
> (with the goal of splitting out non-core dependencies based
> modules from the PDL core distribution and replacing the
> all-in-one install by a bundle install). The "show me the code"
> approach has the plus it will help PDL and provide some
> concrete examples of what I mean in Alien2.
>
> Below I've commented on the current status of dependency
> detection in the PDL-2.007 distribution. This is off the top of
> my head. Clarification and updates welcome.
>
> Cheers,
> Chris
>
>
> On Tue, Apr 15, 2014 at 4:49 PM, kmx <***@volny.cz
> <mailto:***@volny.cz>> wrote:
>
> Hi Chris,
>
> as for the ConsoleZ you are right.
>
> Maybe lets focus on external library detection. Here is my
> observation (based on strawberry-perl-5.19.10.3-...-PDLv02 testing):
>
> PDL::IO::Browser ### (PDL-2.007) ncursesw not detected
>
> Alien::Curses or Alien::NCurses would work but I really want to
> have a better data visualization tool for PDL. Text based should
> be the fallback, not the default standard approach.
>
> PDL::Graphics::PLplot ### (PDL-2.007) Cannot find plplot library (libplplotd.dll), skipping PDL::Graphics::PLplot
>
> I've only just been able to compile the latest PLplot library to
> revisit this detection and use in the module. David Mertens
> had done some work here but discovered Prima and switched
> to his own implementation which is PDL::Graphics::Prima
>
> PDL::IO::HDF ### (PDL-2.007) Cannot find hdf library, libdf.a
>
> Definitely looking for an Alien::HDF4 here. Figured I would
> do it at the same time as the HDF5 work.
>
> PDL::GSL::* ### (PDL-2.007) OK
>
> Detection ok, not sure where Alien::GSL stands but work for
> that was part of the motivation for Joel Berger's Alien::Base
> work. I would like to see a more comprehensive numeric
> processing support in PDL, maybe via additional GSL bindings...
>
> PDL::GIS::Proj ### (PDL-2.007) PROJ4 library found but cannot initialize projection, won't build
>
> PROJ4 pretty much builds out of the box so an Alien::PROJ4
> would work and the current detection using Devel::CheckLib seems
> to work. If the projection is not found in the test, then probably
> a needed environment variable is not set or the datum info has
> not been installed.
>
> PDL::Transform::Proj4 ### (PDL-2.007) ~~ dtto ~~
>
> Ditto
>
> PDL::IO::HDF5 ### Cannot find hdf5 library, libhdf5.so or libhdf5.a.
>
> Alien::HDF5 is in the works to support PDL::IO::HDF5 but
> it is still waiting on the Alien2 push...
>
> PDL::FFTW3 ### Sorry, Alien::FFTW3 doesn't work for Microsoft Windows
>
> Have a rudimentary, incomplete Alien::FFTW3 that would be
> nice to make cross platform but waiting on the Alien2 push...
>
> PDL::FFTW ### OK
>
> This module is no longer part of the PDL core and is
> buggy. Once the PDL::FFTW3 is finished, I plan to
> deprecate it completely.
>
> PDL::Graphics::Gnuplot ### OK
>
> This worked for cygwin and I've had it work for Strawberry
> Perl. I assume Rob could say if that is the case for ASPerl.
>
> Alien::GSL ### does not detect existing library
>
> Alien::FFTW3 ### does not detect existing library
>
>
> These need the Alien2 rework...
>
> Should I try to create patches for related modules? Or do you
> plan to unify lib detection over all PDL related modules?
> (Alien::* approach of Devel::CheckLib approach of other?)
>
> If you gimme a short crash course how a build system for PDL
> modules work I can start with some hacking.
>
> --
> kmx
>
> ---------------------------------------------------------------------------
> _______________________________________________
> PDL-porters mailing list
> PDL-***@jach.hawaii.edu <mailto:PDL-***@jach.hawaii.edu>
> http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
>

[s***@optusnet.com.au]

From: kmx
Sent: Wednesday, April 16, 2014 5:24 PM

> ExtUtils::Depends IMO handles scenario where:
> - you have an XS module - let's say PDL::XS::Core (implemented in XS)
> - you want to provide an interface for other modules to call directly XS
> functions of PDL::XS::Core
> - if you install PDL::XS::Core you will get
> ....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
> ....lib/auto/PDL/XS/Core/Core.a - this extra
> - now you have PDL::Something::Special written in XS where you want to
> directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
> - for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core
> + PDL::Something::Special)

That's pretty much the way I understand it, too.

Alternatively, couldn't you do away with ExtUtils::Depends, and simply have
PDL::Something::Special "use PDL::XS::Core;" (as is the current practice) ?

What are the *advantages* of the ExtUtils::Depends approach ? (That's the
bit about which I'm unclear.)

Cheers,
Rob

[Ed .]

I see the advantage of EU-D being that it already exists and is used. Not
using it (as at the present time) means that modules wanting to use PDL in
XS code must use the PDL-specific code to do so (calling functions from
PDL::Core::Dev).

Must every module make its own incompatible standard for no benefit? ;-)

-----Original Message-----
From: ***@optusnet.com.au
Sent: Thursday, April 17, 2014 2:07 AM
To: kmx ; Ed . ; Chris Marshall
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience


From: kmx
Sent: Wednesday, April 16, 2014 5:24 PM

> ExtUtils::Depends IMO handles scenario where:
> - you have an XS module - let's say PDL::XS::Core (implemented in XS)
> - you want to provide an interface for other modules to call directly XS
> functions of PDL::XS::Core
> - if you install PDL::XS::Core you will get
> ....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
> ....lib/auto/PDL/XS/Core/Core.a - this extra
> - now you have PDL::Something::Special written in XS where you want to
> directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
> - for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core
> + PDL::Something::Special)

That's pretty much the way I understand it, too.

Alternatively, couldn't you do away with ExtUtils::Depends, and simply have
PDL::Something::Special "use PDL::XS::Core;" (as is the current practice) ?

What are the *advantages* of the ExtUtils::Depends approach ? (That's the
bit about which I'm unclear.)

Cheers,
Rob

[kmx]

EU-D might safe some man-days (or maybe not) as it's there and seems to work.

If you look at CPAN you will find also different approaches similar to EU-D

Wx module provides XS functions that are called e.g. from Wx::TreeListCtrl
https://metacpan.org/pod/Wx::build::MakeMaker
https://metacpan.org/source/MDOOTSON/Wx-TreeListCtrl-0.13/Makefile.PL

EV module provides XS functions that are called e.g. from EV::ADNS
https://metacpan.org/pod/EV::MakeMaker
https://metacpan.org/source/MLEHMANN/EV-ADNS-2.2/Makefile.PL

But EU-D seems to be most frequently used approach. But as Chris mentioned
it seems to be too much EU-MM oriented.

--
kmx

On 17.4.2014 3:24, Ed . wrote:
> I see the advantage of EU-D being that it already exists and is used. Not
> using it (as at the present time) means that modules wanting to use PDL
> in XS code must use the PDL-specific code to do so (calling functions
> from PDL::Core::Dev).
>
> Must every module make its own incompatible standard for no benefit? ;-)
>
> -----Original Message----- From: ***@optusnet.com.au
> Sent: Thursday, April 17, 2014 2:07 AM
> To: kmx ; Ed . ; Chris Marshall
> Cc: pdl-porters
> Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience
>
>
> From: kmx
> Sent: Wednesday, April 16, 2014 5:24 PM
>
>> ExtUtils::Depends IMO handles scenario where:
>> - you have an XS module - let's say PDL::XS::Core (implemented in XS)
>> - you want to provide an interface for other modules to call directly XS
>> functions of PDL::XS::Core
>> - if you install PDL::XS::Core you will get
>> ....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
>> ....lib/auto/PDL/XS/Core/Core.a - this extra
>> - now you have PDL::Something::Special written in XS where you want to
>> directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
>> - for this you will need ExtUtils::Depends (on both sides -
>> PDL::XS::Core + PDL::Something::Special)
>
> That's pretty much the way I understand it, too.
>
> Alternatively, couldn't you do away with ExtUtils::Depends, and simply have
> PDL::Something::Special "use PDL::XS::Core;" (as is the current practice) ?
>
> What are the *advantages* of the ExtUtils::Depends approach ? (That's the
> bit about which I'm unclear.)
>
> Cheers,
> Rob
>
>
> _______________________________________________
> PDL-porters mailing list
> PDL-***@jach.hawaii.edu
> http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters
>

[Chris Marshall]

[s***@optusnet.com.au]

-----Original Message-----
From: Ed .
Sent: Thursday, April 17, 2014 11:24 AM
To: ***@optusnet.com.au
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience

> I see the advantage of EU-D being that it already exists and is used. Not
> using it (as at the present time) means that modules wanting to use PDL in
> XS code must use the PDL-specific code to do so (calling functions from
> PDL::Core::Dev).
>
> Must every module make its own incompatible standard for no benefit? ;-)

Well ... that hasn't bothered me, to date.

Don't get me wrong, I've nothing against EU::D.
It did cause me a few headaches a couple of years back trying to build
Gtk2::GladeXML on Windows, but I think those issues have now been addressed.

Cheers,
Rob

-----Original Message-----
From: ***@optusnet.com.au
Sent: Thursday, April 17, 2014 2:07 AM
To: kmx ; Ed . ; Chris Marshall
Cc: pdl-porters
Subject: Re: [Pdl-porters] Strawberry Perl PDL - better user experience


From: kmx
Sent: Wednesday, April 16, 2014 5:24 PM

> ExtUtils::Depends IMO handles scenario where:
> - you have an XS module - let's say PDL::XS::Core (implemented in XS)
> - you want to provide an interface for other modules to call directly XS
> functions of PDL::XS::Core
> - if you install PDL::XS::Core you will get
> ....lib/auto/PDL/XS/Core/Core.dll - as any other XS module
> ....lib/auto/PDL/XS/Core/Core.a - this extra
> - now you have PDL::Something::Special written in XS where you want to
> directly call directly functions from ....lib/auto/PDL/XS/Core/Core.dll
> - for this you will need ExtUtils::Depends (on both sides - PDL::XS::Core
> + PDL::Something::Special)

That's pretty much the way I understand it, too.

Alternatively, couldn't you do away with ExtUtils::Depends, and simply have
PDL::Something::Special "use PDL::XS::Core;" (as is the current practice) ?

What are the *advantages* of the ExtUtils::Depends approach ? (That's the
bit about which I'm unclear.)

Cheers,
Rob

[kmx]

Chris,

I'll cut out only Alien2 part from your last email. Let us discuss the rest
in separate thread.

On 15.4.2014 23:20, Chris Marshall wrote:
> < adding pdl-porters cc to this discussion >
>
> Hi kmx-
>
> My plan is to provide Alien::XXX modules for all the external
> dependencies but, as you are aware, I've been chewing on
> the problem for a while without much progress.
>
> Part of the problem is that the current approach to implement
> Alien modules from the original manifesto is too vague and
> currently results in Alien::XXX modules that often only work
> on a singleplatform or type of platform, don't give a clue in the
> docs ortest results about whether or not it actually works, and
> doesn't even handle consistently, correctly, or at all just
> detecting and using an existing dependency!
>
> I have some concrete ideas for a fix with the start here:
>
> >
> http://blogs.perl.org/users/chris_marshall/2013/12/a-framework-for-alien-modules-the-alien2-manifesto.html
>
> I haven't had a lot of time to finish the second post on an
> example Alien2 implementation with discussion. The hope
> was that we could refactor the existing Alien::Base to
> support the Alien2 improvements.
>
> It might be simpler to finish some example "Alien2::XXX"
> implementations to support the PDL2 -> PDL3 transition
> (with the goal of splitting out non-core dependencies based
> modules from the PDL core distribution and replacing the
> all-in-one install by a bundle install). The "show me the code"
> approach has the plus it will help PDL and provide some
> concrete examples of what I mean in Alien2.

I have quite enough experiences from authoring Alien::SDL Alien::IUP and
Alien::Tidyp

Reading your blog post I agree with most of your suggestion.

But: "On installation, make sure the required package is there, otherwise
install it."

according to my experience the trouble is with "otherwise install it" part
- and by trouble I really mean the big trouble, especially on MS Windows
platform (which I am an expert at)

I would personally split the problem into to parts:
1/ detect whether the required package is there
2/ otherwise install it

Implementing Alien2 covering 1/ + 2/ will be veeeery demanding for
implementation as well as for maintenance. In strawberry perl we have
approx. 30 external libraries - most of them has new versions more than
once a year, some of them require custom patches that has to be updated
with every new release. It is quite a lot of work for one platform and will
be even more for many platforms.

However part 1/ on its own is IMO doable - of course it is far from Alien
idea but we can call it like ExtUtils::LibDetect (or whatever). I would
also suggest to keep all detection submodules (like
ExtUtils::LibDetect::GSL, ExtUtils::LibDetect::FFTW ...) in one
distribution - just to keep central control and avoid many sub-modules with
different quality. So there will be only one ExtUtils-LibDetect-X.Y.tar.gz
on CPAN.

If ExtUtils::LibDetect will turn out to be a good idea whoever will later
start Alien2 implementation he/she can simply use ExtUtils::LibDetect for
detection part and focus only on "otherwise install" part.

--
kmx