[Pdl-porters] linux packaging for Alien::XXX modules

[Chris Marshall]

I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).

It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

--Chris

[Dima Kogan]

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

Hi.

Linux distros manage dependencies well, and Alien::xxx isn't necessary
for them. For OSs that don't manage dependencies (windows, osx, etc),
this may be a good approach; others can chime in here.

In fact, requiring Alien::XXX makes more difficult the job of
distribution packaging, since it's a not-yet-unpackaged dependency to
deal with. So I strongly request that this dependency stay optional.

PDL::FFTW3 does something like this. If Alien::FFTW3 is available, it is
used. Otherwise, pkg-config does its thing. This clearly requires that
libfftw3-dev is installed, and that can be done with one command.

[Chris Marshall]

Post by Dima Kogan

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

Hi.
Linux distros manage dependencies well, and Alien::xxx isn't necessary
for them. For OSs that don't manage dependencies (windows, osx, etc),
this may be a good approach; others can chime in here.
In fact, requiring Alien::XXX makes more difficult the job of
distribution packaging, since it's a not-yet-unpackaged dependency to
deal with. So I strongly request that this dependency stay optional.
PDL::FFTW3 does something like this. If Alien::FFTW3 is available, it is
used. Otherwise, pkg-config does its thing. This clearly requires that
libfftw3-dev is installed, and that can be done with one command.

This was the origin of my question. I thought linux package managers
were able to handle this type of dependency.

How is the direct pkg-config hardwired into the Makefile.PL more difficult
than just having Alien::FFTW3 the dependency on the perl side. That
would directly have a dependency on libfftw3-dev for the same reason
and would appear to be able to handle other platforms symmetrically
rather than with one-off code.

What am I missing? I was hoping by using Alien::XXX we could factor
out all the detection code from the PDL or other module builds and use
the Alien::XXX to pass on the configuration as needed. If this is not
possible, then it would seem that the whole idea of Alien modules
won't ever work.

I tried looking searching for some information on how debian packages
perl modules to get some insight but did not find any helpful references.
I don't have a debian system so I can't easily see what the existing
package dependencies (for something like PDL) would be...

--Chris

[Chris Marshall]

If a transition period were needed until the Alien::FFTW3 were stable
and released, it might be bundled in the PDL::FFTW3 distribution in
the inc/ folder to provide the needed functionality. That would allow the
pkg-config to be factored out of the Makefile.PL as far as the PDL::FFTW3
module and package requirements go...

Post by Chris Marshall

Post by Dima Kogan

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

Hi.
Linux distros manage dependencies well, and Alien::xxx isn't necessary
for them. For OSs that don't manage dependencies (windows, osx, etc),
this may be a good approach; others can chime in here.
In fact, requiring Alien::XXX makes more difficult the job of
distribution packaging, since it's a not-yet-unpackaged dependency to
deal with. So I strongly request that this dependency stay optional.
PDL::FFTW3 does something like this. If Alien::FFTW3 is available, it is
used. Otherwise, pkg-config does its thing. This clearly requires that
libfftw3-dev is installed, and that can be done with one command.

This was the origin of my question. I thought linux package managers
were able to handle this type of dependency.
How is the direct pkg-config hardwired into the Makefile.PL more difficult
than just having Alien::FFTW3 the dependency on the perl side. That
would directly have a dependency on libfftw3-dev for the same reason
and would appear to be able to handle other platforms symmetrically
rather than with one-off code.
What am I missing? I was hoping by using Alien::XXX we could factor
out all the detection code from the PDL or other module builds and use
the Alien::XXX to pass on the configuration as needed. If this is not
possible, then it would seem that the whole idea of Alien modules
won't ever work.
I tried looking searching for some information on how debian packages
perl modules to get some insight but did not find any helpful references.
I don't have a debian system so I can't easily see what the existing
package dependencies (for something like PDL) would be...
--Chris

[Chris Marshall]

Two examples from wheezy are:

* libsdl-perl depends on libalien-sdl-perl
* libwx-perl which depends on libalien-wxwidgets-perl

which match my initial usage. I'm not sure about what
other use cases for perl Alien::XXX modules in debian or
other packaging systems but I would like to keep things
consistent if possible.

--Chris

Post by Chris Marshall
If a transition period were needed until the Alien::FFTW3 were stable
and released, it might be bundled in the PDL::FFTW3 distribution in
the inc/ folder to provide the needed functionality. That would allow the
pkg-config to be factored out of the Makefile.PL as far as the PDL::FFTW3
module and package requirements go...

Post by Chris Marshall

Post by Dima Kogan

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

Hi.
Linux distros manage dependencies well, and Alien::xxx isn't necessary
for them. For OSs that don't manage dependencies (windows, osx, etc),
this may be a good approach; others can chime in here.
In fact, requiring Alien::XXX makes more difficult the job of
distribution packaging, since it's a not-yet-unpackaged dependency to
deal with. So I strongly request that this dependency stay optional.
PDL::FFTW3 does something like this. If Alien::FFTW3 is available, it is
used. Otherwise, pkg-config does its thing. This clearly requires that
libfftw3-dev is installed, and that can be done with one command.

This was the origin of my question. I thought linux package managers
were able to handle this type of dependency.
How is the direct pkg-config hardwired into the Makefile.PL more difficult
than just having Alien::FFTW3 the dependency on the perl side. That
would directly have a dependency on libfftw3-dev for the same reason
and would appear to be able to handle other platforms symmetrically
rather than with one-off code.
What am I missing? I was hoping by using Alien::XXX we could factor
out all the detection code from the PDL or other module builds and use
the Alien::XXX to pass on the configuration as needed. If this is not
possible, then it would seem that the whole idea of Alien modules
won't ever work.
I tried looking searching for some information on how debian packages
perl modules to get some insight but did not find any helpful references.
I don't have a debian system so I can't easily see what the existing
package dependencies (for something like PDL) would be...
--Chris

[Dima Kogan]

Post by Chris Marshall
* libsdl-perl depends on libalien-sdl-perl
* libwx-perl which depends on libalien-wxwidgets-perl
which match my initial usage. I'm not sure about what
other use cases for perl Alien::XXX modules in debian or
other packaging systems but I would like to keep things
consistent if possible.

Yes, some maintainers chose to package their Alien::XXX, but it was
work. You can instead use 'pkg-config' to get the flags (and often you
don't need any at all), which makes Alien::XXX superfluous as a means of
getting build flags.

[Chris Marshall]

Post by Dima Kogan

Post by Chris Marshall
* libsdl-perl depends on libalien-sdl-perl
* libwx-perl which depends on libalien-wxwidgets-perl
which match my initial usage. I'm not sure about what
other use cases for perl Alien::XXX modules in debian or
other packaging systems but I would like to keep things
consistent if possible.

Yes, some maintainers chose to package their Alien::XXX, but it was
work. You can instead use 'pkg-config' to get the flags (and often you
don't need any at all), which makes Alien::XXX superfluous as a means of
getting build flags.

Please describe this work. CPAN does not provide pkg-config
and Alien modules are specifically designed to provide this
configuration and build functionality.

--Chris

[Craig DeForest]

Hmmm, Dima and I disagree on this (I'm pro-Alien all the way, though I wish
there were more general infrastructure for Alien modules. (Alien::Base is great,
but hard to hack for cases that aren't perfectly in line with Joel's initial use
case.)

As it turns out, sweeping install weirdness under the Alien rug works pretty well:
for platforms that one knows how to support, it's very convenient.

I agree that there is some infrastructural overhead -- putting together a CPAN
module is a nontrivial piece of work, at least for us mortals, and I understand
the module-count issues with Debian. But that's really a game-theory / infrastructure
issue. From the standpoint of any one package, having an Alien:: module is a big
win, since it simplifies installation across platforms.

Yes, there's a global cost to Debian if every library has an Alien:: module as well
as a Debian package -- and of course that is a problem common to every packaging
system, including MacPorts, Yum, and whatever Microsoft Windows people use these days.
But the size and effort cost to each distro of having a slightly bloated package
catalog is miniscule compared to the size and effort involved in other things the
distros already carry. (emacs alone has got to be several times bigger than the
whole catalog, and don't get me started on the X libraries or compilers).

The fact is, there's still no truly cross-platform binary distribution solution, and
Alien modules are a nice way to make that work within the context of CPAN.

Post by Chris Marshall

Post by Dima Kogan

Post by Chris Marshall
* libsdl-perl depends on libalien-sdl-perl
* libwx-perl which depends on libalien-wxwidgets-perl
which match my initial usage. I'm not sure about what
other use cases for perl Alien::XXX modules in debian or
other packaging systems but I would like to keep things
consistent if possible.

Yes, some maintainers chose to package their Alien::XXX, but it was
work. You can instead use 'pkg-config' to get the flags (and often you
don't need any at all), which makes Alien::XXX superfluous as a means of
getting build flags.

Please describe this work. CPAN does not provide pkg-config
and Alien modules are specifically designed to provide this
configuration and build functionality.
--Chris
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

[Henning Glawe]

Post by Craig DeForest
Yes, there's a global cost to Debian if every library has an Alien:: module as well
as a Debian package -- and of course that is a problem common to every packaging
system, including MacPorts, Yum, and whatever Microsoft Windows people use these days.
But the size and effort cost to each distro of having a slightly bloated package
catalog is miniscule compared to the size and effort involved in other things the
distros already carry. (emacs alone has got to be several times bigger than the
whole catalog, and don't get me started on the X libraries or compilers).

For one, potentially adding a (architecture-dependant) for each and every
possible library would put quite a load on:
- the developer: most probably, the given 'alien' package needs to have a
dependency on both the library version _and_ the debian package revision
in order to be consistent
- the debian mirror network (most of the alien packages will be small, but
think about fs performance scales not necessarily with the file size, but
also with the number of files)
- the clients, as the full package lists for all the presently selected
architectures have to be searched when performing even trivial apt
operations

Sure, the load of the comparatively low number of PDL dependencies may be
managable, but it is still a significant amount of mostly superfluous
work.
pkg-config exists and configuration is supplied by many libraries, and is
now the standard mechanism to get compiler/link flags on linux machines (most
probably, coverage on other unix-oid platforms is also increasing).

There is even a ExtUtils::PkgConfig module available (but I don't know about
its quality due to a lack of experience with it).

What advantages would the Alien:: approach give to justify the amount of work
necessary?

Shouldn't that efford be better spent on improving the pkg-config solution
(by adding missing library information and porting it to platforms where it
is not yet available) ?

Post by Craig DeForest
The fact is, there's still no truly cross-platform binary distribution solution, and
Alien modules are a nice way to make that work within the context of CPAN.

I would see the Alien approach only as a fallback for the few people that are
willing to build pdl and its dependency chain in a self-contained manner
(installations to /home on systems where you cannot easily get additional
software installed come to my mind).

[Craig DeForest]

Post by Henning Glawe
Sure, the load of the comparatively low number of PDL dependencies may be
managable, but it is still a significant amount of mostly superfluous
work.
pkg-config exists and configuration is supplied by many libraries, and is
now the standard mechanism to get compiler/link flags on linux machines (most
probably, coverage on other unix-oid platforms is also increasing).

The existence of pkg-config makes writing most Alien modules a snap, especially
for those libraries which play nicely with Joel's Alien::Base. The existence of
an Alien module allows for the possibility of hacking in support on platforms
where pkg-config does not exist, and/or providing a clean way to identify and
access a platform's usual package manager to pull in a standard version of a
required library or app.

Post by Henning Glawe
There is even a ExtUtils::PkgConfig module available (but I don't know about
its quality due to a lack of experience with it).
What advantages would the Alien:: approach give to justify the amount of work
necessary?

Alien:: provides a uniform interface for CPAN installation of modules with
external dependencies. That is particularly important for cross-platform modules
such as PDL (or Gnuplot).

Post by Henning Glawe
Shouldn't that efford be better spent on improving the pkg-config solution
(by adding missing library information and porting it to platforms where it
is not yet available) ?

Well, sure. You first.

I'm being facetious here, but the skillset and effort required to make pkg-config
fully general and port it to all CPAN-bearing platforms, plus be able to interface
with every platform's standard package manager is somewhat larger than the skillset
required to get an Alien:: module working. Yes, IWBNI it existed - but it doesn't,
and many module developers (including me) don't have the time or inclination to
work in that direction, but still want their modules to have a decent chance of
working the first time on as many platforms as possible.

Post by Henning Glawe

Post by Craig DeForest
The fact is, there's still no truly cross-platform binary distribution solution, and
Alien modules are a nice way to make that work within the context of CPAN.

I would see the Alien approach only as a fallback for the few people that are
willing to build pdl and its dependency chain in a self-contained manner
(installations to /home on systems where you cannot easily get additional
software installed come to my mind).

The issue isn't so much whether you have root on a particular system, as whether
there is a single-command-line pathway to getting the damned module to work.
We've come a long way since the days when INSTALL instructions were 10kb or more,
but until *all* main contingencies are encapsulated on all major supported
platforms, installation issues will stand in the way of module adoption.

[Chris Marshall]

On Mon, Jan 20, 2014 at 3:40 PM, Craig DeForest

Post by Craig DeForest

Post by Henning Glawe
Sure, the load of the comparatively low number of PDL
dependencies may be managable, but it is still a significant
amount of mostly superfluous work. pkg-config exists and
configuration is supplied by many libraries, and is now
the standard mechanism to get compiler/link flags on
linux machines (most probably, coverage on other unix-oid
platforms is also increasing).

The work is only superfluous on platforms where pkg-config
is standard.

Post by Craig DeForest
The existence of pkg-config makes writing most Alien modules
a snap, especially for those libraries which play nicely with
Joel's Alien::Base.

This is only true for pkg-config platforms.

Post by Craig DeForest
The existence of an Alien module allows for the possibility
of hacking in support on platforms where pkg-config does not
exist, and/or providing a clean way to identify and access
a platform's usual package manager to pull in a standard
version of a required library or app.

The encapsulation of all the platform probing stuff into
Alien modules means that it is possible to keep the same
interface for CPAN XS modules across the supported perl
platforms.

Post by Craig DeForest

Post by Henning Glawe
There is even a ExtUtils::PkgConfig module available (but
I don't know about its quality due to a lack of experience
with it).
What advantages would the Alien:: approach give to justify
the amount of work necessary?

Alien:: provides a uniform interface for CPAN installation
of modules with external dependencies. That is particularly
important for cross-platform modules such as PDL (or
Gnuplot).

I would add that Alien modules have a chance at actually
working across platforms where pkg-config must have an
existing infrastructure. Of particular interest to me,
is that new PDL users should not have to be developers
to use PDL for their research.

Post by Craig DeForest

Post by Henning Glawe
Shouldn't that effort be better spent on improving the
pkg-config solution (by adding missing library information
and porting it to platforms where it is not yet available) ?

Well, "you can't get there from here" where I mean that
if we encapsulate the pkg-config stuff in the Alien modules,
then the porting and improvement of pkg-config can be done
while none-pkg-config platforms can still use the various
XXX libraries by having the Alien modules *detect* what
is there already via Devel::CheckLib as one method.

Post by Craig DeForest
Well, sure. You first.
I'm being facetious here, but the skillset and effort
required to make pkg-config fully general and port it to all
CPAN-bearing platforms, plus be able to interface with every
platform's standard package manager is somewhat larger than
the skillset required to get an Alien:: module working. Yes,
IWBNI it existed - but it doesn't, and many module developers
(including me) don't have the time or inclination to work in
that direction, but still want their modules to have a decent
chance of working the first time on as many platforms as
possible.

Post by Henning Glawe

Post by Craig DeForest
The fact is, there's still no truly cross-platform binary
distribution solution, and Alien modules are a nice way to
make that work within the context of CPAN.

I would see the Alien approach only as a fallback for the
few people that are willing to build pdl and its dependency
chain in a self-contained manner (installations to /home
on systems where you cannot easily get additional software
installed come to my mind).

The current PDL release builds out of the box on all the
PDL platforms. The only difficulty is that the dependencies
need to be installed by hand. Not a problem for folks with
real package managers but it is a *huge* problem for platforms
like windows with no official package manager or even a
generic software install infrastructure.

The use of Alien modules also has the possibility of reducing
packaging bloat since a single Alien module could be used
by multiple CPAN modules to provide access to needed library
functionality.

Post by Craig DeForest
The issue isn't so much whether you have root on a particular
system, as whether there is a single-command-line pathway
to getting the damned module to work. We've come a long way
since the days when INSTALL instructions were 10kb or more,
but until *all* main contingencies are encapsulated on all
major supported platforms, installation issues will stand in
the way of module adoption.

[Chris Marshall]

Post by Henning Glawe

Post by Craig DeForest
Yes, there's a global cost to Debian if every library has an Alien:: module as well
as a Debian package -- and of course that is a problem common to every packaging
system, including MacPorts, Yum, and whatever Microsoft Windows people use these days.
But the size and effort cost to each distro of having a slightly bloated package
catalog is miniscule compared to the size and effort involved in other things the
distros already carry. (emacs alone has got to be several times bigger than the
whole catalog, and don't get me started on the X libraries or compilers).

For one, potentially adding a (architecture-dependant) for each and every
- the developer: most probably, the given 'alien' package needs to have a
dependency on both the library version _and_ the debian package revision
in order to be consistent

I think the fact that an Alien module can first detect an existing
library and provide the configuration information to use that may
simplify the issue of rigid version lock.

Post by Henning Glawe
- the debian mirror network (most of the alien packages will be small, but
think about fs performance scales not necessarily with the file size, but
also with the number of files)
- the clients, as the full package lists for all the presently selected
architectures have to be searched when performing even trivial apt
operations

--Chris

[Henning Glawe]

Post by Chris Marshall

Post by Henning Glawe

Post by Craig DeForest
Yes, there's a global cost to Debian if every library has an Alien:: module as well
as a Debian package -- and of course that is a problem common to every packaging
system, including MacPorts, Yum, and whatever Microsoft Windows people use these days.
But the size and effort cost to each distro of having a slightly bloated package
catalog is miniscule compared to the size and effort involved in other things the
distros already carry. (emacs alone has got to be several times bigger than the
whole catalog, and don't get me started on the X libraries or compilers).

For one, potentially adding a (architecture-dependant) for each and every
- the developer: most probably, the given 'alien' package needs to have a
dependency on both the library version _and_ the debian package revision
in order to be consistent

I think the fact that an Alien module can first detect an existing
library and provide the configuration information to use that may
simplify the issue of rigid version lock.

In the case of a pkg-config-backed-alien, this would be entirely possible;
however, how do you know what linking/compiler options a particular build of
a library would need? Sure, you can always fallback to some default values,
but...

p.s.
to be fair, this case isn't handled by PDLs present build system, AFAIK ;)

[Dima Kogan]

Post by Chris Marshall

Post by Dima Kogan

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?

Hi.
Linux distros manage dependencies well, and Alien::xxx isn't necessary
for them. For OSs that don't manage dependencies (windows, osx, etc),
this may be a good approach; others can chime in here.
In fact, requiring Alien::XXX makes more difficult the job of
distribution packaging, since it's a not-yet-unpackaged dependency to
deal with. So I strongly request that this dependency stay optional.
PDL::FFTW3 does something like this. If Alien::FFTW3 is available, it is
used. Otherwise, pkg-config does its thing. This clearly requires that
libfftw3-dev is installed, and that can be done with one command.

This was the origin of my question. I thought linux package managers
were able to handle this type of dependency.

Hi Chris.

This email ended up way longer than I wanted, so instead of posting a
way-too-long thing, some short questions/comments we can expand later:

- Is Alien::xxx intended to just get the flags or to handle installation
also?

- If it's just the flags, 'pkg-config' already is a standard method to
do that. If it's installation, we shouldn't attempt this because there
are many ways to do that, and an automated method would get it wrong.

- If somebody is building something from source, it's not unreasonable
to expect they know how to install and use dependencies. Otherwise,
prebuild packages should be used (which we should provide)

- What distros give you is a standardized framework for library
organization and distribution. So you know how to easily install
something, and how to use it. And you have a high expectation that it
will work with the rest of the system.

- I favor assuming dependencies are in the "normal" location. Then
things either "just work" on all platforms, or you get an obvious
error message. This may not be viable for osx/windows, and in that
case having an Alien::XXX may be good.

- Alien::XXX should be optional so that distro maintainers don't have to
package it.

I have way more I can say, but I'd like to get input on the above first.
Also, if all you want is to know more about debian things, I can just
answer your questions, and hold back opinions :)

dima

[Chris Marshall]

Post by Dima Kogan

Post by Chris Marshall

Post by Dima Kogan

Post by Chris Marshall
I would like to factor out the explicit detection and
configuration of the PDL build process on external libraries
(such as PROJ4, HDF5, FFTW3,...) into corresponding
Alien::PROJ4 or similar distributions. The job of these
Alien::XXX modules is to check for XXX. or install XXX,
and provide configuration to the perl module that does a
'use Alien::XXX'. The Alien::XXX module would then provide
the needed information on how to build against the XXX
dependency for the perl module (usually an XS based module).
It seems to me that the logical thing would be to have
linux package dependencies for PDL on Alien::XXX and then
the package for Alien::XXX would have a dependency on the
underlying XXX library/dependency. This would be for a
binary install. Is this a reasonable approach from the
considerations of linux packaging?

Hi.
Linux distros manage dependencies well, and Alien::xxx isn't
necessary for them. For OSs that don't manage dependencies
(windows, osx, etc), this may be a good approach; others can
chime in here.
In fact, requiring Alien::XXX makes more difficult the job
of distribution packaging, since it's a not-yet-unpackaged
dependency to deal with. So I strongly request that this
dependency stay optional.
PDL::FFTW3 does something like this. If Alien::FFTW3 is
available, it is used. Otherwise, pkg-config does its thing.
This clearly requires that libfftw3-dev is installed, and
that can be done with one command.

This was the origin of my question. I thought linux package managers
were able to handle this type of dependency.

Hi Chris.
This email ended up way longer than I wanted, so instead of posting a
- Is Alien::xxx intended to just get the flags or to handle installation
also?

On installation it is supposed to detect, configure, and/or install
the needed dependency so that a 'use Alien::xxx;' will provide the user
with enough information to use xxx via class methods. See the original
docs at http://search.cpan.org/~abergman/Alien-0.91/lib/Alien.pm .

Post by Dima Kogan
- If it's just the flags, 'pkg-config' already is a standard method to
do that. If it's installation, we shouldn't attempt this because there
are many ways to do that, and an automated method would get it wrong.

The difference is that Alien::xxx is a standard method for CPAN
modules and not for binary package distribution. Clearly, if the
Alien::xxx is detecting and configuring for xxx on a platform that
supports pkg-config it could use that (see Alien::Base for example).
However, for non-pkg-config capable platforms, Alien::xxx has the
ability to use something that will work.

Post by Dima Kogan
- If somebody is building something from source, it's not unreasonable
to expect they know how to install and use dependencies. Otherwise,
prebuild packages should be used (which we should provide)

The idea is that XS modules can be built and installed via the
CPAN tools without the user having to be a build/install guru for
the platform they are on. I haven't had to hand-edit a makefile
in a while now that the autoconf stuff basically works. I don't
think someone wanting to use PDL::FFTW3 for calculations in perl
should need to know how to build the FFTW3 library and compile and
link to it---we already know what and how.

Post by Dima Kogan
- What distros give you is a standardized framework for library
organization and distribution. So you know how to easily install
something, and how to use it. And you have a high expectation that it
will work with the rest of the system.

Same for perl and CPAN.

Post by Dima Kogan
- I favor assuming dependencies are in the "normal" location. Then
things either "just work" on all platforms, or you get an obvious
error message. This may not be viable for osx/windows, and in that
case having an Alien::XXX may be good.

Normal location only works if you have admin permissions on your
platform. If not, this almost always fails.

Post by Dima Kogan
- Alien::XXX should be optional so that distro maintainers don't have to
package it.

Is pkg-config optional?

Post by Dima Kogan
I have way more I can say, but I'd like to get input on the above first.
Also, if all you want is to know more about debian things, I can just
answer your questions, and hold back opinions :)

Again, I was looking for the reasons why an Alien::xxx
implementation for CPAN would not work as described.

"should be optional so distribution maintainers don't have to
package it" seems to be conflating the linux distro packaging and
packagers with the perl modules packaging and packagers. Clearly,
if a distribution does not wish to build a package for any given
perl module, they have that right. I would like to make sure that
the Alien::xxx standard being evolved to, is workable on both
ends of the packaging pipeline (CPAN and linux distros).

Likewise, if a perl module author wishes to provide direct support
for linux distro(s) in their module, they are welcome to. My
experience is that it tends to be fragile and incomplete as well
as possibly needing to be duplicated for every perl XS module
using libxxx. The Alien::xxx was intended to factor out this
work into a common dependency that looks to CPAN tools like
a typical module dependency but that really implements for perl
modules the how to get and use an external library so that they
don't need to know all the platform specific details.

--Chris

[Dima Kogan]

Post by Chris Marshall

Post by Dima Kogan
- If it's just the flags, 'pkg-config' already is a standard method to
do that. If it's installation, we shouldn't attempt this because there
are many ways to do that, and an automated method would get it wrong.

The difference is that Alien::xxx is a standard method for CPAN
modules and not for binary package distribution. Clearly, if the
Alien::xxx is detecting and configuring for xxx on a platform that
supports pkg-config it could use that (see Alien::Base for example).
However, for non-pkg-config capable platforms, Alien::xxx has the
ability to use something that will work.

OK. I think you may have the wrong idea of what pkg-config is. It's much
simpler/smaller than what you think. A libwhatever-dev debian package
often ships .pc files. These files contain information that a
library-user can query to determine the needed build flags. So, for
instance, a user of libfftw3 can put this into their Makefile to use the
"right" cflags:

CFLAGS += `pkg-config --cflags fftw3f`

So pkg-config doesn't manage or install anything

Post by Chris Marshall

Post by Dima Kogan
- If somebody is building something from source, it's not unreasonable
to expect they know how to install and use dependencies. Otherwise,
prebuild packages should be used (which we should provide)

The idea is that XS modules can be built and installed via the
CPAN tools without the user having to be a build/install guru for
the platform they are on. I haven't had to hand-edit a makefile
in a while now that the autoconf stuff basically works. I don't
think someone wanting to use PDL::FFTW3 for calculations in perl
should need to know how to build the FFTW3 library and compile and
link to it---we already know what and how.
<snip>

OK.

I understand what you want. It's a nice thing to want, but without a TON
of work, I don't see any way to get it done in a reliable way. You're
trying to do this: http://en.wikipedia.org/wiki/Dll_hell

Summary:

I request that any package that uses Alien::XXX uses it as a soft
dependency, with the Makefile.PL assuming "normal" paths, so that a user
that already has the dependencies installed with see the installation
"just work".


Longer version:

Having more than one system for "installing" things is a set of problems
just waiting to happen, and promoting such a setup is a disservice to
one's users.

For instance, let's say I'm running debian, so my libraries go into
/usr. Then I also use Alien::FFTW3 to install to /usr/local. So now I
have two possibly-incompatible copies of this library. So now the
correct one must be picked at build time, and a correct one must be
picked at run time. The copy picked at run time MUST match the copy
picked at build time. There is NO way to resolve this. And this can bite
you at a later point.

Let's say the user had no system-wide libfftw3. Then they used
Alien::FFTW3, and their PDL::FFTW3 works now. Great! Now next week they
install some unrelated program with APT. That program has a dependency
on libfftw3, and APT helpfully installs it. So at this point either the
program they just installed may crash or PDL::FFTW3 may crash, depending
on which takes priority. The user maybe didn't even know the thing they
installed used fftw3 internally, but they have crashes anyway.

So yeah, if you have no package manager (osx, windows), then you chose
this life. But people using Debian (et al) already have ways to install
dependencies, and going around those standardized methods only breaks
things. Sure, Alien::FFTW3 can install with APT instead of dumping to
/usr/local. But are you going to support APT and YUM and emerge and
pacman and everybody else?


Now in the case of Debian, specifically, you don't need to do anything
special. Installing to /usr/local will work as well (or as badly) as it
does on any other OS.

As for distro-specific support inside packages, I'm not advocating that.
I'm happy enough for the PDL::FFTW3 Makefile.PL to use pkg-config or
even just link with -lfftw3. If an error results, it's extremely obvious
what the issue is.

dima

[Chris Marshall]

Post by Dima Kogan

Post by Chris Marshall

Post by Dima Kogan
- If it's just the flags, 'pkg-config' already is a standard method to
do that. If it's installation, we shouldn't attempt this because there
are many ways to do that, and an automated method would get it wrong.

The difference is that Alien::xxx is a standard method for CPAN
modules and not for binary package distribution. Clearly, if the
Alien::xxx is detecting and configuring for xxx on a platform that
supports pkg-config it could use that (see Alien::Base for example).
However, for non-pkg-config capable platforms, Alien::xxx has the
ability to use something that will work.

OK. I think you may have the wrong idea of what pkg-config is. It's much
simpler/smaller than what you think. A libwhatever-dev debian package
often ships .pc files. These files contain information that a
library-user can query to determine the needed build flags. So, for
instance, a user of libfftw3 can put this into their Makefile to use the
CFLAGS += `pkg-config --cflags fftw3f`
So pkg-config doesn't manage or install anything

Yes, and if pkg-config files are not set up correctly, it doesn't
work at all. For windows installs, there is pretty much no pkg-config
support.

Post by Dima Kogan

Post by Chris Marshall

Post by Dima Kogan
- If somebody is building something from source, it's not unreasonable
to expect they know how to install and use dependencies. Otherwise,
prebuild packages should be used (which we should provide)

The idea is that XS modules can be built and installed via the
CPAN tools without the user having to be a build/install guru for
the platform they are on. I haven't had to hand-edit a makefile
in a while now that the autoconf stuff basically works. I don't
think someone wanting to use PDL::FFTW3 for calculations in perl
should need to know how to build the FFTW3 library and compile and
link to it---we already know what and how.
<snip>

OK.
I understand what you want. It's a nice thing to want, but without
a TON of work, I don't see any way to get it done in a reliable way.
You're trying to do this: http://en.wikipedia.org/wiki/Dll_hell

Err, no. DLL hell is a problem generic to the windows operating
system and not a goal of the Alien modules which is to provide
the glue for CPAN as a dependency for external library dependencies.
The libraries don't have to be DLLs.

Post by Dima Kogan
I request that any package that uses Alien::XXX uses it as a soft
dependency, with the Makefile.PL assuming "normal" paths, so that a user
that already has the dependencies installed with see the installation
"just work".

If a package that needs XXX uses Alien::XXX as a dependency (hard),
then if the user already has the dependency libXXX installed then
it will "just work" with the existing libXXX. This doesn't work
with Alien::XXX not being a dependency as far as CPAN tools are
concerned.

Post by Dima Kogan
Having more than one system for "installing" things is a set of problems
just waiting to happen, and promoting such a setup is a disservice to
one's users.
For instance, let's say I'm running debian, so my libraries go into
/usr. Then I also use Alien::FFTW3 to install to /usr/local. So now I
have two possibly-incompatible copies of this library. So now the
correct one must be picked at build time, and a correct one must be
picked at run time. The copy picked at run time MUST match the copy
picked at build time. There is NO way to resolve this. And this can bite
you at a later point.

It is always possible for multiple installs of a library to
cause problems---even on systems with rather robust package systems.

Post by Dima Kogan
Let's say the user had no system-wide libfftw3. Then they used
Alien::FFTW3, and their PDL::FFTW3 works now. Great! Now next week they
install some unrelated program with APT. That program has a dependency
on libfftw3, and APT helpfully installs it. So at this point either the
program they just installed may crash or PDL::FFTW3 may crash, depending
on which takes priority. The user maybe didn't even know the thing they
installed used fftw3 internally, but they have crashes anyway.

Actually, Alien::FFTW3 should provide the information needed to
use the fftw3 library. By default, my preference would be that
Alien::FFTW3 would detect and use an existing system install.
If that were not available, then an install could be provided
and the configuration should provide the information needed to
use that install---whereever it is located.

I would recommend that Alien::XXX packages for platforms with
good package managers use that to install any needed dependency
to avoid exactly this type of problem. If that is the case, I
wouldn't expect any difficulties for debian systems.

Post by Dima Kogan
So yeah, if you have no package manager (osx, windows), then
you chose this life. But people using Debian (et al) already
have ways to install dependencies, and going around those
standardized methods only breaks things. Sure, Alien::FFTW3
can install with APT instead of dumping to /usr/local. But
are you going to support APT and YUM and emerge and pacman
and everybody else?

These issues are always present in any system that attempts
to support multiple platforms. I don't think a switch statement
to choose the install tool to use for a package managed linux
or bsd or whatever system is onerous. At the worst case, the
Alien module could report what needs to be installed and let
the user handle the actual process. This is pretty much what
happens now with the advantage of the Alien module might give
better directions.

Post by Dima Kogan
Now in the case of Debian, specifically, you don't need to do anything
special. Installing to /usr/local will work as well (or as badly) as it
does on any other OS.
As for distro-specific support inside packages, I'm not advocating that.
I'm happy enough for the PDL::FFTW3 Makefile.PL to use pkg-config or
even just link with -lfftw3. If an error results, it's extremely obvious
what the issue is.

The definition of obvious depends on the user. Alien modules are
supposed to reduce the need for everyone to be a guru or developer
and to have intimate knowledge of all the software tools that they
use.

--Chris

[Dima Kogan]

Post by Chris Marshall

Post by Dima Kogan
I understand what you want. It's a nice thing to want, but without
a TON of work, I don't see any way to get it done in a reliable way.
You're trying to do this: http://en.wikipedia.org/wiki/Dll_hell

Err, no. DLL hell is a problem generic to the windows operating
system and not a goal of the Alien modules which is to provide
the glue for CPAN as a dependency for external library dependencies.
The libraries don't have to be DLLs.

Post by Dima Kogan
I request that any package that uses Alien::XXX uses it as a soft
dependency, with the Makefile.PL assuming "normal" paths, so that a user
that already has the dependencies installed with see the installation
"just work".

If a package that needs XXX uses Alien::XXX as a dependency (hard),
then if the user already has the dependency libXXX installed then
it will "just work" with the existing libXXX. This doesn't work
with Alien::XXX not being a dependency as far as CPAN tools are
concerned.

Post by Dima Kogan
Having more than one system for "installing" things is a set of problems
just waiting to happen, and promoting such a setup is a disservice to
one's users.
For instance, let's say I'm running debian, so my libraries go into
/usr. Then I also use Alien::FFTW3 to install to /usr/local. So now I
have two possibly-incompatible copies of this library. So now the
correct one must be picked at build time, and a correct one must be
picked at run time. The copy picked at run time MUST match the copy
picked at build time. There is NO way to resolve this. And this can bite
you at a later point.

It is always possible for multiple installs of a library to
cause problems---even on systems with rather robust package systems.

Post by Dima Kogan
Let's say the user had no system-wide libfftw3. Then they used
Alien::FFTW3, and their PDL::FFTW3 works now. Great! Now next week they
install some unrelated program with APT. That program has a dependency
on libfftw3, and APT helpfully installs it. So at this point either the
program they just installed may crash or PDL::FFTW3 may crash, depending
on which takes priority. The user maybe didn't even know the thing they
installed used fftw3 internally, but they have crashes anyway.

Actually, Alien::FFTW3 should provide the information needed to
use the fftw3 library. By default, my preference would be that
Alien::FFTW3 would detect and use an existing system install.
If that were not available, then an install could be provided
and the configuration should provide the information needed to
use that install---whereever it is located.
I would recommend that Alien::XXX packages for platforms with
good package managers use that to install any needed dependency
to avoid exactly this type of problem. If that is the case, I
wouldn't expect any difficulties for debian systems.

Post by Dima Kogan
So yeah, if you have no package manager (osx, windows), then
you chose this life. But people using Debian (et al) already
have ways to install dependencies, and going around those
standardized methods only breaks things. Sure, Alien::FFTW3
can install with APT instead of dumping to /usr/local. But
are you going to support APT and YUM and emerge and pacman
and everybody else?

These issues are always present in any system that attempts
to support multiple platforms. I don't think a switch statement
to choose the install tool to use for a package managed linux
or bsd or whatever system is onerous. At the worst case, the
Alien module could report what needs to be installed and let
the user handle the actual process. This is pretty much what
happens now with the advantage of the Alien module might give
better directions.

Post by Dima Kogan
Now in the case of Debian, specifically, you don't need to do anything
special. Installing to /usr/local will work as well (or as badly) as it
does on any other OS.
As for distro-specific support inside packages, I'm not advocating that.
I'm happy enough for the PDL::FFTW3 Makefile.PL to use pkg-config or
even just link with -lfftw3. If an error results, it's extremely obvious
what the issue is.

The definition of obvious depends on the user. Alien modules are
supposed to reduce the need for everyone to be a guru or developer
and to have intimate knowledge of all the software tools that they
use.

OK.

I get what you're saying. A side question: do we know what scipy does
here? That may be useful to look at.

[Chris Marshall]

They apparently use binary distribution sites, instructions for users
on systems with package managers, and binaries+instructions for poor
windows folks:

http://www.scipy.org/install.html

--Chris

Post by Dima Kogan

Post by Chris Marshall

Post by Dima Kogan
I understand what you want. It's a nice thing to want, but without
a TON of work, I don't see any way to get it done in a reliable way.
You're trying to do this: http://en.wikipedia.org/wiki/Dll_hell

Err, no. DLL hell is a problem generic to the windows operating
system and not a goal of the Alien modules which is to provide
the glue for CPAN as a dependency for external library dependencies.
The libraries don't have to be DLLs.

Post by Dima Kogan
I request that any package that uses Alien::XXX uses it as a soft
dependency, with the Makefile.PL assuming "normal" paths, so that a user
that already has the dependencies installed with see the installation
"just work".

If a package that needs XXX uses Alien::XXX as a dependency (hard),
then if the user already has the dependency libXXX installed then
it will "just work" with the existing libXXX. This doesn't work
with Alien::XXX not being a dependency as far as CPAN tools are
concerned.

Post by Dima Kogan
Having more than one system for "installing" things is a set of problems
just waiting to happen, and promoting such a setup is a disservice to
one's users.
For instance, let's say I'm running debian, so my libraries go into
/usr. Then I also use Alien::FFTW3 to install to /usr/local. So now I
have two possibly-incompatible copies of this library. So now the
correct one must be picked at build time, and a correct one must be
picked at run time. The copy picked at run time MUST match the copy
picked at build time. There is NO way to resolve this. And this can bite
you at a later point.

It is always possible for multiple installs of a library to
cause problems---even on systems with rather robust package systems.

Post by Dima Kogan
Let's say the user had no system-wide libfftw3. Then they used
Alien::FFTW3, and their PDL::FFTW3 works now. Great! Now next week they
install some unrelated program with APT. That program has a dependency
on libfftw3, and APT helpfully installs it. So at this point either the
program they just installed may crash or PDL::FFTW3 may crash, depending
on which takes priority. The user maybe didn't even know the thing they
installed used fftw3 internally, but they have crashes anyway.

Actually, Alien::FFTW3 should provide the information needed to
use the fftw3 library. By default, my preference would be that
Alien::FFTW3 would detect and use an existing system install.
If that were not available, then an install could be provided
and the configuration should provide the information needed to
use that install---whereever it is located.
I would recommend that Alien::XXX packages for platforms with
good package managers use that to install any needed dependency
to avoid exactly this type of problem. If that is the case, I
wouldn't expect any difficulties for debian systems.

Post by Dima Kogan
So yeah, if you have no package manager (osx, windows), then
you chose this life. But people using Debian (et al) already
have ways to install dependencies, and going around those
standardized methods only breaks things. Sure, Alien::FFTW3
can install with APT instead of dumping to /usr/local. But
are you going to support APT and YUM and emerge and pacman
and everybody else?

These issues are always present in any system that attempts
to support multiple platforms. I don't think a switch statement
to choose the install tool to use for a package managed linux
or bsd or whatever system is onerous. At the worst case, the
Alien module could report what needs to be installed and let
the user handle the actual process. This is pretty much what
happens now with the advantage of the Alien module might give
better directions.

Post by Dima Kogan
Now in the case of Debian, specifically, you don't need to do anything
special. Installing to /usr/local will work as well (or as badly) as it
does on any other OS.
As for distro-specific support inside packages, I'm not advocating that.
I'm happy enough for the PDL::FFTW3 Makefile.PL to use pkg-config or
even just link with -lfftw3. If an error results, it's extremely obvious
what the issue is.

The definition of obvious depends on the user. Alien modules are
supposed to reduce the need for everyone to be a guru or developer
and to have intimate knowledge of all the software tools that they
use.

OK.
I get what you're saying. A side question: do we know what scipy does
here? That may be useful to look at.
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

[Zakariyya Mughal]

Post by Chris Marshall
They apparently use binary distribution sites, instructions for users
on systems with package managers, and binaries+instructions for poor
http://www.scipy.org/install.html

Yes, exactly. To elaborate, the approach they've used to avoid the
complexity of compilers, build options, and conflicting system libraries
is to make the Conda package manager that distributes all of the binary
dependencies (roughly equivalent to apt-get, yum for multiple OSes)[^1].

There was actually a conversation on Reddit comparing Conda to pip (the
normal way of installing packages) just yesterday that that got a
response from several of the core SciPy developers[^2].


[1]: <http://docs.continuum.io/anaconda/pkgs.html>, <http://www.continuum.io/blog/conda>, <http://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html>.

[2]: <http://www.reddit.com/r/Python/comments/1vled5/anaconda_seems_to_have_a_nicely_set_up_library/>.

Regards,
- Zaki Mughal

Post by Chris Marshall
--Chris

[David Mertens]

Hey everyone -

I sense two basic approaches here. There is the "use package managers"
group and the "make it install everywhere via CPAN" group. Personally I
love package managers and I try to use them whenever I can. Sometimes I
can't. Sometimes I'm on a restricted Linux box, other times I'm on a
Windows machine. (Aside: for Windows, I just discovered
Chocolatey<http://chocolatey.org/>while writing this. Maybe we *could*
focus on package managers for
everything.)

The whole point of Alien modules is that some distributions, like Padre,
have *tons* of CPAN dependencies. Some of those dependencies rely on
external libraries (i.e. wxWidgets). Before the concept of Alien modules,
you would say "cpan Padre", come back after 30 minutes and curse that the
build process died two minutes in because the development headers needed
for wxWidgets weren't installed. You'd install libraries, re-invoke "cpan
Padre", come back after thirty minutes, and break your mug because the
build process died *again* at some other dependency. Even on Linux systems
this is a pain in the neck (and it's morel likely to occur on Linux
systems, since you're unlikely to have your X11 development headers
installed).

Ultimately, I disagree with Dima's characterization of our users. I think
we need to consider our users just like FireFox does. Some users like to
install Add-ons to FireFox. In the same way, some users like to install
Perl modules from CPAN. But we should see our CPAN consumers as *users*,
not *developers*, just like FireFox Add-on users are not developers,
either. "Users" just want the damn thing to install so they can get on with
surfing the web, or building a website, or performing their data analysis.
If installation isn't super-simple, *they'll never share it with their
friends*.

Having written Alien::TinyCC so that it Just Works on The Big Three
platforms, I have developed a few opinions.

@debian folks and other package managers: Including separate
libper-alien-XXX packages leads to large package lists. One solution is to
just put the requisite Alien::XXX module in the *development* version of
the upstream package. But of course that sets an arguably bad precedent to
include such build info for *all* languages in *all* dev packages. But this
is really an aside. Perl modules like PDL have to be oriented for
cross-platform distribution.

@dll hell: When Joel was working on Alien::Base, he convinced me of the
idea that Alien modules should *never* install their libraries to any
system-wide location. Instead, your Alien module should (1) ensure that the
library is available, (2) provide compile-time flags for EU::MM, M::B, etc,
and finally, (3) *provide a run-time module to be invoked by consuming
modules that prepares the environment*. I just realized while typing this
that the third point is not part of the Alien manifesto, but I think it
should be. Let me illustrate with an example of how I avoid dependency hell
using this system.

Alien::tcc ensures that you have a working copy of the Tiny C Compiler on
your system. Amusingly, it breaks rule 1 by always installing tcc, because
I hacked it out quickly and didn't want to handle detection issues.
However, it puts tcc in your File::ShareDir location. For example, since I
use perlbrew on a Mac, this installs the tcc compiler and all the headers
and libraries to
"$HOME/perl5/perlbrew/perls/perl-5.18.1-thread-multi/lib/site_perl/5.18.1/auto/share/dist/Alien-TinyCC/"
Let me assure you there is no way to accidentally load those shared
objects! (On a side-note, as far as I am aware, File::ShareDir does not
provide architecture-dependent sharedir locations. This is a problem when
you have machines with different architectures using NFS. And you happen to
install an Alien module on said NFS. Yes, I know from experience.)

"But," you ask, "if you build an XS module against libtcc, how could
Dynaloader possibly find the .a or .so or .dll or .dylib or whatever files?
In Windows, for example, the dynamic library loader searches %PATH% for the
requested library file. On Linux, the $LD_LIBRARY_PATH needs to contain
this info." That's a good question, and I'm glad you asked. The solution to
this problem is to let the package Alien::TinyCC modify your PATH or
LD_LIBRARY_PATH environment variable so that the crazy-long Perl-specific
path *is* in the correct variable. Thus, if your XS module uses libtcc,
your Perl module file (.pm file) must contain the line "use Alien::TinyCC"
before calling Dynaloader (or XSloader). No matter what, installing tcc to
your system won't break Alien::TinyCC, and Alien::TinyCC won't break your
locally installed tcc.

@debian folks: Consider what I just pointed out. If an Alien module is
designed to install its local copy of the library to the File::ShareDir,
then it must provide runtime hooks to properly prepare the environment. Of
course, there's no need for Debian's Alien::XXX to be a replica of the
original: It could provide its own runtime module that properly sets up
everything to link against the system library.

Longwinded as always. Sorry. But I hope this adds some useful perspective
to why I like to do Alien modules this way.

David

P. S. There is a way to use pkgconfig everywhere: we could for example
provide Alien::pkgconfig! Then the Alien modules could depend on it, and
reliably use pkg-config on all platforms. But the Alien author would still
have to ensure the pkg-config works, and user would still have to install
the libraries separately, which defeats the whole point of Alien modules.

Post by Zakariyya Mughal

Post by Chris Marshall
They apparently use binary distribution sites, instructions for users
on systems with package managers, and binaries+instructions for poor
http://www.scipy.org/install.html

Yes, exactly. To elaborate, the approach they've used to avoid the
complexity of compilers, build options, and conflicting system libraries
is to make the Conda package manager that distributes all of the binary
dependencies (roughly equivalent to apt-get, yum for multiple OSes)[^1].
There was actually a conversation on Reddit comparing Conda to pip (the
normal way of installing packages) just yesterday that that got a
response from several of the core SciPy developers[^2].
[1]: <http://docs.continuum.io/anaconda/pkgs.html>, <
http://www.continuum.io/blog/conda>, <
http://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html>.
[2]: <
http://www.reddit.com/r/Python/comments/1vled5/anaconda_seems_to_have_a_nicely_set_up_library/

Post by Chris Marshall
.

Regards,
- Zaki Mughal

Post by Chris Marshall
--Chris

_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

[s***@optusnet.com.au]

-----Original Message-----
From: Chris Marshall

For windows installs, there is pretty much no pkg-config support.

It's a somewhat moot point, but there is actually quite good support
available in windows for pkg-config. It's just that not much use is made of
that support (which is the reason that I've called the point "moot").

Installing Gtk on Windows will provide one with a functioning
pkg-config.exe - and ExtUtils::PkgConfig will then install fine straight out
of the box.
Libraries built for windows in the msys shell (or cross-compiled for windows
in the Cygwin shell) will create and install the pc file if configure so
decrees, and the whole process has worked fine for me whenever perl has
tried to utilise it.
Building proj, netcdf, gsl and fftw3 in the msys shell installs a pc file
that could be used if called upon.

The only trap is that I have to provide a "--prefix=C:/MinGW/msys/1.0/local"
configure arg - otherwise the pc file prefix gets set to the posix-style
"/usr/local". Of course, both of those locations are the same place, but
perl won't be able to make any sense of "/usr/local".

I've never known a build using a Microsoft compiler to create a pc file, but
plplot (built using cmake) seems also to be pc-aware.

Cheers,
Rob

[Chris Marshall]

-----Original Message----- From: Chris Marshall

For windows installs, there is pretty much no pkg-config support.

It's a somewhat moot point, but there is actually quite good support
available in windows for pkg-config. It's just that not much use is made of
that support (which is the reason that I've called the point "moot").
Installing Gtk on Windows will provide one with a functioning pkg-config.exe
- and ExtUtils::PkgConfig will then install fine straight out of the box.
Libraries built for windows in the msys shell (or cross-compiled for windows
in the Cygwin shell) will create and install the pc file if configure so
decrees, and the whole process has worked fine for me whenever perl has
tried to utilise it.
Building proj, netcdf, gsl and fftw3 in the msys shell installs a pc file
that could be used if called upon.
The only trap is that I have to provide a "--prefix=C:/MinGW/msys/1.0/local"
configure arg - otherwise the pc file prefix gets set to the posix-style
"/usr/local". Of course, both of those locations are the same place, but
perl won't be able to make any sense of "/usr/local".
I've never known a build using a Microsoft compiler to create a pc file, but
plplot (built using cmake) seems also to be pc-aware.

Thanks for clarifying, Rob.

I'm fine with pkg-config as the standard default for
Alien::XXX detection, but it needs to be called from
the Alien module so that support for non-pkg-config
implementations can be provided seamlessly.

I did look at the current pkg-config documents and
they seem to have improved the support for win32
platforms. With Alien::XXX in place, the upstream
library builds could phase in more uniform support
for pkg-config for windows while existing windows
can just work with the non-pkg-config implementation.

To a certain extent, this seems like a chicken or the egg
type problem.

Cheers,
Rob

[Jonathan Yu]

Hey Chris,

I think this is actually the usual approach, although it does have a
cost for distributions as an extra package must be built separately.
The costs are to build infrastructure (more packages need to be
built), developer workload, and to mirrors (bandwidth). As an example,
there are 3 packages in Debian [0] that match "libalien-(.*)-perl" -
the small number of packages means that in practice, the cost isn't
too bad. Someone once pointed out to me that Debian in particular is
sometimes used in embedded computing scenarios, and that each package
increases the metadata in the Debian packages list, which must be
stored whether or not the package is installed (this is how Debian
identifies available packages). For each package, it's probably not a
big deal, but in the aggregate, it means more stuff can't be installed
or won't fit on a given CD or DVD.

It would be nice if we had a better solution (one that did not require
the cost of a "dummy" Alien module, which results in a pretty
empty-looking package on distributions with binary-only packages, such
as Debian). But, again, this seems to have been the usual approach we
followed historically.

I suppose one option is to have something similar to (or perhaps even
as an extension of) Module::Install, where the necessary code to find
a module is copied into inc/ and thus available at build time, but not
required to be installed. This code could then put the linker options
somewhere where the Perl module could pick them up later. I suppose
there are other options as well, like a shared library that can do
this sort of thing, I don't know. But given that that does not seem to
be a readily available option at the moment, I would suggest that yes,
you proceed with an Alien module. I would, however, suggest that you
try to coordinate with downstream distributions where you can/have
interest, as you might be able to make some simple changes now to make
their lives easier. Debian in particular has a few PDL modules in use
[1], though I'm not sure if your change would affect it.

Cheers,

Jonathan

[0] http://packages.debian.org/search?keywords=libalien
[1] http://packages.debian.org/search?keywords=libpdl

Post by Chris Marshall
I would like to factor out the explicit detection and configuration of
the PDL build process on external libraries (such as PROJ4, HDF5,
FFTW3,...) into corresponding Alien::PROJ4 or similar distributions.
The job of these Alien::XXX modules is to check for XXX. or install
XXX, and provide configuration to the perl module that does a 'use
Alien::XXX'. The Alien::XXX module would then provide the needed
information on how to build against the XXX dependency for the perl
module (usually an XS based module).
It seems to me that the logical thing would be to have linux package
dependencies for PDL on Alien::XXX and then the package for Alien::XXX
would have a dependency on the underlying XXX library/dependency.
This would be for a binary install. Is this a reasonable approach
from the considerations of linux packaging?
--Chris