[Pdl-porters] PDL::FFTW3 developers release

Discussion:

Dima Kogan

2014-01-20 20:59:43 UTC

Hi all.

I just uploaded the current PDL::FFTW3 code to CPAN as a dev release.
Lot of thanks to Chris Marshall for doing much of the actual work. The
main new feature is better handling of various data alignments. There's
still at least one outstanding issue, but it likely works well for most
people. This is version 0.02.2_01

dima

Chris Marshall

2014-01-20 22:48:35 UTC

Permalink

Works for me and just submitted a PASS report to CPAN Testers.

--Chris

Post by Dima Kogan
Hi all.
I just uploaded the current PDL::FFTW3 code to CPAN as a dev release.
Lot of thanks to Chris Marshall for doing much of the actual work. The
main new feature is better handling of various data alignments. There's
still at least one outstanding issue, but it likely works well for most
people. This is version 0.02.2_01
dima
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2014-01-20 22:51:48 UTC

Permalink

To access the release, you'll need to use the distribution path
since developers releases are not official and a simple PDL::FFTW3
only shows any official releases. For the cpan shell, you can use:

cpan[1]> install DKOGAN/PDL-FFTW3-0.02.2_01.tar.gz

for example. Other tools behave similarly, I believe. Thanks to
Dima for pushing the release and I welcome any feedback on
build, install, and/or usability/docs.

--Chris

Post by Chris Marshall
Works for me and just submitted a PASS report to CPAN Testers.
--Chris

s***@optusnet.com.au

2014-01-21 10:09:00 UTC

Permalink

-----Original Message-----
From: Chris Marshall
Sent: Tuesday, January 21, 2014 9:48 AM
To: Dima Kogan
Cc: pdl-porters
Subject: Re: [Pdl-porters] PDL::FFTW3 developers release

Post by Chris Marshall
Works for me and just submitted a PASS report to CPAN Testers.

Passed on native MS Windows, too - though there were a number of tests
skipped because memory may be unaligned - namely tests (transcribed) 3, 5,
7, 9, 11, 13, 15, 17, 19, 22, 33, 35, 38, 40, 43, 45, 47, 49, 51, 53, 55,
57, 63, 66, 70, 74, 77, 81, 84, 86, 88, 90, 92, 94, 96, 98, 100, 102, 104,
106, 108, 110, 112, 114, 116, 119, 122, 128, 131, 134, 136, 141, 144, 146,
148, 153, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174, 176.

Three minor issues:
##################################### 1 ##
The Makefile.PL specifies:
@$(PERL) -e 'exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module(q{PDL::FFTW3}); }; ';

That is non-portable - ie won't work on Windows.
The following rendition worked ok for Windows (... is it ok on Cygwin/*nix
systems too ?) :
@$(PERL) -e qq{exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module("PDL::FFTW3"); }; };

##################################### 2 ##

On the 64-bit builds of perl (but not 32-bit builds), I had to invoke the
"-std=gnu99" switch - otherwise I got errors in relation to some C 'for'
looping. (Sorry - didn't record those exact errors, but I can reproduce them
if needed.)
Not sure why it was necessary for *only* the 64-bit builds.

##################################### 3 ##

During the running of 'dmake test' there was no mention made of the fact
that some tests had been skipped. It merely reported that all 176 tests were
successful.

##########################################

Tested on 32-bit perls 5.8.8, 5.8.9, 5.10.0, 5.12.0, 5.14.0, 5.16.0, 5.18.0,
and 5.19.8.
Tested on 64-bit perls 5.12.0, 5.14.0, 5.16.0, 5.18.0 and 5.19.8.

(Using the pkg-config options, btw.)

Has any progress been made on the alignment issues affecting native Windows
?

Dima Kogan

2014-01-21 10:22:20 UTC

Permalink

Post by s***@optusnet.com.au
Passed on native MS Windows, too - though there were a number of tests
skipped because memory may be unaligned

Yeah, that's normal.

Post by s***@optusnet.com.au
##################################### 1 ##
@$(PERL) -e 'exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module(q{PDL::FFTW3}); }; ';
That is non-portable - ie won't work on Windows.
The following rendition worked ok for Windows (... is it ok on Cygwin/*nix
@$(PERL) -e qq{exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module("PDL::FFTW3"); }; };

Sure, if you say that's better, I'll take it.

Post by s***@optusnet.com.au
##################################### 2 ##
On the 64-bit builds of perl (but not 32-bit builds), I had to invoke the
"-std=gnu99" switch - otherwise I got errors in relation to some C 'for'
looping. (Sorry - didn't record those exact errors, but I can reproduce them
if needed.)
Not sure why it was necessary for *only* the 64-bit builds.

I'm using C99, so you must tell the compiler to do that. I'm assuming
you are using gcc of some sort. Can you check the value of
"$Config{ccname}", please? Do we support any non-gcc compilers?

Post by s***@optusnet.com.au
##################################### 3 ##
During the running of 'dmake test' there was no mention made of the fact
that some tests had been skipped. It merely reported that all 176 tests were
successful.

That has more to do with 'dmake test' than with PDL::FFTW3. The skipped
tests are normal, and I guess it interprets the skips as successful
completions. That's fine, I think.

Post by s***@optusnet.com.au
Has any progress been made on the alignment issues affecting native
Windows ?

In-PDL alignment is as it was previously. PDL::FFTW3 should handle
unaligned memory now, though (with a perfomance penalty).

Chris Marshall

2014-01-21 11:28:36 UTC

Permalink

Post by Dima Kogan

Post by s***@optusnet.com.au
Passed on native MS Windows, too - though there were a number of tests
skipped because memory may be unaligned

Yeah, that's normal.

Sure, if you say that's better, I'll take it.

I'm using C99, so you must tell the compiler to do that. I'm assuming
you are using gcc of some sort. Can you check the value of
"$Config{ccname}", please? Do we support any non-gcc compilers?

Yes. perl and PDL support non-gcc compilers. The MSVC and
clang compilers are two that come immediately to mind. There is
no current plan to suddenly drop support for non-gcc compilers.

Post by Dima Kogan

That has more to do with 'dmake test' than with PDL::FFTW3. The skipped
tests are normal, and I guess it interprets the skips as successful
completions. That's fine, I think.

I saw the same thing. I think something is up with the output
from Test::More skip that didn't trigger the Test::Harness to
generate the skip count. Maybe calling the SKIP block within
a sub confused things?

Post by Dima Kogan

Post by s***@optusnet.com.au
Has any progress been made on the alignment issues affecting native
Windows ?

In-PDL alignment is as it was previously. PDL::FFTW3 should handle
unaligned memory now, though (with a perfomance penalty).

PDL::FFTW3 now handles alignments that are not a multiple of
16 bytes to the extent that the underlying fftw3 library does. As
Dima mentioned, this will lead to reduced performance and there
may be some issues with threaded computations where the individual
thread loops come into and out of alignment for the plan.

We're still working to sort that out. This has generated a lot of
appreciation (by me) and use cases for PDL3 having better support
for alignment requirements.

s***@optusnet.com.au

2014-01-21 13:30:49 UTC

Permalink

-----Original Message-----
From: Chris Marshall
Sent: Tuesday, January 21, 2014 10:28 PM

As usual, thanks for the checks, Rob.

Well ... it's always fun. (Sometimes more fun than others ;-)

How did you manage the FFTW3 library install for PDL::FFTW3 to build
(pkg-config or hand tweak?).

Didn't have to hand tweak at all - pkg-config performed admirably.
I don't normally have pkg-config.exe in my path, so I had to add C:\Gtk+\bin
to my path.
And, as per EU::PC's requirement, I also had to set the PKG_CONFIG_PATH
environment variable to the path to fftw3.pc.

It then all worked as intended.

As I mentioned earlier, the key to success is to specify a
"--prefix=C:/windows/path" to configure when building FFTW3 in the msys
shell. Otherwise fftw3.pc will specify a POSIX path for prefix ... and perl
won't be able to make sense of that POSIX path.

Given that the skipped tests indicate only reduced performance (and *not*
reduced functionality), I think that congrats to you and Dima are in order.

Well done !!

Cheers,
Rob

Chris Marshall

2014-01-21 14:37:17 UTC

Permalink

I've attached a patch to improve the gcc detection logic.
Does this resolve the problem for your 64bit builds?

--Chris

Dima Kogan

2014-01-21 20:07:31 UTC

Permalink

Post by Chris Marshall
I've attached a patch to improve the gcc detection logic.
Does this resolve the problem for your 64bit builds?

Well, hold on.

The right thing to do here is to always ask for C99. Here we're only
doing that with gcc. Is this the default for other supported compilers?
Can somebody please try to build this with those?

Thanks.

Chris Marshall

2014-01-21 20:24:21 UTC

Permalink

Post by Dima Kogan

Post by Chris Marshall
I've attached a patch to improve the gcc detection logic.
Does this resolve the problem for your 64bit builds?

Well, hold on.

No need. This generalizes the gcc check to match other
ccname strings. I was hoping for verification by Rob before
assuming that my simple fix was the fix. As test results
and information from builds with other compilers occur,
we can add support for those other compilers as they occur.

Post by Dima Kogan
The right thing to do here is to always ask for C99. Here we're only
doing that with gcc. Is this the default for other supported compilers?
Can somebody please try to build this with those?

See above. Given the variations in platform specific compiler
options, it is usually more time effective to implement things
cleanly and add support for new compilers as they arise.
Trying to get correct info out of documentation without a live
bug report to test is almost impossible to get right.

If we require C99 then we should mention that in the installation
information. Is there anything specific that requires C99? We
could document that as well.

Cheers,
Chris

s***@optusnet.com.au

2014-01-22 00:10:52 UTC

Permalink

-----Original Message-----
From: Chris Marshall
Sent: Wednesday, January 22, 2014 1:37 AM
To: Sisyphus
Cc: Dima Kogan ; pdl-porters
Subject: Re: [Pdl-porters] PDL::FFTW3 developers release

Post by Chris Marshall
I've attached a patch to improve the gcc detection logic.
Does this resolve the problem for your 64bit builds?

Yes. It didn't occur to me that the Makefile.PL might be doing that.

The paranoid regex is /\bgcc$/, but to my knowledge /gcc/ has never yet been
inadequate.

Cheers,
Rob

s***@optusnet.com.au

2014-01-21 12:42:30 UTC

Permalink

-----Original Message-----
From: Dima Kogan
Sent: Tuesday, January 21, 2014 9:22 PM
To: pdl-***@jach.hawaii.edu
Subject: Re: [Pdl-porters] PDL::FFTW3 developers release

Post by Dima Kogan

Post by s***@optusnet.com.au
@$(PERL) -e qq{exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module("PDL::FFTW3"); }; };

Sure, if you say that's better, I'll take it.

Just to elaborate a little, in the typical *nix shells we do:
perl -e '....'

but, in the Windows cmd.exe shell, that needs to be:
perl -e "...."

Of course, neither rendition is portable across both systems, so the
solution that came first to my mind was to resort to the quote-like
operators. (Perl to the rescue ;-)

Post by Dima Kogan

I'm using C99, so you must tell the compiler to do that. I'm assuming you
are using gcc of some sort. Can you check the value of "$Config{ccname}",
please?

Yes, they're both gcc compilers ... the 64-bit one is (technically) a
cross-compiler.

32-bit:
C:\>perl -V:ccname
ccname='gcc';

64-bit:
C:\>perl -V:ccname
ccname='x86_64-w64-mingw32-gcc';

Post by Dima Kogan
Do we support any non-gcc compilers?

Depends upon what you mean by "we" ... and by "support".

PDL itself builds straight out of the box with microsoft compilers, though
it's unclear to me whether that constitutes "support" or merely
"accommodation".
FAIK it's also quite possible to readily build the fftw3 library using
microsoft compilers (though I've not done so myself) ... as to how
PDL::FFTW3 would (or even ought to) fare when built with an MS compiler is
anybody's guess ;-)

Post by Dima Kogan

That has more to do with 'dmake test' than with PDL::FFTW3. The skipped
tests are normal, and I guess it interprets the skips as successful
completions. That's fine, I think.

You may be right. I know that, for some module test suites, I see messages
about skipped tests. But perhaps that only happens when *all* tests are
being skipped for a specific test file.

Post by Dima Kogan

Post by s***@optusnet.com.au
Has any progress been made on the alignment issues affecting native
Windows ?

In-PDL alignment is as it was previously. PDL::FFTW3 should handle
unaligned memory now, though (with a perfomance penalty).

Oh, I see.
That's an excellent fallback to have. (Nice work.)

Dima Kogan

2014-01-23 07:02:00 UTC

Permalink

Post by s***@optusnet.com.au
@$(PERL) -e 'exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module(q{PDL::FFTW3}); }; ';
That is non-portable - ie won't work on Windows.
The following rendition worked ok for Windows (... is it ok on Cygwin/*nix
@$(PERL) -e qq{exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module("PDL::FFTW3"); }; };

Hi.

I just tested it, and it doesn't work on unix (this makes sense; the
qq{} contruct is a perl-ism, but here it's being passed to the shell). I
grabbed the analogous code from PDL::Graphics::Gnuplot, since that works
for everybody, presumably. All that does is to replace ' with ". FYI.

s***@optusnet.com.au

2014-01-23 09:57:39 UTC

Permalink

'cc' the list, too.

-----Original Message-----
From: Dima Kogan

Post by Dima Kogan

Post by s***@optusnet.com.au
@$(PERL) -e qq{exit if $$ENV{DESTDIR}; use PDL::Doc; eval {
PDL::Doc::add_module("PDL::FFTW3"); }; };

I just tested it, and it doesn't work on unix (this makes sense; the
qq{} contruct is a perl-ism, but here it's being passed to the shell).

Heh, how dumb was that ... I dunno what I was thinking there ... apparently
wasn't - and mistook lack of error for success.

Post by Dima Kogan
I grabbed the analogous code from PDL::Graphics::Gnuplot, since that works
for everybody, presumably. All that does is to replace ' with ". FYI.

Oh - I had thought that double quotes would not work there on *nix.
Thanks for catching; thanks for correcting.

Cheers,
Rob