[Pdl-porters] Stored IO::Storable data unreadable in new PDL

I took a look at the PDL::IO::Storable code at it seems to
encode the type of the pdl from $pdl->get_datatype which
is a number. As we now have a new data type, PDL_Indx,
the old numbers are not the same as the new ones.

Not sure how to fix but you may be able to correct the
type information to allow restoring on non-binary compatible
PDL versions.

--Chris

Post by Dima Kogan
Hi all.
There's a major regression in the new version of PDL. If I save a data
file with PDL::IO::Storable on an older PDL instance, and try to read it
back with the newer PDL, I get garbage. Not even an error, mind you, but
bogus data.
store.pl
================
#!/usr/bin/perl
use PDL;
use PDL::IO::Storable;
use Storable qw(retrieve store);
my $x = sequence(10);
store $x, "/tmp/tst.dat";
==================
load.pl
==================
#!/usr/bin/perl
use PDL;
use PDL::IO::Storable;
use Storable qw(retrieve store);
use feature ':5.10';
my $x = retrieve "/tmp/tst.dat";
say $x;
===================
[0 0 0 1.875 0 2 0 2.125 0 2.25]
Can somebody who's familiar with that code please take a look?
Thanks
dima
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Dima Kogan

2013-11-15 08:52:05 UTC

Post by Chris Marshall
I took a look at the PDL::IO::Storable code at it seems to
encode the type of the pdl from $pdl->get_datatype which
is a number. As we now have a new data type, PDL_Indx,
the old numbers are not the same as the new ones.
Not sure how to fix but you may be able to correct the
type information to allow restoring on non-binary compatible
PDL versions.

I see. Not only did we get a new type, it's in the middle, so everything
past it was broken. I fixed this and pushed up the fix (and the tests).

After the patch, we can read old files once again. And we can read new
files we save. And (in theory) we can read files saved on a machine with
a different endian-ness. I tested this last bit somewhat, but lacking
access to an actual big-endian box I couldn't test this fully. If
anybody has a big-endian machine running PDL, it'd be great if they
could add a corresponding test to t/storable.t

Chris, can we make a point release with this patch? I consider this to
be a MAJOR problem, so I think it's justified.

Thanks

dima

Ingo Schmid

2013-11-15 09:18:36 UTC

I think it just requires replacing get_datatype with type at line 137 of
storable.pd.

Chris Marshall

2013-11-15 12:06:32 UTC

I see. Not only did we get a new type, it's in the middle, so everything
past it was broken. I fixed this and pushed up the fix (and the tests).

Sigh, the rigid type order requirement was one thing
I was hoping to fix with improved type support. Thanks
for following up with a fix.

Post by Dima Kogan
After the patch, we can read old files once again. And we can read new
files we save. And (in theory) we can read files saved on a machine with
a different endian-ness. I tested this last bit somewhat, but lacking
access to an actual big-endian box I couldn't test this fully. If
anybody has a big-endian machine running PDL, it'd be great if they
could add a corresponding test to t/storable.t
Chris, can we make a point release with this patch? I consider this to
be a MAJOR problem, so I think it's justified.

I'll push a CPAN developers release out this weekend.

--Chris

Dima Kogan

2013-11-15 17:26:49 UTC

Post by Dima Kogan
Chris, can we make a point release with this patch?

I'll push a CPAN developers release out this weekend.

Thank you!

Henning, can you roll a new .deb when that happens?

Thanks again.

David Mertens

2013-11-15 23:31:02 UTC

Dima++

Thanks for the identification and fix!

Post by Dima Kogan
Chris, can we make a point release with this patch?

I'll push a CPAN developers release out this weekend.

Thank you!
Henning, can you roll a new .deb when that happens?
Thanks again.
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan

Chris Marshall

2013-11-16 16:09:59 UTC

I have test failure for the new t/storable.t on cygwin. It looks
like Storable itself is having a problem with longlong data:

perl -Mblib t/storable.t
1..23
ok 1 - use PDL::IO::Storable;
ok 2 - PDL freeze/thaw
ok 3 - The reference isa HASH
ok 4 - PDL in structure
ok 5 - PDL->can('freeze')
ok 6 - PDL has-a works with freeze/thaw
ok 7 - PDL is a hash
ok 8 - Initialization from seraialized object
ok 9 - Initialization from dclone object
Long integer size is not compatible at
/usr/lib/perl5/5.14/i686-cygwin-threads-64int/Storable.pm line 379, at
t/storable.t line 117.
# Looks like you planned 23 tests but ran 9.
# Looks like your test exited with 255 just after 9.

***@tesseract-4 ~/pdl/git/pdl-code $
perl -V
Summary of my perl5 (revision 5 version 14 subversion 2) configuration:

Platform:
osname=cygwin, osvers=1.7.15(0.26053), archname=cygwin-thread-multi-64int
uname='cygwin_nt-5.1 winxp 1.7.15(0.26053) 2012-05-09 10:25 i686 cygwin '
config_args='-de -Dlibperl=cygperl5_14.dll -Dcc=gcc-4 -Dld=g++-4
-Darchname=i686-cygwin-threads-64int -Dmksymlinks -Dusethreads
-Accflags=-g'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc-4', ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -g
-fno-strict-aliasing -pipe -fstack-protector',
optimize='-O3',
cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -g
-fno-strict-aliasing -pipe -fstack-protector'
ccversion='', gccversion='4.5.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++-4', ldflags =' -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--enable-auto-image-base
-fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib /lib
libs=-lgdbm -ldb -ldl -lcrypt -lgdbm_compat
perllibs=-ldl -lcrypt
libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=cygperl5_14.dll
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' --shared -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--enable-auto-image-base -L/usr/local/lib
-fstack-protector'

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_PRESERVE_IVUV
PERL_USE_SAFE_PUTENV USE_64_BIT_INT USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
USE_REENTRANT_API
Locally applied patches:
Bug#55162 File::Spec::case_tolerant performance
CYG07 $vendorarch/auto/.rebase
CYG15 static Win32CORE
CYG17 cyg-1.7 paths-utf8
0c612ce82 Fix building static extensions on cygwin, -UUSEIMPORTLIB
1bac5ecc1 Fix 64-bit threading sv.c: S_anonymise_cv_maybe
Cygwin::sync_winenv added
Built under cygwin
Compiled at Jul 12 2012 14:17:21
%ENV:
PERL5LIB="/home/chm/local/lib/perl5"
PERLDOC_PAGER="less"
@INC:
/home/chm/local/lib/perl5/cygwin-thread-multi-64int
/home/chm/local/lib/perl5
/usr/lib/perl5/site_perl/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/site_perl/5.14
/usr/lib/perl5/vendor_perl/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/vendor_perl/5.14
/usr/lib/perl5/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/5.14
/usr/lib/perl5/site_perl/5.10
/usr/lib/perl5/vendor_perl/5.10
/usr/lib/perl5/site_perl/5.8

Chris Marshall

2013-11-16 17:53:04 UTC

Same failure for Strawberry Perl Portable on win7. The problem appears to
be when ivtype is 64bit but neither int or long are 64bit. Looks like we need
a test case for Storable upstream...

E:\chm\perl\strawberry\spp64_padre\git\pdl-code>perl -V
Summary of my perl5 (revision 5 version 16 subversion 3) configuration:

Platform:
osname=MSWin32, osvers=4.0, archname=MSWin32-x64-multi-thread
uname='Win32 strawberry-perl 5.16.3.1 #1 Tue Mar 12 12:12:07 2013 x64'
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.6.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++.exe', ldflags ='-s
-L"E:\chm\perl\strawberry\spp64_padre\perl\lib\CORE"
-L"E:\chm\perl\strawberry\spp64_padre\c\lib"'
libpth=E:\chm\perl\strawberry\spp64_padre\c\lib
E:\chm\perl\strawberry\spp64_padre\c\x86_64-w64-mingw32\lib
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm
-lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lw
inmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl516.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s
-L"E:\chm\perl\strawberry\spp64_padre\perl\lib\CORE"
-L"E:\chm\perl\strawberry\spp64_padre\c\lib"'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PL_OP_SLAB_ALLOC
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Built under MSWin32
Compiled at Mar 12 2013 12:16:40
%ENV:
PERLDOC_PAGER="less"
@INC:
E:/chm/perl/strawberry/spp64_padre/perl/site/lib
E:/chm/perl/strawberry/spp64_padre/perl/vendor/lib
E:/chm/perl/strawberry/spp64_padre/perl/lib
.

Post by Chris Marshall
I have test failure for the new t/storable.t on cygwin. It looks
perl -Mblib t/storable.t
1..23
ok 1 - use PDL::IO::Storable;
ok 2 - PDL freeze/thaw
ok 3 - The reference isa HASH
ok 4 - PDL in structure
ok 5 - PDL->can('freeze')
ok 6 - PDL has-a works with freeze/thaw
ok 7 - PDL is a hash
ok 8 - Initialization from seraialized object
ok 9 - Initialization from dclone object
Long integer size is not compatible at
/usr/lib/perl5/5.14/i686-cygwin-threads-64int/Storable.pm line 379, at
t/storable.t line 117.
# Looks like you planned 23 tests but ran 9.
# Looks like your test exited with 255 just after 9.
perl -V
osname=cygwin, osvers=1.7.15(0.26053), archname=cygwin-thread-multi-64int
uname='cygwin_nt-5.1 winxp 1.7.15(0.26053) 2012-05-09 10:25 i686 cygwin '
config_args='-de -Dlibperl=cygperl5_14.dll -Dcc=gcc-4 -Dld=g++-4
-Darchname=i686-cygwin-threads-64int -Dmksymlinks -Dusethreads
-Accflags=-g'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
cc='gcc-4', ccflags ='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -g
-fno-strict-aliasing -pipe -fstack-protector',
optimize='-O3',
cppflags='-DPERL_USE_SAFE_PUTENV -U__STRICT_ANSI__ -g
-fno-strict-aliasing -pipe -fstack-protector'
ccversion='', gccversion='4.5.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='off_t', lseeksize=8
alignbytes=8, prototype=define
ld='g++-4', ldflags =' -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--enable-auto-image-base
-fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib /lib
libs=-lgdbm -ldb -ldl -lcrypt -lgdbm_compat
perllibs=-ldl -lcrypt
libc=/usr/lib/libc.a, so=dll, useshrplib=true, libperl=cygperl5_14.dll
gnulibc_version=''
dlsrc=dl_dlopen.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags=' --shared -Wl,--enable-auto-import
-Wl,--export-all-symbols -Wl,--enable-auto-image-base -L/usr/local/lib
-fstack-protector'
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_PRESERVE_IVUV
PERL_USE_SAFE_PUTENV USE_64_BIT_INT USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF
USE_REENTRANT_API
Bug#55162 File::Spec::case_tolerant performance
CYG07 $vendorarch/auto/.rebase
CYG15 static Win32CORE
CYG17 cyg-1.7 paths-utf8
0c612ce82 Fix building static extensions on cygwin, -UUSEIMPORTLIB
1bac5ecc1 Fix 64-bit threading sv.c: S_anonymise_cv_maybe
Cygwin::sync_winenv added
Built under cygwin
Compiled at Jul 12 2012 14:17:21
PERL5LIB="/home/chm/local/lib/perl5"
PERLDOC_PAGER="less"
/home/chm/local/lib/perl5/cygwin-thread-multi-64int
/home/chm/local/lib/perl5
/usr/lib/perl5/site_perl/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/site_perl/5.14
/usr/lib/perl5/vendor_perl/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/vendor_perl/5.14
/usr/lib/perl5/5.14/i686-cygwin-threads-64int
/usr/lib/perl5/5.14
/usr/lib/perl5/site_perl/5.10
/usr/lib/perl5/vendor_perl/5.10
/usr/lib/perl5/site_perl/5.8

Dima Kogan

2013-11-16 19:37:49 UTC

Post by Chris Marshall
Same failure for Strawberry Perl Portable on win7. The problem appears to
be when ivtype is 64bit but neither int or long are 64bit. Looks like we need
a test case for Storable upstream...

Hi Chris.

Thank you for testing this. I don't quite understand the issue and whose
fault it is. Is it that Storable itself has architecture-compatibility
issues?

Thanks
dima

Chris Marshall

2013-11-16 20:49:30 UTC

I just saw the message in the Storable.xs with one for various
cases of data types---and omitting longlong. I assume that is
an indication of why it fails. Don't know enough of the implementation
to go further.

Also, 32bit SPP is broken with the new PDL::IO::Storable because
there is no such pack/unpack code for long long data:

E:\chm\work\PDL-2.007_01>perl -Mblib t\storable.t
1..23
ok 1 - use PDL::IO::Storable;
Invalid type 'q' in pack at
E:\chm\work\PDL-2.007_01\blib\lib/PDL/IO/Storable.pm line 99, at
t\storable.t line 27.
# Looks like you planned 23 tests but ran 1.
# Looks like your test exited with 255 just after 1.

I took a look but no quick fix came to mind...

--Chris

Hi Chris.
Thank you for testing this. I don't quite understand the issue and whose
fault it is. Is it that Storable itself has architecture-compatibility
issues?
Thanks
dima
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Chris Marshall

2013-11-16 21:02:19 UTC

I think you could check $Config{ivsize} to determine if
4byte or 8byte code and/or pack strings should be used.

Can you do an implementation that works with either
32bit or 64bit?

--Chris

Post by Chris Marshall
I just saw the message in the Storable.xs with one for various
cases of data types---and omitting longlong. I assume that is
an indication of why it fails. Don't know enough of the implementation
to go further.
Also, 32bit SPP is broken with the new PDL::IO::Storable because
E:\chm\work\PDL-2.007_01>perl -Mblib t\storable.t
1..23
ok 1 - use PDL::IO::Storable;
Invalid type 'q' in pack at
E:\chm\work\PDL-2.007_01\blib\lib/PDL/IO/Storable.pm line 99, at
t\storable.t line 27.
# Looks like you planned 23 tests but ran 1.
# Looks like your test exited with 255 just after 1.
I took a look but no quick fix came to mind...
--Chris

Dima Kogan

2013-11-16 21:35:59 UTC

Post by Chris Marshall
I think you could check $Config{ivsize} to determine if
4byte or 8byte code and/or pack strings should be used.
Can you do an implementation that works with either
32bit or 64bit?

Sure.

If pack('q') isn't supported then what is sizeof(PDL_Indx)? Is it still
8, or do we fall back to '4' in that case?

dima

Chris Marshall

2013-11-16 21:38:07 UTC

PDL_Indx type and size are ivtype and ivsize respectively.
It didn't seem useful to allow for folks running a 64bit perl
to force only 32bit indexing support.

--Chris

Sure.
If pack('q') isn't supported then what is sizeof(PDL_Indx)? Is it still
8, or do we fall back to '4' in that case?
dima
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Dima Kogan

2013-11-16 23:03:29 UTC

Done and pushed.

I expect this will now work on your SPP setup, but please do test it.
It'd also be good to make sure this works on piddles whose sizes
actually don't fit into 32 bits, but I don't have enough RAM to test
that. If that's easy for you to do, that'd be good also.

dima

Chris Marshall

2013-11-17 12:06:56 UTC

Done and pushed.

The test fails for SPP32 with the following error:

E:\chm\work\PDL-2.007_01>perl -Mblib t\storable.t
1..23
ok 1 - use PDL::IO::Storable;
PDL::IO::Storable tried reading a file with dimensions that don't
fit into 32 bits.
However here PDL_Indx can't store a number so large. Giving up. at
E:/spp_5161/perl/lib/Storable.pm line 417, at t\storable.t line 29.
# Looks like you planned 23 tests but ran 1.
# Looks like your test exited with 255 just after 1.

Since this is a freeze/thaw by this perl+PDL for this perl+PDL,
it should work.

Post by Dima Kogan
I expect this will now work on your SPP setup, but please do test it.
It'd also be good to make sure this works on piddles whose sizes
actually don't fit into 32 bits, but I don't have enough RAM to test
that. If that's easy for you to do, that'd be good also.

Sure, glad to.

--Chris

Chris Marshall

2013-11-17 14:43:54 UTC

This is the byte sequence from the first freeze at line
27 in t/storable.t, maybe you can see if something is
funny about the output:

DB<6> p join ',', map { sprintf "%02x", $_ } unpack "C*", $serialized
04,08,04,31,32,33,34,04,04,04,08,13,00,03,50,44,4c,58,ff,ff,ff,ff,ff,ff,ff,ff,50,44,4c,5f,44,20,20,20,20,20,20,20,20,20,20,20,08,00,00,00,01,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,f0,3f,00,00,00,00,00,00,00,40,00,00,00,00,00,00,08,40

and this is the same from cygwin:

DB<3> p join ',', map { sprintf "%02x", $_ } unpack "C*", $serialized
04,08,08,31,32,33,34,35,36,37,38,04,04,04,08,13,00,03,50,44,4c,58,ff,ff,ff,ff,ff,ff,ff,ff,50,44,4c,5f,44,20,20,20,20,20,20,20,20,20,20,20,08,00,00,00,01,00,00,00,02,00,00,00,00,00,00,00,02,00,00,00,00,00,00,00,02,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,f0,3f,00,00,00,00,00,00,00,40,00,00,00,00,00,00,08,40

Done and pushed.

E:\chm\work\PDL-2.007_01>perl -Mblib t\storable.t
1..23
ok 1 - use PDL::IO::Storable;
PDL::IO::Storable tried reading a file with dimensions that don't
fit into 32 bits.
However here PDL_Indx can't store a number so large. Giving up. at
E:/spp_5161/perl/lib/Storable.pm line 417, at t\storable.t line 29.
# Looks like you planned 23 tests but ran 1.
# Looks like your test exited with 255 just after 1.
Since this is a freeze/thaw by this perl+PDL for this perl+PDL,
it should work.

Sure, glad to.
--Chris

Dima Kogan

2013-11-17 19:21:57 UTC

Post by Chris Marshall
This is the byte sequence from the first freeze at line
27 in t/storable.t, maybe you can see if something is
DB<6> p join ',', map { sprintf "%02x", $_ } unpack "C*", $serialized
04,08,04,31,32,33,34,04,04,04,08,13,00,03,50,44,4c,58,ff,ff,ff,ff,ff,ff,ff,ff,50,44,4c,5f,44,20,20,20,20,20,20,20,20,20,20,20,08,00,00,00,01,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,02,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,f0,3f,00,00,00,00,00,00,00,40,00,00,00,00,00,00,08,40
DB<3> p join ',', map { sprintf "%02x", $_ } unpack "C*", $serialized
04,08,08,31,32,33,34,35,36,37,38,04,04,04,08,13,00,03,50,44,4c,58,ff,ff,ff,ff,ff,ff,ff,ff,50,44,4c,5f,44,20,20,20,20,20,20,20,20,20,20,20,08,00,00,00,01,00,00,00,02,00,00,00,00,00,00,00,02,00,00,00,00,00,00,00,02,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,f0,3f,00,00,00,00,00,00,00,40,00,00,00,00,00,00,08,40

OK.

Try again now.

dima

Chris Marshall

2013-11-17 21:59:05 UTC

Works now, thanks! I've just pushed CHM/PDL-2.007_01.tar.gz
and once the Storable and double conversion issues are resolved,
I recommend pushing another official PDL release.

Thanks for all who helped!
Chris

OK.
Try again now.
dima
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Dima Kogan

2013-11-16 21:06:27 UTC

OK.

Can you run the following experiment for me, please? On a unixy box do

perl -MStorable -e 'Storable::store( [1,2,3], "tst.dat" );'

Then take that file and try to load it on the windows box you're seeing
issues on. This should work:

perl -MStorable -e 'Storable::retrieve( "tst.dat" );'

This tests just Storable (not PDL::IO::Storable) for portability
problems. If it works with no issues, then digging deeper is warranted,
I think.

Post by Chris Marshall
Also, 32bit SPP is broken with the new PDL::IO::Storable because
E:\chm\work\PDL-2.007_01>perl -Mblib t\storable.t
1..23
ok 1 - use PDL::IO::Storable;
Invalid type 'q' in pack at
E:\chm\work\PDL-2.007_01\blib\lib/PDL/IO/Storable.pm line 99, at
t\storable.t line 27.
# Looks like you planned 23 tests but ran 1.
# Looks like your test exited with 255 just after 1.

This is clearly in my domain, so I can fix it. But to be clear, are you
really saying that there's a commonly used perl build with no 64-bit
pack support? I.e. does anybody have a reason to use the "32bit SPP"?

dima

Chris Marshall

2013-11-16 21:36:53 UTC

The closest I have is cygwin vs SPP, the result, no matter
which is the store while the other is the retrieve is the
message:

Byte order is not compatible at
/usr/lib/perl5/5.14/i686-cygwin-threads-64int/Storable.pm line 379, at
-e line 1.

OK.
Can you run the following experiment for me, please? On a unixy box do
perl -MStorable -e 'Storable::store( [1,2,3], "tst.dat" );'
Then take that file and try to load it on the windows box you're seeing
perl -MStorable -e 'Storable::retrieve( "tst.dat" );'
This tests just Storable (not PDL::IO::Storable) for portability
problems. If it works with no issues, then digging deeper is warranted,
I think.

This is clearly in my domain, so I can fix it.

Great! Please do so.

Post by Dima Kogan
But to be clear, are you really saying that there's a commonly used
perl build with no 64-bit pack support? I.e. does anybody have a reason
to use the "32bit SPP"?

Fixes for one PDL platform that break another PDL platform
build aren't useful. PDL-2.007 is currently testing >98% PASS
with CPAN testers and builds out of the box for all the supported
PDL platforms (32bit, 64bit, win32, macosx, linux,...).

Many people rely on PDL to "get the job done" for their research
or work. Having a new PDL release arbitrarily break on platform
X that has worked for years means folks can't upgrade to current
PDL releases with new features. This drags out the entire PDL
development and user community.

--Chris

Dima Kogan

2013-11-16 21:59:34 UTC

Post by Chris Marshall
The closest I have is cygwin vs SPP, the result, no matter
which is the store while the other is the retrieve is the
Byte order is not compatible at
/usr/lib/perl5/5.14/i686-cygwin-threads-64int/Storable.pm line 379, at
-e line 1.

OK... So it seems like Storable itself has portability issues. Although
the byte order itself should be the same, so maybe this is a problem
with one of the builds you're using. Can you please run a few more
experiments?

If it is one of the builds that's "weird", it'd be good to know which
one. I posit that the behavior on UNIX is "right", so the build that
can't read those files is "weird". As the code currently stands, are you
seeing a "Byte order is not compatible at" test failure on both cygwin
and SPP, or just one of them? If just one of them is failing then that's
the "weird" one. So then we should look there for the problem, maybe
filing a bug in the right BTS.

To be clear, the new PDL::IO::Storable didn't break anything here; the
new tests I added are just showing an old problem.

Post by Dima Kogan
But to be clear, are you really saying that there's a commonly used
perl build with no 64-bit pack support? I.e. does anybody have a reason
to use the "32bit SPP"?

Fixes for one PDL platform that break another PDL platform
build aren't useful. PDL-2.007 is currently testing >98% PASS
with CPAN testers and builds out of the box for all the supported
PDL platforms (32bit, 64bit, win32, macosx, linux,...).
Many people rely on PDL to "get the job done" for their research
or work. Having a new PDL release arbitrarily break on platform
X that has worked for years means folks can't upgrade to current
PDL releases with new features. This drags out the entire PDL
development and user community.

Yeah, I certainly don't want to break anything on platforms that people
use. The question was whether "32bit SPP" was one of those. Since it
appears to be, then we're on the same page and I'll try to fix it.

dima

Chris Marshall

2013-11-16 22:05:43 UTC

I just sent you the generated data files for spp64, spp32, and cygwin
along with the test output and the perl -V outputs.

OK... So it seems like Storable itself has portability issues. Although
the byte order itself should be the same, so maybe this is a problem
with one of the builds you're using. Can you please run a few more
experiments?
If it is one of the builds that's "weird", it'd be good to know which
one. I posit that the behavior on UNIX is "right", so the build that
can't read those files is "weird". As the code currently stands, are you
seeing a "Byte order is not compatible at" test failure on both cygwin
and SPP, or just one of them? If just one of them is failing then that's
the "weird" one. So then we should look there for the problem, maybe
filing a bug in the right BTS.
To be clear, the new PDL::IO::Storable didn't break anything here; the
new tests I added are just showing an old problem.

Agreed. The only "new" breakage was using [qQ] in the pack/unpack.
And, yes, it is a bit of a pain to account for back compatibility when
something is actually missing.

Post by Dima Kogan
But to be clear, are you really saying that there's a commonly used
perl build with no 64-bit pack support? I.e. does anybody have a reason
to use the "32bit SPP"?

Fixes for one PDL platform that break another PDL platform
build aren't useful. PDL-2.007 is currently testing >98% PASS
with CPAN testers and builds out of the box for all the supported
PDL platforms (32bit, 64bit, win32, macosx, linux,...).
Many people rely on PDL to "get the job done" for their research
or work. Having a new PDL release arbitrarily break on platform
X that has worked for years means folks can't upgrade to current
PDL releases with new features. This drags out the entire PDL
development and user community.

Thanks. -chm

s***@optusnet.com.au

2013-11-17 00:25:41 UTC

-----Original Message-----
From: Dima Kogan

Post by Dima Kogan
Can you run the following experiment for me, please? On a unixy box do
perl -MStorable -e 'Storable::store( [1,2,3], "tst.dat" );'
Then take that file and try to load it on the windows box you're seeing
perl -MStorable -e 'Storable::retrieve( "tst.dat" );'
This tests just Storable (not PDL::IO::Storable) for portability
problems. If it works with no issues, then digging deeper is warranted,
I think.

For me:

=====
Case 1
=====
Windows: ivsize == 8; ivtype eq 'long long'
Ubuntu: ivsize == 8; ivtype eq 'long'

If "tst.dat" is built on Windows, and retrieved on Ubuntu, then the error
(when trying to retrieve) is:

Long integer size is not compatible at
/home/sisyphus/perl518-multi/lib/5.18.0/x86_64-linux-thread-multi/Storable.pm
line 380, at -e line 1.

If "tst.dat" is built on Ubuntu, and retrieved on Windows, then the error
(when trying to retrieve) is:

Long integer size is not compatible at C:/_64/perl518_470/lib/Storable.pm
line 380, at -e line 1.

(Seems odd that they both complain about "long integer size" - I would have
expected that if one of those perls complained about "long integer size",
then the other should complain about "long long integer size".)

=====
Case 2
=====
Windows: ivsize == 4; ivtype eq 'long'
Ubuntu: ivsize == 8; ivtype eq 'long'

If "tst.dat" is built on Windows, and retrieved on Ubuntu, then the error
(when trying to retrieve) is:

Byte order is not compatible at
/home/sisyphus/perl518-multi/lib/5.18.0/x86_64-linux-thread-multi/Storable.pm
line 380, at -e line 1.

If "tst.dat" is built on Ubuntu, and retrieved on Windows, then the error
(when trying to retrieve) is:

Byte order is not compatible at C:/MinGW/perl516/lib/Storable.pm line 380,
at -e line 1.

So ... it seems that the "Byte order" error indicates that the the ivsizes
don't match, whereas the "Long integer" error indicates that the ivtypes
don't match.

Cheers,
Rob

Dima Kogan

2013-11-17 02:46:36 UTC

Post by s***@optusnet.com.au
-----Original Message-----
From: Dima Kogan

<snip>

Thanks for checking. I sent the below to chm earlier. By mistake I sent
it only to him and not the list.

Reading the Storable manpage, it seems that by default it writes things
in a non-portable way. However if you use 'nstore' instead of 'store'
then it does it portably (at least as far as endianness is concerned;
I'm hoping data sizes are handled too). This seems like an unfriendly
default, but that's what we have.

I'm attaching data files generated with nstore. Can you see if the tests
pass with those files instead of the older ones? The test script itself
doesn't need to be changed because the files carry meta-information
indicating that they're 'nstore' files.

Just copy those two files to t/, and do 'perl -Mblib t/storable.t'

Thanks

dima

Dima Kogan

2013-11-18 06:11:51 UTC

Post by s***@optusnet.com.au
-----Original Message-----
From: Dima Kogan

<snip>

Thanks for checking. I sent the below to chm earlier. By mistake I sent
it only to him and not the list.
Reading the Storable manpage, it seems that by default it writes things
in a non-portable way. However if you use 'nstore' instead of 'store'
then it does it portably (at least as far as endianness is concerned;
I'm hoping data sizes are handled too). This seems like an unfriendly
default, but that's what we have.
I'm attaching data files generated with nstore. Can you see if the tests
pass with those files instead of the older ones? The test script itself
doesn't need to be changed because the files carry meta-information
indicating that they're 'nstore' files.
Just copy those two files to t/, and do 'perl -Mblib t/storable.t'

Hi.

Do the tests pass on windows with the latest code? I imagine they do not
because of issues in Storable itself, right? If this is the case, can
somebody please try the tests with the files attached in the above
message (made with 'nstore' instead of 'store')? If that makes the tests
pass, I'll update the test suite with these.

dima

s***@optusnet.com.au

2013-11-18 07:45:51 UTC

-----Original Message-----
From: Dima Kogan

Post by Dima Kogan
Do the tests pass on windows with the latest code?

Hi Dima,

Yes, the storable tests pass on Windows with PDL-2.007_01.
It makes no difference whether I use the the two .dat files that ship with
the PDL-2.007_01 source, or the two .dat files you posted.

Tested on both 32-bit and 64-bit builds of perl.

The only problem is that perl-5.8's pack() function doesn't understand the
'<' template (forcing little-endian format). Easiest fix for that is to have
PDL no longer support perl-5.8.

Cheers,
Rob

Dima Kogan

2013-11-18 07:57:10 UTC

Post by s***@optusnet.com.au
-----Original Message-----
From: Dima Kogan

Post by Dima Kogan
Do the tests pass on windows with the latest code?

Hi Dima,
Yes, the storable tests pass on Windows with PDL-2.007_01.

OK. That's pretty surprising to me, since earlier in this thread we saw
that Storable itself had issues there. I'm perfectly happy to let it go,
though.

Thanks
dima

s***@optusnet.com.au

2013-11-18 08:54:26 UTC

Post by s***@optusnet.com.au
-----Original Message-----
From: Dima Kogan

Post by Dima Kogan
Do the tests pass on windows with the latest code?

Hi Dima,
Yes, the storable tests pass on Windows with PDL-2.007_01.
It makes no difference whether I use the the two .dat files that ship with
the PDL-2.007_01 source, or the two .dat files you posted.
Tested on both 32-bit and 64-bit builds of perl.
The only problem is that perl-5.8's pack() function doesn't understand the
'<' template (forcing little-endian format). Easiest fix for that is to
have PDL no longer support perl-5.8.
Cheers,
Rob

Ummm ... hang on ... there's some trickery going on here that I hadn't
noticed before.
Using the original .dat files, tests 10-23 are skipped with the message:

# skip LongLong only as IV bug in Storable

It's only when I use the revised .dat files (that you provided) that all
tests are actually run (and pass).
So, I guess we *do* need those revised .dat files after all.

Cheers,
Rob

Dima Kogan

2013-11-18 09:38:08 UTC

Post by s***@optusnet.com.au

Post by s***@optusnet.com.au
-----Original Message-----
From: Dima Kogan

Post by Dima Kogan
Do the tests pass on windows with the latest code?

Hi Dima,
Yes, the storable tests pass on Windows with PDL-2.007_01.
It makes no difference whether I use the the two .dat files that ship with
the PDL-2.007_01 source, or the two .dat files you posted.
Tested on both 32-bit and 64-bit builds of perl.
The only problem is that perl-5.8's pack() function doesn't understand the
'<' template (forcing little-endian format). Easiest fix for that is to
have PDL no longer support perl-5.8.
Cheers,
Rob

Ummm ... hang on ... there's some trickery going on here that I hadn't
noticed before.
# skip LongLong only as IV bug in Storable
It's only when I use the revised .dat files (that you provided) that all
tests are actually run (and pass).
So, I guess we *do* need those revised .dat files after all.

OK; pushed. Thanks.

dima

Chris Marshall

2013-11-18 18:19:49 UTC

The latest push from Dima has resolved all the problems
with SPP32 and SPP64 builds for PDL::IO::Storable.

-chm

s***@optusnet.com.au

2013-11-19 14:25:32 UTC

Hi,

I think Dima (at some stage) asked about the storable behaviour on a
big-endian machine ... or maybe I dreamt it.
Anyway, I was curious to see how PDL would go on such a machine, and finally
got around to trying it out tonight.

On a big-endian machine (powerpc, running Debian wheezy, perl-5.18.1), using
the two .dat files that shipped with the 2.007_01 source, I get:

sisyphus-***@debian-sis:~/comp/PDL-2.007_01$ perl t/storable.t
1..23
ok 1 - use PDL::IO::Storable;
ok 2 - PDL freeze/thaw
ok 3 - The reference isa HASH
ok 4 - PDL in structure
ok 5 - PDL->can('freeze')
ok 6 - PDL has-a works with freeze/thaw
ok 7 - PDL is a hash
ok 8 - Initialization from seraialized object
ok 9 - Initialization from dclone object
ok 10 # skip LongLong only as IV bug in Storable
ok 11 # skip LongLong only as IV bug in Storable
ok 12 # skip LongLong only as IV bug in Storable
ok 13 # skip LongLong only as IV bug in Storable
ok 14 # skip LongLong only as IV bug in Storable
ok 15 # skip LongLong only as IV bug in Storable
ok 16 # skip LongLong only as IV bug in Storable
ok 17 # skip On a big endian machine the old stored files will be bogus
ok 18 # skip On a big endian machine the old stored files will be bogus
ok 19 # skip On a big endian machine the old stored files will be bogus
ok 20 # skip On a big endian machine the old stored files will be bogus
ok 21 # skip On a big endian machine the old stored files will be bogus
ok 22 # skip On a big endian machine the old stored files will be bogus
ok 23 # skip On a big endian machine the old stored files will be bogus

If I replace those two .dat files with the ones that Dima posted to the list
(and ultimately pushed to git), I get:

sisyphus-***@debian-sis:~/comp/PDL-2.007_01$ perl t/storable.t
1..23
ok 1 - use PDL::IO::Storable;
ok 2 - PDL freeze/thaw
ok 3 - The reference isa HASH
ok 4 - PDL in structure
ok 5 - PDL->can('freeze')
ok 6 - PDL has-a works with freeze/thaw
ok 7 - PDL is a hash
ok 8 - Initialization from seraialized object
ok 9 - Initialization from dclone object
ok 10 - Reading from file 't/storable_new_amd64.dat'
ok 11 - Reading an array-ref of size 3 from file 't/storable_new_amd64.dat'
ok 12 - Reading a correct string from file 't/storable_new_amd64.dat'
ok 13 - Reading a piddle from file 't/storable_new_amd64.dat' isa PDL
ok 14 - Reading another piddle from file 't/storable_new_amd64.dat' isa PDL
ok 15 - Read correct data from file 't/storable_new_amd64.dat'
ok 16 - Read more correct data from file 't/storable_new_amd64.dat'
ok 17 # skip On a big endian machine the old stored files will be bogus
ok 18 # skip On a big endian machine the old stored files will be bogus
ok 19 # skip On a big endian machine the old stored files will be bogus
ok 20 # skip On a big endian machine the old stored files will be bogus
ok 21 # skip On a big endian machine the old stored files will be bogus
ok 22 # skip On a big endian machine the old stored files will be bogus
ok 23 # skip On a big endian machine the old stored files will be bogus

Is there any further testing (especially regarding endianness issues) to do
on this machine ?

The test suite runs fine, so long as ExtUtils::F77 doesn't get found -
something not quite right with the gfortran that apt-get delivered.

And it's just a bare-basics build of PDL - none of the optional extras apart
from pthreads support.

Cheers,
Rob

Dima Kogan

2013-11-19 17:46:51 UTC