[Pdl-porters] grammars for PDL

Discussion:

Chris Marshall

2014-11-24 17:22:28 UTC

I've been thinking about ways to improve and future-proof PDL and how
to smooth the PDL-2.x to PDL3 transistion. Does it make sense to
consider developing explicit BNF grammars for various PDL features as
a way to better specify (and quantify!) what PDL "means"? E.g., if we
specify a grammar for NiceSlice syntax, that might make is easier to
make a more portable implementation, say using Marpa...

Thoughts (I'm not a computer scientist)?
Chris

David Mertens

2014-11-25 03:36:24 UTC

Permalink

What needs to be grammarized? All PDL operations in Perl code already fit
within the grammar provided Perl for object methods and functions.
PDL::NiceSlice does not have an explicit grammar, but one could probably be
written. I don't think we'd win much speed over the regex-based
implementation unless we did some sort of keyword-like modification to the
Perl lexer/tokenizer, but that's not really possible with Perl in its
current state.

However, if you are proposing something like modifying Perl's
lexer/tokenizer to make a formal grammar for PDL::NiceSlice, I can think of
some other notation that might be nice to have. Matlab has the "Matrix \
vector" for matrix-inversion-times-vector that I think is nice.

What's left? PDL::PP? That needs more than a grammar... :-)

David

Post by Chris Marshall
I've been thinking about ways to improve and future-proof PDL and how
to smooth the PDL-2.x to PDL3 transistion. Does it make sense to
consider developing explicit BNF grammars for various PDL features as
a way to better specify (and quantify!) what PDL "means"? E.g., if we
specify a grammar for NiceSlice syntax, that might make is easier to
make a more portable implementation, say using Marpa...
Thoughts (I'm not a computer scientist)?
Chris
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

Bryan Jurish

2014-11-25 07:58:49 UTC

Permalink

morning all,

I agree with David that an (additional) explicit grammar is not strictly
necessary -- the well-formedness conditions are implicit in the code, and
the regexes that govern e.g. slice parsing are themselves a species of
grammar (perl regexes are basically Chomsky type-3 grammars on steroids).
Also, an explicit BNF-style grammar would only make the syntax explicit,
not the semantics: it wouldn't in and of itself tell us anything about what
the constituents (terminals and non-terminals) *mean*. That said, an
explicit grammar with an intuitive naming scheme for the constituents tends
to go a long way as far as documentation is concerned, even without
explicit (formal) semantics, which tend to get technical and messy (just
like code does), and can also help to make gaps, omissions, and potential
extensions of the existing syntax more apparent. All in all, I think that
if there's a transparent and self-documenting way to bootstrap and maintain
an explicit grammar (such as Parse::Yapp for context-free LALR parsers),
that can only be a Good Thing. Of course, as a computational linguist, my
opinions on the matter of grammars may be somewhat biased ;-)

marmosets,
Bryan

Post by David Mertens
What needs to be grammarized? All PDL operations in Perl code already fit
within the grammar provided Perl for object methods and functions.
PDL::NiceSlice does not have an explicit grammar, but one could probably be
written. I don't think we'd win much speed over the regex-based
implementation unless we did some sort of keyword-like modification to the
Perl lexer/tokenizer, but that's not really possible with Perl in its
current state.
However, if you are proposing something like modifying Perl's
lexer/tokenizer to make a formal grammar for PDL::NiceSlice, I can think of
some other notation that might be nice to have. Matlab has the "Matrix \
vector" for matrix-inversion-times-vector that I think is nice.
What's left? PDL::PP? That needs more than a grammar... :-)
David

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan
_______________________________________________
PDL-porters mailing list
http://mailman.jach.hawaii.edu/mailman/listinfo/pdl-porters

--
Bryan Jurish "There is *always* one more bug."
***@gmail.com -Lubarsky's Law of Cybernetic Entomology

Chris Marshall

2014-11-25 18:07:53 UTC

Permalink

My thought for PDL::NiceSlice was that by generating
a grammar for the niceslice syntax it would provide
the structure needed to verify and document the
implementation---especially as regards conflicts or
not with existing perl syntax and usage.

There are a number of users not using PDL::NiceSlice
because the current implementation is incomplete and is
known to affect code that is not intended. The additional
structure should make it possible to implement a more
complete set of tests that could be used to verify the
implementation as correct.

Post by Bryan Jurish
morning all,
I agree with David that an (additional) explicit grammar is not strictly
necessary -- the well-formedness conditions are implicit in the code, and
the regexes that govern e.g. slice parsing are themselves a species of
grammar (perl regexes are basically Chomsky type-3 grammars on steroids).
Also, an explicit BNF-style grammar would only make the syntax explicit,
not the semantics: it wouldn't in and of itself tell us anything about what
the constituents (terminals and non-terminals) *mean*. That said, an
explicit grammar with an intuitive naming scheme for the constituents tends
to go a long way as far as documentation is concerned, even without
explicit (formal) semantics, which tend to get technical and messy (just
like code does), and can also help to make gaps, omissions, and potential
extensions of the existing syntax more apparent. All in all, I think that
if there's a transparent and self-documenting way to bootstrap and maintain
an explicit grammar (such as Parse::Yapp for context-free LALR parsers),
that can only be a Good Thing. Of course, as a computational linguist, my
opinions on the matter of grammars may be somewhat biased ;-)
marmosets,
Bryan

--
Bryan Jurish "There is *always* one more bug."

Dystan Hays

2014-11-26 01:27:58 UTC

Permalink

Hi all,

First, I'm always appreciative of your (Chris's) thinking and of the
style in which you've
been handling the huge role you've played for such a long time -- and I
agree that this
ploy might be helpful for NiceSlice.

Second, I'm not sure (like several of the comments) how helpful it'd be
for PDL more
generally, but:

Third, I believe the big payoff for formal grammar (and other
meta-compilation tools
as well -- Marpa and the older lex- and yacc-style ones) would be for
building converters
from IDL, Matlab, R, Pymath or whatever the Python folks use, and so forth
(By the way,
is anyone familiar or working with Julia? I've just come across meetups for
it here in
the L.A. region.) -- to PDL. I think there's a big payoff for the users
of all those tools
to switch to PDL both because it's such an elegant language for specifying
calculations
(and performs so well) and also because it's so smooth a transition to
using the features
of a real full featured, general-purpose language in Perl.

I think there'd be a humongous payoff both for all of them as well as for
the PDL dev
community to bring them in by greasing the skids, making it easier for them
to make
the transition -- and, at least, try PDL to get a taste...

regards, -- John Kirk (267) 882-7777 (Dystan Hays
is a pen-name)

Post by Chris Marshall
My thought for PDL::NiceSlice was that by generating
a grammar for the niceslice syntax it would provide
the structure needed to verify and document the
implementation---especially as regards conflicts or
not with existing perl syntax and usage.
There are a number of users not using PDL::NiceSlice
because the current implementation is incomplete and is
known to affect code that is not intended. The additional
structure should make it possible to implement a more
complete set of tests that could be used to verify the
implementation as correct.

On Mon, Nov 24, 2014 at 12:22 PM, Chris Marshall <