1. Data::Iterator::Hierarchical A module by Brian McCauley Birmingham Perl Mongers

2. Data::Iterator::Hierarchical ● The problem ● My solution ● Experience of making it a CPAN module – Naming – API must be explainable – Documentation – Tests – Packaging

3. The problem ● A rowset from ordered SQL left join query – Is conceptually hierachical agent=X co=B, co_name=Belgium sound=fizz co=D, co_name=Germany sound=bang sound=pow sound=zap agent=Y agent=Z co=B, co_name=Belgium co=E, co_name=Spain sound=bar co=I, co_name=Italy sound=foo

4. Ugly solution ● Keep track of previous value of each variable ● Does not look “natural” ● Does not extend nicely ● Really messy if doing more than printing my ($prev_agent,$prev_co); while (my ($agent,$co,$co_name,$sound) = $sth->fetchrow_array ) { unless ( defined $prev_agent && $prev_agent eq $agent ) { print "agent=$agent\n"; undef $prev_co; } next unless defined $co; unless ( defined $prev_co && $prev_co eq $co ) { print " co=$co, co_name=$co_name\n"; } print " sound=$sound\n"; } my ($prev_agent,$prev_co); while (my ($agent,$co,$co_name,$sound) = $sth->fetchrow_array ) { unless ( defined $prev_agent && $prev_agent eq $agent ) { print "agent=$agent\n"; undef $prev_co; } next unless defined $co; unless ( defined $prev_co && $prev_co eq $co ) { print " co=$co, co_name=$co_name\n"; } print " sound=$sound\n"; }

5. Pretty solution ● I would like some nested loops ● Looks cleaner ● Extends easily ● Just find the “somethings” something $sth; while( my ($agent) = something1 ) { print "agent=$agent\n"; while( my ($co,$co_name) = something2 ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = something3 ) { print " sound=$sound\n"; } something $sth; while( my ($agent) = something1 ) { print "agent=$agent\n"; while( my ($co,$co_name) = something2 ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = something3 ) { print " sound=$sound\n"; }

6. API choices ● Assume I can implement the “somethings” – Evolution of internals would be another talk! ● What should they look like? – Clear – Robust – Easy to explain (document) – Extensible – Appropriately named

7. First attempt ● Personal / in-house use – Single function returning iterator closure ● Hard to document ● Fragile my $it = recursive_iterator [1,2,1],$sth; while( my ($agent) = $it->() ) { print "agent=$agent\n"; while( my ($co,$co_name) = $it->() ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it->() ) { print " sound=$sound\n"; } my $it = recursive_iterator [1,2,1],$sth; while( my ($agent) = $it->() ) { print "agent=$agent\n"; while( my ($co,$co_name) = $it->() ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it->() ) { print " sound=$sound\n"; }

8. Small refinement ● Address some the fragility – Optionally make depth explicit and die if wrong ● Still hard to document ● Pain to add more levels my $it = recursive_iterator [1,2,1],$sth; while( my ($agent) = $it->(1) ) { print "agent=$agent\n"; while( my ($co,$co_name) = $it->(2) ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it->(3) ) { print " sound=$sound\n"; } my $it = recursive_iterator [1,2,1],$sth; while( my ($agent) = $it->(1) ) { print "agent=$agent\n"; while( my ($co,$co_name) = $it->(2) ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it->(3) ) { print " sound=$sound\n"; }

9. I needed help ● Turned to my Perl Monger colleagues ● Showed them what I had so far ● Showed a few API options ● Got useful feedback

10. Naming the module ● Need a name for the module ● Iterator::Recursive – The Iterator namespace – expectation of being based on the Iterator module – Recursive – not really descriptive enough ● DBIx::whatever – This isn't just for DBI ● Data::Iterator::Hierarchical

11. Final API ● Still a function returning iterator closure ● Iteration of outer iterator gets inner iterator ● Too good to be true? – Use Want module to know how many to return – Or use a 2 nd argument in iterator call use Data::Iterator::Hierarchical; my $it = hierarchical_iterator($sth); while( my ($agent) = $it->(my $it_co)) { print "agent=$agent\n"; while( my ($co,$co_name) = $it_co->(my $it_sound) ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it_sound->() ) { print " sound=$sound\n"; } use Data::Iterator::Hierarchical; my $it = hierarchical_iterator($sth); while( my ($agent) = $it->(my $it_co)) { print "agent=$agent\n"; while( my ($co,$co_name) = $it_co->(my $it_sound) ) { print " co=$co, co_name=$co_name\n"; while( my ($sound) = $it_sound->() ) { print " sound=$sound\n"; }

12. DBI is not the only rowset source ● Arrays ● Iterator functions ● Any object with a fetchrow_array sub hierarchical_iterator { my ($input) = @_; my $get = do { if (ref($input) eq 'CODE') { $input; } elsif ( ref($input) eq 'ARRAY' ) { +sub { @{ shift @$input || [] } } ; } else { +sub { $input->fetchrow_array }; }; #.... } sub hierarchical_iterator { my ($input) = @_; my $get = do { if (ref($input) eq 'CODE') { $input; } elsif ( ref($input) eq 'ARRAY' ) { +sub { @{ shift @$input || [] } } ; } else { +sub { $input->fetchrow_array }; }; #.... }

13. Skipping data ● Not using all the data from an iterator – Discards some input lines – Exiting a loop prematurely – Conditionally ignoring inner iterators – Pure dummy iterators while( my ($agent) = $it->(my $it_co)) { next if $agent eq 'Q'; print "agent=$agent\n"; while( my ($co,$co_name) = $it_co->(my $it_sound) ) { last if $co eq 'D'; print " co=$co, co_name=$co_name\n"; } while( my ($agent) = $it->(my $it_co)) { next if $agent eq 'Q'; print "agent=$agent\n"; while( my ($co,$co_name) = $it_co->(my $it_sound) ) { last if $co eq 'D'; print " co=$co, co_name=$co_name\n"; }

14. Iterators not always wanted ● Iterators good for outer loops ● Innermost loops boringly simple – Just build hashes and/or arrays ● Better let the module take the strain my $item = {}; while( my ($item) = $it->( my $it2 ) ) { $items->{$item} = \my @values; while( my ($value) = $it2->() ) { push @values => $value; } my $item = {}; while( my ($item) = $it->( my $it2 ) ) { $items->{$item} = \my @values; while( my ($value) = $it2->() ) { push @values => $value; } $item = $it->slurp( hash_depth => 1, one_column => 1 );

15. Tests ● Even with a simple module there are a lot – Basic load – All the different patterns of NULLs – All the ways of bailing out of loop – All the slurp variants – POD (and POD coverage) – Optional dependency on Want – Examples actually work Code

16. Testing optional dependency ● Test that – Want is used correctly if it is installed – We carp if it's not ● Also test that – We would carp if it were not installed – It would be used if it were installed ● When actually isn't ● Even if it actually is! ● Maybe I should write a module! Code

17. Testing examples ● Couldn't find a module to do it ● Did it all rather ad-hoc ● Maybe I should write a module! Code

18. Packaging ● Metadata ● Choice of archiving program ● Optional dependency problem – Don't need Want.pm – But if it's there need later than 0.12 Bug

19. Thankyou ● Questions?