1. Building open source with Django
#dbbug Django Bath and Bristol Users Group
Hi, my name is Ed Crewe and I work at the University of Bristol, in the IT Services R&D section.
We act in a cross over roll between internal projects, and services, external commercial services (such as BOS), consultancy and research. 
This talk is mainly for web developers, particularly django ones. It starts with a little look at the project that is requiring me to build a web authorisation system. Followed by what I picked up in terms of web authorisation standards. A review of the django third party eggs that are available for this task. Finally what I went with and how I am building it.
Finishing with some python code.
I have used Python ever since it was thrust upon me by an academic enthusiast back in when the University jumped on the zope bandwagon.
Having developed the Universities CMS in zope and Plone - Over the last few years I have jumped off that venerable but rather ancient vehicle and into Django.
(Although I still have to be there for our University's public Plone based web)
Ed Crewe

2. Introduction Ed Crewe * - DBBUG Python and Django developer.
Representing Django Bristol and Bath User Group.
Django is a leading web framework, in Python, used by the likes of Mozilla, Instagram and OpenStack.
It is the framework bundled with Google App Engine - whose team is lead by the inventor of Python, Guido Van Rossum.
I am here as a developer who uses Python - in recent years mainly via one of its leading web frameworks, Django. Hence
10 mins on DBBUG, python, django and open source
* Developer at the University of Bristol (& amateur painter - see slide pics!)
- I used to work at UWE before that!

3. What is python?
Python is a high level programming language, it inherits many of the same principles set out by the first programming language explicitly designed to be object orientated, Smalltalk (1970)
Python was invented at the end of the 1980s and a number of its concepts were shared by Java when it came along a few years later. The main difference being Java's static typing and verbose syntax, rather than Python's / Smalltalk's strong dynamic typing.
The core areas of Python use are scientific / numerical analysis (replacing Fortran), educational tools (Raspberry Pi etc.), config management, and of course web software.
The former two stem from interactive programming (iPython), where Python's ability to use run time, as well as pre-compilation, caters for running programs as you write them.
Useful both for kids controlling robots, or banking quant.s analysing big data streams of share prices.

4. The Zen of Python
The key ethos of Python is quite well adhered to by the creators of Django.
Simple, explicit, DRY (inheritance), less code, RAD, loose coupling tight cohesion.
Python lets you do what you want - procedural or OOP, multiple inheritance or interfaces, pre-compiled or meta / interactive programming ... so unusually diverse in methodology terms, but it has very strict rules of convention - break them and you advertise your failings - but those failings will still compile.
There should ideally only be one syntactic way any line of code is written, and that way is the simplest and most readable way - see the PEPs (Python Enhancement Proposals). However a core principle of Python coding is also
'It is easier to ask forgiveness, than permission' coined by Grace Hopper
Hence exception handling for program flow is OK, programming methodologies or typing should not be enforced - so you can go do what you want, as long as you write for readability, then ask forgiveness ... if it doesn't pass PEP lint tests!
So its not really acceptable to write code in any style you like as you might in Java, C#, PHP etc. Convention is really a requirement for Python, its a bit like valid HTML, its not just a nice to have.10 mins on DBBUG, python, django and open source
There is full documentation of the standards in the - and there is always only one way to do it right.
(ie. these aren't Perl standards - as many standards as there are programmers!)
So documentation and naming standards, no more than 80 characters a line, use of spaces etc.

5. What is django?
Django was created around the same time as Ruby on the Rails and open sourced a few months afterwards in 2005.
Like Rails it is MVC based and one of the top ten most popular web frameworks, alongside ASP MVC, CakePHP, Spring, Flask etc.
Its a medium complexity orientated framework, sitting between Spring and the micro-frameworks (eg. Flask) on the scale of large development vs. productivity.
So for systems that require less than 10 full time developers - but are more than a few weeks work for one.
Like most significant open source projects it has a non-profit foundation to promote it and protect its licences.
Probably the most well known example is the Apache foundation* - which looks after many smaller open source projects, particularly Java based ones, perhaps because due to its commercial heritage Java doesn't have its own language foundation like Python, Perl etc.
10 mins on DBBUG, python, django and open source
25 mins on talk
5 mins qu.s
* Colleague Damian Steer in Bristol Uni ITS R&D is a member

6. #dbbug Django Bath and Bristol Users Group
Started by Dan Fairs in 2008 ~ 100 members.
Mainly developers with a scattering of project
managers.
The Django Bath and Bristol Users Group is a place for local developers and companies to come and talk Django, Python, and web development in general. We run events every couple of months, with a mix of socials and technical meets.
Usually between 10 and 20 members turn up.
Membership is a mix of freelancers and company members, eg. from The House of Omni, ISM Games, Potato, SecondSync, Digital TV labs, Maplecroft, Joinerysoft, Netsight, Bravo6 etc.
plus the odd public sector bod like me from the University of Bristol and Cardiff University

7. What do we do?
#dbbug
We tend to alternate, talk sessions with socials.
So it may be lightning talks or more a more in depth talks from a member.
Even at the talk sessions there will usually be some social lubrication such as beer and pizza - hopefully paid for by a recruitment company!
We have also done a django dash (a competitive sprint), and a group outing to Djangocon.eu, a couple of years ago when it was in Amsterdam.

8. #dbbug Come and join us (...you don't have to be an evangelist)
Please come along to our next meet.
Not that we have a date for it yet, but just the list at Google group search for dbbug
... if you and ask to meet, it will prompt
somebody to suggest a date for the next one - so you can be the inspiration for that gathering!
You don't have to be a developer or a Python user already, but it helps if you aim to be one.
List is mainly for group events (sometimes gets job offers and newbie questions too)
(I guess there are better django lists for technical questions)

9. Django use at Bristol University
Django is being used at the University to provide an integration framework for web applications that need to appear on our public web.
So this integrates with our institutional CMS driven content and central data sources (e.g.
We are also using it to rewrite our national survey system BOS (like survey monkey for academic and cross institutional provision)

10. Background - the survey application
BOS has been going for a decade!
Now rewriting in django - so battle of the python web frameworks, but in our case django is david and pyramid is goliath in the form of survey monkey.
More seriously - we are targeting a more specialist academic niche - where institutions often wish to aggregate results between them.
along with requiring help wrt. producing an academically credible survey / data collection.
But seriously we are in a specific sector which ideally is not in direct competion with our bigger python brother.
As you can see from the stats it is more than qualified as a national HE service - with the majority of the UK HE sector as our main customers.
We are moving from Perl to Python (and Django) and adding a range of new features along the way.
Along with a more modular and exposed API code base.

11. Here is a very conceptual! look at the architecture.
(NB: This slide and the preceding one are taken from the BOS poster - So some of you may recognize it - or have chatted about it at those sessions)
Backend storage - Postgres for structured data storage - survey constructions, accounts etc.
Cassandra for survey responses - so the great thing about Cassandra (like BigTable, et al.) it writes faster than it reads (where conflict resolution occurs)
It is also built around the cluster concept (you need 3 servers as a minimum!) - this means it can happily handle hundreds of thousands of respondents filling in surveys and not fall over like a traditional RDBMS
We use a combination of data access layers, and were in the process of tying them together with tasty-pie
Very simple concept -
survey design in -> Postgres
respond -> Cassandra
analyse <- Cassandra and possibly Postgres for more complex aggregation queries.
BUT .... leaving all that aside ... today I am just talking about federated authorisation ... so we are just looking at the bottom of the second speech bubble.

12. Agile practises
Daily scrum to report progress and decide current work units
Sprints of 3-4 weeks duration
Back log to handle sprint over spill
Product owner to help lead overall work prioritisation
Continuous Integration - nightly builds for various test suites
Maybe not full TDD but good test coverage
Continuous refactor effort
B.C.S. SCRUM talk by Nigel Baker - Agile Bear

13. Packaging functional components
Splitting a large development project into packages
Caters for separate tests, documentation, release management and development cycles.
Encourages increase in modularisation and encapsulation - helps to decouple the code. (Macro level OOP!)
Adding generic functionality - separate package can be used across projects.
Aim for it to be entirely generic then it can be released - this then opens it up to being freely tested and improved and / or influence other components of an open framework
For remote role allocation we use a bespoke standard means of setting entitlements via a parseable string.
the string has the option to set things for a whole class of objects (within an account)
or for specific instances of them.
There is also a means to add include/exlude qualifiers to cater for institutions who do not want to handle these set operations within their group management tools.

14. Python packages In python (and hence django) packages are called eggs.
There is a central repository of all open source eggs called
PyPI (the cheese shop), like CPAN.
PyPI is built into the core language packaging features. Create a folder add a few files including setup.py, to make it an egg, and run python setup.py register to push it to PyPI.
A user of your code then just has to download and run python setup or use the pip add on and run pip install package-name to add it to their django based site.

15. e.g. django-cookieless https://pypi.python.org/pypi/django-cookieless
Django requires cookies to maintain session, and hence for authorisation.
However there are cases when forms are used on public sites, where setting cookies is not desirable (due to privacy legislation), since technically they are not required for anonymous users to respond to forms. So if used, strictly consent is needed from the user to set them.
This package is designed to cater for anonymous user session maintenance, without cookies.
It provides a sessions implementation and decorator class for views to allow for forms to maintain state without using cookies by posting the encrypted session id between forms, or via urls. This also enables a user to run multiple sessions across different browser tabs.
We have handled the first half of our requirements. Now what about the next half...

16. Open source ethos
Python ethos = OS ethos - code should be readable, openly accessible, reusable and adaptable by everyone.
So open source should be well documented and maintained with test suites, release management. To deliver these goals.
Note that the majority of commercial software is also now, at least partially, open source.
The ethos of Python is unsurprisingly closely related to the ethos of open source itself (and of the whole spectrum of the origins of the web - W3C - open formats & knowledge - academic peer review principles, etc.)
Some take this to the extent that just making code public that isn't well documented, and release managed with tests etc., is worse than not releasing - since you dirty the waters with potentially impenetrable rotting code.
At worst this results in the kind of fragile, black box coding that often has to be employed with closed source code.
Since the open principle tends to lead to more robust / financially viable components.
The only significant exceptions were Microsoft and Oracle - but even software companies who still stick to the old licence charging funding model, now want to be seen to be on board too. Microsoft Openness, Oracle MySQL etc.

17. Sharing code To start sharing code, just reply to somebody with some.
Maybe on a platform list or stack overflow or the like.
If its a longer snippet you may want to use your platform's favourite paste bin site, eg. to share it.
If you think its of general use why not make it more permanently public, for django that means
If its more than a snippet then it should be in a versioning system, so why not pick a public one ...
The ethos of Python is unsurprisingly closely related to the ethos of open source itself (and of the whole spectrum of the origins of the web - W3C - open formats & knowledge - academic peer review principles, etc.)
Some take this to the extent that just making code public that isn't well documented, and release managed with tests etc., is worse than not releasing - since you dirty the waters with potentially impenetrable rotting code.
At worst this results in the kind of fragile, black box coding that often has to be employed with closed source code.
Since the open principle tends to lead to more robust / financially viable components.
The only significant exceptions were Microsoft and Oracle - but even software companies who still stick to the old licence charging funding model, now want to be seen to be on board too. Microsoft Openness, Oracle MySQL etc.

18. Community coding Like Facebook, but with a purpose! Site Pros Cons
github.com
market leader
only does git
bitbucket.com
(Atlassian)
git or mercurial *
free private repos
Its written in django :-)
has been accused of plaigarising github
code.google.com
git, mercurial or
svn (for the old folks)
poorly featured,
few signs of new releases
gitorious.org
like github but open source infrastructure
bit flakey and smaller scale
sourceforge.net
the original ...
... and worst.
graveyard of old projects
We have handled the first half of our requirements. Now what about the next half...
* Mercurial is the Python twin of Git (written in C), separated at birth!

19. The open road
Reply to help lists - stack overflow or python and django specific
Reply to project support chat rooms
Reporting bugs
Fixing bugs
Adding new features to existing packages
Contributing enough to warrant a new release
Coagulating snippets, paste bins, stack overflows into an egg
Creating and releasing your own package
Acting as a release manager
Becoming a core contributor of a large project
Starting your own large project / framework / language (Warning 11 is not for people who have a life outside coding)
We have handled the first half of our requirements. Now what about the next half...

20. Creative co-operation
Githubs fun visualisation of contributors to one of my eggs .... hmm need to catch up merging those forks, or I will lose release manager / origin status for it ...
We have handled the first half of our requirements. Now what about the next half...

21. Don't be afraid to commit
Contributing to Django core, or other Python projects
(Boosting you half way along the open road!)
A workshop next week, 8th June, by fellow DBBUG member, Daniele Procida
(Details - room 3 - 1:50pm)
See for more information.
The workshop will take participants through the complete cycle of
identifying a simple issue in a Django or Python project, writing a patch
with tests and documentation, and submitting it.
The workshop is aimed at the first-time committer. Very little experience
is required:
We have handled the first half of our requirements. Now what about the next half...

22. Interested in Django ... home dbbug free book 90 other books
dbbug
free book
90 other books
chat #django IRC
help
core devs
other dbbugs
jobs
and
add ons

23. Ed Crewe django bath and bristol user group #dbbug
Contact
django bath and bristol user group
#dbbug
Django Bath and Bristol Users Group (DBBUG)
The Django Bath and Bristol Users Group is a place for local developers and companies to come and talk Django, Python, and web development in general. We run events every couple of months, with a mix of socials and technical meets.
Building open source with Django
Django is a leading web framework, in Python, used by the likes of Mozilla, Instagram and OpenStack. BOS is a survey system developed by the R&D team over the last decade, and in use by most UK Universities. Find out how we are rewriting BOS from scratch in Django, Cassandra and Postgres, using current agile practises. Along with how to package a component of a larger product, to contribute back to an open source tool set.
PS: Also a member of SW Big Data
/south-west-big-data/
university of bristol - ITS - R&D