Presentation is loading. Please wait.

Presentation is loading. Please wait.

Building open source with Django

Similar presentations


Presentation on theme: "Building open source with Django"— Presentation transcript:

1 Building open source with Django
#dbbug Django Bath and Bristol Users Group Hi, my name is Ed Crewe and I work at the University of Bristol, in the IT Services R&D section. We act in a cross over roll between internal projects, and services, external commercial services (such as BOS), consultancy and research.  This talk is mainly for web developers, particularly django ones. It starts with a little look at the project that is requiring me to build a web authorisation system. Followed by what I picked up in terms of web authorisation standards. A review of the django third party eggs that are available for this task. Finally what I went with and how I am building it. Finishing with some python code. I have used Python ever since it was thrust upon me by an academic enthusiast back in when the University jumped on the zope bandwagon. Having developed the Universities CMS in zope and Plone - Over the last few years I have jumped off that venerable but rather ancient vehicle and into Django. (Although I still have to be there for our University's public Plone based web) Ed Crewe

2 Introduction Ed Crewe * - DBBUG Python and Django developer.
Representing Django Bristol and Bath User Group. Django is a leading web framework, in Python, used by the likes of Mozilla, Instagram and OpenStack. It is the framework bundled with Google App Engine - whose team is lead by the inventor of Python, Guido Van Rossum. I am here as a developer who uses Python - in recent years mainly via one of its leading web frameworks, Django. Hence 10 mins on DBBUG, python, django and open source * Developer at the University of Bristol (& amateur painter - see slide pics!) - I used to work at UWE before that!

3 What is python? Python is a high level programming language, it inherits many of the same principles set out by the first programming language explicitly designed to be object orientated, Smalltalk (1970) Python was invented at the end of the 1980s and a number of its concepts were shared by Java when it came along a few years later. The main difference being Java's static typing and verbose syntax, rather than Python's / Smalltalk's strong dynamic typing. The core areas of Python use are scientific / numerical analysis (replacing Fortran), educational tools (Raspberry Pi etc.), config management, and of course web software. The former two stem from interactive programming (iPython), where Python's ability to use run time, as well as pre-compilation, caters for running programs as you write them. Useful both for kids controlling robots, or banking quant.s analysing big data streams of share prices.

4 The Zen of Python The key ethos of Python is quite well adhered to by the creators of Django. Simple, explicit, DRY (inheritance), less code, RAD, loose coupling tight cohesion. Python lets you do what you want - procedural or OOP, multiple inheritance or interfaces, pre-compiled or meta / interactive programming ... so unusually diverse in methodology terms, but it has very strict rules of convention - break them and you advertise your failings - but those failings will still compile. There should ideally only be one syntactic way any line of code is written, and that way is the simplest and most readable way - see the PEPs (Python Enhancement Proposals). However a core principle of Python coding is also 'It is easier to ask forgiveness, than permission' coined by Grace Hopper Hence exception handling for program flow is OK, programming methodologies or typing should not be enforced - so you can go do what you want, as long as you write for readability, then ask forgiveness ... if it doesn't pass PEP lint tests! So its not really acceptable to write code in any style you like as you might in Java, C#, PHP etc. Convention is really a requirement for Python, its a bit like valid HTML, its not just a nice to have.10 mins on DBBUG, python, django and open source There is full documentation of the standards in the - and there is always only one way to do it right. (ie. these aren't Perl standards - as many standards as there are programmers!) So documentation and naming standards, no more than 80 characters a line, use of spaces etc.

5 What is django? Django was created around the same time as Ruby on the Rails and open sourced a few months afterwards in 2005. Like Rails it is MVC based and one of the top ten most popular web frameworks, alongside ASP MVC, CakePHP, Spring, Flask etc. Its a medium complexity orientated framework, sitting between Spring and the micro-frameworks (eg. Flask) on the scale of large development vs. productivity. So for systems that require less than 10 full time developers - but are more than a few weeks work for one. Like most significant open source projects it has a non-profit foundation to promote it and protect its licences. Probably the most well known example is the Apache foundation* - which looks after many smaller open source projects, particularly Java based ones, perhaps because due to its commercial heritage Java doesn't have its own language foundation like Python, Perl etc. 10 mins on DBBUG, python, django and open source 25 mins on talk 5 mins qu.s * Colleague Damian Steer in Bristol Uni ITS R&D is a member

6 #dbbug Django Bath and Bristol Users Group
Started by Dan Fairs in 2008 ~ 100 members. Mainly developers with a scattering of project managers. The Django Bath and Bristol Users Group is a place for local developers and companies to come and talk Django, Python, and web development in general. We run events every couple of months, with a mix of socials and technical meets. Usually between 10 and 20 members turn up. Membership is a mix of freelancers and company members, eg. from The House of Omni, ISM Games, Potato, SecondSync, Digital TV labs, Maplecroft, Joinerysoft, Netsight, Bravo6 etc. plus the odd public sector bod like me from the University of Bristol and Cardiff University

7 What do we do? #dbbug We tend to alternate, talk sessions with socials. So it may be lightning talks or more a more in depth talks from a member. Even at the talk sessions there will usually be some social lubrication such as beer and pizza - hopefully paid for by a recruitment company! We have also done a django dash (a competitive sprint), and a group outing to Djangocon.eu, a couple of years ago when it was in Amsterdam.

8 #dbbug Come and join us (...you don't have to be an evangelist)
Please come along to our next meet. Not that we have a date for it yet, but just the list at Google group search for dbbug ... if you and ask to meet, it will prompt somebody to suggest a date for the next one - so you can be the inspiration for that gathering! You don't have to be a developer or a Python user already, but it helps if you aim to be one. List is mainly for group events (sometimes gets job offers and newbie questions too) (I guess there are better django lists for technical questions)

9 Django use at Bristol University
Django is being used at the University to provide an integration framework for web applications that need to appear on our public web. So this integrates with our institutional CMS driven content and central data sources (e.g. We are also using it to rewrite our national survey system BOS (like survey monkey for academic and cross institutional provision)

10 Background - the survey application
BOS has been going for a decade! Now rewriting in django - so battle of the python web frameworks, but in our case django is david and pyramid is goliath in the form of survey monkey. More seriously - we are targeting a more specialist academic niche - where institutions often wish to aggregate results between them. along with requiring help wrt. producing an academically credible survey / data collection. But seriously we are in a specific sector which ideally is not in direct competion with our bigger python brother. As you can see from the stats it is more than qualified as a national HE service - with the majority of the UK HE sector as our main customers. We are moving from Perl to Python (and Django) and adding a range of new features along the way. Along with a more modular and exposed API code base.

11 Here is a very conceptual! look at the architecture.
(NB: This slide and the preceding one are taken from the BOS poster - So some of you may recognize it - or have chatted about it at those sessions) Backend storage - Postgres for structured data storage - survey constructions, accounts etc. Cassandra for survey responses - so the great thing about Cassandra (like BigTable, et al.) it writes faster than it reads (where conflict resolution occurs) It is also built around the cluster concept (you need 3 servers as a minimum!) - this means it can happily handle hundreds of thousands of respondents filling in surveys and not fall over like a traditional RDBMS We use a combination of data access layers, and were in the process of tying them together with tasty-pie Very simple concept - survey design in -> Postgres respond -> Cassandra analyse <- Cassandra and possibly Postgres for more complex aggregation queries. BUT .... leaving all that aside ... today I am just talking about federated authorisation ... so we are just looking at the bottom of the second speech bubble.

12 Agile practises Daily scrum to report progress and decide current work units Sprints of 3-4 weeks duration Back log to handle sprint over spill Product owner to help lead overall work prioritisation Continuous Integration - nightly builds for various test suites Maybe not full TDD but good test coverage Continuous refactor effort B.C.S. SCRUM talk by Nigel Baker - Agile Bear

13 Packaging functional components
Splitting a large development project into packages Caters for separate tests, documentation, release management and development cycles. Encourages increase in modularisation and encapsulation - helps to decouple the code. (Macro level OOP!) Adding generic functionality - separate package can be used across projects. Aim for it to be entirely generic then it can be released - this then opens it up to being freely tested and improved and / or influence other components of an open framework For remote role allocation we use a bespoke standard means of setting entitlements via a parseable string. the string has the option to set things for a whole class of objects (within an account) or for specific instances of them. There is also a means to add include/exlude qualifiers to cater for institutions who do not want to handle these set operations within their group management tools.

14 Python packages In python (and hence django) packages are called eggs.
There is a central repository of all open source eggs called PyPI (the cheese shop), like CPAN. PyPI is built into the core language packaging features. Create a folder add a few files including setup.py, to make it an egg, and run python setup.py register to push it to PyPI. A user of your code then just has to download and run python setup or use the pip add on and run pip install package-name to add it to their django based site.

15 e.g. django-cookieless https://pypi.python.org/pypi/django-cookieless
Django requires cookies to maintain session, and hence for authorisation. However there are cases when forms are used on public sites, where setting cookies is not desirable (due to privacy legislation), since technically they are not required for anonymous users to respond to forms. So if used, strictly consent is needed from the user to set them. This package is designed to cater for anonymous user session maintenance, without cookies. It provides a sessions implementation and decorator class for views to allow for forms to maintain state without using cookies by posting the encrypted session id between forms, or via urls. This also enables a user to run multiple sessions across different browser tabs. We have handled the first half of our requirements. Now what about the next half...

16 Open source ethos Python ethos = OS ethos - code should be readable, openly accessible, reusable and adaptable by everyone. So open source should be well documented and maintained with test suites, release management. To deliver these goals. Note that the majority of commercial software is also now, at least partially, open source. The ethos of Python is unsurprisingly closely related to the ethos of open source itself (and of the whole spectrum of the origins of the web - W3C - open formats & knowledge - academic peer review principles, etc.) Some take this to the extent that just making code public that isn't well documented, and release managed with tests etc., is worse than not releasing - since you dirty the waters with potentially impenetrable rotting code. At worst this results in the kind of fragile, black box coding that often has to be employed with closed source code. Since the open principle tends to lead to more robust / financially viable components. The only significant exceptions were Microsoft and Oracle - but even software companies who still stick to the old licence charging funding model, now want to be seen to be on board too. Microsoft Openness, Oracle MySQL etc.

17 Sharing code To start sharing code, just reply to somebody with some.
Maybe on a platform list or stack overflow or the like. If its a longer snippet you may want to use your platform's favourite paste bin site, eg. to share it. If you think its of general use why not make it more permanently public, for django that means If its more than a snippet then it should be in a versioning system, so why not pick a public one ... The ethos of Python is unsurprisingly closely related to the ethos of open source itself (and of the whole spectrum of the origins of the web - W3C - open formats & knowledge - academic peer review principles, etc.) Some take this to the extent that just making code public that isn't well documented, and release managed with tests etc., is worse than not releasing - since you dirty the waters with potentially impenetrable rotting code. At worst this results in the kind of fragile, black box coding that often has to be employed with closed source code. Since the open principle tends to lead to more robust / financially viable components. The only significant exceptions were Microsoft and Oracle - but even software companies who still stick to the old licence charging funding model, now want to be seen to be on board too. Microsoft Openness, Oracle MySQL etc.

18 Community coding Like Facebook, but with a purpose! Site Pros Cons
github.com market leader only does git bitbucket.com (Atlassian) git or mercurial * free private repos Its written in django :-) has been accused of plaigarising github code.google.com git, mercurial or svn (for the old folks) poorly featured, few signs of new releases gitorious.org like github but open source infrastructure bit flakey and smaller scale sourceforge.net the original ... ... and worst. graveyard of old projects We have handled the first half of our requirements. Now what about the next half... * Mercurial is the Python twin of Git (written in C), separated at birth!

19 The open road Reply to help lists - stack overflow or python and django specific Reply to project support chat rooms Reporting bugs Fixing bugs Adding new features to existing packages Contributing enough to warrant a new release Coagulating snippets, paste bins, stack overflows into an egg Creating and releasing your own package Acting as a release manager Becoming a core contributor of a large project Starting your own large project / framework / language (Warning 11 is not for people who have a life outside coding) We have handled the first half of our requirements. Now what about the next half...

20 Creative co-operation
Githubs fun visualisation of contributors to one of my eggs .... hmm need to catch up merging those forks, or I will lose release manager / origin status for it ... We have handled the first half of our requirements. Now what about the next half...

21 Don't be afraid to commit
Contributing to Django core, or other Python projects (Boosting you half way along the open road!) A workshop next week, 8th June, by fellow DBBUG member, Daniele Procida (Details - room 3 - 1:50pm) See for more information. The workshop will take participants through the complete cycle of identifying a simple issue in a Django or Python project, writing a patch with tests and documentation, and submitting it. The workshop is aimed at the first-time committer. Very little experience is required: We have handled the first half of our requirements. Now what about the next half...

22 Interested in Django ... home dbbug free book 90 other books
dbbug free book 90 other books chat #django IRC help core devs other dbbugs jobs and add ons

23 Ed Crewe django bath and bristol user group #dbbug
Contact django bath and bristol user group #dbbug Django Bath and Bristol Users Group (DBBUG) The Django Bath and Bristol Users Group is a place for local developers and companies to come and talk Django, Python, and web development in general. We run events every couple of months, with a mix of socials and technical meets. Building open source with Django Django is a leading web framework, in Python, used by the likes of Mozilla, Instagram and OpenStack. BOS is a survey system developed by the R&D team over the last decade, and in use by most UK Universities. Find out how we are rewriting BOS from scratch in Django, Cassandra and Postgres, using current agile practises. Along with how to package a component of a larger product, to contribute back to an open source tool set. PS: Also a member of SW Big Data /south-west-big-data/ university of bristol - ITS - R&D


Download ppt "Building open source with Django"

Similar presentations


Ads by Google