Presentation on theme: "Experience of Automated Valuation Modelling (AVM) in England"— Presentation transcript:
1 Experience of Automated Valuation Modelling (AVM) in England Tim Eden BSc MRICS IRRVDeputy Director of Council TaxValuation Office Agency
2 Presentation overview A “fly through” of the VOA AVM experience including:BackgroundThe challengeThe extent of data captureChanges in data managementProcess improvement and changeModel developmentTrainingAchievementsTE
3 To be a world class organisation providing valuation and property services for the public sector
4 VOA - Our purposeTo provide a fair and robust basis for taxes, which help to pay for public services; and to help drive better use of property in the public sector by:compiling and maintaining accurate and comprehensive valuation lists for local taxationproviding accurate valuations for national taxesdelivering expert advice on valuations and strategic property managementdeveloping and maintaining a comprehensive and up to date property databaseadvising policy makers on valuation and property issues
5 VOA - Our structure Country VOA VOA Deals with: - No. of Offices Local Domestic Property TaxationLocal Non-domestic Property TaxationNational Property TaxationOther Government Property ServicesEngland67YesWales7Scotland5NoTE
6 “Recent” Taxation History 1956General National Valuation – to Point Rental Value1963General National Revaluation19731982General National Revaluation (Cancelled)Non-DomesticDomestic1990National Revaluation with Antecedent Valuation Date (AVD) 1 April 1988Individual-based Taxation called Community Charge (“Poll Tax”)1993National Property Valuation for Council Tax to Capital Value Bands (AVD 1 April 1991)1995Revaluation AVD 1 April 19932000Revaluation AVD 1 April 19982005Revaluation AVD 1 April 20032007Revaluation AVD 1 April 2005 (Postponed)
7 Council Tax (CT) - Context CT + Non-Domestic Rates together raise c. £40bn paCT raises £18.5bn pa22m domestic properties in England1.3m domestic properties in WalesClientsDepartment for Communities & Local Gov’t (DCLG) in EnglandWelsh Assembly Gov’t (WAG) in Wales
8 CT – 1993 List Bands (England) A - up to £40,000B - £40,001 to £52,000C - £52,001 to £68,000D - £68,001 to £88,000E - £88,001 to £120,000F - £120,001 to £160,000G - £160,001 to £320,000H – over £320,001
9 CT - RevaluationLocal Government Act 2003 included requirement to undertake revaluations in Wales (2005) and England (2007)In Wales AVD 1 April 2005; list came into force on 1 April 2005In England AVD 1 April 2005; list was to come into force on 1 April 2007Draft lists were to be published 1 September 2006
10 The Challenge New banding scheme not known Necessary to produce individual valuations & then overlay new bandsScale of taskComplexity of housing typesVagaries of the marketInconsistent, paper-based records
11 The Challenge – Market Vagaries Market analysis across EnglandFree flow of moneyHistorically low interest ratesFluctuating volumes in sales
12 The Challenge – Housing Types Profile of Housing Stock80% Houses/Bungalows20% Flats/MaisonettesOver 1/3rd of all flats in Greater LondonAround 1/6th of properties at least 100 years oldBUT high proportion of new properties in sales evidence!
13 The Solution! Develop an AVM Skill-up staff Learn from others Cole Layer Trumble (CLT) – computer modelling expertise, AVM softwareKPMG – programme & project management supportIAAO – statistical trainingDigitise data
14 VOA - AVM Journey Began in 2002; worldwide research Mass data capture/enhancementModel developmentMany innovations and lessonsThese aspects have been very important to the success of the projectImproving Business DecisionsAVM designMRA modelsdata collection22 million valuesdata cleansingdata enhancementbusiness casepolitical issuesvalue reviewsstaff planning and allocationTE
15 Attribute Data Investigation Investigation commences in 2002 considering:Use of existing “attribute” codes already available on VOA IT database (“Group”, “Type”, “Area”)Valuer and caseworker engagement to determine useful codesData availability & maintainabilityCT List Maintenance workExternal standards e.g. IAAO (6 year cycle)Impact of differing local practicesAbility to undertake sales investigation activityPre-contractor appointment; so limited AVM expertiseTE
16 Main Attributes to be digitised Architectural Style and construction quality – “Group”Property Type e.g. House Semi-Detached – “Type”Age (coded as era e.g. G – )Area m² - external area (houses); internal area (flats)No. of roomsNo. of bedroomsNo of bathroomsNumber of floors (houses); floor level (flats)ParkingConservatory (type and size)Outbuilding detailsValue Significant CodesAt the start of the AVM project, a programme was already in place to transfer data from paper to database.So the data started to arrive for analysis and was, in principle, made up of one record per property, defining property’s attributes.Most of these attributes are listed above.Here we see a Type HT, Group 20 property.There are about 50 Groups and about 30 Types.The above picture shows examples of terraced houses taken from the VOA Handbook which describes the categories of Group and Type by means of some example pictures.Here we see an early challenge. The properties pictured above were developed by Local Councils for rent but recently were sold to the tenants. The prices paid by the tenants were below market price.There are approximately 3.5 million properties that were built subsidised. They were built in estates ranging from just a few houses (~20) up to large estates with more than 10,000 properties.
17 What about Sales Data?Access to Land Registry and Stamp Duty Land Tax informationAddress irregularities – creates matching problems and manual effortNeed to establish whether transaction at “Market Value”, special circumstances etc.Data understanding significantly assistsAVM geared to support this process“Value” understanding working with AVM is key to the processTEAVM supports sales verification and attribute validation through identification of ‘outlier’ sales…These can be calibration outliers and sales ratio outliers
18 Process Development Consider existing process The impact of AVM – scale and profileSkillsExisting skills and abilities of targeted staffThe need to re-train certain staffResource availabilityNew process to reflect Revaluation needs and business as usualIT constraints and capabilitiesUnderstand relationship between sales and model performanceTE
19 The Analysis Process After Before Sales received Suspect sales coded Property Data(Group, type, floor area, age, bedrooms, rooms, garage)Models(MRA, algorithms and variables)Locality Definition(area from which comparables derived)AVMAVM analysis output reviewed by qualified staffDecision to change, retain or include current data and codingSales Data1)New Sales continually updating analysis set2) Value significance consideredAfterBeforeSales receivedSuspect sales codedSales searched to support each valuation case “on paper”At a Revaluation “Beacon Sales” reviewed and recorded “on paper”PS
20 Initial Issues with Modelling Process No first hand knowledge of saleData time lagData gaps due to“Permissible development” – the public need not inform the Billing Authority (BA)BAs internally not acting in a joined-up way – billing, planning & building controlBAs not acting in joined-up way with VOAAre we maintaining lists or data or both?“Condition” not an attributePSCondition – Statutory hypothesis assumes in ‘reasonable’ repair having regard to property age, character and locality. Condition at sale causes ‘noise’ in the model, notably for older properties and properties towards the bottom end of the market. Condition (where captured) is taken into account in the model during sales analysis (condition at sale %) and duly reflected when used to value other properties. Condition is NOT however taken into consideration when valuing a property in poor repair UNLESS the repairs are exceptional. Lack of modernisation is however reflected both in sales analysis and subject property valuation (by way of a Value Significant Factor)Condition is not routinely recorded – a lesson for the future! But need to balance benefits in model performance against cost of training staff and ultimately collecting the data. Currently no legislative reason to collect this information!
21 Integrating the AVM Investigation work commenced late 2002 Procurement process 2003Collaboration of in-house team and successful contractors/partners - Cole Layer Trumble (CLT)Working on BA areas where data capture is advanced (note had only started in 2003!)Challenge of integrating AVM technology with existing VOA IT systemsPS
22 Getting Started Started with a simple additive model £ = B0 + B1 * size + B2 * Detached * Size + B3 * Terraced * Size + … + Bn * Date of Sale * Size1993 Band initially used to support analysis, but quickly stripped out of valuation modelsPostcode sectors used as proxy for location – VOA developed “localities”Postal areas are too crudeDesigned to support mail delivery, not to reflect influences on the property market!Bi or Bi?
23 Performance Improvement (1) Better understanding of subject/sale data helped to:Determine usability rules for raw dataDetermine usability rules for a saleThis understanding fedComparable selectionModel specificationConsideration of rarely occurring variablesConsideration of locality relationshipsPSUsability rules translated into electronic flags that were used to focus data enhancement and sales verification and validation effort. For instance a house (as opposed to a flat or condominium) cannot have a floor level starting at 10 floors above ground level or the sale of a small starter home in an ‘average’ area cannot be correct at £1,000,000 – more likely £100,000
24 High Correlations Floors/ Floor Level Type Parking Group Age 32 separate codesParkingGroup55 separate codesAge11 separate codesCorrelation among variablesHigh correlation compromises modelling stabilityAreaBathroomsPSFor example total floor area and number of bedrooms and rooms tend to be very highly correlated. Sound model specification required careful consideration of correlations. Typically, Area would be entered into model, whereas bedrooms and rooms would be left out as they tend to explain the same thing. Leaving them in the model lead to counter intuitive coefficients for beds / rooms.Note: Beds was however left in the comparable selection algorithm to help explainability when ultimately discussing the valuation with the taxpayer.VOA property attributes established many years prior to the implementation of an AVM. Some attributes are not exclusive and there are crossovers that lead to modelling challenges. For example Property Group often crosses with Age and even Area.RoomsBedrooms
25 Localities – illustrative photograph Created bespoke localities (neighbourhoods)Local Authority housing estatePrivately built housing estatePSIn England properties of very different quality can be located close together.The example above represents a case where the market behaviour in adjacent areas can be quite different. In this case where Local Authority housing is contiguous to private developments.Another frequent occurrence is where new properties are dropped into the plots of older properties with large plots or into what were once commercial areas. This could be an estate of maybe up to 30 houses, or a small plot of maybe 2 or 3 houses.Even subsidised housing can be built by infill.Since the AVM models are trying to estimate the underlying market behaviour, they can be significantly disturbed if they don’t take account of these location effects.
26 Mapping & Localities In excess of 10,000 localities Regular boundary review requiredThematic mapping as part of processX-Y co-ordinate becomes necessary dataX-Y suitable for comparable selectionIssue of who maintains the dataPSThe calibration process and thematic mapping of sales and subject property data supported the review of locality boundariesXY – used in the comparable selection algorithm. Issues of correctness compounded by the fact that the data is maintained by a third party and not the VOA.
28 Performance Improvement (2) Move to multiplicative (log linear) model structureLog (£) = B0 + B1 * log (size) + B2 * Detached + B3 * Terraced + …+ Bn * Date of Sale + Bp * Log (Locality Adjustment Factor)where Bi determined from sales set using MRAThis enabled improvements in: -Locality Adjustment Factor (time adjusted median price per square metre)Locality Grouping – support for comparable selectionCentral Modelling across the whole countryPSAdopted model structure – Log Linear. Earlier model structure had been additive.LAF = Time & Type Adjusted Sale Price Per Sqm for LocalityTime & Type Adjusted Sale Price Per Sqm for whole Valuation Area
29 Benefits of Central Modelling Central Modelling enabled:Central recognition of national modelling patterns and data issuesModelling “constraints” could be imposed by the centre ensuring model coefficients are consistently appliedEffective direction of effortCalculation of market trend information to support VOA modelling and wider government market appreciationPS
30 Performance Improvement (3) Integrating X-Y & Mapping allowed mass calculation of “plot size”Issues with map plots are:Plot bleedNon-alignment with the property transferredMaintenance of dataPS
31 Model Development - Lessons Create a multi-skilled and focused R&D teamSelect several representative areas to testEnsure proper debate on proposalsPredict overall likely modelling gainsSense check & gather feedback from local staffBusiness model to relate model performance and cost/benefitProduce individual cost/benefit for each proposalPromote external verification e.g. IAAOPSNeed to balance modeller’s wishes for more data with cost benefits & timeliness
32 Supporting Decision Making (1) VOA recognised management of a national valuation delivery required consistency in valuation decisionsProperty level confidence score requiredCOD/COV inadequate existing banding too remoteScore needed to support decisions at several levels:Strategic:Business case developmentProcess definitionData collection & enhancementTactical:Resource planningOperational:Resource allocationValue review or band reviewPSSpecific needs related to:Confirming the existing business case – which was based on 60% properties valued through the AVM and 40% being valued manuallyOptimisation of processes, e.g. allocation of valuation staff to right propertiesAppropriate / targeted investment in data collection, cleansing and enhancementResource planning – recruitment, training, transfersAllocation of properties to bands
33 Supporting Decision Making (2) Confidence estimate for each subject property needed to reflect the aspects which would reduce accuracy:Data qualityData availabilityMarket variabilityModel accuracyAdequate coverage for accuracy in the MRAAdequate comparablesPS
34 Confidence ModelConfidence model based upon indicators from MRA and comparablesRelated actual errors to the dispersion amongst the comparablesComparable sales approach provides a number of other indicators for confidences:comparability distanceweighted estimate (average adjusted) vs. MRA estimatethe overall COV for the model on the sales set which tells us something about the underlying uniformity in the market and sales baseMRA modelling includes a control model which considers current CT bandSo Confidence Model is:Likely Error = A+ B dispersion of comparables+ C x average distance between comparables+ D x absolute value of (ln (mkt est. / control model est.) )+ E x model standard error+ F x absolute value of (ln (weighted est. / MRA est.) )PSDispersion of the 5 comp sales prices computed as (standard deviation of the adjusted sales prices / average of adjusted sales price). In other words, a measure of uniformity of the adjusted sales prices.Average Comp Distance of the 5 comps used in computing the value estimate, i.e. a measure of how good the comps are. Euclidean Distance MeasurementThe absolute value of the natural log of the market estimate / control model, i.e. ABS(ln(MrktEst / Control Model). This is simply a measure of the relative uniformity estimate and the control model.Model Standard Error, i.e. the standard error of the appropriate model used in computing the value estimate. This tells us how well the MRA model is predicting sale price.The absolute value of the natural log of the ratio of the weighted estimate / MRA estimate. This is a measure of the relative uniformity of the MRA estimate and the market estimate. ABS(ln(Weighted Est / MRA Est).In simple terms the Confidence Model adopts the principle that confidence is likely to be high (i.e. the Confidence Interval is narrow) when:The 5 comparable sales prices are similar, i.e. dispersion of the 5 comp sales is low.The comparables are similar to the subject, i.e. Average Comp Distance is low.The Market Estimate and the Control Model Estimate are similar.The MRA model is accurate, i.e. the Model Standard Error is low.The MRA estimate and the weighted estimate are similar.
35 Confidence Model Maintenance Re-calibrated with every calibration iterationPeriodically calibrated to valuer judgement of estimate outputCan be used individually or aggregated for decisions at varying levelsDeveloped between VOA, John Thompson (Cole Layer Trumble), Dr Jim Abbott (EDS)Presented at IAAO CAMA/GIS conference February 2006PS
36 Summary of Models (1) Multiplicative MRA and comparable selection Operational deliveryUndertaken locallySpecified centrallyBroad Based ModelMarket TrendsLinearisations e.g. Group and Type variables to allow wider comparabilityDevelop constraints for local modelsQuality assurance on operational deliveryPS
37 Summary of Models (2) Control Model Confidence Model Operates in background of local modelHighlights data irregularitiesConfidence ModelUses data from all modelsCalibrated to valuer opinionProvides information for use at all levelsPS
38 Managing Business Change Addressing the time lagReview whole data processUse electronic transfer to bring in sales dataConsider how to engage with taxpayerNew skillsControl modelling and train professionalsDevelop support and maintenance staffRigorous application of project management methodologyTE
39 IT DevelopmentVOA has successfully integrated AVM technology with existing IT; this required:Improving data flow and managementEnhanced mapping tools“Customising” 3rd party “off-the-shelf” productIntroducing workflow to manage deliveryLead-in time has created tensionsIT supplier’s understanding of business and new technology does not happen overnight!TE
40 Conclusions/Lessons Learned (1) Other data sources create issues:Often aggregatedCurrency – frequency of updateNiche provider with limited depthDefinition – what creates the dataCost – Is it worth it to you?Your records may be adequate and better than elsewhere anyway!TE
41 Conclusions/Lessons Learned (2) How do you measure data qualitySale data qualityMostly measured by modelDependent upon proper sales review processSubject data qualityAdoption of external standards not always possible e.g. IAAO 6 year cycleData sampling using existing maintenance activities can support the viewTE
42 Conclusions/Lessons Learned (3) Time lagUnderstand the delivery requirementAddress steps in the data processWork closer with data “partners”Central modelling can identify areas of concernProximity to dataYou cannot easily replace local knowledge of market and meaning of sales.TE
43 Conclusions/Lessons Learned (4) ModellingDefine “attributes” with knowledge of AVM techniquesBalance modellers desire for more data of imperfect market and cost to complete/maintainStabilise data prior to commencement of modellingValuing to “Band” does not loosen the rigour of modelling and data managementTE
44 Conclusions/Lessons Learned (5) Use AVM to direct effortRaw data analysisCOD & COV are not the only measures“Frequently used comparables”Thematic mappingUse formal modelling hierarchyDon’t rush to deliver. Inefficiencies resultStabilise Model approach to ensure consistencyData stability and consistency
45 Conclusions/Lessons Learned (6) Create clear lines of communicationLocal ManagementLocal technical/modellingTrainingMake timelyConsider delivery mechanism (e-learning, workshop)Project Management StructureA quality model can be let down by a poorly defined and delivered process!
46 Some VOA achievements Digitisation of over 22 Million records Data completeness approaching 100%Over 120 surveyors trained in AVM techniquesMarket analysis nationwide, including mapping localities across the whole countryModel performance well within recognised standards e.g. median COD of 353 BAs is 9.97Proven IT platform for mass appraisal at national level.
47 Questions? Tim Eden BSc MRICS IRRV Deputy Director of Council Tax