1. Experience of Automated Valuation Modelling (AVM) in England
Tim Eden BSc MRICS IRRV
Deputy Director of Council Tax
Valuation Office Agency

2. Presentation overview
A “fly through” of the VOA AVM experience including:
Background
The challenge
The extent of data capture
Changes in data management
Process improvement and change
Model development
Training
Achievements TE

3. To be a world class organisation providing valuation and property services for the public sector

4. VOA - Our purpose
To provide a fair and robust basis for taxes, which help to pay for public services; and to help drive better use of property in the public sector by:
compiling and maintaining accurate and comprehensive valuation lists for local taxation
providing accurate valuations for national taxes
delivering expert advice on valuations and strategic property management
developing and maintaining a comprehensive and up to date property database
advising policy makers on valuation and property issues

5. VOA - Our structure Country VOA VOA Deals with: - No. of Offices
Local Domestic Property Taxation
Local Non-domestic Property Taxation
National Property Taxation
Other Government Property Services
England 67
Yes
Wales 7
Scotland 5 No TE

6. “Recent” Taxation History
1956
General National Valuation – to Point Rental Value
1963
General National Revaluation
1973
1982
General National Revaluation (Cancelled)
Non-Domestic
Domestic
1990
National Revaluation with Antecedent Valuation Date (AVD) 1 April 1988
Individual-based Taxation called Community Charge (“Poll Tax”)
1993
National Property Valuation for Council Tax to Capital Value Bands (AVD 1 April 1991)
1995
Revaluation AVD 1 April 1993
2000
Revaluation AVD 1 April 1998
2005
Revaluation AVD 1 April 2003
2007
Revaluation AVD 1 April 2005 (Postponed)

7. Council Tax (CT) - Context
CT + Non-Domestic Rates together raise c. £40bn pa
CT raises £18.5bn pa
22m domestic properties in England
1.3m domestic properties in Wales
Clients
Department for Communities & Local Gov’t (DCLG) in England
Welsh Assembly Gov’t (WAG) in Wales

8. CT – 1993 List Bands (England)
A - up to £40,000
B - £40,001 to £52,000
C - £52,001 to £68,000
D - £68,001 to £88,000
E - £88,001 to £120,000
F - £120,001 to £160,000
G - £160,001 to £320,000
H – over £320,001

9. CT - Revaluation
Local Government Act 2003 included requirement to undertake revaluations in Wales (2005) and England (2007)
In Wales AVD 1 April 2005; list came into force on 1 April 2005
In England AVD 1 April 2005; list was to come into force on 1 April 2007
Draft lists were to be published 1 September 2006

10. The Challenge New banding scheme not known
Necessary to produce individual valuations & then overlay new bands
Scale of task
Complexity of housing types
Vagaries of the market
Inconsistent, paper-based records

11. The Challenge – Market Vagaries
Market analysis across England
Free flow of money
Historically low interest rates
Fluctuating volumes in sales

12. The Challenge – Housing Types
Profile of Housing Stock
80% Houses/Bungalows
20% Flats/Maisonettes
Over 1/3rd of all flats in Greater London
Around 1/6th of properties at least 100 years old
BUT high proportion of new properties in sales evidence!

13. The Solution! Develop an AVM Skill-up staff Learn from others
Cole Layer Trumble (CLT) – computer modelling expertise, AVM software
KPMG – programme & project management support
IAAO – statistical training
Digitise data

14. VOA - AVM Journey Began in 2002; worldwide research
Mass data capture/enhancement
Model development
Many innovations and lessons
These aspects have been very important to the success of the project
Improving Business Decisions
AVM design
MRA models
data collection
22 million values
data cleansing
data enhancement
business case
political issues
value reviews
staff planning and allocation TE

15. Attribute Data Investigation
Investigation commences in 2002 considering:
Use of existing “attribute” codes already available on VOA IT database (“Group”, “Type”, “Area”)
Valuer and caseworker engagement to determine useful codes
Data availability & maintainability
CT List Maintenance work
External standards e.g. IAAO (6 year cycle)
Impact of differing local practices
Ability to undertake sales investigation activity
Pre-contractor appointment; so limited AVM expertise TE

16. Main Attributes to be digitised
Architectural Style and construction quality – “Group”
Property Type e.g. House Semi-Detached – “Type”
Age (coded as era e.g. G – )
Area m² - external area (houses); internal area (flats)
No. of rooms
No. of bedrooms
No of bathrooms
Number of floors (houses); floor level (flats)
Parking
Conservatory (type and size)
Outbuilding details
Value Significant Codes
At the start of the AVM project, a programme was already in place to transfer data from paper to database.
So the data started to arrive for analysis and was, in principle, made up of one record per property, defining property’s attributes.
Most of these attributes are listed above.
Here we see a Type HT, Group 20 property.
There are about 50 Groups and about 30 Types.
The above picture shows examples of terraced houses taken from the VOA Handbook which describes the categories of Group and Type by means of some example pictures.
Here we see an early challenge. The properties pictured above were developed by Local Councils for rent but recently were sold to the tenants. The prices paid by the tenants were below market price.
There are approximately 3.5 million properties that were built subsidised. They were built in estates ranging from just a few houses (~20) up to large estates with more than 10,000 properties.

17. What about Sales Data?
Access to Land Registry and Stamp Duty Land Tax information
Address irregularities – creates matching problems and manual effort
Need to establish whether transaction at “Market Value”, special circumstances etc.
Data understanding significantly assists
AVM geared to support this process
“Value” understanding working with AVM is key to the process TE
AVM supports sales verification and attribute validation through identification of ‘outlier’ sales…These can be calibration outliers and sales ratio outliers

18. Process Development Consider existing process
The impact of AVM – scale and profile
Skills
Existing skills and abilities of targeted staff
The need to re-train certain staff
Resource availability
New process to reflect Revaluation needs and business as usual
IT constraints and capabilities
Understand relationship between sales and model performance TE

19. The Analysis Process After Before Sales received Suspect sales coded
Property Data
(Group, type, floor area, age, bedrooms, rooms, garage)
Models
(MRA, algorithms and variables)
Locality Definition
(area from which comparables derived)
AVM
AVM analysis output reviewed by qualified staff
Decision to change, retain or include current data and coding
Sales Data
1)New Sales continually updating analysis set
2) Value significance considered
After
Before
Sales received
Suspect sales coded
Sales searched to support each valuation case “on paper”
At a Revaluation “Beacon Sales” reviewed and recorded “on paper” PS

20. Initial Issues with Modelling Process
No first hand knowledge of sale
Data time lag
Data gaps due to
“Permissible development” – the public need not inform the Billing Authority (BA)
BAs internally not acting in a joined-up way – billing, planning & building control
BAs not acting in joined-up way with VOA
Are we maintaining lists or data or both?
“Condition” not an attribute PS
Condition – Statutory hypothesis assumes in ‘reasonable’ repair having regard to property age, character and locality. Condition at sale causes ‘noise’ in the model, notably for older properties and properties towards the bottom end of the market. Condition (where captured) is taken into account in the model during sales analysis (condition at sale %) and duly reflected when used to value other properties. Condition is NOT however taken into consideration when valuing a property in poor repair UNLESS the repairs are exceptional. Lack of modernisation is however reflected both in sales analysis and subject property valuation (by way of a Value Significant Factor)
Condition is not routinely recorded – a lesson for the future! But need to balance benefits in model performance against cost of training staff and ultimately collecting the data. Currently no legislative reason to collect this information!

21. Integrating the AVM Investigation work commenced late 2002
Procurement process 2003
Collaboration of in-house team and successful contractors/partners - Cole Layer Trumble (CLT)
Working on BA areas where data capture is advanced (note had only started in 2003!)
Challenge of integrating AVM technology with existing VOA IT systems PS

22. Getting Started Started with a simple additive model
£ = B0 + B1 * size + B2 * Detached * Size + B3 * Terraced * Size + … + Bn * Date of Sale * Size
1993 Band initially used to support analysis, but quickly stripped out of valuation models
Postcode sectors used as proxy for location – VOA developed “localities”
Postal areas are too crude
Designed to support mail delivery, not to reflect influences on the property market!
Bi or Bi?

23. Performance Improvement (1)
Better understanding of subject/sale data helped to:
Determine usability rules for raw data
Determine usability rules for a sale
This understanding fed
Comparable selection
Model specification
Consideration of rarely occurring variables
Consideration of locality relationships PS
Usability rules translated into electronic flags that were used to focus data enhancement and sales verification and validation effort. For instance a house (as opposed to a flat or condominium) cannot have a floor level starting at 10 floors above ground level or the sale of a small starter home in an ‘average’ area cannot be correct at £1,000,000 – more likely £100,000

24. High Correlations Floors/ Floor Level Type Parking Group Age
32 separate codes
Parking
Group
55 separate codes
Age
11 separate codes
Correlation among variables
High correlation compromises modelling stability
Area
Bathrooms PS
For example total floor area and number of bedrooms and rooms tend to be very highly correlated. Sound model specification required careful consideration of correlations. Typically, Area would be entered into model, whereas bedrooms and rooms would be left out as they tend to explain the same thing. Leaving them in the model lead to counter intuitive coefficients for beds / rooms.
Note: Beds was however left in the comparable selection algorithm to help explainability when ultimately discussing the valuation with the taxpayer.
VOA property attributes established many years prior to the implementation of an AVM. Some attributes are not exclusive and there are crossovers that lead to modelling challenges. For example Property Group often crosses with Age and even Area.
Rooms
Bedrooms

25. Localities – illustrative photograph
Created bespoke localities (neighbourhoods)
Local Authority housing estate
Privately built housing estate PS
In England properties of very different quality can be located close together.
The example above represents a case where the market behaviour in adjacent areas can be quite different. In this case where Local Authority housing is contiguous to private developments.
Another frequent occurrence is where new properties are dropped into the plots of older properties with large plots or into what were once commercial areas. This could be an estate of maybe up to 30 houses, or a small plot of maybe 2 or 3 houses.
Even subsidised housing can be built by infill.
Since the AVM models are trying to estimate the underlying market behaviour, they can be significantly disturbed if they don’t take account of these location effects.

26. Mapping & Localities In excess of 10,000 localities
Regular boundary review required
Thematic mapping as part of process
X-Y co-ordinate becomes necessary data
X-Y suitable for comparable selection
Issue of who maintains the data PS
The calibration process and thematic mapping of sales and subject property data supported the review of locality boundaries
XY – used in the comparable selection algorithm. Issues of correctness compounded by the fact that the data is maintained by a third party and not the VOA.

27. Mapping Example PS

28. Performance Improvement (2)
Move to multiplicative (log linear) model structure
Log (£) = B0 + B1 * log (size) + B2 * Detached + B3 * Terraced + …
+ Bn * Date of Sale + Bp * Log (Locality Adjustment Factor)
where Bi determined from sales set using MRA
This enabled improvements in: -
Locality Adjustment Factor (time adjusted median price per square metre)
Locality Grouping – support for comparable selection
Central Modelling across the whole country PS
Adopted model structure – Log Linear. Earlier model structure had been additive.
LAF = Time & Type Adjusted Sale Price Per Sqm for Locality
Time & Type Adjusted Sale Price Per Sqm for whole Valuation Area

29. Benefits of Central Modelling
Central Modelling enabled:
Central recognition of national modelling patterns and data issues
Modelling “constraints” could be imposed by the centre ensuring model coefficients are consistently applied
Effective direction of effort
Calculation of market trend information to support VOA modelling and wider government market appreciation PS

30. Performance Improvement (3)
Integrating X-Y & Mapping allowed mass calculation of “plot size”
Issues with map plots are:
Plot bleed
Non-alignment with the property transferred
Maintenance of data PS

31. Model Development - Lessons
Create a multi-skilled and focused R&D team
Select several representative areas to test
Ensure proper debate on proposals
Predict overall likely modelling gains
Sense check & gather feedback from local staff
Business model to relate model performance and cost/benefit
Produce individual cost/benefit for each proposal
Promote external verification e.g. IAAO PS
Need to balance modeller’s wishes for more data with cost benefits & timeliness

32. Supporting Decision Making (1)
VOA recognised management of a national valuation delivery required consistency in valuation decisions
Property level confidence score required
COD/COV inadequate existing banding too remote
Score needed to support decisions at several levels:
Strategic:
Business case development
Process definition
Data collection & enhancement
Tactical:
Resource planning
Operational:
Resource allocation
Value review or band review PS
Specific needs related to:
Confirming the existing business case – which was based on 60% properties valued through the AVM and 40% being valued manually
Optimisation of processes, e.g. allocation of valuation staff to right properties
Appropriate / targeted investment in data collection, cleansing and enhancement
Resource planning – recruitment, training, transfers
Allocation of properties to bands

33. Supporting Decision Making (2)
Confidence estimate for each subject property needed to reflect the aspects which would reduce accuracy:
Data quality
Data availability
Market variability
Model accuracy
Adequate coverage for accuracy in the MRA
Adequate comparables PS

34. Confidence Model
Confidence model based upon indicators from MRA and comparables
Related actual errors to the dispersion amongst the comparables
Comparable sales approach provides a number of other indicators for confidences:
comparability distance
weighted estimate (average adjusted) vs. MRA estimate
the overall COV for the model on the sales set which tells us something about the underlying uniformity in the market and sales base
MRA modelling includes a control model which considers current CT band
So Confidence Model is:
Likely Error = A
+ B  dispersion of comparables
+ C x average distance between comparables
+ D x absolute value of (ln (mkt est. / control model est.) )
+ E x model standard error
+ F x absolute value of (ln (weighted est. / MRA est.) ) PS
Dispersion of the 5 comp sales prices computed as (standard deviation of the adjusted sales prices / average of adjusted sales price). In other words, a measure of uniformity of the adjusted sales prices.
Average Comp Distance of the 5 comps used in computing the value estimate, i.e. a measure of how good the comps are. Euclidean Distance Measurement
The absolute value of the natural log of the market estimate / control model, i.e. ABS(ln(MrktEst / Control Model). This is simply a measure of the relative uniformity estimate and the control model.
Model Standard Error, i.e. the standard error of the appropriate model used in computing the value estimate. This tells us how well the MRA model is predicting sale price.
The absolute value of the natural log of the ratio of the weighted estimate / MRA estimate. This is a measure of the relative uniformity of the MRA estimate and the market estimate. ABS(ln(Weighted Est / MRA Est).
In simple terms the Confidence Model adopts the principle that confidence is likely to be high (i.e. the Confidence Interval is narrow) when:
The 5 comparable sales prices are similar, i.e. dispersion of the 5 comp sales is low.
The comparables are similar to the subject, i.e. Average Comp Distance is low.
The Market Estimate and the Control Model Estimate are similar.
The MRA model is accurate, i.e. the Model Standard Error is low.
The MRA estimate and the weighted estimate are similar.

35. Confidence Model Maintenance
Re-calibrated with every calibration iteration
Periodically calibrated to valuer judgement of estimate output
Can be used individually or aggregated for decisions at varying levels
Developed between VOA, John Thompson (Cole Layer Trumble), Dr Jim Abbott (EDS)
Presented at IAAO CAMA/GIS conference February 2006 PS

36. Summary of Models (1) Multiplicative MRA and comparable selection
Operational delivery
Undertaken locally
Specified centrally
Broad Based Model
Market Trends
Linearisations e.g. Group and Type variables to allow wider comparability
Develop constraints for local models
Quality assurance on operational delivery PS

37. Summary of Models (2) Control Model Confidence Model
Operates in background of local model
Highlights data irregularities
Confidence Model
Uses data from all models
Calibrated to valuer opinion
Provides information for use at all levels PS

38. Managing Business Change
Addressing the time lag
Review whole data process
Use electronic transfer to bring in sales data
Consider how to engage with taxpayer
New skills
Control modelling and train professionals
Develop support and maintenance staff
Rigorous application of project management methodology TE

39. IT Development
VOA has successfully integrated AVM technology with existing IT; this required:
Improving data flow and management
Enhanced mapping tools
“Customising” 3rd party “off-the-shelf” product
Introducing workflow to manage delivery
Lead-in time has created tensions
IT supplier’s understanding of business and new technology does not happen overnight! TE

40. Conclusions/Lessons Learned (1)
Other data sources create issues:
Often aggregated
Currency – frequency of update
Niche provider with limited depth
Definition – what creates the data
Cost – Is it worth it to you?
Your records may be adequate and better than elsewhere anyway! TE

41. Conclusions/Lessons Learned (2)
How do you measure data quality
Sale data quality
Mostly measured by model
Dependent upon proper sales review process
Subject data quality
Adoption of external standards not always possible e.g. IAAO 6 year cycle
Data sampling using existing maintenance activities can support the view TE

42. Conclusions/Lessons Learned (3)
Time lag
Understand the delivery requirement
Address steps in the data process
Work closer with data “partners”
Central modelling can identify areas of concern
Proximity to data
You cannot easily replace local knowledge of market and meaning of sales. TE

43. Conclusions/Lessons Learned (4)
Modelling
Define “attributes” with knowledge of AVM techniques
Balance modellers desire for more data of imperfect market and cost to complete/maintain
Stabilise data prior to commencement of modelling
Valuing to “Band” does not loosen the rigour of modelling and data management TE

44. Conclusions/Lessons Learned (5)
Use AVM to direct effort
Raw data analysis
COD & COV are not the only measures
“Frequently used comparables”
Thematic mapping
Use formal modelling hierarchy
Don’t rush to deliver. Inefficiencies result
Stabilise Model approach to ensure consistency
Data stability and consistency

45. Conclusions/Lessons Learned (6)
Create clear lines of communication
Local Management
Local technical/modelling
Training
Make timely
Consider delivery mechanism (e-learning, workshop)
Project Management Structure
A quality model can be let down by a poorly defined and delivered process!

46. Some VOA achievements Digitisation of over 22 Million records
Data completeness approaching 100%
Over 120 surveyors trained in AVM techniques
Market analysis nationwide, including mapping localities across the whole country
Model performance well within recognised standards e.g. median COD of 353 BAs is 9.97
Proven IT platform for mass appraisal at national level.

47. Questions? Tim Eden BSc MRICS IRRV Deputy Director of Council Tax

48. Achieving World Class