by Dr. Michael Sklarz and Dr. Norman Miller | October 30, 2017
Download a PDF file of this research paper here.
Introduction
Last year we wrote a Collateral Analytics Research Article titled “A New Method to Define Accuracy of Appraisals.” We have now incorporated this methodology to produce an Appraisal Confidence Score in some of our valuation risk products. Having worked extensively over the years with both AVMs and appraisers, we have always felt that there was significant overlap in the two valuation approaches and that the concept of a Confidence Score and High/Low Range required of AVMs should also be applicable to traditional appraisals.
Several research papers in the past year have only reinforced the idea that valuation accuracy disclosure should be integrated into professional output for clients. For example, a study covering millions of mortgages by Calem, Lambie-Hanson and Nakamura (July, 2017) from the Federal Reserve Bank of Philadelphia concludes the current system of price disclosure creates a flaw in the intended risk management system.[1] They suggest that when appraisals exactly hit the purchase prices, as is so often the case, no real information is gained by lenders and that “…we find that appraisals are less predictive of default than automated valuation model estimates.” About 30% of all appraisals in the Calem et al study were exactly equal to the purchase price and over 90% were at, or above the contract price.
The Calem et al study analyzed the predictive power of appraisals versus AVMs with respect to default and losses. Biased appraisals result in over statements of true equity and understatements of true LTVs. When prices go down or when the borrower faces economic distress, there is less equity to protect the mortgage investor. This is why the AVMs do a better job than appraisals, since AVMs more accurately and objectively value the property. This is not to say that objective appraisers could not accurately appraise property or do so better than some AVMs. It is simply that if they do so, they tend to get less business and so they are significantly biased towards hitting the mark (the “mark” being a price target needed for financing) while AVMs are agnostic about value. Even if the AVMs were less accurate than an objective appraiser, they are in the case of residential mortgage loans, more predictive of default and losses. While none of this is new to the informed professionals in this industry, never before has a study from a Fed bank been so explicit about the superior predictive powers of default provide by AVMs.
Another study from researchers at Fannie Mae provided evidence that appraisals exhibit similar estimation errors as AVMs when appraisers are not given the selling prices of the properties.[2] In this study the authors were able to compare the dispersion of appraisal value estimates when price was known and unknown. The results shown below in our Exhibit 1 from the paper’s Figure 3 Panel C. Note that when prices are unknown, the appraisal error is quite significant, and in fact, often greater than the error expected by AVMs, discussed below.
Exhibit 1: Appraisal Accuracy When Price is Given or Unknown
One of the conclusions in the Fannie Mae paper is that “investors should consider releasing appraisers from the unrealistic requirement to arrive at a single value and instead provide a value band within which a home might reasonably transact.”
Other alternatives are to ask an appraiser to provide a reasonably confident high and low value. Another option is to simply calculate the low and high range for appraisers based on the standard deviation of the adjusted comparable values. For example, if the standard deviation of three estimates is 10% then the appraiser is 90% confident the value is between 83.5% of the estimate and 116.5% of the estimate using normal distributions. This type of disclosure may lead to some gaming of the comparable adjustments by appraisers or it may actually improve the result by providing some immediate feedback to the appraisers that suggests over or under adjustments of the comps.
An early study that was never published formally, as it was intended for internal use by a large relocation company, was conducted in 2009-2011. It was an early opportunity to compare the relative accuracy of appraisals and AVMs when the sales prices are unknown. In fact, the purpose of the appraisal was to assist in the determination of list price. Several thousand properties were both appraised by professional residential appraisers with AVMs also run. We see the results, in Exhibit 2, of the AVMs run in 2011 compared to the subsequent selling price of the property where a zero equals an exact match.
Exhibit 2: AVM Accuracy Compared to Subsequent Selling Price
Exhibit 3: Appraisal Accuracy Compared to Subsequent Selling Price
In Exhibit 4, we compare the error of the appraisals to the AVMs over time. The higher the percentage of appraisals or AVM Value estimates the better are the results. The conclusion is that when prices are unknown the average error is less for AVMs than for appraisals in this 2009 through 2011 test suggest that AVMs do a better job of predicting subsequent selling price and since then the accuracy of AVMs has only improved.
Exhibit 4: Appraisal and AVM Error Measured Over 3 Sample Years
Automated valuation models, unlike appraisers, are agnostic about any subject property sale price and they are modeled to be unbiased with a much more even distribution of low and high value estimates around the observed purchase prices. We are not here negating the possibility that manual appraisals are inherently biased based on current economic incentives and regulations. We are only addressing the question of error or uncertainty behind the value estimates and it appears that there is potentially significant uncertainty in manual appraisals, just as there is in AVM reporting ever since the genesis of their use in valuation and valuation risk management. Industry pundits have assumed that AVMs had higher standard deviations and uncertainty than traditional appraisals, but no one has ever empirically compared the two approaches to value until now.
Strategic Lending
There are several ways lenders could use information on value uncertainty, aside from screening out risky loans on the basis of this uncertainty, for example:
- Target marketing programs towards borrowers on homes that have less uncertainty behind the value estimate.
- Make larger loans and accept higher LTVs when the value certainty is higher, subject to regulatory constraints.
- Charge higher interest rates on loans that have less certainty on the LTV, what we call “risk based pricing.”
- Adjust maximum loans to reflect the degree of certainty required.
Third party insurance companies could also, based on value uncertainty, generate price downside risk protection products for home-owners and lenders as well as mortgage insurance companies. The key to enabling innovation in the market is first to make such information available. It could be provided by running AVMs in parallel as one alternative that accompanies traditional appraisals. Alternatively, appraisers could report standard deviations and uncertainty behind their valuations as one additional piece of information that lenders could use to assess risk.
Conclusions
An appraiser’s dream world would be three virtually identical properties to a subject, all selling yesterday, for an assignment to generate a value today. No adjustments would be required and the uncertainty behind the value conclusion would be nil. Alternatively, if perfect adjustments were possible on all the factors considered important by the market, an appraiser would still get three or more very similar adjusted sales prices from which to derive value. That would be the second best scenario for an appraiser. These worlds do not exist and yet, for all of history, we have relied on traditional estimates of value as if this single point were highly probable. In some markets, it is possible and in others, it is not, but we have no information about such uncertainty. In some markets, there are not enough recent sales. In other markets, the properties are all too unique and customized making comparison and property characteristics adjustments difficult.
For the first time, we provide here some evidence that value estimates from traditional appraisals may have a great deal of uncertainty behind them. Such information could be used for strategic lending and or better risk management by buyers of mortgages or portfolio lenders. It may require new regulation to force disclosure of valuation uncertainty and some training of appraisers, but the process suggested here is not technically difficult, nor is it that much extra work for appraisers.
When appraisers recognize that such information is being utilized or disclosed, it is very likely that the standard deviations of the appraisal’s adjusted comparables will decline. This is because it is fairly easy to re-think prior adjustments and reconcile them towards a tighter range of value indicators using the large range of subjective adjustments available to appraisers.[3] Still, greater disclosure behind value estimates would enhance and improve the working of the mortgage markets for buyers, lenders and investors and perhaps mitigate the extent of the next housing bubble burst.
[1] See Working Paper No. 17-23, Appraising Home Purchase Appraisals, by Paul S. Calem, Laurie Lambie-Hanson, and Leonard Nakamura, July, 2017.
[2] See “Contract Price Confirmation Bias: Evidence from Repeat Appraisals” by Michael Eriksen, Hamilton Fout, Mark Palim and Eric Rosenblatt, July, 2016. Eriksen is at the University of Cincinnati while the other authors are with Fannie Mae.
[3] Such re-tweaking might also improve the quality of appraisals.