« Pretty Fences Make Good….
Interactive Billboard Advertising: Bringing Web 2.0 Offline »

Computer Generated Home Values: Does Knowing the Median Error Rate Make the Zestimate More Reliable?

Published by Joseph Ferrara (2) January 16th, 2009 in zillow. Comments

zestimate-error-rates

There is an interesting debate in the comment section of this post:  Zillow Ain’t Got Nothin on Cyberhomes…. (theFrontsteps.com)

The post says Cyberhomes has more accurate home valuations than Zillow, at least in the San Francisco area.  Cyberhomes pulls its information from Fidelity National Title, which has one of the largest databases of home sales and tax information in the country.  Comments from Z-men say No Way.

Do AVM Error Rates Really Matter?

One element of the debate concerns the publication of error rates on the computer generated (AVM) home values.  The Zillow forces contend that Cyberhomes does not publish the area error rates of their home values implying that this raises the value of the zestimates. Zillow publishes median error rates for geographic areas.

I say knowing the median error rate in an area does not give Zillow’s zestimates an advantage when looking at a particular home because knowing the median error rate will NOT reveal the error rate for that specific home.  Therefore, an AVM from Zillow or Cyberhomes (or anyone else) is equally suspect. 

Zillow’s median error rate, in other words, is a red herring. What say you?

Here are the relevant comment threads (go to the site for comments by others– Sophie’s is worth a read:

David G:

Hi guys, it’s David G from Zillow.com,

As far as I know, Cyberhomes don’t publish their AVM’s accuracy and so claims about being more accurate than Zillow are unsupported (and from all anecdotal indications, probably false.) If you do have any info on this, please let me know.

I wonder if you considered the inherent inaccuracy of estimates that are only based on public records? In my experience, value estimates of listings need special attention and that’s simply not Cyberhomes’ expertise.

The challenge that you will face with Cyberhomes (and all other AVM’s beside Zillow) is that estimates on listings can be inaccurate due to public records that are incomplete, out of date or incorrect. An experienced Realtor will understand the challenge with public records when it comes to home facts like beds, baths and sq.ft. Despite the fact that Cyberhomes has good access to public records their estimate accuracy suffers greatly whenever that raw data is flawed. You will find that this scenario can be extremely frustrating for sellers, buyers and agents because the listing info is usually correct yet the estimate is way off because of public facts.

Zillow is the only site of its kind that incorporates listing information in estimates. So, even ignoring algorithm differences, Zestimates on listings are inherently more accurate.

Since you’ve already done the work to integrate this stuff I am sure that this comment is poorly timed and my apologies for that. This important issue of a listings’ estimate accuracy seemed to be overlooked. Good luck with Cyberhomes and let me know if you’d like to learn more about Zillow’s API’s or Zestimates.

Reggie from Cyberhomes:

@David G. – Let me help you get your facts straight…the Cyberhomes valuation does include public record data…to be specific our database of over 110 million properties, ownership, sales and mortgage records…YES. But it also includes analysis of local market trends, recent sales, and local economic factors all meshed together using our proprietary AVM process. Not to mention it’s enhanced with 10 years of appraisals and appraisal review information. I would expect you to know this?

Spencer Rascoff (Zillow):

Frontsteps,
Have you been able to find accuracy statistics on Cyberhomes’ valuations? If so, please tell me! We’ve looked and can’t find them anywhere, nor can we figure out how a consumer would contact Cyberhomes when their valuation is incorrect.

Reggie,
How accurate are Cyberhomes’ valuations?

theFrontSteps:

Spencer,

We’ve found Cyberhomes to be as much, if not more, accurate than Zillow with their data and valuations, which is why we chose them. We have yet to search a home on Zillow for SF that came back anywhere near accurate in valuation. The couple we searched on Cyberhomes were much closer to what we would value the homes at. Not 100% accurate, but closer.

But just like Zillow, there is an inaccuracy factor due to insufficient data. We’re sure both companies are working towards perfection, we just like Cyberhomes vibe better. We’re all about the vibe man and going with the underdog. Truth be told, we’ve never been fans of Zillow, but we’re open for them proving us wrong.

Reggie (Cyberhomes):

Spencer,
I can’t help but feel like you are trying to stir up trouble with untruths…but regardless let me help you get the facts straight. Fidelity National Real Estate Solutions (FNRES) has taken on the name Cyberhomes, LLC after the successful Cyberhomes.com. One of the Cyberhomes, LLC holdings is the rDesk line of agent and broker websites solutions, are you talking about this? Nothing has changed except the name. (I’m sure you wish that were different.)

As for your statement about Consumers not being able to contact Cyberhomes regarding their valuation…another complete and total untruth! Our customer support center takes great pride in listening to our customers and their concerns. We also have a link on every page designed to help us collect feedback from our users, thus helping us meet our goals of continuous site improvement.

Spencer Rascoff (Zillow):

….I noticed that eppraisal updates their valuations once a month, and that your valuation of my home is date-stamped 12/31/08. How frequently does Cyberhomes update valuations? (I could probably find this out on your site somewhere, so my apologies for asking what might be an obvious question.)

My comment:

Great comments here. I especially love Sophie’s spot on observations (the vacant lot example is classic)

David G’s first comment reminds me of a debate we had in 2006 on a post called Mining the Elusive Unzillowable. http://tinyurl.com/2ffov5

His quote here “… estimates on listings can be inaccurate due to public records that are incomplete, out of date or incorrect.” is eerily similar to my objection back in 2006: “(assumption) 2. To be valuable, data must be fresh, accurate and complete. (and public data isn’t)”

David’s reply back then was:

“Assumption #2 is generally incorrect but can be correct in some edge cases — remember that the public record is actually being double-checked by many human beings. I’ve toured the assessor and recorder’s processes in Seattle (highly recommended) and there’s not much room for an error to make it the whole way through that system.”

Oh my, how he has come around to the truth about public data. Thus, to the extent zillowfied homes rely only on public data (and not listing info), they are equally suspect.

Re: Error rates

Zillow’s reliance on error rates to assuage the fundamental zestimate flaws is misguided, since a median error rate is simply the middle number in the error rate findings (calculated ex post facto, and that;s a whole ‘nother can of worms) meaning 50% of the Z error rates are ABOVE the median and can be huge.

For example, using error rates of 3% 4% 5% 7% 18% 39% 150%, the median error rate is 7% and looks good. But there are 2 problems–
1. the average error rate is over 30% &
2. (the most relevant) what error rate applies to the specific house I am looking at? Is it 7% 30% 150% or something else? It is UNKNOWABLE.

SO, knowing the median error rate in an area is USELESS when dealing with a specific house.

Someone please do a mashup of all the AVMs so their unreliability can be readily apparent.

Technorati Tags: , , ,

Share This Post
  • Let me get this straight ... are you really saying that having ZERO reporting and disclosure on accuracy is preferable to using a statistic that you don't understand? This post makes no sense ... unless of course your only motive is to bait me ... but I thought you said you were getting over those impulses?
  • I'm saying that median error rates are red herrings when trying to assess the accuracy of a zestimate on a specific home. They are worthless. Totally doo-doo (it's an industry term)

    Perhaps a simple example will show you what I mean. If the median error rate in New York is 10%, how accurate is the zestimate on 1250 Victory Blvd, SI, NY? Are you saying it's within 10%-- no. Could it be off by 65%-- yes. Get it?

    PS. No I am not trying to bait you but I do appreciate the humor.
  • Toss Homegain into the mix. HomeGain was the first company to put web based instant home valuations on line-back in 1999.

    HomeGain's provides a range instead of an exact valuation (we could provide an exact number but we think that would indicate that a computer could accuratly provide a precise valuation.)

    We believe that only an experienced reator can accurately place a value on a home as after all she the one who is going to get the price for the seller.

    That's why along with our instant home valuation, in more than 2100 zip codes, we also provide access to a realtor to contact to get a more accurate assessment of a home's value. We also feature that realtor's blog, profile and listings. Here is an example

    http://www.homegain.com/homevalues/Merritt-Isla...

    The focus is on the featured Realtor's expertise, not the accuracy of the HomeGain instant home valuation tool.

    We also provide access to our Home Sale Maximizer tool,which based on a survey of over 2000 real estate agents, let's you know which home improvements will provide the best ROI (http://www.homesalemaximizer.com)
  • Zillow does concede the zestimate is a starting point, which should suggest to folks Zillow does not really know and you ought call an experienced real estate pro to have a look see. Home condition alone (not to mention the unzillowables: http://tinyurl.com/26vx5z has a wide ranging effect on home value and since no company offering a AVM comes by to inspect your home, well then, how reliable can this be? It's poop in a nice bag.

    Zillow would be better off aligning themselves with homeowners (instead of ad men) and work with them to match them with their directory of appraisers & real estate agents, to get discounted appraisals & CMAs. Then they would have something to crow about. Rich, are you listening?
  • Sometimes an answer is so obvious and just get missed.

    C'mon, if you really know the error rate for geographic areas why haven't you fixed your algorithm?

    I love the argument..........."My error rate is better than your error rate!"

    Hey Mr and Mrs Consumer - CALL YOUR LOCAL REALTOR!
  • If you knew the error rate that would indicate that you actually knew what the CORRECT price should be. If you do indeed know what the correct price is.................ummmmmmmmm why aren't you providing that price?

    I know.................you can't.
  • A specific home's error rate cannot be computed until AFTER the house
    sells. Even then, Zillow will put the zestimate, not the sold price, into
    its Zindex report before shipping it off to newspapers for publication on
    the state of home "values".
  • Laurie,

    You are correct, these guess estimates are USELESS!!! Get in contact with someone on the ground, in the neighborhood that can accurately compare the useful comps and tell give you an accurate value!! A Realtor or appraiser. As jefsellsius say's it so nicely, everything else is poop in a nice bag."
  • Joe,

    If it is true that Zillow is reporting the zestimate and not the sold price, after the sold price is published, and sending it to the media, then they are part of the problem. How unprofessional! It's a shame the media isn't held to a higher standard as well.

    Marti,

    Yep.............USELESS!
  • I had this conversation with Spencer Rascoff and expressed my surprise and displeasure that sold prices are trumped by zestimates (albeit recalculated), meaning that ONLY zestimates are the data points Zillow puts in the Zindex report. http://www.zillow.com/howto/WhatsaZindex.htm Why a sold price is not a data point is beyond my understanding.
  • Marvin..........sorry. Had too many windows open multi-tasking.
  • Actual sold price is the ONLY thing that should be reported once it is available, to report anything but that is misleading. Geez Joe, that sounds like another post. I can see the title now; "When Branding and Drinking Kool-ade Collide"..................
  • You would think SOLD prices should be data points instead of zestimates but
    not according to the Zindex, which is ONLY zestimates. Even the mainstream
    media, which publishes the Zindex report seems to overlook this.

    Yes, there should be a post... and there is : http://tinyurl.com/4d9rpa
    (May 2008)
  • Laurie -

    Joe's got you a bit confused. Understandable. This stats stuff can be confusing but I think it's important for a Realtor to understand how housing data is aggregated - especially those indexes that are commonly cited in the MSM. Please bear with me as I explain what is a rather dry topic ...

    First, Zillow certainly does report recent sales prices on the site. It's EXTREMELY important data at the home level and recently sold homes on Zillow get flagged as such. The Zillow Home Value Index that is widely reported in the media is a function of sales prices. Recent sales prices are the most important input to Zestimate calculations.

    Now here's what Zillow does not do ... when we aggregate home value statistics, we don't only consider the value of those homes that have recently sold. Instead, we use the (estimated) value of all of the homes in the area and report the median of those values as the Home value Index. We don't report median sales values because that metric only considers the values of those homes that have recently sold. Obviously today's sale price is the best measure of the value of a home but in the aggregate, the price of that handful of homes that recently sold is actually not the most useful information because it leaves out all far more homes than it considers. A median sales analysis is therefore a view into just one slice of the market and is heavily skewed by current demand. So, median sales is a relatively inaccurate measure of the trend in value of all homes - specially in a changing market. Anyone who worked in Real Estate over the past 5 years will remember how use of this metric by NAR largely caused David Lereah to continue to report "a great time to buy" when markets were already declining.

    I hope I've adequately explained the need for something other than median sales analysis. Enter the home value index. Just like other economic indexes, a home value index tries to normalize sales data to remove market noise. There are currently two approaches to this; either extrapolate the values of a normalized "basket" of sales of representative homes types ... or ... estimate and aggregate the value of all homes in the geography.

    The Case-Schiller Index, OFHEO and Zillow's Home Value Index (HVI) are the three indexes that you currently see cited by the MSM. Noticeably absent from this group in recent years is NAR's median sales analysis. While the usefulness of aggregate home values to your current buyers is arguable, a Realtor will be asked about the trends that are reported by the media and so I believe it's a good idea to understand how the popular indexes are calculated.

    The CSI is probably the most popular today. Very simply speaking, it uses a normalized set of home sales, just like the CPI uses a standard basket of groceries to measure inflation. The challenge with the CSI is that it needs lots of recent sales data and so can't report very granularly on home values in local geographies. In the city-level geographies that they do report on, we've seen a very close correlation between Zillow's HVI and the CSI. OFHEO uses a similar approach to CSI but since it only considers sales eligible for conforming mortgages, it has grossly under-reported the values lost in high-value states like California. Zillow's approach is to calculate the Home Value Index in a bottom-up approach that considers the estimated value of all homes. This simple yet thorough aggregation allows us to report home value trends at a much finer granularity than the other indexes (all the way down to the ZIP level.)

    Zillow uses the median Zestimate value to calculate the Home Value Index. This calculation gives us two benefits:
    1) Considering the value of all homes gives us a lot of data to work with and thereby enables us to report on local home value trends. This approach considers way more home values than either the median sales or basket index approaches do.
    2) Selecting the median estimate gives us a really robust way of negating the variance in accuracy in individual homes' estimates. Since Zestimates that are inaccurate have as much random chance of being high as they do of being low, selecting the median Zestimate ensures an acurrate measure of home value trends in the aggregate.

    I told you it was going to be boring but I hope this topic is clearer now - if you read all the way through my comment, thank you!
  • Joe -

    Your suggestion is not only ridiculous, it's impossible. To use median sales to calculate the HVI, two thing would need to happen ...
    1) We would need all homes in the country to be sold at the same time ...
    2) We would need the sales price of all of those sales to be instantaneously and accurately made public.

    When both of these things happen, it will become possible to use the median sales price to understand home value trends. I give you my word that on that day, I will join you in the campaign to have Zillow change the HVI calculation. I'm ready. Just let me know when it happens ...
  • I blame myself for failing to explain to you a very simple point. I will try once again.

    Zillow only uses zestimates as data points in the Zindex, NOTsales prices.. My suggestion is : when Zillow has a recent sales price on a house, use the sales price as the data point in the Zindex, NOT the zestimate. To do otherwise is ridiculous and I can prove it to you-- here goes (hold on to your hat and concentrate)-- what value does Zillow use to calculate an error rate when measuring accuracy of a zestimate? Answer: THE SALES PRICE. Thus, if the sales price is the gold standard to measure zestimate accuracy, it should be the standard when counting data points in the Zindex.

    So, lets say a home recently sold for $300K and the zestimate for that home is $295K, the data point used is the $295K zestimate, NOT the sales price. I say it should be 300K.

    Your comments confirm this method:

    "The Zillow Home Value Index that is widely reported in the media is a function of sales prices" ->Notice the words "is a function of"

    "Zillow's approach is to calculate the Home Value Index in a bottom-up approach that considers the estimated value of all homes. " ->Yes, considers only the estimates.

    "Zillow uses the median Zestimate value to calculate the Home Value Index". ->Right- uses the median zestimate.

    Perhaps this example will help you understand what I mean:

    house 1: zestimate = 300K
    house 2 : zestimate = 325K
    house 3: recent sales price: 350K ; zestimate 330K
    house 4 : zestimate 375K
    house 5: zestimate 425K

    Zindex median is 330K because it's the middle value zestimate. If the sale price on house 3 was used as a data point the median would be 350.
  • First of all, I am not in the least bit "confused".

    Let me make this very simple. All your calculations are useless to consumers. Appraisers are barely going back 1 month when pulling comparables to values homes. None of your findings are relevant to today's seller.

    Zillow and other entities that are purporting to provide consumers will home values are doing more harm at the street level than they are doing good. Once a homeowner sees their value at $350,000 on one of these sites, it is almost impossible to explain to them that they cannot possibly sell for more than $300,000. They do not want to hear it. They will find an agent to list it high, will chase the market down and will lose value, probably selling at $250,000. Zillow will have misled that consumer and have cost them $50,000 or more. I won't be that agent.

    When a buyer sees an incorrect price valuation on Zillow at say $250,000, you try telling them that today's market value is $300,000. They will continue to use Zillow as the barometer for their search, as a buyer I worked with did. He currently lives in rent. He will be staying in rent until lenders loosen up and he can once again qualify for a loan or he will have to wait for the next down market. But he believed!

    There are reasons that Realtor's exist. There are reasons that we are licensed and trained. We not only have immediate access to valid local records, we know how to analyze those results in our local markets.

    There is NO replacement for the local Realtor.

    David, I understand that Zillow wants to be able to report home sale statistics. That cannot be done accurately without all of the information. Zillow just does not have all of the information and much of what it has is not accurate. If the information that you in possession of is not accurate then the numbers you are reporting aren't either, its just that simple.
  • David,

    Your comment: "Since Zestimates that are inaccurate have as much random chance of being high as they do of being low, selecting the median Zestimate ensures an acurrate measure of home value trends in the aggregate."

    Only works if the ratio of error on the low side is the same as the ratio of error is on the high side.

    Any differential would skew the numbers to inaccuracy.
blog comments powered by Disqus
Blog Widget by LinkWithin