You can’t escape the fact that zestimate accuracy is a function of comparison to actual sales. The difference from the actual sales is the error rate. If actual sales are the standard to which zestimates are measured, how can zestimates be more valuable?

BTW, Please, please tell us if Z knows the “average” error rates of its zestimates? If so, what are they? We have a feeling, the average error rate is “shocking” and that is why Z only discloses the median. Prove us wrong.

1) ZERO percent of Zillow’s quarterly reports cover actual sales.
2) Zestimate error rate is of no consequence to the median calculation because Zestimate errors are unbiased.

It really is just that simple.

“A starting point” explains that there are outliers in INDIVIDUAL Zestimates. That issue is irrelevant in the aggregate because again, medians effectively exclude all outliers when there’s no bias in the error.

A housing report which does not reveal the percentage of the data which is actual sales vs. unsold zestimates (of undetermined inaccuracy) cannot be said to be better by virtue of those unreliable zestimates. If you agree with Surowiecki, as you seem to, if the median of “actual sales” is suspect, how can the median of guestimates be better? The Zindex Housing Report is a mixed bag—only we don’t know the mixture. If most of the data is zestimates of UNKNOWN error rates, what you have is a Zestimate Report. And since a Zestimate Report contains data of uncertain accuracy, it is not likely to be a true indicator—it looks more like a STARTING POINT for a report. My logic goes like this—if a zestimate is a starting point, a Housing Report containing those same zestimates must also be only a Starting Point. No?
(If, on the other hand, most of the data in Z’s Report is actual sales, you have a more traditional report, which as Jim says, is also flawed. Pick your poison.) To repeat myself once again, medians of actual sales are flawed and medians of guestimates are more flawed.

The Zindex Drinking Water

I have used the analogy of drinking water. Actual sales are pure water. Zestimates are contaminates because they may be way off (highly toxic). I do not want to drink from the Zindex Report unless I know the mixture. Why doesn’t Z reveal the % of data in the Report that comes from actual sales and those from zestimates. Theorectically, if a market had no sales, the entire report is based on contaminated (in various degrees)zestimates, guesses, starting points, first steps. In a word, bupkes to the serious. As you have stated:
“Zestmates are a useful starting point in researching house values, but serious buyers will get to a point where they move beyond the Zestimate…” A serious housing report must do the same.

Why Advertise Median Error rates? Why calculate them against actual sales? Because actual sales are the standard the zestimate is measured against.

The reason why Z publishes their median error rates is to make the zestimate seem fairly accurate. And the zestimate’s accuracy is ultimately measured against “actual sales”. The actual sales, the reality, triggers the error rate which measures the value of the zestimate POST FACTO. If the error rate was zero you would in fact have the equivalent of ACTUAL sales. Thus, housing reports with zestimates, which aspire to be as accurate as actual sales, cannot logically be more valuable than the actual sales themselves. This concept may be hard to grasp but once you do, you’ll realize why zestimates are always compared to actual sales to measure their value.

A final question. We always talk about median, but rarely average. Does Zillow know their “average” error rates? If so, will they reveal them? If not, why not?

Sorry to put you on the spot David, but we really want to know.

My quest to explain Zillow’s quarterly reporting continues … good news is that we have middle ground to work from — thanks for linking to the Surowiecki article. The quote that explains the author’s criticism of traditional house value reporting is … “Because nominal median prices compare completely different groups of homes (all those sold in August, 2005, say, and all those sold in August, 2006), they can overstate how much prices go up during booms and understate how much they go down during busts.”

That is why reporting on house value trends using median sales is flawed. In the article he says this is how NAR statistics are reported — it’s frankly the most common method. As Surowiecki explained, this approach becomes increasingly flawed with increasing fluctuation in the state of the housing market. Few people would contest that we’re in a fluctuating market.

Surowiecki’s point is that the only way to accurately measure a trend is to normalize the sample. Only Zillow does this.

Your other criticism seems to be that “zestimated homes are mixed with sold homes”. This is not the case. In fact, we don’t consider any sales and only look at Zestimates (they are inputs to Zestimates only). We’re trying to understand where the market IS not where it WAS at various times (sale dates) during the past 3 months.

Now you know we’re only reporting Zestimates the obvious question is “what about the Zestimates that are way off?”

That’s why we use a median to calculate the Zindex; it excludes inaccurate outliers. Obviously it’s only effective if the outliers are evenly distributed. If you read this week’s WSJ analysis of Zestimate accuracy, you’ll notice they confirm our internal accuracy measurements and that their study showed no Zestimate bias … i.e. when we’re wrong, we’re as likely to be “high” as we are to be “low”.

So, Zestimate inaccuracies are ignored when we calculate the Zindex (trend).

Bottom line … Zillow is the only source of house value trend reporting that is immune to shifts in local markets. That said, all statistics benefit from an expert’s interpretation — which is why we now release our raw data to local experts to add their local flavor. Zillow’s Q4 reports in 75 Metro areas can be found here: http://www.zillow.com/quarterlies/QuarterlyReports.htm

I know that these statistical concepts are hard to grasp. Let me know if you have any questions about my comments.