Data Miners Blog

Very helpful for my final year examination! Thanks...

2012-05-19T14:27:48.780-04:00

Very helpful for my final year examination! Thanks so much!

It is certainly not always true that the hotels wi...

2012-04-19T14:54:30.512-04:00

It is certainly not always true that the hotels with the most reviews have higher ratings, but it is true often enough to be true on average.

This is an interesting observation. I would be cur...

2012-04-19T13:58:21.269-04:00

This is an interesting observation. I would be curious to see if this trend holds when you examine more specific groups of hotels. For example, if you took all the hotels in a particular city, do the most reviewed hotels always get the higher ratings. I could see it going the other way if a particular city really did have crumby hotels as a whole compared to cities like Las Vegas. I could also imagine a cute little BnB getting higher reviews in a small town, even though its size prevents it from having the number of reviews as a big hotel in that same city.

This is an interest observation. I would be curiou...

2012-04-19T13:55:35.307-04:00

This is an interest observation. I would be curious to see if the trend holds when you look at more specific groups of hotels. For example, if I look at just the hotels in a specific city, do the most reviewed hotels still get the higher ratings? I could see it going the other way if we talk about a particular city as a whole really does have crumby hotels compared to cities like Las Vegas.

This explanation makes a lot of sense, and the con...

2012-04-19T12:39:54.936-04:00

This explanation makes a lot of sense, and the concept is great. I have a question, though, about bias. On one hand, it seems that the between-group variance, as measured here, includes a bias of number of zip codes per state, ascribing more between-group variance to states with a greater number of zip codes, and that the bias could be removed by using only a single data point for each state’s average zip code population. On the other hand, that bias may be necessary, because the original, not-averaged variance seems to have the same bias. And states with more zip codes likely have more total variance contributed, though not necessarily. My conceptual problem is that the set of properties that govern how many zip codes a state has may or may not intersect the set of properties that influence between-state variance in average zip code population. So, while this method does separate the variance into two parts, how much is the separation actually an isolation of between-group vs. within-group influences?

In case of Unbalanced data set, how reliable is C ...

2012-04-19T05:25:12.868-04:00

In case of Unbalanced data set, how reliable is C value? Can C value be treated as measure of Model fit as mostly in unbalanced data set models number of ties are higher.

It occurs to me that testing for cannibalization m...

2012-04-18T20:28:49.219-04:00

It occurs to me that testing for cannibalization might be done in different ways for different applications. Your approach for this scenario makes sense, you wouldn't want to test using a longitudinal study because hotels change over time and you have data to map hotels together to do the comparison. However, lets say that a library wanted to find out if enhancements to their online offerings was cannibalizing their physical offerings. A longitudinal study would make sense in that case. Assuming you had the data, you would be able to determine physical usage of library resources before the online resources were offered and then their usage after. The only thing that has changed is the presence of online resources.

It seems that the solution to this problem is to c...

2012-04-18T15:48:55.498-04:00

It seems that the solution to this problem is to compute averages weighted by bucket size and/or variance. In statistics, this is the same idea as correcting for non-independent data points. For instance, if I'm fitting a linear model over a set of observations of human activity, and multiple observations may be derived from the same individual, then I may want a random effect in my model to account for individuals as a grouping factor. In the case of hotel scores, if you were actually interested in the average hotel score across all hotels, then you would certainly want to consider the hotel as a grouping factor that impacts individual ratings. One simple solution is to use an average of averages, but you can get more fancy by taking into account things like variance within each bucket.

One place that I see this type of use that causes ...

2012-04-18T00:22:45.200-04:00

One place that I see this type of use that causes the same problem, and that is in top x from a graduating class. That top is not always a good indicator of the actual performance, since each group class weighs differently. This is both a blessing and a curse since a top performer might perform better with other more aggressive classmantes, as it could also fall behind the pack. School systems need to adjust their metrics.

Interesting. Your point that “you can't extrap...

2012-04-17T16:16:04.905-04:00

Interesting. Your point that “you can't extrapolate from one level of aggregation to another without knowing how much weight to give each unit” is well taken. I wish we could spread the word--it might change the way news outlets report polling results. I wonder, though, about the implications of performing these analyses on ordinal data. Is the distance between a 1 and a 2 really the same as between a 4 and a 5? If 1 through 3 were collapsed into 2 buckets, would that remove some the disparity?

Interesting analysis nonetheless, and the fact that the data is ordinal doesn’t affect the validity of your point about extrapolation. I’m just curious whether you have looked at the ordinal issue, and, if so, what you have found.

what are best universities to have MS in data mini...

2012-04-16T21:15:40.508-04:00

what are best universities to have MS in data mining in new york state?

This blog post focuses on one of the aspects of da...

2012-04-13T17:16:42.648-04:00

This blog post focuses on one of the aspects of data mining that interests me the most, ethics. The example of whether data should be collected in order to improve medical health is an interesting one. I strongly believe that pharmaceutical companies should be able to collect relevant data on their drugs in a manner that isn’t hindering them from improving their products; however, I also see how that kind of information can be abused. I think it comes down to how much people care.

For example, I’ve resigned myself to the fact that, thanks to social networks such as Facebook, privacy is fast becoming a thing of the past. A lot of information about me is available to companies willing to pay for it. There are almost a billion members on Facebook, but how many are aware that information about them is being sold to companies. Do people not care because they are ignorant? Or do they not care because the information they are sharing is what they want to share?

It is interesting just how much people are willing to share about their lives, even with anonymity stripped away. In the case of Facebook, there are no apparent adverse affects to sharing content about oneself. While there have been stories about people being fired for something they wrote, it was the customers choice to share the information, so we don’t throw up our arms in outrage. It is when companies use our data in ways we didn’t expect, things we didn’t explicitly share, that people are enraged (and rightly so).

For example, if pharmaceutical companies were able to access more information about their consumers, their abuse of the data could reflect negatively on the consumer. So, in order for there to be any progress, the data that companies have access to should be data that can be used to benefit the customer without any apparent negative effects. Although even then I’m sure we’d spark debate on what people perceive as negative effects.

It seems that the original challenge stems from a ...

2012-04-13T13:14:50.911-04:00

It seems that the original challenge stems from a lack of public education in current machine learning techniques. R measure is mainly used in statistics and not really applicable to a decision tree. In a world where data mining is more prevalent we need to educate and inform more about how it works and how to apply it in a business setting. If the people needing this information are more educated on data mining and machine learning practices and metrics then they might be more apt to accept the model without trying to apply a qualitative metric that while valid, does not fit the model.

It seems that the original challenge stems from a ...

2012-04-13T13:12:32.070-04:00

THAAAAAAANK YOOUU very much! I would never have ma...

2012-04-13T12:30:09.490-04:00

THAAAAAAANK YOOUU very much! I would never have managed to calculate this on my own

65.4 65.9 66.9 Seems like a wash! ;-)

2012-03-15T09:24:37.398-04:00

65.4
65.9
66.9

Seems like a wash! ;-)

Interesting read.

2012-03-11T19:59:29.104-04:00

Interesting read.

Thanks so much for posting these instructions. The...

2012-02-18T05:39:11.642-05:00

Thanks so much for posting these instructions. They have been enormously helpful. I have got to point 7 in your list above but can't ping the linux guest from the host. Hence when it comes to setting up eclipse there seems to be no way to get eclipse on the host to communicate with the VMWare linux guest.

Thanks for sharing, I have circulated to my collea...

2012-02-12T16:23:24.349-05:00

Thanks for sharing, I have circulated to my colleagues

that also doesnt work

2012-02-07T09:51:32.092-05:00

that also doesnt work

Can you please explain how the iteration works in ...

2012-01-05T02:23:54.663-05:00

Can you please explain how the iteration works in MapReduce? I mean how to
input map->output->reduce->input map->output->reduce and so on. It is soo complicated :(

Hi Michael, great post, thx for sharing with us! ...

2011-12-26T17:13:26.022-05:00

Hi Michael,

great post, thx for sharing with us!

Abs!

Flavio

In my testing, the formula above was incorrect for...

2011-12-10T09:24:06.292-05:00

In my testing, the formula above was incorrect for numeric items at least. It needed to be something like

{=SUM(IF(data!B$2:B$55>=data!B2, 1, 0)) + 1 + SUM(IF(data!B$2:B2=data!B2, -1, 0))}

to give a result where 1 was the first sorted entry

Good stuff... been looking for how to accomplish t...

2011-10-31T06:35:41.140-04:00

Good stuff... been looking for how to accomplish this exact task.

Thanks!!!!!!

I didn’t have space to discuss which sorts of bran...

2011-10-11T03:46:53.178-04:00

I didn’t have space to discuss which sorts of brands are more or less threatened by reviews sites, but it’s worth pointing out that consumer product brands like hotels and motels are safe.