vendredi 19 septembre 2014

Scotland the day after: how did the polls fare?

Hi,

Here is the graph that I published in my previous message on Wednesday afternoon. It shows the likely result when I attribute 67% of non-disclosers to the No side. The results of the vote (55.3% No, 44.7% Yes) tell us that the No vote is underestimated by almost two points, even with this non-proportional attribution. Two polls were published on Election Day: YouGov put the No at 54% and Ipsos-Mori at 53% (this poll was conducted before Election Day however). This may mean that there was some last minute movement. I checked whether my estimate would have been much different if I had given 75% of non-disclosers to the No side. It does not change enough in the prediction to warrant publication.




Were opt-in panel polls responsible for the underestimation of the No side? If I keep only the polls published since the beginning of August, opt-in panels do not differ at all on average from polls conducted using other methods (telephone and face-to-face) in their prediction of the final results. They do differ in the average proportion of non-disclosers: more than five points less on average than other polls. Finally, it is interesting to notice that Survation and ICM  who used to conduct opt-in panel polls both conducted a telephone poll at the end of the campaign. ICM conducted its telephone poll at about the same time (Sept 9-11) as an opt-in panel poll. The telephone poll gave 40% Yes, 42% No and 17% non-disclosers (49% Yes - 51% No after proportional allocation of non-disclosers) while the opt-in panel gave 49% Yes, 42% No and 9% non-disclosers (54% Yes, 46% No). The telephone poll is clearly better. However, we cannot base our evaluation of opt-in panels on only one case.

Conclusion

As with the Quebec 1995 referendum, the polls underestimated the No vote and this, even when we allocate two-thirds of non-disclosers to the No side. This seems like a systematic bias that we will have to take into account in future referendum votes. However, the opt-in panels did not fare better nor worse than other polls at the end of the campaign.

mercredi 17 septembre 2014

Scotland Independence: update with the last polls

Hi,

Tomorrow is voting day. I include in my analysis all the polls published yesterday and today -- before 2 PM Montreal time (I do not have the YouGov and the TNS yet, I may update later on if I have time).

Let us examine what the polls tell us of the likely portrait of support for Scottish independence.  First, a graph including the non-disclosers, i.e., don't knows and those who say they will not vote.The graph  shows that the two sides are still close to each other but with the No side still ahead. In addition, the proportion of non-disclosers has stopped declining.


Now, if we use the pollsters' attribution of non-diclosers, i.e., proportional attribution, here is what we get. Voting intentions seem to have frozen. They have been quite stable for the last two weeks, with about five points difference in favor of the No side.

  
 

If I perform a non-proportional attribution of non-disclosers, giving 67% of them to the No side, which, according to many political scientists is realistic, I get the following graph. The analysis estimates the difference between Yes and No at more than seven points.



Finally, the following graph is quite interesting. I focus on the difference between support for independence overall and support among young people aged 16-34 years old. As you can see, we could attribute part of the increase in support for the Yes side to young people in recent weeks but polls show that this age group seems to have backed off during the last few days.



Conclusion

If polls are not incredibly wrong, my analysis shows that the No side is likely to win with a good margin. If I am wrong, it will be the pollsters' fault :). And political scientists and survey methodologists will have to revise all what they said in the last 30 years.

lundi 15 septembre 2014

Yes Scotland: Ceiling reached?

Hi,

Now, here is the situation, when the polls conducted during the week-end are included. It is interesting to notice that two pollsters -- Survation and ICM -- that used to do only opt-in panel polls conducted telephone polls. The situation right now is that the two sides are close to one another, which explains why we get contradictory estimates from different pollsters. This is quite normal in such circumstances. The graph shows that the Yes side has gained some points and the No side together with the non-disclosers have lost some.The No side is still ahead (barely) but it seems to have lost two points since the beginning of August, while the Yes side has gained close to five points.


Now, if we attribute the non-disclosers proportionally to each side, we get the following graph. It shows that the increase in support for the Yes side seems to have stalled. Support for Yes Scotland is around 48% and support for Better Together at 52%.



If we use a more realistic attribution of non-disclosers' voting intentions, giving 67% of those to the No side, support for the Yes side is closer to 46.5%. The difference between the two sides is estimated at around seven points. The preceding graph and the next one both show rather clearly that support for the Yes side has stopped increasing last week. Voting intentions seem to have stabilized.



A last graph shows the part played by younger people -- 16 to 34 years old -- in the increase in support for the Yes side. Young people's support for the Yes side, already higher than in the rest of the population, has also increased more than that of older Scottish residents. This situation is very similar to what has been observed in the 1995 Quebec referendum. Support for Yes Scotland among young people is around 55% according to the polls.



In conclusion: Are the polls reliable?

The Yes side has gained an average of 1.15 points per week since the beginning of August. Even if it would take another 1.15 points during the last week, it is not sufficient to win. But are the polls reliable? First, let us notice that the difference in estimation between opt-in panel polls and other polls has disappeared. Since August at least, there is no significant difference anymore between web polls and other polls. However, as in Quebec in 1995, the polls may overestimate the Yes side. Why is that? First, it seems that Yes supporters are more likely to answer surveys and to reveal their vote and second, as the Yes becomes more likely to win, some people may back off and finally decide not to "take the risk". As I said before, saying that you will vote Yes to a pollster has much less consequences than doing it for real. A very good analysis of the possible vote of non-disclosers -- by Stephen Fisher -- can be found here. Therefore, I still think that all possible errors of the polls would result in an overestimation of the Yes side.

vendredi 12 septembre 2014

Scottish independence polls: Update & plus

Hi,

***Please note that I made a mistake. All the analysis are conducted since the beginning of January and not August as I stated in my post. *** Please see my next post for an update.
From now on, I concentrate my analysis on the polls published since the beginning of August in order to give more weight to the recent polls in the estimation of change. In this post, I also come back on the generation gap that I have analyzed in a previous post.

The next graph shows the recent evolution of the situation. It includes the YouGov and ICM polls published in the last hours. We see a concentration of estimates in the last few days: Most of the polls have put the two sides very close.


If we attribute non-disclosers proportionally to each side, we get the following portrait. It is slightly modified compared to my preceding post. Because of the two last posts, it no longer shows the two sides at par. The analysis estimates the Yes side at about 47.5% and the no at 52.5%. The spread of estimates has decreased and the estimates seem more similar across pollsters.

However, there is still a very significant difference between the opt-in web polls and the other polls. Using regression analysis, I have the estimate of support for the Yes side (with proportional attribution of non disclosers) at 42.8% at the beginning of August, and almost three points more when the polls are  opt-in panel polls, and gaining on average .136 points per week for the last 6 weeks (0.8 points total). Therefore, opt-in panels have the Yes at about 46.6% and the other polls at 43.6%.


Like in my preceding post, I estimate the results with a non proportional attribution of non-disclosers, giving 67% of them to the No side and 33% to the Yes side (see here for an explanation). This gives the following graph. With this more realistic attribution, the Yes side is closer to 45%


The generation gap

I have looked at the generation gap in a preceding post. It is very interesting to update on this gap for comparative purposes. In the 1995 referendum in Quebec, people less than 55 years old were more favorable to sovereignty at the beginning of the campaign and they were also the only one to move towards the Yes side during the campaign. The 55 years and older age group did not change its mind -- at least on average -- during the campaign.

The following graph allows to compare the 16-34 age old group to the total. Unfortunately, since pollsters do not use the same age categories, it is the only comparison that I can perform (Notice that for YOUGOV, the age group is in fact 16-39 and was 18-39 in the polls conducted before the August 12-15 poll). The graph confirms that, on average, support for independence is higher and has increased slightly more rapidly among the younger age group. However, notice that the estimates (in light blue) vary much. In August, for example, some polls have put support for independence in the younger age group as low as 37% while others put it at 60%.


Conclusion

We have to be very careful not to interpret random variation as movement. There is no doubt that there has been an increase in support for the Yes side during the last three weeks. However, if we take all the polls into account, the Yes side does not have the support of a majority of voters right now. The next polls will be crucial to estimate whether the increase in support has reached a ceiling or not.

In addition, let us stress the fact that when the difference between the two sides is estimated at less than six points, it is not appropriate -- and even a false assertion -- to say that "The Yes side is ahead for the first time" or "the No side is still ahead". This is what the margin of error means. And it is even more so when polls are not conducted using random samples of respondents like opt-in panels do. In this latter case, some speak about a credibility interval that would be similar to the margin of error. However, empirically, there is evidence that, in addition to that random error, there is some systematic bias in opt-in panels. In this case, the bias is likely to produce an overestimation of the Yes side. If the results show that it is not the case, it will be a very interesting topic for research.

mardi 9 septembre 2014

Scotland last stretch oh la la



Hi,

One word to come back on my last post. I ended that post saying that the Scottish referendum campaign looked more like the 1980 Quebec campaign, which ended with a 59.6% No vote, than like the 1995 campaign (50.5% No). Obviously, it is not the case anymore. In the last two weeks, we went from a Quebec1980-like campaign to a Quebec1995-like campaign. This has a number of implications in terms of poll analysis, one of them being the question of how to attribute the voting intentions of the so-called undecideds. I prefer to call this group the non-disclosers, since in all the polls, they include both those who say that they don't know how they are going to vote and those who say they are not going to vote. 

Let us first look at whether there has effectively been a change in voting intentions in the last weeks and to what extent. Since the last debate on August 25, which was won by Alex Salmond according to most observers and polls, five new polls were carried, one by Survation, one by Panelbase, two by YouGov and one, published today, by TNS-Sofres. All these polls tend to show that the two sides are coming closer, almost at par. The following graph gives an idea of what it means using all the polls conducted since the beginning of 2014. It shows that the proportion of non-disclosers is decreasing as the proportion of Yes voters is going up and the proportion of No voters is stable.



However, the preceding graph takes into account all the polls published since January. If I analyze only the polls conducted since the beginning of August, it gives the next graph. This analysis gives all the chances to the Yes side. It shows an impressive movement towards the Yes and away from the No concurrent with a decrease in non-disclosure to the point that the two lines cross at 45%. However, we have to remind ourselves that almost all the last polls are opt-in panels and that these polls have given the Yes five points more than the other polls during last year*. In short, this portrait is likely to be inaccurate.




What about the attribution of preferences to the non-disclosers?

As the Yes and No camps become closer, the question of the non-disclosers becomes important. How should we attribute their vote? John Curtice has shown here that it is not likely that these respondents be more favorable to the Yes side than to the No side. 

First, let us stress that if the "undecideds" were just people who still do not know how they will vote,  that proportion would tend to be the same whoever the pollster. However, it is not the case at all. The proportion of non-disclosers varies widely between pollsters, from an average of  9.5% for Ipsos Mori (telephone) to 26.9% for TNS-Sofres (face to face). In between, opt-in panels have average rates that vary from 11% (YouGov) to 18% (ICM). This shows that the proportion of non-disclosers is a feature of the way polls are conducted by each firm at least as much as a feature of the respondents.

Let us first look at what happens when we use a proportional distribution of voting intentions to the non-disclosers, i.e., the method used by all pollsters in Scotland right now. When we use all the polls published since January, we get the following graph. The analysis estimates the Yes at close to 47% with the No at more than 53%.


However, if we use only the polls published since the beginning of August, here is what we get. The two sides are almost equal.




Is there a relationship between the proportion of non-disclosers in the polls and the proportion of Yes or No? I checked and there is none. This does not mean however that the non-disclosers can be attributed to each side proportionally. Why is that? Because the profile of the non-disclosers is usually more similar to the profile of No voters than to the profile of Yes voters.

I did not performed non-proportional attribution of voting intentions before because since the No side was quite ahead, it did not change anything in the estimates. However, in the actual situation, we have to consider non-proportional attribution for at least two reasons. One is that the No voters may be less likely to be reached by pollsters -- and even more by opt-in panels -- or to accept to cooperate with them. The second reason is that, when they are reached, they tend more to hide their opinion. This is even more likely when the Yes side gets ahead because the Yes side tends to be seen as more "popular".

The situation was about the same in the Quebec 1995 referendum campaign so that the media used to publish estimates that gave 75% of the non-disclosers -- around 13% of the respondents at that time -- to the No side. With such a non proportional attribution, the polls' estimates gave 50.5% to the No side and 49.5% to the Yes side, i.e., the exact result of the referendum. In the case of Scotland, it seems relevant to estimate what it would mean to perform a non-proportional attribution. I attributed 67% of undecideds to the No side and 33% to the Yes side because I think that the possible tendency to hide preferences for the No may be less substantial in Scotland than it was in Quebec because of the ethnic concentration of voters. This is the portrait that I get. It estimates the Yes side at 45% and the No side at 55%.


If I use only the polls conducted since the beginning of August, here is what I get. The two sides are no longer at par like in the comparable graph using proportional attribution. The Yes side is at about 47% and the No side at 53%.





Conclusion


Many commentators, particularly in Quebec, have an impression of déjà vu when they see the reaction to a possible change in the polls from the Better Together camp. Polls do not serve only to measure public opinion. At times, they may be used by people to send a message that they want something on the table if they are to stay. Saying that you will vote Yes in the polls has no consequence besides having the other side make moves.  However, the change in voting intentions may also be very real. Only the next polls will tell us how likely it is that the Yes side wins this referendum. In my next post, I will come back on voting intentions according to age groups because this also seems to have changed recently. As in Quebec in 1995, the younger age cohorts are the more likely to move towards the Yes side.


* About methods: The method that I use takes into account the timing of the polls. It may have its drawbacks but it is certainly preferable to taking the average of the polls published during the last two weeks, for example. This is certainly not appropriate when there is a substantial level of change with time. I also see graphs that do not respect the time line i.e., polls that are two weeks apart are put at the same distance than polls that are two days apart. This is also very misleading and it should not be published.

lundi 18 août 2014

One month before, Scottish Independence & Quebec Sovereignty

Hi,

This is the last month before the referendum on Scottish independence, an appropriate moment to compare the current situation in Scotland and the situation that prevailed in 1995 in Quebec.

Scotland
First, let's examine the current situation in Scotland. With 49 polls published in 2014, one month to go, the likely situation right now is a gap of more than 10 points in favor of the No (12 points if we distribute the non-disclosers proportionally), as the following graphs show. Some have maintained that a rise in support for the Yes side have occurred last week. This is not obvious right now. As we can see, there is variation in the estimates published by the different pollsters. If we take that into account, we are still in a situation of no change in either direction. The impact of the debate held on August 5 may have been to reduce the proportion of non-disclosers (including mostly don't knows).

In evaluating the situation, we have to take into account the fact that half the polls, i.e.,  those conducted using opt-in panels by ICM, Survation and Panelbase, put support for the Yes side 4.5 points higher on average than the other polls (*see methodological notice at the end). If we look at all the elections where opt-in panels were compared to election results, we may conclude that these polls likely overestimate support for the Yes side. Let us now compare this situation with what happened in Quebec during a similar period to see if it can teach us something.



Quebec 1995
What was the situation in Quebec during about the same period? I kept all the polls -- including those conducted by academics, governmental agencies and political parties -- conducted between February 15, 1995 and Referendum Day, October 31th, 1995. I kept the results only when the question asked about vote for Sovereignty with an association with the rest of Canada. Before June 12 (first vertical line on the graph), when an agreement between Parti Québécois, Action Démocratique du Québec and Bloc Québécois -- called the 3-party agreement -- was signed, it was not clear what the question would be and whether a partnership with the rest of Canada would be part of it. Therefore, there are not many polls asking about this option before June 12. I have shown elsewhere that support differs according to the constitutional option -- independence, separation, sovereignty, sovereignty with partnership -- and that only the last option managed to reach majority support during some periods.

The first graph shows the change in support from February 15, 1995 to Referendum Day, including the non-disclosers. It shows that support for sovereignty with an association with the rest of Canada managed to get the support of a plurality of Quebeckers until two months before referendum Day. However, after the official launching of the campaign on September 7 (second vertical line on the graph), Support for the Yes side decreased to 40% and the proportion of non-disclosers increased until the last month. Then support for the Yes side went back up until referendum Day. The final two months are characterized by a roller-coaster type of movement and an increase in the proportion of non-disclosers (including don't knows).



The second graph shows the same information after proportional attribution of non-disclosers (**see methodological notice at the end).  Again, it shows a situation favorable to the Yes side, up until the launching of the campaign on September 7 and a roller-coaster ride afterwards.



Let's now focus on the last stretch. The third graph shows that, when the campaign was launched, the two sides were at par. Support for the Yes side went down during the first month so that one month before referendum Day, support for the Yes side was five to six points lower than support for the No. However, support for the Yes side went back up to finish 6 points higher than support for the No. However, as we can see, with proportional attribution of non-disclosers, support for the Yes was seriously overestimated. The referendum ended with 49.5% Yes and 50.5% No. With realistic (*) attribution of non-disclosers, the prediction of the results was almost perfect.



Conclusion - Scotland vs Quebec
There are a number of differences between the Quebec 1995 campaign and the Scottish 2014 campaign. For instance, the Quebec question pertained to sovereignty accompanied by an offer of  partnership to the rest of Canada and it was officially revealed less than two months before referendum Day. Independence per se (the Scottish option) never managed to get the support of a majority of Quebeckers in the polls.

In terms of poll results, the Quebec question managed to get the support of a short majority of Quebeckers during most of the period, which has not been the case in Scotland. When the Quebec Yes side increased its support during the last month in 1995, it was regaining support from people who had already been inclined to vote Yes at some point during the preceding months. Scotland's Yes side cannot rely on this possibility. It has to convince voters to change a position they held for months. Everything is still possible but, since no movement in support has occurred since March, the Scottish Yes side needs a very major event in order to gain the support of a majority of Scottish people. As the Quebec situation shows, it is possible to reverse the trend. In Quebec, a change in strategy -- i.e. the nomination of Lucien Bouchard as negotiator in chief on October 7th -- seemed to have helped close the gap with the No side. However, Scotland's Yes side would probably need more than that. The current situation in Scotland resembles more to what happened in the first Quebec referendum in 1980 that ended with 59.6 for the No side than to what happened in the second referendum in 1995.

P.S. I will be away from August 24 to September 8. I will be back to monitor the last stretch.

* The other pollsters are TNS-BRMB (face-to-face), Ipsos-Mori (telephone) and YouGov (opt-in panel). These three pollsters had similar estimates since January. Note however that You Gov has just changed some features of its methodology (see here). It announced that it now includes respondents that are 16 and 17 year old (which means that they were not included in preceding polls!!) and that it modified its weighting scheme to take into account country of birth. YouGov states that its results for the last poll are not comparable with previous results and that the proportion of support for Yes Scotland is a bit higher with these new features.
** At that time, in Quebec, media and researchers alike tended to use a "realistic" attribution of non-disclosers that gave 75% of  them to the No side. This practice tended to put both sides almost equal and gave a very good prediction of the final result. Some Media published three figures, i.e., voting intention before attribution of non-disclosers, with proportional attribution and with "realistic" attribution.


lundi 11 août 2014

Polls on Scottish Independence: questions of methods



In this post, I first examine the current state of support for Scottish independence and second, I examine whether different methodological poll features are related to estimates, all else being equal.

The current state of support for Scottish independence

The following graphs show the change in support since January 2014. The first graph includes non-disclosers (don't knows & will not vote). Support seems rather stable since March 2014. The first graph shows a possible recent small increase in support for the No concurrent with a decrease in the proportion of non-disclosers. However, when non-disclosers are attributed proportionally to both sides (second graph), no such increase in  support for the No side is present. There is no significant relationship between the proportion of non-disclosers and support for each side (see below). Before attribution of non-disclosers, Yes is at 38% and No at 50%. After proportional attribution, the difference between the two sides is close to 15%, with the Yes slightly over 43% and the No at 57%.

Notice that in the following graphs, the time line is respected, which is not the case usually in the graphs presented on the different web sites and in the media.






Questions of methods

There was an interesting debate in Scotland on the weighting of opt-in web polls according to the recall of previous votes. In short, the question pertained to which election(s)  should be used for weighting. John Curtice's appraisal of the debate -- that took place between Survation and YouGov -- can be found here: Who is right? Survation or YouGov?  I quite share his appraisal that a)  the difference in estimates between YouGov and the three other pollsters using opt-in panels (ICM, Survation and Panelbase) cannot be due only to weighting and b) we do not have sufficient information to assess the cause(s) of this difference.

However, the interesting point here is that we have a systematic difference within the opt-in panel method. Therefore, we may think that there are specific features of opt-in panels (recruitment, selection, administration) that lend to different estimates. Usually, I check whether there is a difference between opt-in panels and other methods (I did find some in most elections held in Canada --see here and here). In the Scottish situation, I will also check whether there is a significant difference between YouGov, Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) on one side and the opt-in panels carried by ICM, Survation and Panelbase on the other side.

I also address the question of whether the wording of the question makes a difference and whether the proportion of non-disclosers is related to support and to methods.



a) Administration mode

Controlling for time, opt-in web polls in general (including YouGov) estimate support for the Yes side, 3.1 points higher, on average, than the telephone and face-to-face polls. However, if we group YouGov with the telephone and face-to-face polls --which is justified by analysis of variance -- we conclude that the "other" opt-in web polls (Survation, ICM and Panelbase), estimate the proportion of Yes 4.6 points higher than the other polls.

In short, there is clearly a difference according to administration mode but YouGov's estimates are closer to the telephone and face-to-face polls' estimates than to the other opt-in panels.

b) Question wording

Question wording is related to mode of administration but it differs also between the different pollsters. I checked for two possibilities, i.e., whether the fact that Yes or No is specifically mentioned in the question makes a difference and whether the fact that the question asks how the respondent would vote now vs on September 18 makes a difference.

Mentioning specifically Yes or No does not make any difference, most probably because it is not relevant in Web Polls and because both Ipsos-Mori (telephone) and TNS-Sofres (face-to-face) mention Yes or No in their question.

The impact on the level of support of asking how the respondent would vote "now" vs "on September 18th varies depending on the variables that are present in the analyses. For now, I conclude that the difference noticed is more related each pollster's "cluster of methods" than to question wording itself.

c) Non disclosers

If there is one main difference between the pollsters, it is in the proportion of non-disclosers. All the firms differ significantly from each other in this regard. TNS-Sofres (face-to-face) has the highest proportion of non-disclosers -- an average of 28% -- while Ipsos-Mori (telephone) has the lowest -- 9.5%. One may ask whether TNS could reduce its very high proportion by improving its methods using for example, a secret ballot or a leaning question. The web polls  have proportions that are in between, from YouGov at 11.7% to ICM at 18.6%. Notice that, during the last month of the campaign in the Quebec 1995 referendum, the level of non-disclosers (telephone polls only) was around 13% on average and that the final turnout reached more than 93.5% (!).

It is also interesting to notice that:
  • There is no relationship between time and the proportion of non-disclosers. This proportion is solely related to the pollsters. It is a question of methods.
  • However, asking for voting intention "today" instead of "in September" is related to a decrease in the proportion of non-disclosers of more than 5 points. This is related to the different pollsters but is nonetheless very significant.

Conclusion

It is interesting to notice that the difference between opt-in web polls and other methods, and particularly between some web polls and other methods, is systematic and goes in the anticipated direction, i.e., a higher estimation of support for the Yes side by the opt-in web polls. It should also be noticed that there is some research to be carried on why some web polls' estimates differ systematically from others.The other important point is the substantial difference in the proportion of non disclosers. This is clearly a question of methods since the proportion does not decrease with time. However, since it is not related to the proportion of support for each side, it has no clear impact on the estimate of support. Finally, as we get closer to the date of the referendum, the impact of asking voting intention for today vs for referendum day may fade out.

We are getting to the last month of the campaign. I will go on monitoring whether the impact of methods changes in the last stretch. However, in my next post, the focus will be on comparing Scotland with Quebec, and specifically with what happened during the last month (the official campaign) of the Quebec 1995 referendum.