Evening Standard/YouGov plan new flawed poll to seek to influence election on Thursday
You Gov is reportedly planning a new poll to be published on Thursday, the day of the London election for Mayor.
YouGov's controversial Evening Standard internet polls, which suggest big leads for Boris Johnson, have consistently been far out of line with conventional telephone polls carried out by reputable companies like Ipsos-MORI and ICM which have all shown Ken Livingstone and Boris Johnson to be neck and neck.
The Evening Standard/YouGov polls have been used to try to suggest big leads for Boris Johnson, and generate momentum for his campaign, and it is therefore overwhelmingly likely that this is the purpose of any poll they publish on the day of the election.
Fundamental faults identified in YouGov's methods in these polls for the Mayoral election range from getting the age profile of London's population completely wrong, over-estimating those groups most likely to vote for Johnson, up to including some people outside London who obviously do not have a vote in the Mayoral election.
YouGov have even admitted that they have made basic mistakes in their polls. One of the founders of YouGov, Stephan Shakespeare, has prepared the ground for the company's explanation of how they got it wrong after the election when he said on the ‘Conservativehome' web site on possible reasons for error could be ‘there is one considerable uncertainty: turn-out.' http://conservativehome.blogs.com/centreright/2008/04/yougovs-mayoral.html
The Ken Livingstone campaign have made an official complaint about the fundamental flaws in YouGov's methods and the way they have allowed their polls to be used by the Evening Standard to the Market Research Society.
A spokesperson for the Ken Livingstone Campaign said:
‘The YouGov/Evening Standard internet polls have been transparently used to try to influence the election by suggesting incredible leads for Boris Johnson when all conventional polls by companies like Ipsos-MORI and ICM have shown the election to be neck and neck. A YouGov/Evening Standard poll on Thursday would be just another attempt to deflect voters from the enormous stakes for London in this election by suggesting a Johnson lead which simply does not exist.'
Notes to editors
The text of the complaint to the Market Research Society is as follows:
1. The first ground for complaint is on the basis of Paragraph B49 of the Code of Practice of the Market Research Society:
"B49 Members must ensure that research conclusions disseminated by them are clearly and adequately supported by the data."
a. This is clearly breached if members allow conclusions to be drawn from data from a survey which is not representative. YouGov provided to the Evening Standard and allowed it to publish three surveys purportedly measuring the voting intentions of Londoners, conducted on 19-20 February, 12-14 March and 20-25 March 2008. In each case the YouGov raw sample was approximately half composed of those in their panel aged 55+ This is demonstrably unrepresentative of London adults. The Office of National Statistics (ONS) in 2006 reported that the true figure should be 26%. (See attached tables)
These samples were then weighted so that approximately 37% were aged 18-34, 24% were aged 35-54 and 39% were aged 55+. This is still demonstrably unrepresentative whether the target audience is London adults or Londoners who vote. The correct figures for the whole resident adult population are approximately (from the latest ONS mid-year population estimates for London) 37%, 37% and 26% respectively. (See attached tables)
The weighted sample therefore grossly over-represents those aged 55+ in the population and underestimates those aged 35-54 to the same degree; whereas if the sample is intended to resemble those who vote, those aged 18-34 (the age group with much the lowest turnout) should have been considerably down-weighted from 37%.
Coincidentally the age group most over-represented in relation to its size in the population is the one that (according to YouGov's tables) most strongly supports Boris Johnson for Mayor of London, while the group most under-represented is the one which most strongly supports Ken Livingstone.
b. There was similar discrepancy in an earlier poll conducted for the London Policy Institute on 7-9 November 2007, in which 45-54 year olds made up only 8% of the weighted sample and those aged 55+ made up 37%. However, for the next two polls (those for ITN conducted on 19-21 December 2007 and 22-24 January 2008), YouGov used a different age profile for its weighted sample, considerably nearer to the ONS profile but under-representing those aged 55+. (See attached tables) Yet in the next poll, for the Evening Standard (19-20 February, as discussed above) they reverted to their original unrepresentative profile.
c. After the question of their weighting for age was raised publicly at the start of April, YouGov admitted (in a statement to a blog - attached) that there was an error in its age profile as used in its previous three polls for the Evening Standard, and has adopted for its two most recent polls (2-4 April and 9-11 April) a weighted age profile much closer to the ONS figures. YouGov also stated that, nevertheless, reweighting of its earlier polls to an accurate age profile "made no material difference to the overall results".
d. But this raises further grounds for complaint under this heading: YouGov has not published tables of the reweighted data to demonstrate that the overall voting intentions in these three polls are unaffected by weighting to a different age profile, and continues to show on its website tables weighted to the inaccurate profile, in which the percentages of the vote shown for subgroups must be inaccurate. (Grossing up the percentages shown as supporting each candidate in each age group in the three polls shows that the difference between assuming the ONS age profile and the YouGov weighted profile makes a difference of between three and four percentage points in Boris Johnson's first preference lead. Therefore if the effect of the other weighting factors used has been to ensure that overall voting intention is unchanged when the age profile changes, mathematically the split of the votes within each age group must have changed.) There is apparently no notification given on YouGov's website that it now recognises that these tables are inaccurate. It is therefore continuing to disseminate research conclusions that it knows to be misleading.
2. A second ground for complaint under the same rule is that some of YouGov's London polls have apparently included a small number of respondents from other regions of the country. These should plainly not be included in a poll that claims to measure the voting intentions of London voters, unless they have a vote in London even though living elsewhere. Whether YouGov has ascertained that this is the case for some or all of these respondents is not clear, but the issue was raised on the Polling Report blog by a regular contributor (Mark Senior, on 8 April) who stated that he had been asked to complete a YouGov London voting poll despite not living in London. The pdf version of the tables for at least one poll on YouGov's website (the January poll for ITN) now no longer includes the regional breakdown in the statement of weighting variables which was included when the poll was first posted. (Copies of both versions of the tables for this poll are attached.)
3. The analysis of voting intention by age in YouGov's poll for ITN on 22-24 January is extraordinary, and entirely at odds with that shown in every other poll: in particular, it gives Ken Livingstone a substantial lead among 18-24 year olds and those aged 55+ while Boris Johnson is strongest among 35-54 year olds. Such a finding when taken in conjunction with YouGov's other polls showing completely the opposite pattern cries out for comment. In the absence of any more convincing explanation, it inevitably raises the suspicion of observers that the reporting of the poll may have confused the two candidates and that the wrong candidate was publicly credited with the lead. Even assuming that this is not the case, YouGov should have taken steps to allay this suspicion, which can only be damaging to the credibility of polls. (see attached graph)
4. It is certainly arguable that any poll of London opinion, especially one of Mayoral voting intention, ought to be weighted by or least controlled by ethnic group and car ownership. YouGov has not weighted its earlier polls by either (though it agreed to weight by the former in future, and has done so in its more recent polls), nor published figures to allow observers to judge whether its samples are representative on this basis or not.
5. Given the high profile of the London Mayoral election, and the prominence of the Evening Standard in coverage of this election, producing polls for publication in the Evening Standard which are flawed in the ways detailed above inevitably risks damaging the reputation of the polling industry. (see Evening Standard, 14.4.08, attached)
This would be a further breach of the Code of Practice of the Market Research Society:
"A8 Members must not act in a way which might bring discredit on the profession, MRS or its Members."