Category Archives: Inflation news

CPIH: The impact of adding housing to CPI

On the 21 March 2017, the UK will have a new headline inflation measure of CPIH. It is similar to the Consumer Prices Index (CPI) but attempts to add a measure of owner occupiers’ housing costs, hence the H.

In a separate post I have discussed the issues with this measure. In this one I want to specifically focus on the impact of adding housing to CPI. ONS have calculated what the CPIH rate would have been back to 2006. It is therefore possible to see whether it would have increased or decreased CPI. An initial look at the data suggests that sometimes one is higher, and sometimes the other. However looking more closely there is a strong correlation between the impact and the absolute level of CPI.

When CPI is low (below 2%), CPIH is usually higher than CPI. However when it is above the target 2% level, it usually adds to it. A simple correlation predicts that when CPI is 3%, CPIH may deduct nearly 0.3% from it. At 4% CPI, it could deduct 0.5%.

OOH scatterplot

The reason for this is that the Owner Occupiers’ Housing measure based on rents is a fairly stable number which has averaged just under 2%. Therefore when CPI is above that level, it acts as a drag on it downwards (and vice versa when CPI is below 2%).

Currently OOH is about 2.5%, but data from the Countryside Letting Index implies it will fall over the coming year. That is at the same time as the Bank of England is predicting that CPI will rise to nearly 3%. The net effect is likely to be that by this time next year, the new headline CPIH measure may well be 0.3% lower than had we stuck with CPI as the headline.

Why CPIH will be a poorer measure of headline UK inflation

On the Tuesday 21st March, the UK will be switching the headline inflation measure from CPI to CPIH. This is the first change in the main inflation measure for 14 years. This article examines what CPIH is, how it differs from CPI, why we are changing and what the implications might be.

CPIH is similar to the Consumer Prices Index (CPI) but attempts to add a measure of owner occupiers’ housing costs. The “H” effectively stands for “Housing”.

You might have expected that all our inflation indices include housing costs, as they are a significant part of most people’s lives. Indeed they are included in the UK’s original inflation statistic called RPI (Retail Prices Index). RPI fully weights in house prices, mortgage costs, council tax, water charges, repairs and maintenance, DIY costs, home insurance and ground rents.

However all housing-related costs are specifically excluded from CPI. The reason goes back to the EU. They invented a measure called the Harmonised Index of Consumer Prices (HICP) in the early 1990s. It was developed with the sole purpose of allowing comparisons to be made to determine if countries met the requirements of the Maastricht Treaty to join the Euro. At that time, European statisticians could not agree how to measure house prices, so they just compromised by leaving the whole area out of the measure. They argued it not matter, as HICP was never intended as a cost-of-living index, but was just a standardised benchmark for international comparisons.

The problem came when some countries, including the UK, started adopting HICP as their main inflation measure. The Bank of England did so in 2003 and renamed it CPI. The government may have been happy to use CPI because it usually produced lower inflation numbers. In the UK, CPI is on average approximately 1% lower than our original RPI measure. Lower inflation would have indicated good control over the economy and therefore reflected well on the government. Furthermore, as inflation is a key variable used to calculate GDP, lower inflation rates thus resulted in apparently higher economic activity, which again appearing to enhance any government’s prowess.

Nearly a decade after the introduction of CPI as the UK’s main measure and there was a general agreement that something needed to be done about the missing housing elements. Both ONS in the UK and Eurostat started to examine the options.

In late 2012 while the EU was reviewing the best solution, ONS unilaterally decided to act. It created a version of a measure called “Owner Occupiers’ Housing” costs, or OOH, and then added this to CPI to create a new index called CPIH. ONS decided not to follow the method established in RPI which includes all housing costs (see above list). In particular, they rejected the idea of including house prices or mortgage costs on theoretical grounds. They argued that house prices should be excluded as they were assets, and that mortgage costs were not really consumption costs. Both might be reasonable arguments from an academic point of view, but the alternative method they selected has a number of problems.

OOH calculates housing inflation using something called “Rental Equivalence” or RE for short. To calculate this, ONS have tried to estimate the average rental price of a UK home. They argued this might be a better way to calculate housing inflation; allowed relatively easy data collection; and that it is a reasonable proxy for the sum of all housing costs experienced by home owners. However there are reasonable grounds to doubt most of these assumptions – see discussion later.

As a further twist in the saga, ONS amended their definition of CPIH on the eve of it becoming the headline rate. On 13th March 2017, they decided to include Council Tax as well as OOH into CPI. Although useful to add in a genuine housing cost, the logic for not adding in all other housing expenditure such as ground rent, stamp duty, maintenance, etc was not explained. CPIH has therefore become an index based on “Rental Equivalence” but with one elements of what is called the “Payments Approach”.

The EU is now adopting a completely different method for calculating housing costs. This method is about to be proposed as a standard for all member states to collate. The EU thought it important that real house prices were included in some way and that all costs related to buying and maintaining a home should be included in their housing index. They have therefore adopted a method called “Net Acquisitions”, or NA for short.

There are reasons to think that even the EU approach leaves a certain amount to be desired. It is called “Net” because its main focus is on changes in the house prices of additional new homes. It ignores the price of the bulk of the housing sales i.e. of existing dwellings – hence the net bit. Ideally, they would have preferred to monitor the building costs of a house, having removed the land or asset price, but pragmatically decided this was too difficult to do. Despite these weaknesses, NA as a measure is significantly closer to the public’s expectations of a housing inflation statistic, as it at least includes a number of housing costs.

It is not just the EU who have taken issue with ONS’s method. The UK Statistics Authority (UKSA), have challenged it on a number of grounds. They were so concerned about ONS’s introduction of CPIH that they removed its designation as a national statistic back in 2014 and on 10th March 2017 re-affirmed that they are still remain unhappy with the approach ONS has taken.

UKSA has three main issues with CPIH:

  1. The most significant is that the absolute values and trends derived from the ONS numbers appear not to reflect other published measures of housing costs. For example, if you compare OOH with the housing price index produced as part of RPI (see graph), you can see that two are very different. Not only does OOH usually underestimate true housing costs, the pattern of changes seen is different and additionally appears to suffer a significant lag.
    OOH vs RPI HousingRPI’s measure of rental costs has risen almost half as much again as OOH since 2005. Indeed when you look at all the individual elements of housing costs, all but mortgage interest costs have risen substantially more than ONS’s OOH measure. In addition, the OOH measure seems relatively impervious to the gyrations of real housing related costs.OOHPart of the reasons for rents being a poor measure of all housing costs become obvious with a bit more thought. Rents charged are impacted by many other factors than just input costs e.g. by supply and demand. Most rents exclude housing costs such as water and council tax and so can never represent the impact of changes in these factors. Many rental agreements get set once a year and as such are always going to be a lagging measure of the impact of price rises that have impacted in the intervening period. There are also other issues with the sample and weighting of those rental properties which ONS is using.
  2. The UKSA wanted ONS to improve the quality standards in to the data collection. The rental figures they use come from the Valuation Office Agency (VOA) in England, Rent Offices Wales, Rent Services Scotland and from ONS’s own data collectors in Northern Ireland. This is a problem as ONS does not own most of this data and has only secured access to it on aggregate levels. This means it cannot work out weights of property types by region for example. Furthermore the total sampling frame of all rental properties in the UK is unknown and agencies such as the VOA do not have a random sample of all of them. The data cannot therefore be totally representative of all UK rents.
    To make it worse, should there be a change in government housing benefit policy, the relevant data may no longer be collected. Similarly, should Jeremy Corbyn be elected, he has promised to bring in rent controls. At that point, the variation in the data will merely reflect government policy and it will become irrelevant for measuring housing inflation.
  3. Finally the UKSA has is a broader issue, namely the degree of distrust users have in ONS’s concept of CPIH. To quote their latest report “the degree of user scepticism and disagreement is unusual for a national statistic”. They claim that the ONS has not engaged with users to determine the way they would like to see a housing index calculated. It is unclear if the UKSA mean Eurostat particularly or the large volume of hostile voices from consultations on the subject, or both.

In spite of the UKSA’s concerns, ONS chose to press ahead with adopting CPIH. They had stated, even as late as 28 October 2016, that they would only make CPIH the main headline measure once it had been designated a national statistic again by the UKSA. It was therefore a surprise two weeks later, before receiving an update from the UKSA, that they took the decision to do so. The launch date was announced on 10th November 2016 – the day after Trump’s election. It therefore received little media coverage at the time and few are even aware of the change and what it might mean.

The change to CPIH as the headline measure is therefore going happen on 21st March 2017. So what are the main implications of the change to CPIH. There are five main ones:

  1. It risks under-reporting the true level of inflation in the UK in the future. If CPIH had been the headline inflation rate for the last 10 years, it would have averaged about 0.15% lower. The under-reporting problem appears worse when inflation is high i.e above 2.5% – a period we just about to enter again.
    Note though that during the recent period of very low inflation rates, CPIH has sometimes been slightly higher, as it was in January 2017 (1.9% vs 1.8%).
  2. It risks reducing standards of living in the UK. Benefits, tax allowances and many pensions are all linked to the headline inflation rate, as are many wage rises. Should inflation be lowered by adopting CPIH, this will gradually erode the standards of living of many people over time – particularly of the poor and elderly.
  3. It could affect monetary policy if/when the Bank of England’s target is changed to CPIH. Inflation may be made to appear lower than it really is, delaying action on increasing interest rates. Also were the BoE to assume OOH reflects UK housing cost inflation, they could under-estimate housing’s impact on the economy.
  4. It risks invalidating international inflation comparisons. Despite Brexit, it makes little sense for the UK to go it alone with a different method of calculating its headline inflation rate versus the EU.
  5. Finally it risks severely undermining trust in UK statistics. If the public sees a housing index being reported that bears little resemblance to their experiences and other measures, this will further erode faith in government. It might also result in considerable anger, as it might appear to some to be an underhand way to reduce benefits and pensions.

Concluding thoughts

Back in 2003 when RPI was dropped in favour of CPI, the impact of changing housing costs was lost from the UK’s main inflation measure. There is thankfully now a consensus that housing needs to be included again. However where there is no agreement is in how it should be measured. As is often the case with statistics, there are conflicting views on how the numbers should be compiled because of the difference in the way different stakeholders perceive them and want to use them.

But ONS must remember that the most important stakeholder is the public. They seek an accurate measure of how much the cost-of-living is increasing. They use these numbers to ensure wages, pensions and other contracts are correctly adjusted.
Given this audience, there are three tests that an important statistic such as inflation must pass:

  • It must be worked out using a simple method that is easy for the public to understand
  • It must have validity i.e. it measures what is claims to measure
  • It has to reliably and accurately portray the price rises actually experienced by the total population.

OOH arguably fails all of these tests and as a result CPIH is going to be a poor measure of headline inflation.

Bank of England Inflation Report – November 2016


Listen to Pete talk about the report on Share Radio:

Latest inflation predictions

The latest BoE Inflation Report today showed that the Bank now expects inflation to be higher in the coming years and particularly in 2017. They are now predicting CPI will rise to 2.7% this time next year as opposed to 2% which they said in August. They also increased their inflation predictions for 2018 to 2.7% (2.4%) and do not think it may return to the 2% target until 2020.

Yesterday we had the National Institute for Economic & Social Research (NISER) also raise its forecasts with headlines of 4% predicted inflation next year.

The factors affecting future inflation

There is no doubt that the continued decline in sterling is going to have a significant impact on UK prices. According to the BoE today, the sterling weighted index was down 20% in a year (and it is even more so versus its peak last August). Furthermore, some traders think the current level is a flag on a continued downtrend towards $1.1 and €1 = £1 i.e. a further decline of over 10% at some point during the Brexit negotiations.

It is worth noting though that declines in the pound do not translate simply into rises in the CPI inflation rate. Firstly only a proportion of the prices in the UK are directly influenced by exchange rates – primarily those that are imports or commodity related. These represent about a third of the index. However even here, many importers and companies either choose not to pass all the increases through or may delay doing so (especially if they have forward contracts fixed at pre-Brexit levels).

Pass through depends a lot on the state of the economy and the sectors affected. Usually for motor fuels and gas/electricity prices the pass through is quite quick and in full. In recent days we’ve seen some companies that feel they have bargaining power also pass through the full impact eg Apple and Microsoft. It remains to be seen if the same happens in supermarkets for foods and consumer goods – the Marmite wars indicate pressure is increasing though.

According to Mark Carney today, consumer spending has been so far more resilient than expected post Brexit. That would imply a higher and faster pass through may be attempted on imported goods.

So returning to inflation predictions, next year’s outturn will depend not just on what happens to exchange rates in 2017 but also global commodity prices. Historically, the latter is probably the most influential factor on short term UK inflation rates. Therefore if the oil price was to manage to rise towards $60/barrel this might put even more pressure on CPI than economists are currently factoring in.

Of the two inflation predictions out in the last 24 hours, I expect we’ll see inflation closer to NISER’s 3.8% in 2017 than the Bank’s 2.7%. The Bank has a long track record of predicting inflation two years out will be either exactly at their 2% target or near it. For the vast majority of the decade after 2004, they consistently under-estimated inflation, sometimes by a very wide margin.

Money supply effects on inflation

Another reason to believe inflation is going to run higher in the coming years has not been talked about much in the media. UK inflation over the very long run (i.e. 100 years) has had a strong relationship with the money supply – see here. On Monday we saw figures published showing that the broad UK money supply (M4x) had soared 10% from July-Sept (and 7.7% over the year). Although the pass through of money supply changes is complex, not least because they often go into asset prices first, they will add to continued pressure on UK prices in the years to come.

Implications of the increase in inflation

It is worth considering who is going to win and lose with this burst of inflation.

The key winners are:

  • Government: Tax revenues will rise in line with real inflation (RPI) yet the low interest rates and QE from BoE has ensured that cost of servicing government debt is significantly lower. Furthermore inflation will erode the real value of government borrowings and decrease important debt-to-GDP levels.
  • Other debtors: Major debtors such as corporates and mortgage holders will see the value of their debts decline in real terms. This may well put further pressure on house prices with resulting misery for first-time buyers.

However, for every pound gained by a winner, there is corresponding pound lost by someone eventually. Some of the key losers are:

  • Cash holders: The BoE has ensured interest rates are near zero. If CPI runs at 3-4% CPI for four years (i.e. closer to 4-5% real RPI inflation), purchasing power could be eroded by up to a fifth. That is a staggering level of stealth tax on British companies and savers.
  • Workers: Wages are currently growing at 2.3% a year. That equates to about the current level of RPI inflation (2%). If we see CPI at 4% (RPI = 5%) in 2017, many workers will see a significant decline in their standards of living as wage increases will not keep up. Again compound effects over the coming period of high inflation, could start to have a marked impact on consumer spending – a fact the BoE was highlighting today.
  • Pensions: The impact here is via real bond yields which are the bedrock of most pension savings. The rise in inflation will erode real value of defined contribution pension pots for millions of workers. It may also impact on real annuity rates and more importantly the real value of pensions people receive if they have not inflation protected their annuity.

Share Radio interview: Inflation Report

On 5th November, Simon Rose interviewed Pete Comley to talk about the latest Bank of England Inflation Report. Some of the issues covered included:

  • What was new in the latest Inflation Report?
  • Why do we have an inflation target?
  • Should we scrap the inflation target?
  • Chancellor said zero inflation was “welcome news” today in letter to MPC – the issues with this.
  • Why has inflation been declining since the 1980s?
  • The problems with CPI.

You can hear the full interview here:

IMG_20151105_190617 IMG_20151105_193125



ONS predict UK population to grow the most in the EU

The latest population report from the ONS created a lot of headlines related to immigration and how it was going to be responsible for the countries’ size increasing 0.7% a year in the coming decade. But how credible are these forecasts and what are the key assumptions behind them?

First a few of the facts that ONS published:

  • The UK population is projected to increase by 9.7 million over the next 25 years from an estimated 64.6 million in mid-2014 to 74.3 million in mid-2039.
  • Population will continue to rise over the whole of the next century and will reach 95 million by 2114.
  • Assumed net migration accounts for 51% of the projected increase over the next 25 years, with natural increase (more births than deaths) accounting for the remaining 49% of growth.
  • The population is projected to continue ageing, with the average (median) age rising from 40.0 years in 2014 to 40.9 years in mid-2024 and 42.9 by mid-2039.
  • The UK is projected to have the largest population in the EU by 2047 and to have grown the most in absolute size, i.e.

 Population projections comparison for countries in the European Union (Eurostat)

2014 2039 Increase % increase
European Union
(28 countries)
507.2 523.1 15.9 3%
United Kingdom* 64.2 73.5 9.3 14%
France 65.9 72.6 6.7 10%
Italy 60.7 66.0 5.3 9%
Belgium 11.2 13.8 2.6 23%
Sweden 9.6 11.7 2.0 21%
Austria 8.5 9.6 1.1 13%
Netherlands 16.8 17.7 0.8 5%
Denmark 5.6 6.3 0.6 11%
Finland 5.5 6.0 0.6 11%
Czech Republic 10.5 10.9 0.4 4%
Luxembourg 0.5 0.9 0.4 67%
Cyprus 0.9 1.0 0.1 10%
Ireland 4.6 4.7 0.1 1%
Malta 0.4 0.5 0.0 9%
Slovenia 2.1 2.1 0.0 1%
Estonia 1.3 1.2 -0.2 -11%
Croatia 4.3 4.0 -0.3 -7%
Slovakia 5.4 5.1 -0.3 -5%
Hungary 9.9 9.5 -0.3 -3%
Latvia 2.0 1.5 -0.5 -24%
Lithuania 2.9 2.0 -0.9 -32%
Portugal 10.4 9.4 -1.0 -10%
Bulgaria 7.2 6.1 -1.1 -16%
Greece 11.0 9.6 -1.4 -13%
Romania 20.0 18.5 -1.5 -7%
Spain 46.5 44.5 -1.9 -4%
Poland 38.5 36.4 -2.1 -6%
Germany 80.7 78.1 -2.6 -3%

* Note: The latest ONS projections show even higher UK population growth i.e. 74.6m by 2039/40 not 73.5m.

Key assumptions

There is no doubt that the UK population will continue to rise in the short-term. However the extent of that rise and how long it will continue for are more debatable. It is therefore important to look at some of the key assumptions made by ONS and how valid they might be.

Net Immigration. The ONS is assuming that the current level of migration inflows of 653,000 people in 2014/15 will decline only slightly by 2020/21 to 518,000 and then persist for the rest of the next 100 years. It strikes me as highly unlikely given our acute housing shortage and the rise of the ‘politics of identity’ that such levels of immigration will be allowed to continue and certainly not in the long-run i.e. for more than another decade. The EU comparison table above projects that the UK will be the net recipient of the largest number of EU migrants, which seems politically unlikely. Indeed it is not impossible that the UK might even vote in 2017 to come out of the EU and that would bring these projected inflows to an abrupt halt.

These assumptions are absolutely key to ONS’s projections. If you were to assume net migration grown to halt tomorrow, ONS predict that the population would only rise 3.1m by 2039 and not 9.7m. Moreover even those growth estimates may be too optimistic.

Fertility rates. The other key assumption made by ONS is that UK fertility rates (i.e. the number of children each woman has) will rise from the current level of 1.81 to 1.89 by 2032/33 and then remain at that level. Across the world from South America to SE Asia, fertility rates have been collapsing over the last few decades. They are 1.5 in Russia, 1.4 in Germany, 1.3 in Korea and 1.8 in Brazil for example. They are even declining in India – now just 2.5. ONS are assuming that immigrants to the UK will have significantly higher fertility and that is what is going to increase our rate. Although there is some evidence for this, it is also possible that newer migrants may not be as fertile as some might have expected. Most come here to work and so fertility will decline, just as it is doing in their home countries.

What does this mean?

Like I pointed out recently, ONS data quality can be an issue. In this case, their projections are probably significantly exaggerating the growth of the UK population. That being the case, the UK working population is going to eventually decline during the 21st Century.  Overall spend of UK consumers will also decline as the proportion of older people grows more than is projected in this analysis. That decreased demand will create pressure to keep inflation low. See full argument here.


Replacing the inflation target: Time for a new era of Naked Central Bankers?

This Thursday sees the publication of the Bank of England’s 92nd Inflation Report since it first started targeting inflation back in late 1992. This article (based on one published in the ERC B&O Journal Vol 45, No 3) examines how monitoring inflation targets about and questions whether they serve any purpose now in a low inflation world. It shows that targets have a limited influence over inflation in the short-term. They are also associated with a misleading sense of financial security. It is suggested that we learn from experiments with traffic management that show removal of formal controls results in better outcomes. The MPC should therefore adopt the “Naked Streets” concept and be freed of all formal targets like inflation and instead just focus on the higher level objective of ensuring financial and economic stability.

Purpose of central banks

Central banks have many functions but a key one is to maintaining monetary and financial stability. They have always played such a role right back to the foundation of the very first one; the Amsterdam Wisselbank in 1609. For the Bank of England it became more formalised in the late 19th Century following their help in bailing out Barings Bank.

Emergence of targets in the 1970s

Since the Bank was nationalised in 1946 governments have had a greater impact on its work and role.  In the 1970s, concerns were expressed over the effectiveness of the Bank and there were calls for greater transparency in its workings. These coincided with a period of high inflation and economic problems. Therefore in 1976 Dennis Healey, under pressure from the IMF, imposed the first target for the Bank of England; one of monetary growth (M3).

Its objective was to bring back financial stability by curtailing the money supply and with it inflation, which at that time, had just peaked at 27%. It is debatable how effective such monetary targets were and it was clear from the diaries of Healey that he himself never expected them to work. Part of the problem might have been the Bank’s decision to monitor M3 – other central banks were targeting M1. Nevertheless the target was continued for a decade.

In 1987, the government switched to a target of shadowing the Deutschmark and stopped monitoring monetary goals. That regime persisted until such an aim became unsustainable as Britain was ejected from the ERM on 16 September 1992.

History of inflation targets

In the same era, on the other side of the world in New Zealand, what might have even been almost a joke started a different trend for central banks. The country had sustained inflation rates in excess of 15% for most of the 1980s. On live television on April Fool’s Day 1988, without consulting anyone, finance minister Roger Douglas said his central bank was going to completely stabilise prices and set a target of near-zero inflation.

In the aftermath of this TV program, his government, the Reserve Bank of New Zealand and other interested parties, actually agreed to set a target of under 2% inflation to be achieved by the early 1990s. Amazingly, the Reserve Bank delivered and inflation was reduced to below 1% by 1992 and remained on target for three more years.

This experience was inspirational to other governments and set a trend that was quickly followed by many. In 1991, Canada, Chile and Israel all adopted inflation targets and the UK did in October 1992. Currently they are operated by nearly seventy countries worldwide.

Effectiveness of inflation targets

There is fair amount of evidence that target set for inflation does indeed have an impact the broad long-term inflation rate experienced in a country. For example, Switzerland which sets a target of 0-2% achieved an average of 0.7% between 2000-2013. In contrast, we averaged 2.3% during this period in the UK with our target of 2%, and Russia averaged 10% with its target of 5%. However care must be taken with causality here and governments possibly set targets which they feel they can achieve.

Despite this caveat, inflation rates across the whole world tumbled during the 1990s, as targets were implemented in more and more countries. Central bankers took all the credit for this and their power corresponding increased – especially the Fed. However, it might be argued that a lot of this disinflation was due to the impact of China’s devaluation in the 1994 and the resulting flood of cheap goods around the world. The growing impact of technology and the Internet also reduced prices and the Great Moderation was also starting to have an influence. So it is probably fair to say that central bankers had a strong tailwind behind their efforts at achieving inflation targets.

Indeed their track record since, particularly that of the Bank of England, has been far from impressive. For large parts of the time from 2008 until 2013, inflation was significantly above the 2% target and in 2015 it has varied in the opposite direction and been close to zero. This is in spite of BoE projections throughout this period stating that it would always return to the 2% target within two years.

It looks like influencing inflation with targets in the short-term is far more difficult than central bankers might have us believe. This brings into question the whole purpose of maintaining such targets in a generally low inflation world. The reason that small differences in the inflation rate are difficult to influence is that much of headline rate observed in the UK is actually outside of the Bank’s control. Merryn King used to frequently bemoan that half of the overshooting of inflation during his latter years was due government controlled price rises e.g. train fares, tuition fees and utility prices. Also 2015 has again highlighted the significant impact that changing world commodity prices and currency exchange rates have on CPI – again largely outside the Bank’s influence.

To make matters worse, the whole point of trying to control the inflation rate was to bring economic and financial stability. The financial crisis of 2007-8 clearly highlighted that a dogged focus on controlling the inflation rate was of no help in averting this crisis. Indeed I would argue that it was a hindrance and it prevented the Bank from acting earlier to defuse the credit time-bomb that was being created during the early noughties. Had the Bank just a broad remit of economic and financial stability (and the FSA under its control), it would have acted much earlier with regard to the imbalances and problems with financial institutions. Instead, right up to the end of 2007, it was reporting that inflation was spot on target and that it did not need to intervene.

Naked Streets Experiments

There is a very interesting analogy here in town planning. Following the work of the late Dutch traffic engineer Hans Monderman, many towns have experimented with the concept of shared space roads or “Naked Streets”, as they have come to be known. Monderman showed that accidents could be significantly reduced by removing all the road signs and markers, traffic control measures like lights and removing the clear division between cars and other road users’ space.

The psychology behind it was that drivers would be forced to pay more attention to their environment. They would have to switch off their auto-pilots and really look at what was going on around them. Instead of racing through traffic light junctions because the light said green, they would have to slow and actually check what was really going on.

Since then many towns have replicated his findings, though it has been found that they tend to apply most when traffic volumes and speeds are low. Therefore a good example of a successful one is the new Exhibition Road in London.

Naked Central Bankers?

It strikes me that the current inflation target is very much like a traffic light. The Bank monitors it and a very stop/go like fashion. However for most of the recent time the light has been on amber i.e. inflation outside the 2% +/-1% range. Like most drivers who see such a light, they just keep their foot down, focus on the road ahead and speed on through the junction and hope no-one hits them.

We need to remove the inflation target traffic light and stop it being a simple crutch which the Bank uses to determine the success, or not, of the economy. Inflation is luckily not a major issue now and has not been in the UK for over a couple of decades. In my latest book, Inflation Matters, I suggest that because of world demographic changes, that we are probably headed towards a long era of very low inflation.

Inflation targets did have a place in the world economy a few decades ago, as countries needed help to deal with the aftermath of the 1970s. That period has passed and it probably makes sense to move on. Instead we should instruct the MPC to return to their original remit. The MPC needs to open their eyes and become fully aware of everything going on the economy and finance. If they spot a problem, they must be prepared to take action(s), even if there is no evidence of it triggering inflation. Welcome to the new world of Naked Central Bankers.

Schoolboy error by ONS in web scraping trial

On September 1st, ONS published the results of a trial to scape food prices daily from the internet (see discussion here). These resulted in headlines such as: “costs rocketing”; “cost of basic items has risen by 8% in last year”; and “spaghetti up 20% in a year”. Yesterday, ONS put out a correction saying that they had got the scale inverted and that food prices had actually fallen by 3% a year – a figure now almost the same as the -2% decline in prices seen in CPI.

Pete Comley, of the website Inflation Matters, said: “I pointed out this error to the ONS in early June after they published some initial findings on the web data trial. It looked so obviously wrong to me. Despite this, ONS went on to publish more data with the error in September and have only now admitted to the mistake.”

He continued: “This miscalculation comes after other major revisions to the results between June and September. ONS’s initial analysis created headlines of inflation being potentially over-estimated versus CPI. In September they created the opposite headlines. Such mistakes undermine the public confidence with other pricing statistics such as CPI and RPI. There should be better quality control procedures in place at the ONS. They should at least apply the old sense check of: if it is interesting, it is probably wrong.”

Comley was concerned what the long-term impact of this error might be: “The collection of prices online could have a lot of benefits not just to the ONS but to the public by providing more timely and detailed price indices. It would be a shame if this opportunity for development of our price indices is wasted due to basic errors in a trial.”

The trial referred to involved ONS collecting prices of 35 foods and drinks automatically from Tesco, Sainsburys and Waitrose websites between June 2014 and June 2015. Some 6500 prices were checked every day and a daily chained-linked price index created and compared with changes in average unit prices paid (of the items that could be checked each week). The error occurred in the chain-linking causing price declines to be stated as rises, and vice versa. In addition, ONS also appear to have made some other revisions to their chain-linked data.

A summary of some of the key data published in September (and reported widely) and the latest revisions are:

Original chain-linked inflation est. Revised chain-linked inflation est. Unit price change inflation est.
All food and non-alcoholic drinks +8% -3% -3%
Dry spaghetti/pasta +19% -10% +1%
Red wine bottle +14% -11% -4%
Cola drink 2 litre +14% -2% 0%
Cheddar per kg +14% -10% -6%
Fresh orange juice +9% -4% -1%
Plain biscuits 2-300g +8% -6% -1%
New potatoes per kg +7% -6% -2%
Semi-skimmed milk +2% -2% -2%
Tea bags -4% +4% +3%


Link to the full ONS corrected report:

Link to a copy of the original ONS report can be found at:

Is inflation really 8%? ONS web scraping trial problems

*** UPDATE: on 26 October 2015, ONS admitted to a significant error in this report. The revised report can be found here. ***

On 1 September 2015, ONS published a report on “Research indices using web scraped data” (original ONS report here and data here). It was an update on their initial analysis published in June on a trial using web scraped data to compile price indices.

The objectives of this trial appear to be to determine if and how the ONS might be able to use web scraping as an alternative method of data collection. Their hope, like many trying to use big data, is to reduce costs (by cutting out manual data collectors) and to improve quality (by increasing the number of prices checked and their frequency).

The results show that the ONS seems a very long way from perfecting the use of big data. Their report also somewhat confusingly initially focuses on differences between calculating prices indices using some statistical wizardry called “chain linking” and by comparing the average unit prices of products. These have led to a number of media headlines of inflation being severely underestimated e.g. Cost of everyday items for sale in supermarkets rockets 8 per cent in last year (Daily Mirror).

These reports are completely misleading, as I’ll discuss below. ONS should bear some responsibility for the confusion. Their report should probably have majored on what you have to wait until section 4.3 to read i.e. a “comparison with CPI”. The latter shows that when you scrape prices in a limited way and try and replicate what you are doing off-line as best you can, you can end up with fairly similar results. (Though strangely the results published using this method in September differ markedly to what they published in June – with there being no reference in the report to how they have done things differently to make the fit better).

ONS june rep

June report showing a big difference between CPI and its web scraped version

ONS sept rep

 September report showing the two be very similar

But let us return to hype being reported in the media. It is related to the results of the ONS’s attempts to produce price indices on a more frequent basis (e.g. daily or weekly) from the web scraped data. The problem with scraping as ONS have done it (i.e. letting a computer collect the price of everything with the label “whisky” on it from Tesco, Sainsburys and Waitrose’s websites), is that you can end up with a load data that is difficult to analyse and sometimes misleading (e.g. Tesco include rum in with whisky!) There are also problems that supermarkets frequently change their labels for products, products go out of stock or get delisted, and sometimes no data is collected due to computer problems. All of this means that being able to compare prices consistently over a period of time is almost impossible without some sort of major compromise.

The compromises you make then impact the results to get back. The two main ways ONS looked at it were just find the subset of products that they could track at least every month at some point and compare those – called unit price index. This subset is less than a quarter of the original one and sometimes is down to just one line per supermarket (in the case of bananas) – hardly big data then!

The alternative* is to look at each pair of days over the year and match every day and link the results to the previous day with some clever stats – called chain linking. The latter is a ingenious idea, but in practise means that price indices so created drift ever higher – possibly because supermarkets bring in new lines on promotion at a discount price and then when they return to full price, many then get delisted as people stop buying them.

ONS sept rep chain2

Neither of the above is a viable solution for creating a price index and so the ONS goal of maybe producing daily price indices seems a long way off.

Instead what ONS need to take out of this trial is basic issue with big data. To make it work accurately requires a lot of data cleaning/manipulation (called data wrangling). Without that you risk the “rubbish in, rubbish out” scenario that has dogged so many such projects.

Indeed ONS might find that it is actually cheaper and better quality just to send out the interviewers to collect the prices as they do now. Having said that, the ideal solution is probably a mixture of the two i.e. set up an online interface so an intelligent human data collector can go online and collect the prices from supermarkets that have online shops. They can then deal with the product renamings and when required propose substitutions, if products genuinely become delisted. You then end up with similar data to now but collected a bit quicker hopefully. (Note the word “bit”. I suspect the challenging usability of many online shops means that it might actually be quicker to visit the vegetable isle of Tesco and deal with the substitutions in person than try and do it online).

Finally to re-iterate, inflation for food is not running at 8%. The latest CPI figures today estimate it to be -2.4% (RPI -2.0%).

* Note, ONS also report a variant of chain-linking called GEKS which appears to suffer less from higher prices but its results appear inconsistent and so appears no solution either.

Will ONS scraping and wrangling lower CPI inflation in the future?

On 8th June 2015, ONS published the results of a trial in which they measured inflation online for just under a year by directly sourcing price data from three UK supermarkets websites. The purpose of the trial was:

  1. To develop and test methods of automatically collecting price information (“web scraping”)
  2. To determine its quality by looking at how much cleaning and manipulation it needed (“data wrangling”)
  3. To see how the results compared with the traditional CPI method.

In a nutshell, the results showed that most of the price indices so created showed lower rates of inflation, i.e.

Scraping trial

Source: ONS. Food, beverages and tobacco index. June 2014=100.

The trial covered 35 items in the food and alcoholic drinks sector. Some 2886 products were included. Prices were collected daily and about 1.5 millions price quotes in total were scraped. However ONS discovered that the wrangling process distilling these down to ones which could be correctly compared over time was very high, and so much so that about a half of the data had to be excluded.

Therefore any cost savings from not sending out real price collectors might be significantly consumed by paying for back office data wranglers sorting out the mess of the scrapings. For example, they said that rum often appeared in shops listings for whisky, as did apple juice in their listings for apples. Add to that supermarkets often changed their identifier codes for lines making linking up data automatically fraught with error.

The concept of web scraping prices is not new. Arguably ONS is doing a bit of catch up with this trial. In the US, MIT set up the Billion Prices Project (BPP) which has been successfully scraping prices there for seven years now and is also doing so in some South American countries. In the EU, the Dutch seem ahead of the game and are now using not only direct scraping but collector assisted web scraping interfaces to collect prices from places such as cinemas and for driving lessons – see recent research paper here.

The BPP data has not shown lower prices with scraping. Indeed their indices in Argentina are higher than official stats and in the US, they also tend to be slightly higher. It is therefore probable that the lower prices seen in the UK trial are related to some other factor.

The most likely reason is that of sample differences. The trial covered just three supermarkets – which more than likely included Britain’s top retailers such as Tesco and Sainsburys.  It is likely that price reductions here were markedly higher than seen elsewhere in the retail network. (I have asked ONS to quickly check this hypothesis out, but apparently they don’t have resource to do so!).

Add to that, it is wrong to assume that the prices online for UK retailers are necessarily the same as that in-store. They can vary by category of store (small metros being higher) and by region to some extent. The web price is most likely to exhibit the largest price reductions of all of them because of the competition faced online.

That said there are many other differences between the two methods that might also have an impact. For example, in CPI price collectors visit maybe a 100 stores but they only collect the price of one item of each line – normally the most frequently bought. In the scraping trial, the prices of all lines are included – which could be dozens of items or even a 100+. However logically to me this might argue for the normal CPI to be lower (not higher), as the main price reductions over the last year have probably focused on just a few key popular lines.

Having said that, ONS have only included in their main monthly analysis products that they scraped consistently over the trial period. In correspondence to me over it, they noted that they had to exclude a lot of products which were delisted periodically. A number of these then were relisted at higher prices. Indeed it is possible that retailers normally follow a policy of delisting temporarily before they raise prices. If true, this may be another significant factor in why the scraping trial appeared to show deflation (as items with price rises were systematically excluded).

Note, the method for dealing with out-of-stock is quite different in CPI. When a product cannot be found by a price collector, they carry on using the old price for up to three months. After that period, they start again by tracking a different product.

Day of collection is another possible factor. CPI is normally collected on the second or third Tuesday of the month. The web scraped data was being collected every day. But again there is no logical reason for this to affect the numbers. (Though apparently when ONS tried to compile a daily price index, they ended up with much higher inflation rates due to chain-linking prices so often).

All the above said, moving towards more collection of data online seems a sensible direction of travel for ONS in the 21st Century. It has the potential to harness a much broader sets of prices that might more accurately represent the multiplicity of choice the consumer faces today. Also collecting across the month has the potential to remove some of the weird quirks you sometimes see in the monthly data due to the timing of Easter, for example.

However rather than scraping massive amounts of data automatically the answer is probably to create systems that a human price collector can use online so that the process is more efficient and representative, but the quality of the data is maintained – rather than trying to wrangle it into some shape later. Also any move to reduce the number of sampling points for each item should be resisted.

So returning to the initial question of whether scraping will result in lower inflation rates in the future, I sincerely hope not. However one is reminded of the fact that even slight differences in the way you calculate inflation can have a profound effect on the numbers that come out the other end.

As Paul Johnson, director of the Institute for Fiscal Studies, noted in the FT, “the real finding of the initial research was not that inflation is too high, but the method of collecting prices matters rather a lot”.

UK consumer perceptions of inflation vs the official CPI stats

Last week, the Bank of England published their latest “Inflation Attitudes” survey results for May 2015. It showed that the median Brit thought that prices had risen 2.2% over the last year. That is very different to the latest official CPI figure of -0.1% published in April 2015. Just 6% of the UK population agrees with the official numbers and thought that prices had actually declined in the last year.

Consumer perception of inflation vs CPI

The above chart compares the historical data on median inflation perceptions since November 1999 with the ONS’s estimate of CPI. It shows that consumers almost always think that the real inflation rate is above CPI. On average over the last 15 years, they have guessed it to be 1% higher. However there is some evidence that the difference might be getting larger more recently. The average difference since 2012 has been 1.75%.

So why is there such a big disparity?

ONS have looked into this before and there are a number of articles published about it (for example here in 2010). They like to highlight that consumers’ perceptions are often inaccurate for a number of reasons, i.e.

  1. Individuals’ inflation rates may be different as they consume different amounts of products to the average CPI index. Indeed as the CPI weights are expenditure related, they reflect the spending of the rich far more than the average person –  see here.  If you look at the detailed tables on inflation perceptions, you’ll notice it is the C2DE’s that generally perceive higher rates of inflation, so there might be some slight effect here.
  2. ONS also highlight that people notice price increases more than price reductions (Brachinger, 2005). In addition, individuals are also more likely to remember price changes for items bought frequently (Antonides, Heijman and Schouten, 2006). Although there maybe something in the former, the latter has now been disproved. In the last year food prices have declined 3% and petrol is down 16%. Both of these are frequently bought items and probably key in framing inflation opinions. However consumer perceptions of inflation are still above 2%.
  3. Finally, ONS say that the media is key in forming inflation perceptions with over half of all respondents of the Bank of England Inflation Attitudes Survey (2010) saying it was either ‘very important’ or ‘important’ when forming their views. How is it then when the media is full of headlines of CPI deflation (and even RPI is 0.9%), that most people think inflation is still above 2%?

The most obvious answer is that 94% of the UK population are probably not wrong and inflation is actually higher than reported in CPI.

As is clear from the graph above, most people think inflation is at least 1% higher than the published stats. Interestingly, this is almost exactly the same error that I have calculated using a completely different method – see here.

To briefly recap the real reasons that CPI under-estimates true inflation. These are:

My guess at the underestimation of true inflation is around 1 to 1.5 per cent – see table below. The exact amount is variable and there is no simple consistent adjustment that can be made to determine it.

CPI underestimation

Source: ONS for housing and geometric means effects. Author best estimates for substitution effects.