Monthly Archives: June 2015

Latest UK inflation data – May 2015

The latest UK inflation data published this morning shows annual CPI inflation at 0.1% and RPI at 1%. Both are increases over the last month. However it should be remembered that this result was expected as prices dropped slightly last May and those falls dropped out of the index today. In contrast this May, they rose 0.2% – partly an effect of rising petrol prices (now 121p – up from 115p in Feb), some food price rises (eg diary spreads) and some quirks due to the timing of Easter affecting the cost of flights.

But the latest figures provide some support to the view that we have seen a low in inflation and the direction is now higher – see graph below. Certainly as the year progresses the falls in food and fuel prices will gradually drop out of the calculation. Furthermore, underlying inflation is 1.1% and services inflation is up to 2.3% again. It is prices of things like restaurants/hotels and education that are still pushing service inflation upwards.

cpi trend

These figures are also consistent with the trends in Europe where inflation was also up a similar amount to 0.3% in May. Indeed Germany’s own inflation stats out this morning, show it rising at 0.7% year-on-year now.

You can read the full ONS report here.

Data wrangle

Will ONS scraping and wrangling lower CPI inflation in the future?

On 8th June 2015, ONS published the results of a trial in which they measured inflation online for just under a year by directly sourcing price data from three UK supermarkets websites. The purpose of the trial was:

  1. To develop and test methods of automatically collecting price information (“web scraping”)
  2. To determine its quality by looking at how much cleaning and manipulation it needed (“data wrangling”)
  3. To see how the results compared with the traditional CPI method.

In a nutshell, the results showed that most of the price indices so created showed lower rates of inflation, i.e.

Scraping trial

Source: ONS. Food, beverages and tobacco index. June 2014=100.

The trial covered 35 items in the food and alcoholic drinks sector. Some 2886 products were included. Prices were collected daily and about 1.5 millions price quotes in total were scraped. However ONS discovered that the wrangling process distilling these down to ones which could be correctly compared over time was very high, and so much so that about a half of the data had to be excluded.

Therefore any cost savings from not sending out real price collectors might be significantly consumed by paying for back office data wranglers sorting out the mess of the scrapings. For example, they said that rum often appeared in shops listings for whisky, as did apple juice in their listings for apples. Add to that supermarkets often changed their identifier codes for lines making linking up data automatically fraught with error.

The concept of web scraping prices is not new. Arguably ONS is doing a bit of catch up with this trial. In the US, MIT set up the Billion Prices Project (BPP) which has been successfully scraping prices there for seven years now and is also doing so in some South American countries. In the EU, the Dutch seem ahead of the game and are now using not only direct scraping but collector assisted web scraping interfaces to collect prices from places such as cinemas and for driving lessons – see recent research paper here.

The BPP data has not shown lower prices with scraping. Indeed their indices in Argentina are higher than official stats and in the US, they also tend to be slightly higher. It is therefore probable that the lower prices seen in the UK trial are related to some other factor.

The most likely reason is that of sample differences. The trial covered just three supermarkets – which more than likely included Britain’s top retailers such as Tesco and Sainsburys.  It is likely that price reductions here were markedly higher than seen elsewhere in the retail network. (I have asked ONS to quickly check this hypothesis out, but apparently they don’t have resource to do so!).

Add to that, it is wrong to assume that the prices online for UK retailers are necessarily the same as that in-store. They can vary by category of store (small metros being higher) and by region to some extent. The web price is most likely to exhibit the largest price reductions of all of them because of the competition faced online.

That said there are many other differences between the two methods that might also have an impact. For example, in CPI price collectors visit maybe a 100 stores but they only collect the price of one item of each line – normally the most frequently bought. In the scraping trial, the prices of all lines are included – which could be dozens of items or even a 100+. However logically to me this might argue for the normal CPI to be lower (not higher), as the main price reductions over the last year have probably focused on just a few key popular lines.

Having said that, ONS have only included in their main monthly analysis products that they scraped consistently over the trial period. In correspondence to me over it, they noted that they had to exclude a lot of products which were delisted periodically. A number of these then were relisted at higher prices. Indeed it is possible that retailers normally follow a policy of delisting temporarily before they raise prices. If true, this may be another significant factor in why the scraping trial appeared to show deflation (as items with price rises were systematically excluded).

Note, the method for dealing with out-of-stock is quite different in CPI. When a product cannot be found by a price collector, they carry on using the old price for up to three months. After that period, they start again by tracking a different product.

Day of collection is another possible factor. CPI is normally collected on the second or third Tuesday of the month. The web scraped data was being collected every day. But again there is no logical reason for this to affect the numbers. (Though apparently when ONS tried to compile a daily price index, they ended up with much higher inflation rates due to chain-linking prices so often).

All the above said, moving towards more collection of data online seems a sensible direction of travel for ONS in the 21st Century. It has the potential to harness a much broader sets of prices that might more accurately represent the multiplicity of choice the consumer faces today. Also collecting across the month has the potential to remove some of the weird quirks you sometimes see in the monthly data due to the timing of Easter, for example.

However rather than scraping massive amounts of data automatically the answer is probably to create systems that a human price collector can use online so that the process is more efficient and representative, but the quality of the data is maintained – rather than trying to wrangle it into some shape later. Also any move to reduce the number of sampling points for each item should be resisted.

So returning to the initial question of whether scraping will result in lower inflation rates in the future, I sincerely hope not. However one is reminded of the fact that even slight differences in the way you calculate inflation can have a profound effect on the numbers that come out the other end.

As Paul Johnson, director of the Institute for Fiscal Studies, noted in the FT, “the real finding of the initial research was not that inflation is too high, but the method of collecting prices matters rather a lot”.

Publications Banner

UK consumer perceptions of inflation vs the official CPI stats

Last week, the Bank of England published their latest “Inflation Attitudes” survey results for May 2015. It showed that the median Brit thought that prices had risen 2.2% over the last year. That is very different to the latest official CPI figure of -0.1% published in April 2015. Just 6% of the UK population agrees with the official numbers and thought that prices had actually declined in the last year.

Consumer perception of inflation vs CPI

The above chart compares the historical data on median inflation perceptions since November 1999 with the ONS’s estimate of CPI. It shows that consumers almost always think that the real inflation rate is above CPI. On average over the last 15 years, they have guessed it to be 1% higher. However there is some evidence that the difference might be getting larger more recently. The average difference since 2012 has been 1.75%.

So why is there such a big disparity?

ONS have looked into this before and there are a number of articles published about it (for example here in 2010). They like to highlight that consumers’ perceptions are often inaccurate for a number of reasons, i.e.

  1. Individuals’ inflation rates may be different as they consume different amounts of products to the average CPI index. Indeed as the CPI weights are expenditure related, they reflect the spending of the rich far more than the average person –  see here.  If you look at the detailed tables on inflation perceptions, you’ll notice it is the C2DE’s that generally perceive higher rates of inflation, so there might be some slight effect here.
  2. ONS also highlight that people notice price increases more than price reductions (Brachinger, 2005). In addition, individuals are also more likely to remember price changes for items bought frequently (Antonides, Heijman and Schouten, 2006). Although there maybe something in the former, the latter has now been disproved. In the last year food prices have declined 3% and petrol is down 16%. Both of these are frequently bought items and probably key in framing inflation opinions. However consumer perceptions of inflation are still above 2%.
  3. Finally, ONS say that the media is key in forming inflation perceptions with over half of all respondents of the Bank of England Inflation Attitudes Survey (2010) saying it was either ‘very important’ or ‘important’ when forming their views. How is it then when the media is full of headlines of CPI deflation (and even RPI is 0.9%), that most people think inflation is still above 2%?

The most obvious answer is that 94% of the UK population are probably not wrong and inflation is actually higher than reported in CPI.

As is clear from the graph above, most people think inflation is at least 1% higher than the published stats. Interestingly, this is almost exactly the same error that I have calculated using a completely different method – see here.

To briefly recap the real reasons that CPI under-estimates true inflation. These are:

My guess at the underestimation of true inflation is around 1 to 1.5 per cent – see table below. The exact amount is variable and there is no simple consistent adjustment that can be made to determine it.

CPI underestimation

Source: ONS for housing and geometric means effects. Author best estimates for substitution effects.