Research into the data, measurement and methodologies of ESG rating agencies has resulted in the Aggregate Confusion Project, a research project led by academics at MIT to develop a set of ESG tools and methodologies that will become best practice.
A paper by the Society of Sloan Fellows Professor of Management, Roberto Rigobon, called Aggregate Confusion: The divergence of ESG ratings, that examined ESG ratings based on data from six rating agencies, characterised the problem by bringing to light the fact ESG data is noisy and unreliable. This in turn, means investors don’t trust the data which often leads to inaction. And perhaps more worryingly, it also means that corporates can game the system.
Director of the MIT Sloan Sustainability Initiative, Jason Jay, who leads the project with Rigobon, says the aim of the consortium is to make ESG integration more defensible and more workable for the industry. MIT is looking for asset managers and asset owners to join the research-led consortium, with MassPRIM signing as a founding member.
Executive director and chief investment officer of MassPRIM, Michael Trotsky, says having robust tools was an important evolution for the fund’s ESG journey.
“We’re in the infancy of ESG and the data we rely on is not robust and so it is hard to have robust investments. There is a proliferation of products and we have analysed a lot but we haven’t been particularly happy with their robustness,” Trotsky says. “We like the intent and want to do ESG investing. But we want to do it better than anyone. As it stands we don’t think ESG investing is having a big impact. We want to impactful and want to form robust portfolios that reflect our views.”
In order to develop tools for the industry the consortium first explored the state of play with regard to the quality of measurement. Rigobon’s paper found that correlation among prominent ratings agencies was 0.62 which compares to 0.92 for credit rating agencies.
“It depends on what you think the correct comparison is, if you think it is to credit rating agencies, then this looks really bad. If you think ESG ratings agencies are more like stock analysts which are supposed to have a variety of perspectives then it makes sense to have an array of different viewpoints. Our view is that variety would be ok if it was transparent,” Jay says. “The issue is ratings agencies diverge and people don’t trust the information and become inactive and don’t know what to do with it.”
Jay points to the exponential growth in assets under management that claim to have an ESG tilt, which has grown by 34 per cent since 2016.
“We are building a skyscraper as fast as we can, but it’s being built on a shaky foundation full of gaps and holes. That foundation is the quality of measurement – it’s foundational because it’s the basis of any ESG strategy and if we can’t trust the data then we have a problem,” he says.
Rigobon’s paper revealed the main sources of divergence were that different agencies have different indicators for the same underlying attribute; and that agencies have different scope in what they include in their ESG ratings.
But the problems don’t stop at divergence, for example the data is also very noisy Jay says.
“Multiple measurements of the same firm at the same time yield different results. So we have to do some things to increase the signal to noise ratio and that’s one of our key projects here – to get a better signal to measure returns and other outcomes,” he says.
Another problem is that ratings agencies use weighted linear averaging, and Jay says corporates know this and are gaming the system. He points to tobacco firms as an example.
“Firms who are always going to do poorly on certain indicators such as tobacco companies have learned if they are good at all the other factors then they can be top of the class on ESG. Or they’ve learnt that indicators are related to disclosure so they disclose everything to get a good rating,” he says. “Tobacco companies are the most vulnerable to be screened out in ESG, so they are the companies that have learned to find their way in however they can, which means making this weighted linear average work for them. The industry needs to get more complex and richer in the functional forms of the index construction.”
In addition the agencies vary on their weightings of various factors, with very few paying any attention to investor preferences. This is an area the MIT researchers believe needs attention.
“We think that no one is understanding the preferences of investors and using those preferences to assign the weights, we are pushing for ratings to become more customised.”
So the paper points out the measurement problem and the consortium aims to fix the problem, and has developed four work streams to focus on:
- Improving the signal to noise ratio
- Fixing the functional form of aggregation
- Assessing preferences and weights and customisation
- Understanding how capital flows effect return on investment to sustainability.
In addition to MassPRIM the consortium is looking for three or four investors to join the intellectual partnership with Jay describing the attributes of partners as people: who are deeply committed to ESG as their roadmap as an institution: are critical thinkers who are willing to explore the limitations as well as the possibilities of different strategies; have patience in the research process; and have humility.
“People who are up for collaboration and up for learning together and don’t think they’ve got it all figured out.”
