Reputation and corruption in public procurement
Interview with Francesco Decarolis, Associate Professor at Einaudi Institute for Economics and Finance in Italy. Francesco is currently a Research Fellow of the National Bureau of Economic Research. He was recently awarded ERC Starting Grant to investigate reputation and corruption in public procurement.
Hello, Francesco, welcome to the podcast.
Hello, thank you for this invitation.
Francesco, I usually like to start the interviews by getting the guests to speak a little bit about their background. Could you tell us a little bit more about yourself?
Yes. So I have been working in the last 12 years in the USA, so as you were saying, I completed my PhD in Economics at the University of Chicago, where I focused on the analysis of auction and procurement markets, which is a rapidly growing and very successful area of economics, with a broad range of applications, from auctions and procurement systems that government use to various private sector applications.. Examples include spectrum, allocation for telephone operators, the allocation of oil and gas exploration permits, and really almost every other area you can think of. My origin is Italian, and while studying in the US, and learning about this new area of economics, I realised that there was a big discrepancy between many of the economic models and the practice. The actual way in which certain procurement, important procurement markets, were arranged, and that it was particularly interesting to try to bridge what was in our economic textbooks with what was happening in the practice, because these textbooks were missing some key features of the real-world environment. But also these real-world environments, and the rules under which they developed, were sort of addressing in an imperfect way, in a potentially problematic way, some of the real problems that they were facing. And so this is how I began to build a career as a researcher in the area of auctions and procurement. My studies are based mainly on the empirical analysis of auction and procurement data, but also with some focus on the underlying theory.
What brought you back to Italy, then?
So, in essence, the ERC grant. I worked in several universities in the US. Most recently, I was at Boston University, in the Department of Economics, and while I was there, more precisely I was on sabbatical at Stanford University during the Fall 2015, I learned that I was the recipient of one of the ERC StartingGrants for economics. This grant is an absolutely fabulous opportunity to allow me to conduct research on areas of public procurement that I did not have the time or the resources to explore before, but that I always found very interesting and fascinating. Especially the use of past reputation in public procurement and the role of corruption in procurement. And the reason why I mentioned that I didn’t have the resources is also that for doing some of these studies, especially the one that I’ll be glad to tell you about in a few minutes, I collaborate closely with contracting authorities that implemented these rules, and working with the contracting authorities to try out different procurement formats, evaluating from an ex-ante perspective how to design the new auction formats, then doing the work of collecting the data, analysing, possibly having multiple meetings, both with the contracting authorities and with the suppliers, are very time consuming activities. And so thanks to the ERC grant, I have now all the financial means to perform these activities. Hence, I was very happy to relocate back to Italy. Right now, I am in the Einaudi Institute for Economics and Finance, which is an institute created by the Italian Central Bank to be an excellence in the study of Economics, and here in this institute, several important researchers for the area of public procurement have affiliations, as well. Giancarlo Spagnolo and Elisabetta Iossa for instance.
Very well. So, if you could drill a little bit into the details of the grant, and the research, what are you actually going to be looking at, more specifically?
The premise is that when we think about contract procurement, two features are absolutely crucial. The first is that it is unavoidable that there is some cost uncertainty at the time of bidding. So regardless of whether the procurement is a very complex contract, for instance, for the construction of new military equipment, or whether it is a contract for something easier like repaving a road, there is intrinsic uncertainty. Even in repaving a road, typically the bidding takes place several months before contract execution, and what the weather condition, what the cost condition will be at the time of the execution of the contract, cannot be known for certainty by the firms at the time of bidding. Second, coupled with this intrinsic uncertainty with contract procurement, there is typically a difficulty to verify ex-post performance and to eventually enforce penalty in all those cases where things have not followed the original contract specification. This is a feature that makes contract procurement very different from the type of auction markets that the economics literature has extensively analysed, and for which a lot of important results have been established. For instance, if the auction entails a transaction that will clear right after the auction, think about the auction for a painting, in which individuals or firms bid, but then the transaction clears immediately, then the kind of problems and the kind of solution are extremely different, relative to those of contract procurement, and in particular, in contract procurement, this issue of the ex-post life of the contracts becomes crucial, and makes competition - which is the typical tool that we see as being so important and so effective in auction - a double-edged sword. Things can go pretty bad if competition is pushed and exacerbated in an environment with cost uncertainty and difficulty to verify performance, ex-post, because firms that are unreliable are willing to offer a low price at the time of the auction, might not perform as they should afterwards.
So what does the private sector do in the face of this? Well, a series of things. It can require financial guarantees, like bonding, letters of credit, and it can embed the decision maker to use some form of discretion in selecting bidders. So discretion in the form of contract, or awarding method, negotiations vs auction, or also - and this is typically the case - discretion in the use of past performance. Just think of a very simple example: when you’re renovating your apartment, the past performance of the firm that work with will certainly play a key role in your choice, and this is widespread in the private sector. Now, what is remarkable about public procurement, and especially European public procurement, is that the use of past performance is strictly limited. To be more precise, until very recently, until the latest round of European directives, the use of past performance was strictly forbidden in Europe. And this is peculiar, not just because it makes the public sector so different, relative to the private sector, and potentially it limits a great tool to prevent the problems that I was mentioning at the beginning, of poor performance, but also because it puts the European system at odds with the US public procurement system. In the US, in a nutshell, since 1994, there was a major reform of the Federal Acquisition Regulation, that put past performance of contractor at the heart of the system for selecting suppliers in federal procurement, and the main idea was exactly that of mimicking the good practices of private sector. Now, Europe is gradually moving towards something similar, but very, very slowly, and we are still far away. So what I want to study with this piece of research that I am now conducting is to what extent this reform tried by the US, and other possible reforms, based on the role of past performance in public procurement, can be an effective way to combine improvements in performance together with, still, the objective of limiting prices. And in particular, what I wanted to understand, and study with this research, is how to combine the use of past performance within system of awardings based still on auctions, on transparent auctions, and in particular, how to change from price only auctions, or scoring rule auctions, that do not include reputation, to price plus reputation auctions. So scoring rule auctions that included reputation. So how to measure reputation and how to include reputation into the scoring rule, and then quantify how this matter, how this approach could impact both the performance delivered and the cost of this potential improved performance. So this was the key idea of this piece of research, and the implementation of it is something that is still ongoing, but I have some preliminary results that, if you are interested, that I am happy to discuss.
And we will do that in a minute. Before we get to that, I would like to go back to something that you said about taking into account past performance being strictly limited. Yes, and no. It’s true that you cannot, in EU procurement, use past performance to assess the quality of a tender. You can use it, however, to assess the quality of a tenderer in a previous moment to the actual bidding. So, for example, if you have a restricted procedure, if you use a restricted procedure, you can use, and you can allocate significant points, number of points, to the performance of the economic operators coming to the auction, or coming to the procedure. With the open procedure, it’s a little bit more complex, especially now with the changes that have been introduced, but it can also be done in a certain way. So, from my perspective, and obviously I’m coming from a legal background, so from my perspective, what you’re suggesting could fit within the current rules, and the current rules exist for a very good reason, or a very reasonable reason, at least, which is to ensure equal access to the markets, not only to companies with a huge trading history, but also to companies that do not have a lot of trading history with the public sector. So that’s why you have this clear separation, and this clear rule based on the principle of equality, to make sure that companies or that tenders, at the tender stage, are analysed on as much of an equal ground as possible.
I have two answers. One is about what the current rules say, but the second is even more important, because you have mentioned what I think is really the greatest departure in the view between economists and legal scholars. So let’s jump absolutely into this, and great that you asked. Why you said, essentially, that the rule to limit the use of past performance is there for a good reason? Because you want to keep an open market. Now, this is absolutely, absolutely at odds with how economists see the problem. Why? Don’t we care about open market and new firms and preferential treatment? Of course we do, but the issue is that you are sort of tying the hands of these public administration in a way that is not realistic. In a sense, whenever you are specifying how past performance shall be used in a contract, you must also specify how firms with little or no reputation must be treated. Now, the fact that you are using a system that is based on reputation does not necessarily mean that you are going to say, “We are going to use past performance for those who have, and for those that they don’t have it, they cannot enter the auction.” You can say something completely different. You can say, “We are going to use past performance for the firms that have a past performance, and for those that don’t have it, we are going to give the maximum points, or the minimum points, or the average points, or points estimated in a certain way.” This is what we call a design issue. It’s completely in the hands of the designer., You can think of the designer as whoever writes the law, the rules of the game. So, what happens to firms with no reputation, to new entrants, to firms that also have, maybe, little reputation, is the result of a designer’s choice. And so it’s something that can be optimised and tailored to the market. The most important thing, the only message that I want to communicate is that you don’t have to see that there is a necessary block created for new firms from the use of past reputation, because it’s fully a designer choice, what happens to new firms. So tell me if this sounds convincing to you or not, because this is really something which seems to be misunderstood in many discussions about past performance, like those on the potential implementation of a reputation system in the European Directives, and I’ve seen this in the debate around Article 57.
In which the European Directives 24 2014 has introduced some use of past performance, along the lines you were mentioning at the beginning of your remark, and I see that the point that you mentioned, the criticism that you mentioned, was really prominent there. And tell me if I manage to be clear or not, because this is a key point. It’s really a designer’s issue. So things should be decoupled. We should first decide if we want to use past performance, because, if we think it’s important. Then we can argue, “What’s the best way to treat new firms?”
I gave you the opinion, the general opinion from legal scholars, and why the rules are why they are. In my opinion, as time goes on, I think that you are on the right track in the sense that there is a lack of reputation being actually taken into account in a good way. That doesn’t mean that it’s quite easy to do within the logic of the system, which is to ensure that both new entrants, and older entrants, are not discriminated against, because even with the design, and yes, it’s a design issue, even with the design, it may be very difficult, in practice, to actually making sure that the compensation given to the new firms actually does not create arbitrage, because if it makes life a lot easier for new entrants, then what’s going to happen is that the existing companies, or existing economic operators, will create shell companies to actually be as fresh entrants into the market.
There’s a risk then if you think about how the system works in general, and I’ve mentioned this in previous podcasts, the public procurement rules in Europe exist not to enable great procurement. They exist to avoid really, really bad procurement. So avoiding corruption, avoiding making stupid mistakes, those are two of the key objectives of procurement, not to be economically efficient, because if you’re designing a system to be economically efficient in terms of public procurement, we might end up with something very different from what we have now. So my perspective, and my personal view, is that yes, we should be moving towards a system where reputation is taken into account. So, about a year, or two years ago, I remember writing a blog post suggesting that perhaps we needed something like the eBay ratings system, or Uber, where both parties of a transaction, at the end of a transaction, are compelled to provide feedback on the counterparty, on some sort of exchange that is public, and that anyone can consult, and you end up with a registry of reputation for both the contracting authority and the supplier. So we actually address the reputation issue from both ends. I haven’t written anything more specific about it, but that was something that I was thinking about, because even for a supplier, it’s important to know how reliable the contracting authority is, because…
Let me tell you a story, if you want, exactly on this. So in Italy, following this directive, and the implementation of Article 57, in the Italian Public Procurement Code, there has been a lot of debate, and essentially the authority in charge of supervising the public procurement sector, ANAC, Anti-Corruption Authority, put forward a proposed system to monitor past performance by suppliers, and at the heart of this system, there was essentially the contractual performance measured in terms of percentage delay in time of execution, relative to contractual time, and percentage discrepancy in final cost of the procurement, relative to the contractual price. Now, the Italian firms complained, and complained in a very reasonable way, saying that many times, it’s the fault of the public administration if things cannot be completed at the condition that was originally promised.
And this was extremely reasonable, and I completely agree with this point, and in this sense, they were saying, “We need also a rating system for public administration, and we need something different.” So the quarrel around this proposed implementation of this part of the directives in Italy was so strong that the supervising authority ended up blocking these proposal.
I think that is a pain point that we could address in public procurement going forward. Focusing back on your research, you said that you had some preliminary data to talk about. Can you tell us a little bit more about it?
Absolutely. So it all started with a very interesting experiment. Essentially, in Italy, we have a very large public utility company that provides water and electricity to Rome and central Italy. This company is owned 51% by the municipality of Rome, and so, because of this, it has to follow the public procurement regulations, but also since it’s a public utility company, it is in a group of contracting authorities the Italian public procurement code calls “special sectors,” and special sectors have some degree of freedom in how they design procurement rules. In how they fine-tune the procurement regulation that they apply. So the CEO of this company was extremely worried about the performance in the contracts that they were awarding. They award, every year, about 300 million Euros’ worth of contracts to perform maintenance of their water pipe system and electricity network, and with these contracts, they were considering the prices that they were getting from bidders in the auction quite good, but they thought that quality was really terrible, and you have to think that, especially for electricity, this is one of the really dangerous areas. If safety standards are not respected, people might die, because they work with high tension and you can easily get electrocuted, and the consequences are devastating. So what they did is they started to think about a system, or a way to improve performance, and then they consulted Giancarlo Spagnolo, and through Giancarlo also me, and we started a collaboration on this very fascinating project about how to improve the performance while still being leading within the system of the Public Procurement Court. And so if you think about introducing reputation, there are really two separate pieces. The first answers to the question: what/how do I monitor? How do I construct a measure of past performance, of reputation? And you can think that you could use publicly available measures. For instance, you can use some ratings that are already out there, I don’t know, ISO 9000 certification, or some other publicly available ratings, or you can construct your own rating measure. The company, with regard to this first question, decided that they wanted to build their internal rating system, and in particular, they said that they wanted to experiment two sectors. They picked electricity, and they picked two sub-sectors within electricity. Public illumination, and the maintenance of electrical substations, and they said, “For these contractual classes, we can write down a list, that is exhaustive, of all the things that need to be done properly in the contract, both in terms of safety parameters and in terms of quality parameters, and what we’ll do is, essentially, we’ll have teams of our engineers from the contracting authorities that then will go and inspect the execution of these contracts along this list of parameters.” And in particular, they chose a list of 136 parameters, and they also decided that these engineers that were going to do the inspections were randomly drawn. So their pool of engineers was changing every time, according to this random draw, to limit the risk of corruption. And also which work sites were inspected was also the result of a random draw. This is because it’s costly, it would be costly to monitor many, many contracts. By introducing a drawing system, you have a smart way to give to every contractor a positive probability of being monitored, but without having the cost of monitoring every contract.
And so they sent these people, and they started to monitor, and the results were devastating, because the results of the first three months of monitoring revealed that noncompliance was overwhelming. So these parameters were scored with a zero if they were found to be noncompliant and a one, if compliant. When the scores arrived for the first three months and they were aggregated up, noncompliance was 75%. So only on 25% of the parameters, on average, they were doing something that was as written in the contract, and this was very bad, because again there were, like, safety measures that were violated, putting people at risk of very serious dangers, and this was across all contractors, across all types of works in these two categories, across all parameters. So it was a very widespread phenomenon and this was very much confirming the fear of this company, that noncompliance to contractual elements was very common. These were things for which in principle these firms should have been, or could have been brought to court. But this was not happening. These were things for which, in principle, penalties that were written down in these contracts could have been enforced, but this was not happening. And so, the reason why it’s not happening would open a separate chapter, it’s related to deficiency of the court system that, for instance, in Italy is quite bad, but it’s also related to the general phenomenon that in business, you try to maintain a good attitude with your suppliers, as well as with your customer. And so this firm was feeling that if you started to bring suppliers to court, and to enforce penalties, it would have acquired a bad reputation, and it would have been harder for this firm to conduct business. It was not doing this, it was not bringing them to court, it was not enforcing penalties, but it was very worried about this poor performance.
Moving to step two, after we have designed a reputation system and you have some numbers that you can use to quantify reputation, you have to decide how to incorporate these reputation measures into your decision of future awardings. Now, this is where things become tricky with respect to the regulation. As we were discussing before, under the current EU directives, you could use this information in the stage of selecting which firms are admitted to participate in the auction. Still, you could not use this to select among the bids that are received. So this firm was a little bit torn about how to use the reputation system that it had designed for the selection of future contracts. What people at this firm did was that they announced to their suppliers, they were going to switch to a scoring rule auction that was giving 75% weight to price, and 25% to past performance, where past performance was a weighted average of those zero/one scores that I was mentioning before. So the system clearly was living in a grey area between what it was allowed and what was not allowed by European directive, and this forced the firm to delay the implementation as they were collecting opinions from legal experts, but this made the experiment even more fascinating. Why? Because for a year and a half, basically, the suppliers of the firm had been informed about this intended switch, but the switch was not happening. Why is this so interesting? Because as we were discussing at the very beginning, and your main concern about openness of the system, if they were to switch immediately to the new scoring rule system, we would have observed, in the data, a mix of effects, in part coming from the selection of new firms that are kicked out by the new system, and in part, like, price and quality effects linked to the response to this new system. Instead, what we have here in this beautiful experiment is this one year and a half in which suppliers have already incorporated that eventually the scoring rule system will arrive, and so earning reputation matters, because if they earn reputation today, when the new system will arrive, they can offer higher price and still win the contract because they have earned a good reputation that will be valuable to earn points, in the scoring rule formula. But today, they still don’t face any limitation to their entry and to their probability of winning related to their past performance, because for the space of a year and a half, they still have a first price rule that is in place.
So this allowed us, essentially, to study the effect of the announcement of this switch to reputation, so not the actual switch, but the announcement of the switch, on the behaviour of the contractors in terms of quality delivered, and price, and the results were quite stunning. So basically, in a nutshell, I told you 25% compliance before. After one year post the first announcement in which suppliers were informed about the intended switch, and they were shown the scoring rule formula that was going to be in place, quality, overall, and performance, increased to about 80% compliance on the parameters. So from 25% to 80%, which was absolutely great, and the change involved, essentially, all parameters and all firms that were part of this market.
Has that maintained over time, and has it also affected new suppliers that came in after the announcement, so they were not actually in the market during the announcement?
It maintained over time, and this 80% has remained more or less flat since then. Now, in terms of the selection of suppliers, what is interesting is certainly that we see that even the suppliers that were not performing well before the announcement of the switch to reputation, changed their behaviour. What we learned from this experiment is that we don’t have, necessarily, to think of markets with firms that are intrinsically good and bad, but there is very much a possibility for firms to be responsive to different incentives and to change, their, for instance, managerial practices in how much they value safety, and safety practices, and so on, and so we saw a change that is across the whole spectrum of firms participating in these auctions. In terms of entry and exit, we saw an exit of some firms, but in a way that, when we tried to benchmark with what happened in terms of the auction for the exact same sectors, but taking place in other multi-utility companies in Italy, and the exit rate that we observed for this firm running the experiment is an exit rate that is quite similar to the one of the other multi-utility company that did not run any experiment like this. In terms of entry, we see that there is some entry, but not much, in the period that we observed. Since only three new firms entered, from a statistical perspective, when you have such low numbers, it’s difficult to draw conclusions, because they don’t have any statistical power. So I would say with reasonable certainty that there is a great effect on behaviour, with the same firms changing. There is no effect on exit of firms. There is potentially some effect on entry. Potentially limiting a little bit entry, but it’s not clear how much.
Okay, so we need to wrap up the interview, we’re reaching the time limit. What’s next for you? So what are the next steps with your research?
There is one part that is still related to this experiment, that is, in a second part of the experiment, we were able to relate this performance also to the prices, and we observed that prices, in the initial phase, after there was the switch, they declined, probably because firms were competing very intensively to win contracts, because only by winning a contract you could be monitored and you could earn reputation, and afterwards, prices started to increase, which is compatible with firms passing through this higher cost of performing better, of delivering higher quality. But the passing of cost was not major, in the sense that, compared to the great increase in quality from 25 to 80%, the increase in cost was about 7%, relative to the price they were paying before.
And you could argue that is actually not an increase in price, it’s the price that it should have been, originally, if the quality was supposed to be the one that is in the contract.
We cannot say this for sure. We are not able, as economists, to measure what would be the cost of this quality. From the perspective of the firm we work with, we know that the firm is happy. It thinks that this is a very reasonable price increase for the higher quality. What we have done is a sort of exercise of trying to quantify the welfare produced by this reform, in terms of lives saved, so reduced probability of accidents, and by weighting this reduced probability of accidents with the value of statistical lives, which is a quantity that economists and also social scientists often use, and we saw that, indeed, the policy produced great benefits. So the benefits exceeded the cost by an order of several million Euros per year. So all this suggests that the policy was positive, but we don’t want to stop here, we want to keep on studying and understanding this system. So what comes next is that this company is trying to now understand how to restructure its system to be compatible with the new directive, again, Article 57, as we were discussing, and so how to switch to a system in which the reputation index is used to select firms that will participate, not to select bidders, and it will be very interesting to see how these two completely opposite systems perform, which are the pros and cons, but in terms of prices paid, and in terms of quality delivered. And so this will be, we believe, a fascinating way to learn about different methods of regulating this system. Of course, there are many other questions, because it’s quite arbitrary. Why 75% on price and 25% on past performance? Why those specific measures of past performance? This is really a broad area, and we have several interesting projects, mostly with Giancarlo Spagnolo, that we are pursuing in this area. On a slightly, still connected, but different part of my ERC grant my current research focus, is on the area of corruption, and the problems of criminal infiltrations into public procurement auction, and these are projects with Ray Fisman and Paolo Pinotti.
Thank you very much, Francesco, it was a pleasure having you.
Absolutely, same for me.