Research methods in public procurement (law)
Interview with Dr. Richard Craven, Lecturer at the University of Leicester.His research interests lie at the crossroads between law and social sciences, with a particular emphasis in public procurement research methods. Today’s talk will have more of a shared approach to it than usual as we both have something to say about research methods in public procurement.
It’s great to have you here, we’ve known each other for a few years but I think it’s the first time that we are getting back together and talking about procurement and procurement research methods.
Let’s start with your PhD, what research did you do for your PhD and what methods did you use?
Okay so my PhD completed in 2012, looked at the legal framework in the UK for regulating public private partnerships, so what we know as competitive dialogue, and then as you said, it wasn’t just a case of looking at the law in books. So looking at case law, looking at what the legislation said about how people are supposed to behave, it was more we went out into practice and conducted interviews with lawyers interpreting and applying the rules in practice, policy makers and procurement officers and contracting authorities, so I suppose just to see what sort of legal uncertainties were being interpreted in practice and how difficult areas may be where we might not want to comply with the rules, how that was approached in practice and, yes so really just that sort of, yes law in books to law in action-type approach.
I mean that’s very similar to the research I’ve done and I did in 2010, that I finished in 2010 for my PhD. I also was looking forward to see how competitive dialogue had been implemented and was being used in practice in Portugal and Spain. What I find very interesting is that it’s very different from watching or seeing the law being deployed in action by agents that are actually using and sometimes abusing the law in comparison with what you can read in the books, what you can read let’s say in the directives or in the laws and regulations.
Yes same here really, I think probably in the UK I think there was a lot of say reporting in the trade press that competitive dialogue was causing problems in practice, but I think it was about seeing overall, why were those problems being caused, how was it being interpreted maybe to cause those problems and what strategies were adopted to get around it. So yes, it was fascinating really coming from sort of a law background, you know you’re always studying law as in, ‘Or what are these legal rules?” and then sort of taking a completely different perspective that actually the law is what people are doing in practice, not necessarily what it says in the books yes.
Yes, I think that’s a very good point to remember, back in 2004, 2005 let’s say or 2006 maybe, the commission say in interpretive communication or whatever about competitive dialogues using some made-up examples of where the competitive dialogue could be used for example if [?? 3.25] authority wasn’t sure to use a bridge or a tunnel to cross a river, those kind of examples which made perhaps perfect sense in theory and for the people that were involved in drafting first the Directive 2004/18 and then obviously that communication or interpretation. Then in practice there was no correspondence with reality I mean, I have never seen an example where that kind of problem actually happened in practice.
Yes definitely, I agree, and hopefully then I suppose that’s what our PhDs contributed to good policy-making, because we’ve seen the law reformed now to something that’s maybe a bit more commercially sensible. I think originally without those commission interpretations, the law wasn’t being interpreted in sort of a way that was relevant for practice, and it was because of that sort of research, and we weren’t the only ones, but sort of highlighting that sort of miss-match between the, yes you know maybe for transparency reasons we might like to have these rules, but for commercially sensible reasons it’s just not going to work necessarily. So yes I completely agree there.
Well that takes me to the second question which is, what can we say that is wrong or limited with purely black letter or doctrinal analysis?
Well I don’t want to be too critical of doctrinal analysis, you know I enjoy reading doctrinal analysis sort of, obviously it sets up my work because I do empirical research, but I think it’s always that it can only take us so far, and particularly with public procurement in the UK as well where we have limited case law, we have legislation that’s uncertain in areas and complicated, and also a lot of guidance documents, so what we might call soft law so you don’t have to comply with it but you know, is authoritative in some sense. So a black letter legal analysis can, yes it can tell you what the law might be but actually it can’t tell you what the reality of the law is, what’s actually happening out there so yes, it can only take it so far really.
Yes but I think that is a very important point to make in a very important limitation to highlight, which is it only takes us this far but then it doesn’t allow us to actually know how the law is being used in practice and how to improve it, because at the end of the day to a certain extent, like with the analysis, it’s a little more than just opinion yes, it’s a legal opinion, but it’s only that.
Yes, I can only agree yes, it’s just one side of it and you know, there’s sort of great academic debates that go in say public procurement, public law but in practice just isn’t happening in line with that so it’s, as I say, just limited value isn’t it.
Yes, personally I’m always much more interested perhaps because I used to be a lawyer because I became an academic, I’m always more interested in looking at what is the job to be done that a certain piece of legislation is trying to solve and see if it’s actually solving it in practice, looking at in reality what it is doing instead of what people wanted it to do from the outside.
Definitely, and I suppose it’s in the UK, I suppose it’s just a modern way of doing things as well, so we’ve always, sort of legal researchers have always favoured doctrinal analysis and it’s only more recently that we’re sort of embracing social science methods which can only be a good thing.
Well to be honest, I mean it’s much easier and to a certain extent cheaper as well and less time-consuming to be just black letter analysis instead of doing empirical research.
Yes definitely, but I mean I have colleagues complain that you know, with doctrinal analysis they can’t apply for funding and what have you whereas actually empirical researchers, we can always make case for funding which I think is always viewed quite positively.
Yes that’s true, I hear the same thing both here in Swansea where I am now and also at Bangor University where I was before, the colleagues with the more traditional skillset in terms of legal academia, they find it very difficult in this day and age to find funding.
I think it just shows you what kind of you know, policy-makers, they want that, they want to know what’s happening out there, they don’t just want to know the particular sort of view point or analysis of one person necessarily.
Yes, agree with you. Now moving onto another question, why should we be using mixed methods for legal research, what are the pros and cons of doing it?
I suppose, I mean we’ve outlined the pros to a certain extent.
Well some of them at least.
Yes, and I suppose some of the cons as well in terms of the, sort of the expense and the time, I think also without wishing to be critical of black letter analysis [?? 8.15] or overly critical, I think in the same way black letter legal analysis can be, empirical research can be done very badly.
Yes that’s true.
So I think it’s only good if it’s done sort of quite robustly you know, you’ve got strong methodology there, and I think if it’s not done that, it’s very easy to pull apart and it’s just time wasted, so you know I don’t think it’s something people should just, or researchers should just jump into, I think it is quite hard to do and it needs to be done correctly. But the pros there I mean, as we’ve said in terms of what it tells you about the law and what is happening out there, and the way it can inform policy and show where legal rules might not be achieving what they were intended to achieve, I don’t think there’s any, well the alternative being black letter would not be a suitable approach for that.
I think you’ve made a very good point there about the difficulty of doing empirical research well. That is one of the limitations that we have in legal academia, is that by and large in the undergrad studies and even at the postgrad level, by and large we do not foster in the students the development of research skills other than the traditional research skills.
And I think that is a very important constraint that we have then further down the pipeline when people start doing their own research in their PhD or even after the PhDs when they become early career academics, it becomes very hard to retrain and regain those skills that were not gained perhaps in the moment they should have been gained.
I notice that for example, not exactly as much with empirical research but certainly with more economics-minded research, when I try to blend law and economics I can easily get to the limits of my knowledge and the limits of my ability of doing that kind of mixed-method research which I’m really interested in doing, but because I do not have the right background in terms of quantitative skills it becomes really really hard to do it well.
Mmm, I think there’s a lot to be said there, but at the same time though I mean, I think empirical research is never going to be perfect is it, it’s just about sort of justifying what you’ve done, there’s always, if you don’t write the findings of particular empirical findings I’m sure you can find some faults in the methodology somewhere and pull it apart, but I suppose that’s just the nature of it. So it’s never going to be perfect but it does need to have that sort of, that strength behind it as the sort of solid foundations and, yes I think the more academics are trained in it and I think universities do provide that sort of training now more and more, so they’re better.
On that note what would you like to have known at the beginning of your PhD that you know now about research methods, especially empirical research methods?
I think it’s probably that, that it’s not going to be perfect and things are going to go wrong, you know maybe interviews that are arranged are going to fall through and, or you know, the interviewees aren’t going to say the sort of things that you expect them to say or maybe you want them to say if it’s going to fit with a hypothesis.
Did you have problems with that?
I didn’t but I think you’re just worrying the whole way through because it feels so messy you know it doesn’t feel, you know I think doctrinal research, I think it often feels like you’re in control whereas empirical research doesn’t always feel that way because it’s so dependent, you know so sort of spanning over maybe a couple of years and it’s so, yes just hoping that people contribute and participate in your research but you know, I’ve sort of read around you know afterwards about other sort of empirical studies or famous empirical studies in different areas, and that’s really what they talk about. So you know, you have this book at the end that sort of details this amazing research but actually, getting those findings it was not an easy process and it did feel messy at times, so it would have been good to know that, would have been reassuring to know that.
I can certainly relate to that experience, I remember effectively discovering the main finding of my empirical research literally at the dying stage of the research stage when I was doing my final interviews, and it was actually my last interview in person that I did, I still had a few other ones to do over Skype or phone or something like that, but I remember going into that interview and still thinking “I still need something, there’s something here that I’m not getting”. And I had just had that eureka moment after the final interview, the final in-person interview.
Yes, I think any empirical research will sort of relate with that, and I suppose another thing that I would do differently is just keep on top of all my data a bit better, I think as I was going through I was sort of taking these interviews down and wasn’t sort of transcribing them straightaway and that, you know.
Yes, well occasionally and I think it is important to sort of just interview, transcribe and let that feed into the next interview.
Yes that’s actually one of the things I learned to do right from the start, which was to transcribe the interview as soon as possible after the interview has taken place where everything is still fresh.
I never had any technical problems but if you try to transcribe the interview much later after it was done and you find out for example whatever technology, piece of technology you were using wasn’t working in that scenario, you’re pretty much screwed.
You are yes, I was quite fortunate though, it was all on a digital recorder and, thank goodness, lesson learned.
Yes, what about let’s say other mixed methods of research like economics for example, have you ever been attracted to that?
No not really just because I suppose it just hasn’t been something that I’ve really sort of encountered in my studies, so it’s just been a sort of natural progression to get where I am, and I see myself more as a qualitative researcher, I think I’m more interested in people you know and how people respond to legal rules, so that’s what, sort of just regulation more broadly. So I think just following my interests, it’s got me to there but we still sort of go off an sort of continued the empirical line of things, so rather than law and economics we’ve been recently researching into litigation behaviour. So for example why case law and public procurement are so limited in the UK and that was more sort of questionnaires and numbers involved with also, interviews.
What did you find on that one?
A number of findings actually so looking at behaviour, we’re looking at sort of numbers. I think an overall finding that was limited case law because actually many suppliers didn’t see any reason for litigating, but actually they were happy with the process which actually was quite an interesting finding and you’re you know, maybe not feeling a need to pursue a challenge where there was only a slight technical breach, but then a long with that just the usual sort of reasons for not bringing a charge, be it sort of cost of litigation and fear of upsetting, you know biting the hand that feeds, upsetting the public sector who you might be working with, you might be bidding for further contracts from.
The famous fear of blacklisting?
In a sense, that doesn’t seem very different from the research that Despina [?? 16.04] did maybe fifteen years ago or twelve years ago.
Yes, yes, so it was sort of building on her research really, but it was sort of a modern, you know Despina’s research was sort of pre- I suppose…
…remedies directive, pre-sort of financial fallout if that had any impact, but yes, I was just basically seeing “well has it all changed now?” and actually it hasn’t changed, those are the main deterrents to litigation like sort of cost, time, actually there not being appropriate remedies in place or people not feeling that there are appropriate remedies in place or attainable remedies in place, those are all sort of limiting in the same way. Another big factor being sort of the approach the UK has to interim relief and sort of cross-undertaking in damages being a requirement, [?? 16.57] seem to be a real inhibitor.
I find this fascinating because I trained and worked in a completely different jurisdiction, in Portugal is the polar opposite. I mean as a lawyer, doing litigation of public procurement cases was my bread and butter, and it’s probably dozens in a couple of years, and I was obviously only one lawyer, or only one trainee at time, whatever, but that certainly surprised me when I came to the UK and people say, “No, no, we don’t litigate in UK in public procurement”.
Well I suppose you know, the research is basically, you know these remedies, rules, we’re looking at the effectiveness of those remedies why people won’t be suing, but I don’t think that you know, think “Oh, where [?? 17.41]” obviously it’s just me I don’t think but we’re both of the opinion that actually the best system in place wouldn’t be that you know, you’d have something like maybe you have in Portugal or even, I was in Denmark recently and they were talking about how legislatives there, and you know it just doesn’t seem like the right system, but you have just suppliers suing for every sort of minor breach but that’s what we seem to be trying to achieve you know, here.
Yes but the problem with the system like the one we have in UK where there’s limited enforcement in the courts, it means that the contracting authorities can do whatever they want as there’s a limited risk that they’re going to be caught, and that changes the way that people behave.
Well yes and no, I think with these technical rules it’s not always the case though that you know, the example, I was talking to someone in Denmark recently and they were saying how you know just these minor sort of infringements mean that a procurement is just getting shut down for sort of a matter of months as it progresses through the course. You think “Well that can’t be the right approach,” you know where somebody’s not doing something to get around the rules and maybe it’s just because you know procurement, like we said with empirical research you know, it’s not always so exact and you can say that this is, you know it’s not going to flow easily is it, it’s just sort of, it can be quite I don’t want to say messy but you know they’re difficult aren’t they.
And then you know, especially when they stretch over time, so you know whether it should be that a supplier is going to jump on you every step wrong you make just can’t be the right system I think, but it’s the system we have.
And I have to say that I’m a huge fan of the Canadian system of having a procurement ombudsman. I heard Frank Brunetto was the outgoing Canadian procurement ombudsman on a podcast at an early stage and I think it’s a great system actually, it helps defuse a number of potential issues before they reach the court.
In the UK you’ve got the Mystery Shopper service which may evolve in that direction, although I doubt it that they will ever be called procurement ombudsman type of system, but it’s a shame, it’s a shame I think it’s a very good system and in Europe you should look more closely into it. Speaking about your projects what else are you doing?
Well in addition to that we are just starting some further empirical research into the protection of workers on government contracts so I’d like to interview procurement officers in relation to that, and lawyers if they happen to be listed in the UK. Yes just looking at, I suppose just compliance with legal requirements on public contracts, the way in which that might be achieved through the procurement process and on the procurement contract.
Are you’re doing it by yourself or are you collaborating with other colleagues?
That’s just me at the moment yes.
Okay, why do you think public procurement is a good area to mixed-method research?
I suppose because there’s so much to do, you’ve got uncertain rules that you know leave a lot of scope for interpretation, so how are those rules being interpreted? I think just, it’s an important area so it matters to people, so if you’re thinking about public services and having value for money public services, well, and how that effects people whether it be procurement officers, suppliers or the general public, I think those views are important. And it’s an area where it’s just not really been touched by empirical legal researchers at least, so those who actually know something about this complicated area of law and can go and look at it in practice, I know there’s a few business academics who’ve had a look at it, and economists and that’s very interesting, but actually from legal academics it’s you know, sort of, there’s a lot to do because it’s not been covered.
Well, one final question. How do you break down the barriers between different disciplines?
Yes well, as I mentioned before we sort of started recording, I’d be very interested to hear your views on this but I suppose for me I think collaboration between individuals is a good starting point, and I think that’s very difficult but you know, just putting yourself with say a social scientist and economist, and you can both bring something to the table there, but it’s difficult. And you can see actually with some of these research projects that people are forming teams and I think you know, everybody bringing something in from their particular discipline, that definitely seems like the way forward.
Yes I think that’s the way to go. There’s only so much we can do to retrain ourselves to do research that we haven’t been trained before to do, especially as you go and on and onwards with your career and you have certain, let’s say [?? 22.43] costs on a certain career path, you have also other commitments that you don’t have for example while you were obviously an undergrad or a PG student, so it makes really difficult for you to pick up new research skills as you go onwards with your career. So I think in a way I mean, collaboration is certainly a way forward, but the problem I have with collaboration or the problem I have seen with collaboration is that it’s very difficult to break down the silos between different disciplines, because social scientists look at us when you try to do legal research using social scientists’ methods as being second-class citizens because we don’t have as much experience and knowledge about the methods as they have, and it’s a fair comment to make to many of us and it’s [?? 23.27] with economics. So if you try to do stuff with economists, what we can have and I’ve heard this from colleagues of mine which is “What is in there for us?” Because if we want to public in top-ranking economics journals, by and large all of them are pure economics journals that do not touch mixed-methods research so the opportunity cost of doing mixed-methods research becomes very very costly, and it’s very hard to find the right people with the right skills and the right attitudes to break down that silos, because if you think about how we are assessed in the UK in terms of research, we are still assessed by and large in a very traditional silo way. I mean if you’re going to be put forwards for the Research Excellence Framework we’re going to go in the lower panel, and by and large again in general, the people there will have a traditional legal background although actually the law panel is more mixed than others. But on the economics side is just economists, economists and no one else.
Yes I agree, so it needs some sort of fundamental change somewhere definitely, but I think things are changing slowly you know, when you look at what funders and you know, we’re guided by that now impact and funders [?? 24.41] universities, it is just pushing you towards I think collaboration and you know mixed methods yes, working with the different disciplines. So it’s only a matter of time I think.
Okay, one final question, what kind of research or type of research would you like to do in the near future that you haven’t been able to do yet?
Well I suppose without giving too much away, I’m sort of linking back to maybe one of your other podcasts actually, there was, I think you’ve done a podcast with Amy Ludlow.
Yes, so I was really intrigued by that sort of, the ethnography angle, I don’t know what she meant to say sort of [?? 25.20], just observation you know, just putting yourself in the environment so you’re just going beyond say interviews with individuals and for example you know, seeing a procurement process first-hand and, so I’d quite like to do research there, maybe looking at sort of the way in which central governments have tried to achieve some broader policies like localism or big society through procurement and try and bring in some ethnography there, observation.
I think that’s a very good way to finish the podcast, thank you very much Richard.
Oh thank you very much.
You can find me at my blog Telles.eu or on Twitter where I use two handles, @Detig for general discussion and @publicprocure for public procurement related topics. As ever I am grateful for the support of the British Academy Rising Star Engagement Awards which made possible this project.